We report on follow-up observations of 20 short-duration gamma-ray bursts (GRBs; T90 < 2 s) performed in g′r′i′z′JHKs with the Gamma-Ray Burst Optical Near-Infrared Detector (GROND) between mid-2007 and the end of 2010. This is one of the most comprehensive data sets on GRB afterglow observations of short bursts published so far. In three cases, GROND was on target within less than 10 min after the trigger, leading to the discovery of the afterglow of GRB 081226A and its faint underlying host galaxy. In addition, GROND was able to image the optical afterglow and follow the light curve evolution in five further cases: GRBs 090305, 090426, 090510, 090927, and 100117A. In all other cases, optical/near-infrared upper limits can be provided on the afterglow magnitudes. After shifting all light curves to a common redshift, we find that the optical luminosities of the six events with light curves group into two subsamples. GRBs 090426 and 090927 are situated in the regime occupied by long-duration events (collapsars), while the other four bursts occupy the parameter space typical for merger events, confirming that the short-burst population is contaminated by collapsar events. Three of the aforementioned six bursts with optical light curves show a break: GRBs 090426 and 090510 (Papers I and II) as well as GRB 090305. For GRB 090927, no break is seen in the optical/X-ray light curve until about 150 ks/600 ks after the burst. The GROND multi-color data support the view that this burst is related to a collapsar event. A decay slope of the optical afterglow of GRB 100117A could be measured. For all six GRBs a lower limit on the corresponding jet opening angle can be set. Using these data supplemented by about ten events taken from the literature, we compare the jet half-opening angles of long and short bursts. We find tentative evidence that short bursts have wider opening angles than long bursts. However, the statistics are still very poor.