We present a study on the clustering of a stellar mass selected sample of galaxies with stellar masses M* > 1010M⊙ at redshifts 0.4 < z < 2.0, taken from the Palomar Observatory Wide-field Infrared Survey. We examine the clustering properties of these stellar mass selected samples as a function of redshift and stellar mass, and find that galaxies with high stellar masses have a progressively higher clustering strength than galaxies with lower stellar masses. We also find that galaxies within a fixed stellar mass range have a higher clustering strength at higher redshifts. We further estimate the average total masses of the dark matter haloes hosting these stellar-mass selected galaxies. For all galaxies in our sample the stellar-mass-to-total-mass ratio is always lower than the universal baryonic mass fraction and the stellar-mass-to-total-mass ratio is strongly correlated with the halo masses for central galaxies, such that more massive haloes contain a lower fraction of their mass in the form of stars. The remaining baryonic mass is included partially in stars within satellite galaxies in these haloes, and as diffuse hot and warm gas. We also find that, at a fixed stellar mass, the stellar-to-total-mass ratio increases at lower redshifts. This suggests that galaxies at a fixed stellar mass form later in lower mass dark matter haloes, and earlier in massive haloes. We interpret this as a ‘halo downsizing’ effect.