Nearby galaxies come in two flavors: red quiescent galaxies (QGs) with old stellar populations, and blue young star-forming galaxies (SFGs). This color bimodality seems to be already in place at z = 2 − 3, presenting also strong correlations with size and morphology. Surprisingly, massive QGs at higher redshifts are ~5 times smaller than local, equal mass analogs. In contrast, most of the massive SFGs at these redshifts are still relatively large disks. The strong bimodality in both SFR and sizes indicates that some SFGs must experience strong structural transformations accompanied by a rapid truncation of the star-formation to match the observed properties of QGs. Using high-resolution HST/WFC3 F160W imaging from the CANDELS survey in GOODS-S and UDS, along with multi-wavelength ancillary data, we analyze stellar masses, SFRs and sizes of a sample of massive (M* > 1010M⊙) galaxies at z = 1.4 − 3.0 to identify a population of compact SFGs with similar structural properties as compact QGs at z~2. We also find that the number density of QGs increases rapidly since z = 3. Among these, the number of compact QGs builds up first, and only at z < 1.8 we do start finding a sizable number of extended QGs. This suggests that the bulk of these galaxies are assembled at late times by both continuous migration (quenching) of non-compact SFGs and size growth of cQGs. As a result of this growth, the population of cQGs disappears by z~1. Simultaneously, we identify a population of compact SFGs (cSFGs) whose number density decreases steadily with time since z = 3.0, being almost completely absent at z < 1.4. The number of cSFGs makes up less than 20% of all massive SFGs, but they present similar number densities as cQGs down to z~2, suggesting an evolutionary link between the two populations.