Confocal microscopy has been used to study the crystallization of two colloidal model systems that are geometrically frustrated in a completely different way: (I) hard colloidal polyhedrals, where crystallization is frustrated due to the incommensurate particle shape and (II) large spherical impurities in a sea of monodisperse colloidal hard spheres, where crystallization is frustrated by the introduction of impurities. As a reference system, we analyzed the crystallization of pure monodisperse colloidal hard spheres. We show that although the crystal structures of both systems are highly dissimilar on the individual particle level, both sources of geometrical frustration have a similar effect on the structure on the grain level. We quantitatively characterize the polycrystalline structures and study the crystallization process in time. Whereas grain boundaries persist in the frustrated systems due to structural arrest, the majority of grain boundaries anneals out quite rapidly in the reference system. Therefore, we argue that both sources of geometrical frustration cause a polycrystalline structure.