Mercury-manganese (HgMn) stars were considered to be non-magnetic, showing no evidence of surface spots. However, recent investigations revealed that some stars in this class possess an inhomogeneous distribution of chemical elements on their surfaces. According to our current understanding, the most probable mechanism of spot formation involves magnetic fields. Taking the advantage of a newly-built polarimeter attached to the HARPS spectrometer at the ESO 3.6m-telescope, we performed a high-precision spectropolarimetric survey of a large group of HgMn stars. The main purpose of this study was to find out how typical it is for HgMn stars to have weak magnetic fields. We report no magnetic field detection for any of the studied objects, with a typical precision of the longitudinal field measurements of 10 G and down to 1 Gauss for some of the stars. We conclude that HgMn stars lack large-scale magnetic fields typical of spotted magnetic Ap stars and probably lack any fields capable of creating and sustaining chemical spots. Our study confirms that alongside the magnetically altered atomic diffusion, there must be other structure formation mechanism operating in the atmospheres of late-B main sequence stars.