Maternal supraphysiological estradiol (E2) environment during pregnancy leads to adverse perinatal outcomes. However, the influence of oocyte exposure to high E2 levels on perinatal outcomes remains unknown. Thus, a retrospective cohort study was conducted to explore the effect of high E2 level induced by controlled ovarian stimulation (COH) on further outcomes after frozen embryo transfer (FET). The study included all FET cycles (n = 10,581) between 2014 and 2017. All cycles were categorized into three groups according to the E2 level on the day of the human Chorionic Gonadotropin trigger. Odds ratios (ORs) and their confidence intervals (CIs) were calculated to evaluate the association between E2 level during COH and pregnancy outcomes and subsequent neonatal outcomes. From our findings, higher E2 level was associated with lower percentage of chemical pregnancy, clinical pregnancy, ongoing pregnancy, and live birth as well as increased frequency of early miscarriage. Preterm births were more common among singletons in women with higher E2 level during COH (aOR1 = 1.93, 95% CI: 1.22–3.06; aOR2 = 2.05, 95% CI: 1.33–3.06). Incidence of small for gestational age (SGA) was more common in both singletons (aOR1 = 2.01, 95% CI: 1.30–3.11; aOR2 = 2.51, 95% CI: 1.69–3.74) and multiples (aOR1 = 1.58, 95% CI: 1.03–2.45; aOR2 = 1.99, 95% CI: 1.05–3.84) among women with relatively higher E2 level. No association was found between high E2 level during COH and the percentage of macrosomia or large for gestational age. In summary, oocyte exposure to high E2 level during COH should be brought to our attention, since the pregnancy rate decreasing and the risk of preterm birth and SGA increasing following FET.