Infertility affects 10–15% of couples at the age of conception. Mounting evidence supports that not only are paternal factors crucial during fertilization, but also for embryogenesis. This review aims to provide some clues about the contribution of male factors to reproductive success and live birth, as such contributions can be as important as that of the female. Semen is composed of two fractions: sperm and seminal plasma. Regarding the former, the integrity of sperm components (i.e., centrioles, DNA integrity and methylation, histone-to-protamine ration, specific proteins, etc.) has been proven to be essential for some of the events occurring upon engulfment of the spermatozoon into the oocyte cytoplasm. The metabolic status of sperm also seems to shape their potential fertilizing capacity. Furthermore, seminal plasma appears to modulate the female reproductive tract, and has been suggested to support embryo implantation. In spite of the aforementioned, it remains largely unaddressed how paternal factors interact with maternal ones, and whether the latter may mask the former. While assisted reproductive techniques (ART) are useful to rescue infertility, a better understanding about the contribution of semen to fertilization, embryo development and implantation can increase the efficiency of these techniques, and address further the causes of total fertilization failure, implantation deficiency and recurrent miscarriage.