Sex differences in mental rotation ability have been observed in many countries. A previous study of Finnish participants reported that genetic and environmental influences on mental rotation ability differ between sexes. In this study, we assessed genetic and environmental influences on variance in mental rotation ability in 649 Japanese twins using a mental rotation test. To explain the influence of sex on variance in mental rotation ability, we applied genetic analysis using the sex limitation model. The following two factors explained variance in mental rotation ability: (1) the additive genetic factor, which reflects the accumulated small influence of many genes, and (2) the unique environmental factor, which is a type of environmental factor that differs between co-twins. The shared environmental factor, a type of environmental factor common for co-twins, could not explain the variance in mental rotation ability. Furthermore, the additive genetic factor was the same between sexes (i.e., not qualitative sex differences for the additive genetic factor), indicating that the same genes affect mental rotation ability in both sexes. Despite this observation, the additive genetic influence was greater in males than in females. In contrast, the unique environmental influence was not different between sexes. Considering the current results and those of a previous study, the quantitative sex difference for the additive genetic influences in mental rotation ability may be universal, while the unique environmental differences may depend on the characteristics of specific populations.