The ventral surface of the human occipital lobe contains multiple retinotopic maps. The most posterior of these maps is considered a potential homolog of macaque V4, and referred to as human V4 (“hV4”). The location of the hV4 map, its retinotopic organization, its role in visual encoding, and the cortical areas it borders have been the subject of considerable investigation and debate over the last 25 years. We review the history of this map and adjacent maps in ventral occipital cortex, and consider the different hypotheses for how these ventral occipital maps are organized. Advances in neuroimaging, computational modeling, and characterization of the nearby anatomical landmarks and functional brain areas have improved our understanding of where human V4 is and what kind of visual representations it contains.