Perceptual information includes sensorimotor and emotional experience regarding the multimodality of the perceptual system. The current study provides an image-based visual analysis on the embodiment of color metaphors through the investigation of (i) the perceptual (dis)similarities between the literal and metaphorical meanings of the Chinese color terms hēi ‘black’ and bái ‘white’ and (ii) the influence of emotional valence on the degree of their perceptual (dis)similarities. Specifically, 24 concepts in three semantic domains were represented as eight-dimensional vectors based on the color information extracted from online images, including two color concepts of black and white, 20 abstract concepts referring to 8 metaphorical meanings of hēi and 12 metaphorical meanings of bái, and two abstract concepts referring to positive and negative affective polarity. Statistical analyses show that (i) the literal and metaphorical meanings of hēi and bái are perceptually distinguishable given their significant perceptual (dis)similarities and (ii) the observed perceptual distinguishability cannot be solely attributed to the (in)consistency of emotional valence associated with the senses. The present study provides nonlinguistic evidence for the embodiment of color metaphors in the Chinese context with an empirical approach that can simultaneously capture the metaphorical mappings and affective associations among cross-domain concepts with sensory data.