This study aims to determine if there are differences in color–emotion association between monolingual speakers of Spanish and Mandarin, depending on how colors are presented (verbally or visually). We tested two groups of 25 speakers of these two languages in two different tasks using the Geneva Emotion Wheel, which encompasses 20 types of emotions. In Task 1, 13 colors were presented to participants as color terms in their native language, whereas in Task 2 the same colors were presented as color patches from the Munsell chart. Participants were then asked to associate color terms or color patches to the set of emotion concepts (and intensities of emotion) in the Geneva Emotion Wheel. Overall, differences between languages were not significant, regarding either the type of emotion or individual dimensions of emotion (valence, arousal or power), although significant differences were observed for specific colors. Also, Spanish speakers tended to attribute higher intensity values and higher numbers of emotion values to colors. At the same time, speakers of both languages reacted similarly to color presentation, with color terms being associated with the same emotions as color patches, but eliciting stronger reactions with respect to intensity and the number of emotion values. Finally, we found less variability in color–emotion associations within the Spanish-speaking group. Overall, our study points to a mixed pattern of universality and culture-specificity regarding how colors are used for conveying emotions by people.