Persons with autism (n = 28) or Down syndrome (n = 30) took part in a study of the ability to detect intermodal correspondence between facial and vocal/linguistic information for affect. Participants viewed 24 split-screen images of an individual talking and displaying a different affect on each side of the display (happy, sad, angry, surprised, or neutral). The vocal track, matching one affect (i.e., one side of the split-screen) but not the other, was played from a central speaker. Subjects were asked to point to the side matching the vocal track. The vocal track was desynchronized with both sides, so that rhythmic synchrony was greatly reduced and subjects must use affect to make their choices. In the first control condition, rhythmic synchrony information was restored. In a second control condition, inanimate objects and their sounds were presented. In the experimental condition, when verbal mental age and IQ were taken into account, the autism group performed more poorly than did the Down syndrome group in detecting intermodal correspondence of face and voice. When rhythmic synchrony information was available, both groups' performances improved, with the Down syndrome group performing slightly better than the group with autism. There were no group differences in the condition using inanimate objects. Results suggest that persons with autism may have difficulty detecting intermodal correspondence of facial and vocal/linguistic affect.