In a case-control study of cognitive performance, tests of intelligence, reading, spelling, and pragmatic language were administered to the parents and siblings of 90 community-ascertained probands with autism (AU group) and to the parents and siblings of 40 similarly ascertained probands with trisomy 21 Down syndrome (DS group). The two samples were comparable for age and parents' education; both groups were well-educated and had above-average intelligence. AU parents scored slightly but significantly lower on the WAIS-R Full Scale and Performance IQ, on two subtests (Picture Arrangement and Picture Completion), and on the Word Attack Test (reading nonsense words) from the Woodcock-Johnson battery. There were no differences between AU and DS siblings. As in earlier studies, AU parents, more often than DS parents, reported a history of early language-related cognitive difficulties; we were not able to replicate this in siblings. AU parents who reported such difficulties scored significantly lower on Verbal IQ, spelling, and the nonsense reading test. AU parents without a history of early language-related cognitive difficulties often had a Verbal IQ that exceeded Performance IQ by more than one standard deviation. AU siblings with early language-related difficulties had similar findings: lower Verbal IQ, poorer spelling, and poorer reading scores, compared to AU siblings without such a history. Parents with a positive history also scored worse on a measure of pragmatic language,the Pragmatic Rating Scale, but not on measures of social-related components of the broader autism phenotype. We propose that cognitive differences in a subset of autism family members are manifestations of the language-related component of the broader autism phenotype, and separate from the social-related component. This is consistent with the hypothesis that there are several genes that may interact to cause autism which segregate independently and have distinguishable manifestations in family members. The hypothesis would be further supported by finding different patterns of genetic loci linked to autism in families where one or both parents has language difficulties.