Generic psychometric instruments are frequently used in psychiatric practice. When a respondent provides an affirmative reply to two contrasting items in such a questionnaire (e.g. “I am reserved” and “I am outgoing”), serious questions need to be asked about the respondent, the instrument, and the interaction between the two.

The research aims to identify reasons which could explain the contradictory answers provided by respondents to a well-established, and seemingly psychometrically sound instrument.

World Values Survey data, collected in South Africa (N = 3 531), were analysed, focusing on the personality survey, where contrasting response to matching items were identified. Exploratory factor analyses were used to inspect the factorial structure of the instrument across groups, after which measurement invariance tests were done.

The theorised factorial structure of the personality survey did not mirror the structure in the South African sample. This was demonstrated in the inspection-report, as well as in the tests of measurement invariance. However, in some groups, specifically those who were well-versed in English and possessed higher levels of education, the structures were replaceable.

The assumption that well-established instruments are valid in settings different to the one where they were initially developed, should be questioned, and such instruments should not be used unless thoroughly tested. This presentation exposes the extent of measurement non-invariance when using an instrument in a foreign setting and shows how this can be detected and addressed. Those working with foreign individuals or conducting cross-cultural research should be particularly aware of these threats to validity.

No significant relationships.