Skip to main content Accessibility help

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings.

Close cookie message
×
Hostname: page-component-6bf8c574d5-pdxrj Total loading time: 0 Render date: 2025-03-09T14:50:52.505Z Has data issue: false hasContentIssue false

2 - Hard-to-survey populations in comparative perspective

Published online by Cambridge University Press:  05 September 2014

By
Tom W. Smith
Edited by
Roger Tourangeau ,
Brad Edwards ,
Timothy P. Johnson ,
Kirk M. Wolter  and
Nancy Bates
Tom W. Smith
Affiliation:
University of Chicago
Roger Tourangeau
Affiliation:
Westat Research Organisation, Maryland
Brad Edwards
Affiliation:
Westat Research Organisation, Maryland
Timothy P. Johnson
Affiliation:
University of Illinois, Chicago
Kirk M. Wolter
Affiliation:
University of Chicago
Nancy Bates
Affiliation:
US Census Bureau
Get access

Summary

Introduction

It is clear that surveying difficult-to-reach populations is a universal problem (Maranda, 2004; Statistics Canada, 2004).

It is universal in several senses. First, some populations, subgroups, and individuals will always be hard to survey. Second, there are diverse and complex reasons that populations and individuals are challenging. Difficulty is neither simple nor unidimensional. Finally, difficulty spans the globe. Wherever surveys are done, difficulties arise. But the specific mix of impediments can be society-specific and the steps needed to overcome these impediments must be geared to the realities and complications that prevail in each survey in every country.

The total survey error paradigm can be used to examine the challenges in surveying hard-to-survey populations. Major components of total survey error that specifically relate to these populations include (1) sampling problems, such as the need for special sample frames, (2) noncoverage or undercoverage, (3) nonresponse, and (4) measurement errors, such as misreporting.

First, sampling problems include subgroups that are part of the target population, but that are not covered or undercovered by available sample frames. If the subgroup is the focus of the study, an alternative sampling design often must be developed. Even if the target population is not undercovered, using a sampling frame for the general population and screening down to the targeted subgroup may be inefficient and impractical due to cost. Dual or multiple frames or various referral sampling methods will often be needed to sample adequately and efficiently rare populations (Christman, 2009; Elliott, McCaffrey, Perlman, Marshall, & Hambarsoomians, 2009; Ericksen, 1976; Johnston & Sabin, 2010; Kalsbeek, 2003; Kalton, Chapter 19 in this volume; Kalton & Anderson, 1986; Reed, 1975–76; Rothbart, Fine, & Sudman, 1982; Sudman, 1972). What can be done for a given target population will vary greatly from country to country depending on what information is available for developing sampling frames (McKenzie & Mistianen, 2009; Salganik & Heckathorn, 2004; Treiman, Lu, & Qi, 2009). For example, if a national census or administrative records fail to identity a particular subgroup or rare population, then sample frames based on those sources cannot target these groups.

Type
Chapter
Information
Hard-to-Survey Populations , pp. 21 - 36
Publisher: Cambridge University Press
Print publication year: 2014

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Afrobarometer Network (2007). Afrobarometer Round 4 Survey Manual.
Bates, N., & Edwards, B. (2010). International Conference on Methods for Surveying and Enumerating Hard-to-Reach Populations. Unpublished report.
Benstead, L. (2010). Effects of Interviewer Gender and Religious Dress on Survey Responses: Findings from a Nationally-Representative Field Experiment in Morocco. Paper presented to the Society for Political Methodology, Iowa City, IA.
Billiet, J., Philippen, M., Fitzgerald, R, & Stoop, I. (2007). Estimation of nonresponse bias in the European Social Survey: using information from reluctant respondents. Journal of Official Statistics, 23(1), 135–62.Google Scholar
Bloch, A. (1999). Carrying out a survey of refugees: some methodological considerations and guidelines. Journal of Refugee Studies, 12(4), 367–83.CrossRefGoogle Scholar
Brackertz, N. (2007). Who Is Hard to Reach and Why? ISR Working Paper. Swinburne University of Technology.
Bradburn, N., & Sudman, S. (1979). Improving Interview Methods and Questionnaire Design. San Francisco, CA: Jossey-Bass.Google Scholar
Christman, M. C. (2009). Sampling rare populations. In Pfeffermann, D. & Rao, C. R. (eds.), Handbook of Statistics, Volume 29a. Amsterdam: North Holland.Google Scholar
Cleland, J., & Scott, C. (eds). (1987). The World Fertility Survey: An Assessment. New York: Oxford University Press.PubMed
Cornelius, W. A. (1982). Interviewing undocumented immigrants: methodological reflections based on fieldwork in Mexico and the U.S., International Migration Review, 16(2), 378–411.CrossRefGoogle ScholarPubMed
Couper, M. P., & de Leeuw, E. D. (2003). Nonresponse in cross-cultural and cross-national surveys. In Harkness, J. A., van de Vijver, F. J. R., & Mohler, P. Ph. (eds.), Cross-Cultural Survey Methods (pp. 157–77). Hoboken, NJ: John Wiley & Sons.Google Scholar
D’Alessio, G., & Faiella, I. (2002). Non-response behaviour in the Bank of Italy’s Survey of Household Income and Wealth. Bank of Italy Report No. 462. Rome: Bank of Italy.
de Leeuw, E., & de Heer, W. (2002). Trends in household survey nonresponse: A longitudinal and international comparison. In Groves, R. M., Dillman, D. A., Eltinge, J. L., & Little, R. J. A. (eds.), Survey Nonresponse (pp. 41–54). New York: John Wiley & Sons.Google Scholar
Des Jarlais, S. C., Perlis, T. E., Stimson, G., & Poznyak, V. (2006). Using standardized methods for research on HIV and injecting drug use in developing/transitional countries: case study from the WHO Drug Injection Study Phase II. BMC Public Health, 6, 54.CrossRefGoogle ScholarPubMed
Elliott, M. N., McCaffrey, D., Perlman, J., Marshall, G. N., & Hambarsoomians, K. (2009). Use of expert ratings as sampling strata for a more cost-effective probability sample of a rare population. Public Opinion Quarterly, 73(1), 56–73.CrossRefGoogle ScholarPubMed
Elmore-Meegan, M., Conroy, R. M., & Agala, C. B. (2004). Sex workers in Kenya, numbers of clients and associated risks: an exploratory survey. Reproductive Health Matters, 12(23), 50–7.CrossRefGoogle Scholar
Ericksen, E. P. (1976). Sampling a rare population: a case study. Journal of the American Statistical Association, 71(356), 816–22.CrossRefGoogle Scholar
Feld, K. (2009). Research in a danger zone. Research World, 9(3), 49–51.Google Scholar
Gelberg, L., & Siecke, N. (1997). Accuracy of homeless adults’ self-reports. Medical Care, 35(3), 287–90.CrossRefGoogle ScholarPubMed
Haeder, S., & Gabler, S. (2003). Sampling and estimation. In Harkness, J. A., van de Vijver, F. J. R., & Mohler, P. Ph. (eds.), Cross-Cultural Survey Methods (pp. 117–36). New York: John Wiley & Sons.Google Scholar
Hagan, J. (2011). Voices of the Darfur genocide. Contexts, 10(3), 22–9.CrossRefGoogle Scholar
Hajioff, S., & McKee, M. (2000). The health of the Roma people: a review of the published literature. Journal of Epidemiology & Community Health, 54(11), 864–69.CrossRefGoogle ScholarPubMed
Heckathorn, D. D. (2002). Respondent-driven sampling II: deriving valid population estimates from chain-referral samples of hidden populations. Social Problems, 49(1), 11–34.CrossRefGoogle Scholar
Hewett, P. C., Erulkar, A. S., & Mensch, B. S. (2004). The feasibility of computer-assisted survey interviewing in Africa. Social Science Computer Review, 22(3), 319–34.CrossRefGoogle Scholar
Hochchild, A. (1989). The Second Shift: Working Families and the Revolution at Home. New York: Viking Penguin.Google Scholar
Hornbeck, S. (n.d.). Conducting research in Papua New Guinea. Unpublished report. D3 Systems. Retrieved from .
Hornbeck, S., Peng, D., Studner, C. & Jodice, D. (n.d.). Ensuring data quality in conflict zones. Unpublished report. D3 Systems.
Iachan, R., & Dennis, M. L. (1993). A multiple frame approach to sampling the homeless and transient population. Journal of Official Statistics, 9(4), 747–64.Google Scholar
Irwin, C. (2011). Peace polls. Retrieved from .
Johnston, L. G., & Sabin, K. (2010). Sampling hard-to-reach populations with respondent driven sampling. Methodological Innovation Online, 5(2), 38–48.CrossRefGoogle Scholar
Kalsbeek, W. D. (2003). Sampling minority groups in health surveys. Statistics in Medicine, 22(9), 1527–49.CrossRefGoogle ScholarPubMed
Kalsbeek, W. D., & Cross, A. R. (1982). Problems in sampling nomadic populations. In Joint Statistical Meetings Proceedings, Survey Research Methods Section (pp. 398–402). Alexandria, VA: American Statistical Association.Google Scholar
Kalton, G., & Anderson, D. W. (1986). Sampling rare populations. Journal of the Royal Statistical Society: Series A (General), 149(1), 65–82.CrossRefGoogle Scholar
Kanouse, D. E., Berry, S., Duan, N., Lever, J., Carson, S., Perlman, J. F., et al. (1999). Drawing a probability sample of female street prostitutes in Los Angeles County. Journal of Sex Research, 36(1), 45–51.CrossRefGoogle Scholar
Kelly, C. B. (1977). Counting the uncountable: estimates of undocumented aliens in the United States. Population and Development Review, 3(4), 474–81.CrossRefGoogle Scholar
Kennickell, A. B. (1998). Analysis of nonresponse effects in the 1995 Survey of Consumer Finances. Unpublished report, Federal Reserve System.
Kuebler, D., & Hausser, D. (1997). The Swiss Hidden Population Study: practical and methodological aspects of data collection by privileged access interviewers. Addiction, 92(3), 325–34.CrossRefGoogle ScholarPubMed
Lohr, S. L. (2010). Sampling: Design and analysis. Boston, MA: Brooks/Cole.Google Scholar
Loosveldt, G., & Storms, V. (2008). Measuring public opinions about surveys. International Journal of Public Opinion Research, 20(1), 74–89.CrossRefGoogle Scholar
McKenzie, D. J., & Mistianen, J. (2009). Surveying migrant households: a comparison of census-based, snowball, and intercept point surveys. Journal of the Royal Statistical Society: Series A (Statistics in Society), 172(2), 339–60.CrossRefGoogle Scholar
Maranda, F. (2004). Opening remarks. In Proceedings of the Symposium 2004: Innovative Methods for Surveying Difficult-to-Reach Populations (pp. 1–2). Ottawa: Statistics Canada.Google Scholar
Marpsat, M., & Razafindratsima, N. (2010). Survey methods for hard-to-reach populations: introduction to the Special Issue. Methodological Innovation Online, 5(2), 3–16.CrossRefGoogle Scholar
Michaels, S., & Lhomond, B. (2006). Conceptualizations and measurement of homosexuality in sex surveys: a critical review. Cadernos de Saude Publica, 22(7), 1365–74.CrossRefGoogle Scholar
Norris, F. H. (2006). Disaster research methods: past progress and future directions. Journal of Traumatic Stress, 19(2), 173–84.CrossRefGoogle ScholarPubMed
Pedersen, J. (1995). Drought, migration, and population growth in the Sahel: the case of the Malian Gourma, 1900–1991. Population Studies, 49(1), 111–26.CrossRefGoogle Scholar
Peng, D. (2011). Overcoming Challenges to Sample Design in Iraq. Paper presented at the Joint Statistical Meetings, Denver, CO.
Pennell, B.-E., Harkness, J., Levenstein, R., & Quaglia, M. (2010). Challenges in cross-national data collection. In Harkness, J. A., Braun, M., Edwards, B., Johnson, T. P., Lyberg, L., Mohler, P. Ph., Pennell, B.-E., & Smith, T. W. (eds.), Survey Methods in Multinational, Multiregional, and Multicultural Contexts (pp. 269–98). Hoboken, NJ: John Wiley & Sons.CrossRefGoogle Scholar
Pennell, B.-E., Levenstein, R., & Lee, H. J. (2010). Data collection: Cross-cultural survey guidelines. Retrieved from .
Pescosolido, B. (2012). Personal communication.
Platt, L., Wall, M., Rhodes, T., Judd, A., Hickman, M., Johnston, L. G. et al. (2006). Methods to recruit hard-to-reach groups: comparing two chain referral sampling methods for recruiting injecting drug users across nine studies in Russia and Estonia. Journal of Urban Health, 82(Suppl. 1), 39–53.CrossRefGoogle Scholar
Reed, J. S. (1975–76). Needles in haystacks: studying “rare” populations by secondary analysis of national sample surveys. Public Opinion Quarterly, 39(4), 514–22.CrossRefGoogle Scholar
Rodriquez, H., Quarantelli, E. L., & Dynes, R. (eds.). (2007). Handbook of Disaster Research. New York: Springer Verlag.CrossRef
Rossi, P. H. (1989). Down and Out in America: The Origins of Homelessness. Chicago, IL: University of Chicago Press.Google Scholar
Rothbart, G. S., Fine, M., & Sudman, S. (1982). On finding and interviewing needles in the haystack: the use of multiplicity sampling. Public Opinion Quarterly, 46(3), 408–21.CrossRefGoogle Scholar
Salganik, M. J., & Heckathorn, D. D. (2004). Sampling and estimation in hidden populations using respondent-driven sampling. Sociological Methodology, 34, 193–239.CrossRefGoogle Scholar
Skjak, K. K., & Harkness, J. (2003). Data collection methods. In Harkness, J. A., van de Vijver, F. J. R., & Mohler, P. Ph. (eds.), Cross-Cultural Survey Methods (pp. 179–93). Hoboken, NJ: John Wiley & Sons.Google Scholar
Smith, T. W. (1992). Discrepancies between men and women in reporting number of sexual partners: a summary from four countries. Social Biology, 39(3–4), 203–11.Google ScholarPubMed
Smith, T. W. (2007). Survey non-response procedures in cross-national perspective: the 2005 ISSP non-response survey. Survey Research Methods, 1(1), 45–54.Google Scholar
Smith, T. W. (2010). The globalization of survey research. In Harkness, J. A., Braun, M., Edwards, B., Johnson, T. P., Lyberg, L., Mohler, P. Ph., Pennell, B.-E., & Smith, T. W. (eds.), Survey Methods in Multinational, Multiregional, and Multicultural Contexts (pp. 477–84). Hoboken, NJ: John Wiley & Sons.Google Scholar
Smith, T. W. (2011a). Lessons on Developing Laws for Studying Societal Change. Paper presented to the European Survey Research Association, Lausanne, Switzerland.
Smith, T. W. (2011b). The report on the international workshop on using multi-level data from sample frames, auxiliary databases, paradata, and related sources to detect and adjust for nonresponse bias in surveys. International Journal of Public Opinion Research, 23(3), 389–402.CrossRefGoogle Scholar
Smith, T. W. (2011c). Trends in support for civil liberties. GSS Social Change Report No. 59. Chicago, IL: NORC.Google Scholar
Statistics Canada. (2004). Proceedings of Symposium 2004: innovative methods for surveying difficult-to-reach populations. Ottawa: Statistics Canada.Google Scholar
Stoop, I., Billiet, J., Kohn, A., & Fitzgerald, R. (2010). Improving survey response: lessons learned from the European Social Survey. Chichester: John Wiley & Sons.CrossRefGoogle Scholar
Sudman, S. (1972). On sampling of very rare human populations. Journal of the American Statistical Association, 67(338), 335–39.CrossRefGoogle Scholar
Titterton, M., & Clark, C. (2000). Working with socially excluded Romani communities in central and eastern Europe: lessons from Bulgaria. Social Work in Europe, 7(3), 38–45.Google Scholar
Tortora, R. D. (2012). Personal communication.
Tortora, R. D., Srinivasan, R., & Esipova, N. (2010). The Gallup World Poll. In Harkness, J. A., Braun, M., Edwards, B., Johnson, T. P., Lyberg, L., Mohler, P. Ph., Pennett, B.-E., & Smith, T. W. (eds.), Survey Methods in Multinational, Multiregional, and Multicultural Contexts (pp. 535–44). Hoboken, NJ: John Wiley & Sons.CrossRefGoogle Scholar
Treiman, D. J., Lym, Y., & Qi, Y. (2009). New Approaches to Demographic Data Collection. California Center for Population Research On-Line Working Paper Series. Los Angeles, CA: University of California–Los Angeles.Google Scholar
United Nations. (2009). UN Development Program, 2009. Retrieved from .
Van der Reis, P. (1997). Transportation Surveys among Illiterate and Semiliterate Households in South Africa. Paper presented to the International Conference on Transport Surveys, Grainau, Germany.
Vandepitte, R., Lyerla, R., Dallabetta, G., Crabbe, F., Alary, M., & Buvé, A. (2006). Estimates of the number of female sex workers in different regions of the world. Sexually Transmitted Infections, 82: Supplement 3, 18–25.CrossRefGoogle Scholar
Vigneswaran, D. (2007). Lost in Space: Residential Sampling and Johannesburg’s Forced Migrants. Paper presented to the African Migrations Workshop, Accra, Ghana.
WAPOR. (2011). Pollsters missing in Michoacan, Mexico. Retrieved from .
Weinreb, A. A. (2006). The limitations of stranger-interviewers in rural Kenya. American Sociological Review, 71(6), 1014–39.CrossRefGoogle Scholar
Williams, Linda A. 1984. The classic rape: when do victims report?Social Problems, 31, 459–67.CrossRefGoogle Scholar
Young, C. (2012). Personal communication.

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats
×

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats
×