¹ In their conclusion, the judges wrote, “[Patrick’s] expert report provides evidentiary support for the following reasons: a) It shows in statistical form that from April 1982 to July 1983, the army killed 5.5% of the indigenous people in the Ixil area. b) It confirms, in numerical form, what the victims said. c) It explains thoroughly the equation, analysis, and the procedure used to obtain the indicated result. d) The report establishes that the greatest number of indigenous deaths occurred during the period April 1982 to July 1983 when José Efraín Ríos Montt governed. e) The expert is a person with extensive experience in statistics.” Translation provided by Patrick Ball. A link to the complete opinion, in Spanish, is available via HRDAG’s website, https://hrdag.org/hat-tip-from-guatemala-judges/.

² Hagan, John, Schoenfeld, Heather, and Palloni, Alberto, “The Science of Human Rights, War Crimes, and Humanitarian Emergencies,” Annual Review of Sociology 32 (2006): 329–49.CrossRefGoogle Scholar Also see Helge Brunborg about the ways in which a demographer can contribute to a war crimes trial based on his experiences at the International Criminal Tribunal for the Former Yugoslavia in “Contribution of Statistical Analysis to the Investigations of the International Criminal Tribunals,” Statistical Journal of the United Nations (2001).

³ Patrick Ball, Wendy Betts, Fritz Scheuren, Jana Dudukovic, and Jana Asher, “Killings and Refugee Flow in Kosovo, March-June 1999 (A Report to ICTY),” published by the American Association for the Advancement of Science and the American Bar Association Central and East European Law Initiative (2002).

⁴ Hagan, Schoenfeld, and Palloni, “The Science of Human Rights,” 329–49. Note that Milošević died before his trial was complete, so there was no verdict.

⁵ For many examples, see Patrick Meier’s blog: http://irevolution.net/.

⁶ As just one example, see Mark Danner’s book The Massacre at El Mozote (New York: Vintage, 1994).

⁷ Susanne Karstedt proposes similar questions in her section on contextualizing extreme violence in “Contextualizing Mass Atrocity Crimes: Moving Toward a Relational Approach,” Annual Review of Law and Social Science, 9 (2013): 383–404.

⁸ Such as the data collected by Rwandan organization IBUKA in the Kibuye Prefecture as described in Philip Verwimp, “Machetes and Firearms: The Organization of Massacres in Rwanda,” Journal of Peace Research 43 (2006): 5–22; the Bosnian Book of the Dead as prepared by the Research and Documentation Center in Sarajevo; the Kosovo Memory Book database, as prepared by the Humanitarian Law Center, available at http://www.hlc-rdc.org/.

⁹ Verwimp, Philip, “Death and Survival During the 1994 Genocide in Rwanda,” Population Studies, 58 (2010): 233–45CrossRefGoogle Scholar; Romesh Silva and Patrick Ball, “The Profile of Human Rights Violations in Timor-Leste, 1974–1999: A Report by the Benetech Human Rights Data Analysis Group to the Commission on Reception, Truth and Reconciliation of Timor-Leste,” published in 2006 and available at https://hrdag.org/content/timorleste/Benetech-Report-to-CAVR.pdf, see particularly their discussion of the incompleteness of the cemetery records.

¹⁰ Patrick Ball, Ewa Tabeau, and Philip Verwimp, “The Bosnian Book of Dead: Assessment of the Database (Full Report)” HiCN Research Design Note 5, June 17, 2007.

¹¹ Jule Krüger and Patrick Ball, “Evaluation of the Database of the Kosovo Memory Book,” published in 2014 and available at https://hrdag.org/wp-content/uploads/2013/01/Evaluation_of_the_Database_KMB-2014.pdf.

¹² The Households in Conflict Network provides an in-depth description of different types of surveys and proposes ways to improve existing questionnaires to better understand violent conflict; see Tilman Brück, Patricia Justino, Philip Verwimp, and Alexandra Avdeenko, “Identifying Conflict and Violence in Micro-Level Surveys,” HiCN Working Paper 79 (2010).

¹³ It is worth noting here the discussion in chapter 8 of the Human Security Report 2009/2010 on the limitations and challenges of calculating excess mortality, particularly when data on baseline mortality rates may be out of date or unavailable.

¹⁴ An illustrative and by no means complete or representative list of examples includes: Paul B Spiegel and Peter Salama, “War and Mortality in Kosovo, 1998–99: An Epidemiological Testimony,” The Lancet 355 (2000); D. de Walque and P. Verwimp, “The Demographic and Socio-Economic Distribution of Excess Mortality during the 1994 Genocide in Rwanda,” Journal of African Economies 0 (2010): 1–22; John Hagan and Joshua Kaiser, “A Separate Peace: Explaining War, Crime, Violence, and Security During and After the Surge in Iraq,” May 16, 2013; Damien de Walque, “Selective Mortality During the Khmer Rouge Period in Cambodia,” Population and Development Review 31 (2005): 351–68; John Hagan, Wenona Rymond-Richmond, and Patricia Parker, “The Criminology of Genocide: the Death and Rape of Darfur,” Criminology 43 (2005); and Silva and Ball, “The Profile of Human Rights Violations in Timor-Leste.”

¹⁵ See Human Security Report 2009/2010 and Johnson, N., Spagat, M., Gourley, S., Onnela, J., and Reinert, G., “Bias in Epidemiological Studies of Conflict Mortality,” Journal of Peace Research 45 (2008): 653–64.CrossRefGoogle Scholar

¹⁶ Many introductory statistics texts cover this material in depth. Both Gerald van Belle, Statistical Rules of Thumb (New York: Wiley, 2002) and Sergey Dorofeev and Peter Grant, Statistics for Real-Life Sample Surveys (Cambridge: Cambridge University Press, 2006) are good places to start. The classic, and in our opinion still the most readable work on sampling is Leslie Kish, Survey Sampling (New York: Wiley, 1995), still in print after fifty years.

¹⁷ It is important to note that “bias” is not meant to connote judgement, but rather is used here in the statistical sense, meaning an empirical difference between what is observed and the complete picture of all events, were that knowable.

¹⁸ As an example, according to Vinck et al,’s 2011 study (available at http://www.peacebuildingdata.org/liberia/map), nationwide, only 2 percent of the population took part in the Liberian Truth Commission.

¹⁹ See Price, M. and Ball, P., “Big Data, Selection Bias, and the Statistical Patterns of Mortality in Conflict,” SAIS Review 34 (2014): 9–20.Google Scholar

²⁰ Biderman, A.D. and Reiss, A.J. Jr., “On Exploring the ‘Dark Figure’ of Crime,” Annals of the American Academy of Political and Social Science 374 (1967): 1–15.CrossRefGoogle Scholar Also see Morrison, William Douglas, “The Interpretation of Criminal Statistics,” Journal of the Royal Statistical Society 60 (1897): 1–32.CrossRefGoogle Scholar

²¹ See also Matthieu de Castelbajac, “Brooding Over the Dark Figure of Crime,” British Journal of Criminology 54 (2014): 928-45, and Mosher, C.J., Miethe, T.D., and Hart, T.C., The Mismeasure of Crime (Thousand Oaks, CA: Sage, 2011)CrossRefGoogle Scholar for more in-depth analyses of this topic in criminology.

²² See Castelbajac, “Brooding Over the Dark Figure of Crime.”

²³ Duren Banks, Lance Couzens, Caroline Blanton, and Devon Cribb, “Arrest-Related Deaths Program Assessment,” A technical report published by RTI International (2015) available at http://www.bjs.gov/content/pub/pdf/ardpatr.pdf.

²⁴ Patrick Ball, Jana Asher, David Sulmont, and Daniel Manrique, “How Many Peruvians Have Died?” Report published by the American Association for the Advancement of Science (2003).

²⁵ Ibid.

²⁶ See the Syria Regional Refugee Response Inter-Agency Information Sharing Portal at http://data.unhcr.org/syrianrefugees/regional.php.

²⁷ Megan Price, Jeff Klingner, and Patrick Ball, “Preliminary Statistical Analysis of Documentation of Killings in the Syrian Arab Republic,” published by the Benetech Human Rights Program, commissioned by the United Nations Office of the High Commissioner for Human Rights (2013); Megan Price, Jeff Klingner, Anas Qtiesh, and Patrick Ball, “Full Updated Statistical Analysis of Documentation of Killings in the Syrian Arab Republic,” published by the Human Rights Data Analysis Group, commissioned by the United Nations Office of the High Commissioner for Human Rights (2013); Megan Price, Anita Gohdes, and Patrick Ball, “Updated Statistical Analysis of Documentation of Killings in the Syrian Arab Republic,” published by the Human Rights Data Analysis Group, commissioned by the United Nations Office of the High Commissioner for Human Rights (2014).

²⁸ http://www.csr-sy.com/.

²⁹ http://www.syrianhr.org/.

³⁰ http://syrianshuhada.com/.

³¹ http://www.vdc-sy.info/.

³² On “macro-truth,” see Chapman, Audrey and Ball, Patrick, “The Truth of Truth Commissions: Comparative Lessons from Haiti, South Africa, and Guatemala,” Human Rights Quarterly 23 (2001): 1–42.CrossRefGoogle Scholar

³³ As reported by The Guardian (http://www.theguardian.com/world/2011/aug/07/syrian-government-forces-storm-central-town) and Al Jazeera (http://www.aljazeera.com/news/middleeast/2011/08/20118613846890458.html, http://www.aljazeera.com/news/middleeast/2011/08/20118785421245255.html).

³⁴ As reported by Agence France-Presse, see http://english.alarabiya.net/articles/2012/12/17/255673.html.

³⁵ See http://www.mcclatchydc.com/2013/02/01/181784/syrian-government-offensive-forces.html.

³⁶ See Patrick Ball, Wendy Betts, Fritz Scheuren, Jana Dudukovic, and Jana Asher, “Killings and Refugee Flow in Kosovo, March-June 1999 (A Report to ICTY),” (AAAS, 2002). It is worth noting that the estimate of total killings published by AAAS is largely consistent with results produced via more traditional survey methods (Spiegel and Salama 2000) and full enumeration (Kosovo Memory Book) (see Nicholas P. Jewell, Michael Spagat, and Britta L. Jewell. “MSE and Casualty Counts: Assumptions, Interpretation, and Challenges,” in Counting Civilian Casualties edited by Taylor B. Seybolt, Jay D. Aronson, and Baruch Fischhoff [Oxford: Oxford University Press, 2013] and Michael Spagat, “A Triumph of Remembering: Kosovo Memory Book” available at http://www.kosovskaknjigapamcenja.org/wp-content/uploads/2015/02/Michael-_Spagat_Evaluation_of_the_Database_KMB_December_10_2014.pdf).

³⁷ See “De la Locura a la Esperanza: La guerra de 12 años en El Salvador,” (United Nations, 1993).

³⁸ For the CIIDH, see P. Ball, P. Kobrak, and H.F. Spirer, State Violence in Guatemala (AAAS, 1999). For CCJ, see “Cómo procesa su información la Comisión Colombiana de Juristas,” (no date) available online at http://www.coljuristas.org/documentos/adicionales/como_procesa_su_informacion_la_ccj.pdf, as well as the CCJ’s periodic iterations of the “Informe sobre la situación de derechos humanos” reports published throughout the late 1990s and 2000s.

³⁹ See for example, Hagan et al., “Neighborhood Sectarian Displacement and the Battle for Baghdad: A Self-Fulfilling Prophecy of Crimes against Humanity in Iraq,” and Philip Verwimp, “Testing the Double-Genocide Thesis for Central and Southern Rwanda,” Journal of Conflict Resolution 47 (2003).

⁴⁰ For example, see the Iraq Body Count, https://www.iraqbodycount.org/, and the Computational Event Data System, http://eventdata.parusanalytics.com/index.html.

⁴¹ For reviews of the problem see Winkler, William E., “Overview of Record Linkage and Current Research Directions,” a technical report for the Statistical Research Division, US Census Bureau (2006) and Herzog, Thomas N., Scheuren, Fritz J., and Winkler, William E., Data Quality and Record Linkage Techniques (New York: Springer, 2007).Google Scholar

⁴² Peter Christen’s 2012 book Data Matching—Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection (Springer) is currently the canonical reference for this class of problems.

⁴³ See Matthew Michelson and Craig A. Knoblock, “Learning Blocking Schemes for Record Linkage,” in Proceedings of the 21st National Conference on Artificial Intelligence (2006).

⁴⁴ It is worth noting that there are many different ways to compare names, and many of these ways are project specific. For example, different documentation efforts may record a different number of names (family name, father’s name) in a different order, so name comparisons may be made across different combinations of recorded names. Additionally, “Muhammad” (with various spellings) is a very common name in Syria, so comparisons may be calculated both including and excluding this name.

⁴⁵ Sunita Sarawagi and Anuradha Bhamidipaty, “Interactive Deduplication Using Active Learning,” in Proceedings of the eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2002).

⁴⁶ See Price et al., “Full Updated Statistical Analysis of Documentation of Killings,” and Price, Gohdes, and Ball “Updated Statistical Analysis of Documentation of Killings,” for an in-depth description of record linkage for this case study, including the high level of agreement between different individuals reviewing records in different languages (English and Arabic).

⁴⁷ MSE is also called capture-recapture, or mark-recapture in the ecology literature (see the overview of modern applications in ecology in Handbook of Capture-Recapture Analysis, edited by S.C. Amstrup, T.L. McDonald, and B.F.J. Manly [Princeton: Princeton University Press, 2005]).

⁴⁸ Petersen, C.G.J., “The Yearly Immigration of Young Plaice Into the Limfjord from the German Sea,” Report of the Danish Biological Station(1895) 6(1896): 5–84.Google Scholar

⁴⁹ Lincoln, F.C., “Calculating Waterfowl Abundance on the Basis of Banding Returns,” Circular 118 (1930): 1–4.Google Scholar

⁵⁰ Chandra Sekar, C. and Edwards Deming, W., “On a Method of Estimating Birth and Death Rates and the Extent of Registration,” Journal of the American Statistical Association 245 (1949): 101–15.CrossRefGoogle Scholar

⁵¹ Seber, G.A.F., “A Note on the Multiple Recapture Census,” Biometrika 52 (1965): 249–59.CrossRefGoogle ScholarPubMed

⁵² Marks, Eli S., Seltzer, William, and Krótki, Karol J., Population Growth Estimation: A Handbook of Vital Statistics Measurement (published by The Population Council, 1974).Google Scholar

⁵³ Darroch, John N., Fienberg, Stephen E., Glonek, Gary F. V., and Junker, Brian W., “A Three-Sample Multiple-Recapture Approach to Census Population Estimation with Heterogeneous Catchability,” Journal of the American Statistical Association 88 (1993): 1137–48.CrossRefGoogle Scholar

⁵⁴ Wittes, Janet T. and Sidel, Victor W., “A Generalization of the Simple Capture-Recapture Model with Applications to Epidemiological Research,” Journal of Chronic Disease 21 (1968): 287–301.CrossRefGoogle ScholarPubMed

⁵⁵ Wittes, Janet T., Colton, Theodore, and Sidel, Victor W., “Capture-Recapture Methods for Assessing the Completeness of Case Ascertainment When Using Multiple Information Sources,” Journal of Chronic Disease 27 (1974): 25–36.CrossRefGoogle ScholarPubMed

⁵⁶ International Working Group for Disease Monitoring and Forecasting, “Capture-Recapture and Multiple-Record Systems Estimation I: History and Theoretical Development,” American Journal of Epidemiology 142 (1995): 1047–58.

⁵⁷ International Working Group for Disease Monitoring and Forecasting, “Capture-Recapture and Multiple-Record Systems Estimation II: Applications in Human Diseases,” American Journal of Epidemiology 142 (1995): 1059–68.

⁵⁸ Hook, E.B. and Regal, R.R., “Accuracy of Alternative Approaches to Capture-Recapture Estimates of Disease Frequency: Internal Validity Analysis of Data from Five Sources,” American Journal of Epidemiology 152 (2000): 771–79.CrossRefGoogle ScholarPubMed

⁵⁹ Seber, George A.F., Huakau, John T., and Simmons, David, “Capture-Recapture, Epidemiology, and List Mismatches: Two Lists,” Biometrics 56 (2000): 1227–32.CrossRefGoogle ScholarPubMed

⁶⁰ Ball et al., “Killings and Refugee Flow in Kosovo, March-June 1999.”

⁶¹ Ball et al., “How Many Peruvians Have Died?”

⁶² Brunborg, Helge, Lynstad, Torkild Hovde, and Urdal, Henrik, “Accounting for Genocide: How Many Were Killed in Srebrenica?” European Journal of Population 19 (2003): 229–48.CrossRefGoogle Scholar

⁶³ Lum, Kristian, Price, Megan, Guberek, Tamy, and Ball, Patrick, “Measuring Elusive Populations with Bayesian Model Averaging for Multiple Systems Estimation: A Case Study on Lethal Violations in Casanare, 1998–2007,” Statistics, Politics, and Policy 1 (2010).CrossRefGoogle Scholar

⁶⁴ Jan Zwierzchowski and Ewa Tabeau, “The Global Costs of Conflict,” paper presented at the International Research Workshop, Berlin (2010).

⁶⁵ Manrique-Vallier, D. and Fienberg, S., “Population Size Estimation Using Individual Level Mixture Models,” Biometrical Journal 50 (2008): 1051–63.CrossRefGoogle ScholarPubMed

⁶⁶ See chapter 6 in Bishop, Y.M.M., Fienberg, S.E., and Holland, P.W., Discrete Multivariate Analysis (Cambridge, MA: MIT, 1975).Google Scholar

⁶⁷ Results presented in the rest of this paper first appeared in Megan Price, Anita Gohdes, and Patrick Ball, “Documents of War: Understanding the Syrian Conflict,” Significance 12 (2015): 14–19.

⁶⁸ Tamy Guberek, Daniel Guzmán, Megan Price, Kristian Lum, and Patrick Ball, “To Count the Uncounted: An Estimation of Lethal Violence in Casanare,” (2006); Kristian Lum, Megan Emily Price, and David Banks, “Applications of Multiple Systems Estimation in Human Rights Research,” The American Statistician 67 (2013): 191–200; and other projects referenced earlier. It is worth noting that in addition to the casualty estimates in Kosovo discussed in Jewell, Spagat, and Jewell, “MSE and Casualty Counts,” and Spagat, “A Triumph of Remembering,” our MSE estimates in Timor Leste were also consistent with other estimation methods, see Silva, Romesh and Ball, Patrick, “The Demography of Conflict-Related Mortality in Timor-Leste (1974-1999): Empirical Quantitative Measurement of Civilian Killings, Disappearances & Famine-Related Deaths,” in Statistical Methods for Human Rights edited by Asher, J., Banks, D., and Scheuren, F. (New York: Springer, 2007).Google Scholar