In the focal article “Getting Rid of Performance Ratings: Genius or Folly? A Debate,” two groups of authors argued the merits of performance ratings (Adler et al., 2016). Despite varied views, both sides noted the importance of including multiple raters to obtain more accurate performance ratings. As the pro side noted, “if ratings can be pooled across many similarly situated raters, it should be possible to obtain quite reliable assessments” (Adler et al., p. 236). Even the con side noted, “In theory, it is possible to obtain ratings from multiple raters and pool them to eliminate some types of interrater agreement” (Adler et al., p. 225), although this side was certainly less optimistic about the merits of multiple raters. In the broader industrial–organizational psychology literature, authors have repeatedly heralded the benefits of adding additional raters for performance ratings, some even treating it as a panacea for inaccurate ratings. Although these authors extol the virtues of multiple raters, an important question is often omitted from relevant discussions of performance ratings: To what extent do additional raters actually improve performance ratings? Does adding an additional rater double the validity of performance ratings? Does an additional rater increase the validity of performance ratings by a constant value? Or is the answer something else altogether?

Within the context of the 50th anniversary of the Civil Rights Act of 1964, and specifically as it pertains to the tenets of Title VII, Lindsey, King, Dunleavy, McCausland, and Jones (2013) state: “This focal article raises and addresses critical issues regarding a yet unanswered question: How can organizational researchers and practitioners contribute to the ultimate goal of eradicating employment discrimination” (p. 391). We argue that in the context of employment testing and selection, at least as per the disparate impact theory of discrimination, this question is the wrong one—certainly as framed by Lindsey et al. To the contrary, instead of holding up the "eradication of employment discrimination" as our ultimate goal, perhaps we should continue to focus on the development, implementation, and support of the best (i.e., most job-related and valid) employment practices possible.

Research on gender differences in the allocation and evaluation of three main components of academic work (i.e., research, teaching, and service) is very informative, and we argue that it may point to potential sources of the disparity in academic industrial and organizational (I-O) psychologists’ experiences. We also propose the addition of a few issues to the research agenda set forth by Gardner, Ryan, and Snoeyink (2018) with the belief that attention to these issues will help address the gender disparity.

Through learning about and doing job analysis, industrial–organizational (I-O) psychologists likely already possess skills and knowledge relevant to doing and understanding qualitative research. We'll illustrate this by showing similarities between common job analysis practices and one particular qualitative research approach likely to be relevant to organizational research: grounded theory. Grounded theory was “discovered” in 1967 by Barney Glaser and Anselm Strauss (Glaser & Strauss, 1967). Though Glaser and Strauss later split in their methodologies (an occurrence not unlike the varied approaches to job analysis), the core idea of grounded theory is to develop a new theory of some process or phenomenon from the “ground” up. In the grounded theory approach, researchers typically collect mostly qualitative data—often including interviews (Creswell, 2007)—and simultaneously develop increasingly abstract codes, concepts, and categories from the data. In the final step of analysis, researchers develop a theory that subsumes all categories from the data. If researchers follow the Straussian tradition, categories can be fit into a theoretical framework that details a central phenomenon underlying the process of interest and the conditions that precede it, result from it, and shape the resulting categories (Creswell, 2007). We illustrate this framework in Figure 1. Grounded theory is particularly useful for developing an accurate understanding of many organizational processes and phenomena that I-O psychologists study.

Much ink has been spilled on the scientist–practitioner gap, that is, the apparent divide between knowledge published in academic peer-reviewed journals and the actual business practices employed in modern organizations. Most prior papers have advanced meaningful theories on why the gap exists, ranging from poor communication skills on the part of academics to paywalls and other obstacles preventing the public from accessing research in industrial-organizational psychology (I-O). However, very few papers on the scientist–practitioner gap have taken an empirical approach to better understand why the gap exists and what can be done about it. In our focal article, we specifically discuss the gap as it pertains to small businesses and present empirical data on the topic. Drawing from our experiences working with and in small businesses before entering a PhD program, we suggest that a primary reason for the existence of this gap is the differences between large and small businesses, and we advance two theory-driven reasons for why this is the case. Next, we compiled abstracts and practical implications sections from articles published in top I-O journals in the past 5 years, then we collected ratings and open-ended text responses from subject matter experts (i.e., small business owners and managers) in reaction to reading these sections. We close by recommending several potential perspectives, both for and against our arguments, that peer commentators can take in their responses to our focal article.

The company I work for is one of the leading providers of performance management technology (Jones & Wang-Audia, 2013). This technology is used by more than 3,000 organizations worldwide, including several of the companies mentioned in Adler et al. (2016). The technology is highly configurable. It is currently being used to support performance management processes with no annual manager ratings, processes with traditional annual rating evaluations, processes that only evaluate competencies, processes that only evaluate goal accomplishment, processes that mix goals and competencies, processes that require forced-ranked comparisons between employees, processes that make no direct comparisons between employees, and much more. The capabilities of this and other human resources (HR) technology systems are allowing companies to radically rethink performance management because they enable companies to do things far differently from what was possible when they were constrained to more fixed electronic or paper forms (Hunt, 2011, 2015a). The result is an explosion in the diversity of approaches being taken toward performance management design.

The focal article (Grand et al., 2018) provides an exemplary roadmap for improving the practice of robust science in industrial-organizational (I-O) psychology, which includes recommendations for authors such as practicing robust science, even in the absence of reward. However, authors are faced with practical constraints; to succeed in our field, we must publish, at least occasionally, in prestigious journals to secure scholarships, awards, employment, grants, and tenure. We can practice robust science (commonly referred to as “open science” or “reproducible research”) without reward—but only to some extent, if there is no support from our gatekeepers. After all, it is difficult to influence the field without a job and research funding.