We can make better decisions when we have a better understanding of the different sources of variance that impact job performance ratings. A failure to do so cannot only lead to inaccurate conclusions when interpreting job performance ratings, but often misguided efforts aimed at improving our ability to explain and predict them. In this paper, we outline six recommendations relating to the interpretation of predictive validity coefficients and efforts aimed at predicting job performance ratings. The first three focus on the need to evaluate the effectiveness of selection instruments and systems based only on the variance they can possibly account for. When doing so, we find that it is not only possible to account for the majority of the variance in job performance ratings that most select systems can possibly predict, but that we’ve been able to account for this variance for years. Our last three recommendations focus on the need to incorporate components related to additional sources of variance in our predictive models. We conclude with a discussion of their implications for both research and practice.

Much ink has been spilled on the scientist–practitioner gap, that is, the apparent divide between knowledge published in academic peer-reviewed journals and the actual business practices employed in modern organizations. Most prior papers have advanced meaningful theories on why the gap exists, ranging from poor communication skills on the part of academics to paywalls and other obstacles preventing the public from accessing research in industrial-organizational psychology (I-O). However, very few papers on the scientist–practitioner gap have taken an empirical approach to better understand why the gap exists and what can be done about it. In our focal article, we specifically discuss the gap as it pertains to small businesses and present empirical data on the topic. Drawing from our experiences working with and in small businesses before entering a PhD program, we suggest that a primary reason for the existence of this gap is the differences between large and small businesses, and we advance two theory-driven reasons for why this is the case. Next, we compiled abstracts and practical implications sections from articles published in top I-O journals in the past 5 years, then we collected ratings and open-ended text responses from subject matter experts (i.e., small business owners and managers) in reaction to reading these sections. We close by recommending several potential perspectives, both for and against our arguments, that peer commentators can take in their responses to our focal article.

Anti-work philosophy holds that work, in and of itself, tends to be harmful for most people. Some anti-work theorists even advocate for the abolition of paid employment altogether. We argue that, while endorsement of the radical ideology of anti-work is in no way necessary for I/O psychologists, considering the thinking behind these ideas can be beneficial. In fact, reviewing the tenets of anti-work may prompt some to a broad reconsideration of the nature and purpose of the I/O field and its role, nested as it is in potentially problematic power dynamics both within organizations and in broader society. In this article, after describing anti-work’s core tenets, we outline a number of research directions and practical applications inspired by this perspective. While in some cases these may involve the creation of new theory, constructs, and interventions, they often simply entail the repurposing or refocusing of existing ones that are more attuned to the problematic nature of work. Possibilities for research include, but are not limited to, the examination of the prevalence and nature of “managerialism,” how we might better understand the psychological character of organized labor and its outcomes, and how to encourage healthier manifestations of employee engagement. In terms of practice, we bring to the reader’s attention how anti-work might inspire extensions or adjustments in how we recruit and onboard, train managers, improve job characteristics, measure performance and work with unions and other political advocates. Ultimately, consideration of anti-work’s assertion of the inevitable authoritarian character of employment, combined with I/O psychology’s emphases on objectivity and the translation of science into practice, can spark inquiry and innovation.

Although assessment centers (ACs) are usually designed to measure stable competencies (i.e., dimensions), doubt about whether or not they reliably do so has endured for 70 years. Addressing this issue in a novel way, several published Generalizability (G) theory studies have sought to isolate the multiple sources of variance in AC ratings, including variance specifically concerned with competencies. Unlike previous research, these studies can provide a definitive answer to the AC construct validity issue. In this article, the historical context for the construct validity debate is set out, and the results of four large-scale G-theory studies of ACs are reviewed. It is concluded that these studies demonstrate, beyond reasonable doubt, that ACs do not reliably measure stable competencies, but instead measure general, and exercise-related, performance. The possibility that ACs measure unstable competencies is considered, and it is suggested that evidence that they do so may reflect an artefact of typical AC design rather than a “real” effect. For ethical, individual, and organizational reasons, it is argued that the use of ACs to measure competencies can no longer be justified and should be halted.

As organizations are becoming increasingly aware of the importance of selecting inclusive leaders, this paper proposes measuring inclusive leadership proficiency as an assessment center (AC) dimension. We propose that ACs present a novel way to effectively assess inclusive leadership using interpersonal behavioral simulation exercises, such as role plays. It is argued that AC-measured inclusive leadership can provide incremental prediction of leader performance beyond commonly assessed AC dimensions; and that it positively predicts follower performance and follower demographic diversity. We conclude by suggesting ways future research might empirically investigate the validity and reliability of AC-measured inclusive leadership in organizational settings.

This study generates new insights on the role of initial impressions in assessment centers. Drawing from the “thin slices” of behavior paradigm in personality and social psychology, we investigate to what extent initial impressions of assessees—based on different slices of assessment center exercises (i.e., two minutes at the beginning, middle, and end of AC exercises)—are consistent across and within AC exercises, and are relevant for predicting assessment center performance and job performance. Employed individuals (N = 223) participated in three interactive assessment center exercises, while being observed and evaluated by trained assessors. Based upon video-recordings of all assessment center exercises, a different, untrained group of raters subsequently provided ratings of their general initial impressions of assessees for the beginning, middle, and end of each exercise. As criterion measure, supervisors rated assessees’ job performance. Results show that initial impressions in assessment centers are (a) relatively stable, (b) consistently predict assessment center performance across different slices of behavior (i.e., across the three time points and exercises), and (c) mostly relate to job performance.

Evidence from previous research suggests that frame-of-reference (FOR) training is effective at improving assessor ratings in many organizational settings. Yet no research has presented a thorough examination of systematic sources of variance (assessor-related effects, evaluation settings, and measurement design features) that might influence training effectiveness. Using a factorial ANOVA and variance components analyses on a database of four studies of frame-of-reference assessor training, we found that (a) training is most effective at identifying low levels of performance and (b) the setting of the training makes little difference with respect to training effectiveness. We also show evidence of the importance of rater training as a key determinant of the quality of performance ratings in general. Implications for FOR training theory and practice are discussed.