In their focal article, Chamorro-Premuzic, Winsborough, Sherman, and Hogan (2016) provide an overview of a number of new technologies with potentially significant implications for talent management related practices of industrial–organizational (I-O) psychology (both challenges and opportunities) that they label as “new talent signals.” These signals, they argue, are part of a revolution, due to increasing levels of social activity online as well as data-collection and mining techniques that have overtaken the conventional practice of talent identification in organizations. Their position is that these trends are leaving I-O psychologists in the dust in terms of our existing traditional theory, research, and methodologies. Their optimistic tone, however, also seems to suggest that while these approaches “have not yet demonstrated validity comparable with old school methods, they tend to disregard theory, and they pay little attention to the constructs being assessed (Chamorro-Premuzic et al., p. 634)” they may, in fact, have some legitimate basis in the identification of talent. Their point, of course, is that momentum in this area has surpassed human resources (HR), let alone I-O psychology, so that any concerns one might have about these trends are essentially “irrelevant.” This reminds us of a reference to lemmings going off a cliff together.

During the fall 2015 semester, I (i.e., the last author of this response) taught a doctoral seminar on performance appraisal. Although this course was a general survey of research and theory regarding work performance and performance appraisal processes and methods, we also talked extensively about the value of performance ratings to organizations, raters, and ratees. It was indeed serendipitous that this focal article came out when it did. As part of the final examination requirements (and, admittedly, as a pedagogical experiment), I asked the six PhD students in this course (i.e., the first six authors of this response) to read and respond to the Adler et al. (2016) debate regarding the relative merits of performance ratings. To highlight the perspectives of this next generation of industrial and organizational psychologists, I have collected here various representative comments offered by each of these emerging scholars on this issue.

The focal article by Bergman and Jean (2016) raises an important issue by documenting the underrepresentation of nonprofessional and nonmanagerial workers in industrial and organizational (I-O) research. They defined workers as, “people who were not executive, professional or managerial employees; who were low- to medium-skill; and/or who were wage earners rather than salaried” (p. 89). This definition encompasses a wide range of employee samples: from individuals working in blue-collar skilled trades like electricians and plumbers to police officers, soldiers, and call center representatives to low-skill jobs such as fast food, tollbooth operators, and migrant day workers. Because there is considerable variability in the pay, benefits, skill level, autonomy, job security, schedule flexibility, and working conditions that define these workers’ experiences, a more fine-grained examination of who these workers are is necessary to understand the scope of the problem and the specific subpopulations of workers represented (or not) in existing I-O research.

We agree with Bracken, Rose, and Church (2016) and others that a critical design feature of any 360° feedback process is accountability, where the goal is “creation of sustainable individual, group, and/or organizational change in behaviors valued by the organization” (p. 764). Though we acknowledge the important roles that the organization and raters play in holding leaders accountable for their development, the goal of our commentary is to expand on how the leader's boss and other key individuals can serve as powerful sources of accountability in the 360° feedback process and throughout a leader's development journey. We also want to note that although the Center for Creative Leadership (CCL) encourages leaders to share what they have learned from their 360° feedback with their bosses and other accountability partners (e.g., peers), it is the leader's choice as to whether he or she shares key feedback with others. This practice ensures confidentiality of the data, helping leaders trust the process and increasing the likelihood that individuals accept difficult feedback and use it for performance improvement (Fleenor, Taylor, & Chappelow, 2008; King & Santana, 2010).

In 2008, we saw an emerging business need to accurately identify and develop individuals early in their careers who have the potential to be effective leaders in later organizational positions. We decided early on to take a comprehensive and systematic approach to the challenge in order to build effective solutions with real organizational impact that are sustainable over time. We knew that this was a complex problem and that there was little agreement in industrial–organizational (I-O) psychology on how to approach this critical business need and what approaches would be most effective. Since then, we moved through the following process stages to arrive at effective solutions that are now being used in numerous organizations.

Science and technology often work hand in hand to drive innovations in the practice of industrial–organizational (I-O) psychology. Historically, psychological science applied to workforce management allowed us to weather the storm of human resource trends and technological advances that attracted organizations to new tools and methods. But what happens when the lure of technology trumps the science? The rise of technological advances challenges our thinking and leads to psychological research and theory designed to keep pace with these advances. But in some instances, the adoption of new technology has outpaced scientific best practices. Organizations will adopt new technology in the absence of best practices if there are perceived benefits. Has the array of new talent identification tools described by Chamorro-Premuzic, Winsborough, Sherman, and Hogan (2016) presented a tipping point for I-O in terms of how we approach our science, research, tools, and methods?

Bergman and Jean's (2016) focal article decries the limited research attention of industrial and organizational (I-O) psychologists on “workers”—that is, employees such as wage earners, frontline workers, and contractors, who do not fill professional, managerial, or executive positions. We agree. In addition to the scientific and moral benefits of studying workers, there is a practical imperative. An academic discipline that comes across as being disinterested in workers may leave itself open to charges of being the “handmaiden” of management (Hulin, 2002, p. 12). Moreover, such an academic discipline may be ill prepared to provide evidence-based contributions to important societal debates on topics such as income inequality and immigration.

Adler et al. (2016) open with a summary of the business case driving our field to change and close by providing principles for accomplishing that change, where they conclude that “there is no right answer to the ratings question” (p. 244). Lying between the opening and closing sections is a series of arguments for and against today's performance rating status quo, arguments illustrating just what happens when too many years are spent seeking answers along too narrow a path. In this commentary, we provide additional support for the strategy- and outcome-driven approach to performance management advocated in the article. In addition, we offer ideas for what has contributed to getting us and keeping us where we are. Unless we understand what has driven performance ratings research and practice to be the object of an intense and lengthy debate, these same forces may well drive us to carry out years-long experiments of questionable value along similarly narrow paths. We want to offer our views on how to foster outcome-based practice more broadly.

Chamorro-Premuzic, Winsborough, Sherman, and Hogan (2016) describe a variety of new selection approaches (e.g., “scraping” of social media information, gamified assessments) in the staffing domain that might provide new sources of information about people. The authors also mention advantages and downsides of these potentially “new talent signals.”

In their debate about whether companies should keep traditional, numerical ratings of employee performance in an ever-changing world of work, Colquitt and colleagues argued that performance ratings are too hard to do correctly, while Adler and colleagues (Adler et al., 2016) countered that “‘too hard’ is no excuse for I-O psychology.” I would like to build on this by suggesting that “too hard” is not only no excuse but also a complete dismissal of the central aspect of what our science seeks to achieve.

Two recent focal articles in this journal have addressed issues related to sample selection and generalizability of results (Bergman & Jean, 2016; Landers & Behrend, 2015). If Bergman and Jean are correct, gone are the days of the Hawthorne studies in which research focused on the majority of the human workforce: the working class. Instead, researchers are allegedly two to three times as likely to exclusively sample managers as they are to exclusively sample workers. Assuming this is true, Bergman and Jean are correct to address why this occurs and how it may impact the field. However, there are two critical issues that must be considered alongside these questions: ongoing changes in how work is conducted and temporal trends in work. A consideration of these issues should yield additional insights that may supplement the recommendations made by Bergman and Jean.

Chamorro-Premuzic, Winsborough, Sherman, and Hogan (2016) note that new talent signals recently adopted by organizations are related to older selection and assessment methods. Drawing this connection between old and new technologies is helpful; however, viewing new technology as either shiny new objects or a brave new world creates a false dichotomy. Recent technology-enhanced human resources (HR) processes like the widespread use of gamified practices and video-recorded interviewing are not just fads or the beginning of a transformation in HR but rather natural evolutions of methods that differ across specific dimensions that can be identified and measured. It is important to view these recent advances as extensions of the existing methods. That is, we need to focus on how these new methods are different and not on that they are different.

Adler et al. (2016) raise some controversial issues about whether performance rating systems should be eliminated or not. We strongly believe that the decision to do away with performance ratings is premature because more research needs to be done, as suggested by “the better questions” that Adler et al. listed at the end of the focal article. We propose that those questions can be extended further by testing them in other cultures and supplemented with these questions: When, how, and why do cultural values influence performance management? Given the nature of our increasingly diverse workforce in which employees with different cultural values work together within the same organization, it is crucial to identify and document the influence of culture on performance appraisal practices. In this commentary, we briefly summarize research to date on the influence of cultural values on performance management and identify some important future research directions.

Bergman and Jean (2016) have contributed an important essay to the continuing self-reflection and maturation of the field of industrial–organizational (I-O) psychology—or as it is known in much of the world outside the United States, work psychology.1 They clearly and adequately document that the field has relatively neglected to study the world of (largely lower-level) workers who are not managers, executives, professionals, or students and that this has affected adversely the validity of our science and the relevance of our professional practice in a number of not-so-intuitively obvious ways. But as critical as those observations are, I believe the most important aspect of their piece has to do with the inferences they offer as to why our published literature is so skewed. They suggest six potential, not mutually exclusive, explanations, including the possibility of personal biases among I-O psychologists. However, before focusing on those explanations, it should be informative to place the Bergman/Jean thesis in context. There is a growing, recent body of critical evidence and/or commentary concerning this and similar issues—although less consideration generally has been given to their likely causes.

Few would dispute that the nature of work, and the workers who perform it, has evolved considerably in the 70 years since the founding of the Society for Industrial and Organizational Psychology (SIOP) as the American Psychological Association's (APA's) Division 14, focused on industrial, business, and organizational psychology. Yet, in many ways the populations sampled in industrial–organizational (I-O) psychology research have failed to keep pace with this evolution. Bergman and Jean (2016) highlight how I-O research samples underrepresent (relative to the labor market) low- or medium-skill workers, wage earners, and temporary workers, resulting in a body of science that is overly focused on salaried, professional managers and executives. Though these discrepancies in the nature of individuals’ work and employment are certainly present and problematic in organizational research, one issue that should not be overlooked is that of adequately representing nationality and culture in I-O research samples.

In their focal article, Adler and his colleagues (2016) elaborate on the pros and cons of abolishing the performance appraisal process in organizations. Sherman-Garr (2014) contends that this trend is on the rise because both managers—the raters—and their subordinates—the ratees—disdain performance scores. Employees feel that performance ratings do not reflect their actual performance, and therefore they do not gain the rewards they merit. Conversely, their supervisors/managers experience a great deal of frustration because the improvement of employee performance does not match up to the excessive time and effort invested in the appraisal process, making the whole process ineffective and inefficient. We agree that performance appraisals, specifically the practice of assigning performance ratings, appear to be a disliked and ineffective human resource function. However, we do not agree that goal attainment should be used in place of performance ratings; rating format and rater training represent “disappointing interventions” and, most of all, only “weak” criteria exist for performance ratings.

Describing the current state of gamification, Chamorro-Premuzic, Winsborough, Sherman, and Hogan (2016) provide a troubling contradiction: They offer examples of a broad spectrum of gamification interventions, but they then summarize the entirety of gamification as “the digital equivalent of situational judgment tests.” This mischaracterization grossly oversimplifies a rapidly growing area of research and practice both within and outside of industrial–organizational (I-O) psychology. We agree that situational judgment tests (SJTs) can be considered a type of gamified assessment, and gamification provides a toolkit to make SJTs even more gameful. However, the term gamification refers to a much broader and potentially more impactful set of tools than just SJTs, which are incremental, versatile, and especially valuable to practitioners in an era moving toward business-to-consumer (B2C) assessment models. In this commentary, we contend that gamification is commonly misunderstood and misapplied by I-O psychologists, and our goals are to remedy such misconceptions and to provide a research agenda designed to improve both the science and the practice surrounding gamification of human resource processes.

Adler et al. (2016) provide a discussion of the pros and cons surrounding the issue of “Getting Rid of Performance Ratings.” Yet neither the pro nor the con side of the debate appears to fully consider the central role of performance ratings outside the realm of performance management.