1.1 Old Habits Die Hard

Several factors in the past combined to create conditions that encouraged researchers to avoid open science practices. First, incentives in academic contexts have not historically rewarded such behaviors and, in some cases, may have actually punished them (Reference Smaldino and McElreathSmaldino & McElreath, 2016). To get ahead in an academic career, publications are the coin of the realm, and jobs, promotions, and accolades can sometimes be awarded based on number of publications, rather than publication quality.

Second, human biases conspire to fool us into thinking we have discovered something when we actually have not (Reference BishopBishop, 2020). For instance, confirmation bias allows us to selectively interpret results in ways that support our pre-existing beliefs or theories, which may be flawed. Self-serving biases might cause defensive reactions when critics point out errors in our methods or conclusions. Adopting open science practices can expose researchers to cognitive discomfort (e.g., pre-existing beliefs are challenged; higher levels of transparency mean that critics are given ammunition), which we might naturally seek to avoid.

Finally, psychology uses an apprenticeship model of researcher training, which means that the practices of new researchers might only be as good as the practices of the more senior academics training them. When questionable research practices are taught as normative by research mentors, higher-quality open science practices might be dismissed as methodological pedantry.

Given the abundant evidence of flaws in psychology’s collective body of knowledge, we now know how important it is to overcome the hurdles described here and transition to a higher standard of practice. Incentives are changing, and open science practices are becoming the norm at many journals (Reference Nosek, Alter, Banks, Borsboom, Bowman, Breckler, Buck, Chambers, Chin, Christensen, Contestabile, Dafoe, Eich, Freese, Glennerster, Goroff, Green, Hesse, Humphreys and YarkoniNosek et al., 2015). A new generation of researchers is being trained to employ more rigorous practices. And although the cognitive biases just discussed might be some of the toughest problems to overcome, greater levels of transparency in the publishing process help fortify the ability of the peer review process to serve as a check on researcher biases.

1.2 Benefits of Open Science Practices

A number of benefits of open science practices are worth emphasizing. First, increases in transparency make it possible for errors to be detected and for science to self-correct. The self-correcting nature of science is often heralded as a key feature that distinguishes scientific approaches from other ways of knowing. Yet, self-correction is difficult, if not impossible, when details of research are routinely withheld (Reference Vazire and HolcombeVazire & Holcombe, 2020).

Second, openly sharing research materials (protocols), analysis code, and data provides new opportunities to extend upon research and adds value above and beyond what a single study would add. For example, future researchers can more easily replicate a study’s methods if they have access to a full protocol and materials; secondary data analysts and meta-analysts can perform novel analyses on raw data if they are shared.

Third, collaborative work becomes easier when teams employ the careful documentation that is well-honed for followers of open science practices. Even massive collaborations across time and location become possible when research materials and data are shared following similar standards (Reference Moshontz, Campbell, Ebersole, IJzerman, Urry, Forscher and ChartierMoshontz et al., 2018).

Finally, the benefits of open science practices accrue not only to the field at large, but also to individual researchers. Working openly provides a tangible record of your contributions as a researcher, which may be useful when it comes to applying for funding, awards, or jobs.

Reference MarkowetzMarkowetz (2015) describes five “selfish” reasons to work reproducibly, chiefly: (a) to avoid “disaster” (i.e., major errors), (b) because it’s easier, (c) to smooth the peer review process, (d) to allow others to build on your work, and (e) to build your reputation. Likewise, Reference McKiernan, Bourne, Brown, Buck, Kenall, Lin, McDougall, Nosek, Ram, Soderberg, Spies, Thaney, Updegrove, Woo and YarkoniMcKiernan et al. (2016) review the ample evidence that articles that feature open science practices tend to be more cited, more discussed in the media, attract more funding and job offers, and are associated with having a larger network of collaborators. Reference Allen and MehlerAllen and Mehler (2019) review benefits (along with challenges) specifically for early career researchers.

All of this is not to say that there are not costs or downsides to some of the practices discussed here. For one thing, learning and implementing new techniques takes time, although experience shows that you’ll become faster and more efficient with practice. Additionally, unsupportive research mentors or other senior collaborators can make it challenging to embrace open science practices. The power dynamics in such relationships may mean that there is little flexibility in the practices that early career researchers can employ. Trying to propose new techniques can be stressful and might strain advisor-advisee relationships, but see Reference Kathawalla, Silverstein and SyedKathawalla et al. (2021) for rebuttals to these issues and other common worries.

In spite of these persistent challenges and the old pressures working against the adoption of open science practices, I hope to convince you that the benefits of working openly are numerous – both to the field and to individual researchers. As a testament to changing norms and incentives, open science practices are spreading and taking hold in psychology (Reference Christensen, Freese and MiguelChristensen, Freese, et al., 2019; Reference Tenney, Costa, Allard and VazireTenney et al., 2021). Let us consider in more detail what we actually mean by open science practices.

2.1 Managing the Open Science Workflow: The Open Science Framework

The Open Science Framework (OSF; https://osf.io) is a powerful research management tool. Using a tool like OSF allows you to organize all stages of the research process in one location, which can help you stay organized. Using OSF is also not tied to any specific academic institution, so you won’t have to worry about transferring your work when you inevitably change jobs (perhaps several times). Other tools exist that can do many of the things OSF can (some researchers like to use GitHub, figshare, or Zenodo, for instance), but OSF was specifically created for managing scientific research and has a number of features that make it uniquely suited for the task. OSF’s core functions include (but are not limited to) long-term archival of research materials, analysis code, and data; a flexible but robust pre-registration tool; and support for collaborative workflow management. Later in the chapter, we’ll discuss the ins and outs of each of these practices, but here I want to review a few of the ways that OSF is specialized for these functions.

The main unit of work on OSF is the “project.” Each project has a stable URL and the potential to create an associated digital object identifier (DOI). This means that researchers can make reference to OSF project pages in their research articles without worry that links will cease to function or shared content will become unavailable. A sizable preservation fund promises that content shared on OSF will remain available for at least 50 years, even if the service should cease to operate. This stability makes OSF well-suited to host part of the scientific record.

A key feature of projects is that they can be made public (accessible to all) or private (accessible only to contributors). This feature allows you to share your work publicly when you are ready, whether that is immediately or only after a project is complete. Another feature is that projects can be shared using “view-only” links. These links have the option to remove contributor names to enable the materials shared in a project to be accessible to peer reviewers at journals that use masked review.

Projects can have any number of contributors, making it possible to easily work collaboratively even with a large team. An activity tracker gives a detailed and complete account of changes to the project (e.g., adding or removing a file, editing the project wiki page), so you always know who did what, and when, within a project. Another benefit is the ability to easily connect OSF to other tools (e.g., Google drive, GitHub) to further enhance OSF’s capabilities.

Within projects, it is possible to create nested “components.” Components have their own URLs, DOIs, privacy settings, and contributor list. It is possible, for instance, to create a component within a project and to restrict access to that component alone while making the rest of the project publicly accessible. If particular parts of a project are sensitive or confidential, components can be a useful way to maintain the privacy of that information. Similarly, perhaps it is necessary for part of a research group to have access to parts of a research project and for others to not have the access. Components allow researchers this fine-grained level of control.

Finally, OSF’s pre-registration function allows projects and components to be “frozen” (i.e., saved as time-stamped copies that cannot be edited). Researchers can opt to pre-register their projects using one of many templates, or they can simply upload the narrative text of their research plans. In this way, researchers and editors can be confident about which elements of a study were pre-specified and which were informed by the research process or outcomes.

The review of OSF’s features here is necessarily brief. Reference SoderbergSoderberg (2018) provides a step-by-step guide for getting started with OSF. Tutorials are also available on the Center for Open Science’s YouTube channel. I recommend selecting a project – perhaps one for which you are the lead contributor – to try out OSF and get familiar with its features in greater detail. Later, you may want to consider using a project template, like the one that I use in my lab (Reference CorkerCorker, 2016), to standardize the appearance and organization of your OSF projects.

2.2 Pre-Registration and Registered Reports

Learning about how to pre-register research involves much more than just learning how to use a particular tool (like OSF) to complete the registration process. Like other research methods, training and practice are needed to become skilled at this key open science technique (Reference Tackett, Brandes, Dworak and ShieldsTackett et al., 2020). Pre-registration refers to publicly reporting study designs, hypotheses, and/or analysis plans prior to the onset of a research project. Additionally, the pre-registered plan should be shared in an accessible repository, and it should be “read-only” (i.e., not editable after posting). As we’ll see, there are several reasons a researcher might choose to pre-register, along with a variety of benefits of doing so, but the most basic function of the practice is that pre-registration clearly delineates the parts of a research project that were specified before the onset of a project from those parts that were decided on along the way or based on observed data.

Depending on their goals, researchers might pre-register for different reasons (Reference Da Silva Frost and Ledgerwoodda Silva Frost & Ledgerwood, 2020; Reference LedgerwoodLedgerwood, 2018; Reference NavarroNavarro, 2019). First, researchers may want to constrain particular data analytic choices prior to encountering the data. Doing so makes it clear to the researchers, and to readers, that the presented analysis is not merely the one most favorable to the authors’ predictions, nor the one with the lowest p-value. Second, researchers might desire to specify theoretical predictions prior to encountering a result. In so doing, they set up conditions that enable a strong test of the theory, including the possibility for falsification of alternative hypotheses (Reference PlattPlatt, 1964). Third, researchers may seek to increase the transparency of their research process, documenting particular plans and, crucially, when those plans were made. In addition to the scientific benefits of transparency, pre-registration can also facilitate more detailed planning than usual, potentially increasing research quality as potential pitfalls are caught early enough to be remedied.

Some of these reasons are more applicable to certain types of research than others, but nearly all research can benefit from some form of pre-registration. For instance, some research is descriptive and does not test hypotheses stemming from a theory. Other research might feature few or no statistical analyses. The theory testing or analytic constraint functions of pre-registration might not be applicable in these instances. However, the benefits of increased transparency and enhanced planning stand to benefit many kinds of research (but see Reference Devezer, Navarro, Vandekerckhove and BuzbasDevezer et al., 2021, for a critical take on the value of pre-registration).

A related but distinct practice is Registered Reports (Reference ChambersChambers, 2013). In a registered report, authors submit a study proposal – usually as a manuscript consisting of a complete introduction, proposed method, and proposed analysis section – to a journal that offers the format. The manuscript (known at that point as “stage 1”) is then peer-reviewed, after which it can be rejected, accepted, or receive a revise and resubmit. Crucially, once the stage 1 manuscript is accepted (most likely after revision following peer review), the journal agrees to publish the final paper regardless of the statistical significance of results, provided the agreed upon plan has been followed – a phase of publication known as “in-principle acceptance.” Once results are in, the paper (at this point known as a “stage 2” manuscript) goes out again for peer review to verify that the study was executed as agreed.

When stage 1 proposals are published (either as stand-alone manuscripts or as supplements to the final stage 2 manuscripts), registered reports allow readers to confirm which parts of a study have been planned ahead of time, just like ordinary pre-registrations. Likewise, registered reports limit strategic analytic flexibility, allow strong tests of hypotheses, and increase the transparency of research. Crucially, however, registered reports also address publication bias, because papers are not accepted or rejected on the basis of the outcome of the research. Furthermore, the two-stage peer-review process has an even greater potential to improve study quality, because researchers receive the benefit of peer critique during the design phase of a study when there is still time to correct flaws. Finally, because the publication process is overseen by an editor, undisclosed deviations from the pre-registered plan may be less likely to occur than they are with unreviewed pre-registration. Pragmatically, registered reports might be especially worthwhile in contentious areas of study where it is useful to jointly agree on a critical test ahead of time with peer critics. Authors can also enjoy the promise of acceptance of the final product prior to investing resources in data collection.

Table 11.1 lists guidance and templates that have been developed across different subfields and research methods to enable nearly any study to be pre-registered. A final conceptual distinction is worth brief mention. Pre-registrations are documentation of researchers’ plans for their studies (in systematic reviews of health research, these documents are known as protocols). When catalogued and searchable, pre-registrations form a registry. In the United States, the most common study registry is clinicaltrials.gov, because the National Institutes of Health requires studies that it funds to be registered there. PROSPERO (Reference Page, Shamseer and TriccoPage et al., 2018) is the main registry for health-related systematic reviews. Entries in clinicaltrials.gov and PROSPERO must follow a particular format, and adhering to that format may or may not fulfill researchers’ pre-registration goals (for analytic constraint, for hypothesis testing, or for increasing transparency). For instance, when registering a study in clinicaltrials.gov, researchers must declare their primary outcomes (i.e., dependent variables) and distinguish them from secondary outcomes, but they are not required to submit a detailed analysis plan. A major benefit of study registries is to track the existence of studies independent of final publications. Registries also allow the detection of questionable research practices like outcome switching (e.g., Reference Goldacre, Drysdale, Dale, Milosevic, Slade, Hartley, Marston, Powell-Smith, Heneghan and MahtaniGoldacre et al., 2019). However, entries in clinicaltrials.gov and PROSPERO fall short in many ways when it comes to achieving the various goals of pre-registration discussed above. It is important to distinguish brief registry entries from more detailed pre-registrations and protocols.

3.1 Open Lab Notebooks

One way to track your research as it unfolds is to keep a detailed lab notebook. Recently, some researchers have begun to keep open, digital lab notebooks (Reference CampbellCampbell, 2018). Put briefly, open lab notebooks allow outsiders to access the research process in its entirety in real time (Reference Bradley, Lang, Koch, Neylon, Elkins, Lang, Koch and NeylonBradley et al., 2011). Open lab notebooks might include entries for data collected, experiments run, analyses performed, and so on. They can also include accounts of decisions made along the way – for instance, to change an analysis strategy or to modify the participant recruitment protocol. Open lab notebooks are a natural complement to pre-registration insofar as a pre-registration spells out a plan for a project, and the lab notebook documents the execution (or alteration) of that plan. In fact, for some types of research, where the a priori plan is relatively sparse, an open lab notebook can be an especially effective way to transparently document exploration as it unfolds.

On a spectrum from completely open research to completely opaque research, the practice of keeping an open lab notebook marks the far (open) end of the scale. For some projects (or some researchers) the costs of keeping a detailed open lab notebook in terms of time and effort might greatly exceed the scientific benefits for transparency and record keeping. Other practices may achieve similar goals more efficiently, but for some projects, the practice could prove invaluable. To decide whether an open lab notebook is right for you, consider the examples given in Reference CampbellCampbell (2018). You can also see an open notebook in action here: https://osf.io/3n964/ (Reference Koessler, Campbell and KohutKoessler et al., 2019).

3.2 Open Protocols and Open Materials

A paper’s Method section is designed to describe a study protocol – that is, its design, participants, procedure, and materials – in enough detail that an independent researcher could replicate the study. In actuality, many key details of study protocols are omitted from Method sections (Reference ErringtonErrington, 2019). To remedy this information gap, researchers should share full study protocols, along with the research materials themselves, as supplemental files. Protocols can include things like complete scripts for experimental research assistants, video demonstrations of techniques (e.g., a participant interaction or a neurochemical assay), and full copies of study questionnaires. The goal is for another person to be able to execute a study fully without any assistance from the original author.

Research materials that have been created specifically for a particular study – for instance, the actual questions asked of participants or program files for an experimental task – are especially important to share. If existing materials are used, the source where those materials can be accessed should be cited in full. If there are limitations on the availability of materials, which might be the case if materials are proprietary or have restricted access for ethical reasons, those limitations should be disclosed in the manuscript.

3.3 Reproducible Analyses, Open Code, and Open Data

One of the basic features of scientific research products is that they should be independently reproducible. A finding that can only be recreated by one person is a magic trick, not a scientific truism. Here, reproducible means that results can be recreated using the same data originally used to make a claim. By contrast, replicability implies the repetition of a study’s results using different data (e.g., a new sample). Note that also, a finding can be reproducible, or even replicable, but not be a valid or accurate representation of reality (Reference Vazire, Schiavone and BottesiniVazire et al., 2020). Reproducibility can be thought of as a minimally necessary precursor to later validity claims. In psychology, analyses of quantitative data very often form the backbone of our scientific claims. Yet, the reproducibility of data analytic procedures may never be checked, or if they are checked, findings may not be reproducible (Reference Obels, Lakens, Coles, Gottfried and GreenObels et al., 2020; Reference Stodden, Seiler and MaStodden et al., 2018). Even relatively simple errors in reporting threaten the accuracy of the research literature (Reference Nuijten, Hartgerink, Van Assen, Epskamp and WichertsNuijten et al., 2016).

Luckily, these problems are fixable, if we are willing to put in the effort. Specifically, researchers should share the code underlying their analyses and, when legally and ethically permissible, they should share their data. But beyond just sharing the “finished product,” it may be helpful to think about preparing your data and code to share while the project is actually under way (Reference Klein, Hardwicke, Aust, Breuer, Danielsson, Mohr, IJzerman, Nilsonne, Vanpaemel and FrankKlein et al., 2018).

Whenever possible, analyses should be conducted using analysis code – also known as scripting or syntax – rather than by using point-and-click menus in statistical software or doing hand calculations in spreadsheet programs. To further enhance the reproducibility of reported results, you can write your results section using a language called R Markdown. Succinctly, R Markdown combines descriptive text with results (e.g., statistics, counts) drawn directly from analyses. When results are prepared in this way, there is no need to worry about typos or other transcription errors making their way into your paper, because numbers from results are pulled directly from statistical output. Additionally, if there is a change to the data – say, if analyses need to be re-run on a subset of cases – the result text will automatically update with little effort.

Reference Peikert and BrandmaierPeikert and Brandmeier (2019) describe a possible workflow to achieve reproducible results using R Markdown along with a handful of other tools. Reference RouderRouder (2016) details a process for sharing data as it is generated – so-called “born open” data. This method also preserves the integrity of original data. When combined with Peikert and Brandmeier’s technique, the potential for errors to affect results or reporting is greatly diminished.

Regardless of the particular scripting language that you use to analyze your data, the code, along with the data itself, should be well documented to enable use by others, including reviewers and other researchers. You will want to produce a codebook, also known as a data dictionary, to accompany your data and code. Reference Buchanan, Crain, Cunningham, Johnson, Stash, Papadatou-Pastou, Isager, Carlsson and AczelBuchanan et al. (2021) describe the ins and outs of data dictionaries. Reference ArslanArslan (2019) writes about an automated process for codebook generation using R statistical software.

3.4 Version Control

When it comes to tracking research products in progress, a crucial concept is known as version control. A version control system permits contributors to a paper or other product (such as analysis code) to automatically track who made changes to the text and when they made them. Rather than saving many copies of a file in different locations and under different names, there is only one copy of a version-controlled file, but because changes are tracked, it is possible to roll back a file to an earlier version (for instance, if an error is detected). On large collaborative projects, it is vital to be able to work together simultaneously and to be able to return to an earlier version of the work if needed.

Working with version-controlled files decreases the potential for mistakes in research to go undetected. Reference Rouder, Haaf and SnyderRouder et al. (2019) describe practices, including the use of version control, that help to minimize mistakes and improve research quality. Reference Vuorre and CurleyVuorre and Curley (2018) provide specific guidance for using Git, one of the most popular version control systems. An additional benefit of learning to use these systems is their broad applicability in non-academic research settings (e.g., at technology and health companies). Indeed, developing skills in domain general areas like statistics, research design, and programming will broaden the array of opportunities available to you when your training is complete.

3.5 Working Openly Facilitates Teamwork and Collaboration

Keeping an open lab notebook, sharing a complete research protocol, or producing a reproducible analysis script that runs on open data might seem laborious compared to closed research practices, but there are advantages of these practices beyond the scientific benefits of working transparently. Detailed, clear documentation is needed for any collaborative research, and the need might be especially great in large teams. Open science practices can even facilitate massive collaborations, like those managed by the Psychological Science Accelerator (PSA; Reference Moshontz, Campbell, Ebersole, IJzerman, Urry, Forscher and ChartierMoshontz et al., 2018). The PSA is a global network of over 500 laboratories that coordinates large investigations of democratically selected study proposals. It enables even teams with limited resources to study important questions at a large enough scale to yield rich data and precise answers. Open science practices are baked into all parts of the research process, and indeed, such efforts would not be feasible or sustainable without these standard operating procedures.

Participating in a large collaborative project, such as one run by the PSA, is an excellent way to develop your open science skillset. It can be exciting and quite rewarding to work in such a large team, but in so doing, there is also the opportunity to learn from the many other collaborators on the project.

4.1 Writing Well and Transparently

Reference GernsbacherGernsbacher (2018) provides detailed guidance on preparing a high-quality manuscript (with a clear narrative) while adhering to open science practices. She writes that the best articles are transparent, reproducible, clear, and memorable. To achieve clarity and memorability, authors must attend to good writing practices like writing short sentences and paragraphs and seeking feedback. These techniques are not at odds with transparency and reproducibility, which can be achieved through honest, detailed, and clear documentation of the research process. Even higher levels of detail can be achieved by including supplemental files along with the main manuscript.

One issue, of course, is how to decide which information belongs in the main paper versus the supplemental materials. A guiding principle is to organize your paper to help the reader understand the paper’s contribution while transparently describing what you’ve done and learned. Reference GernsbacherGernsbacher (2018) advises having an organized single file as a supplement to ease the burden on reviewers and readers. A set of well-labeled and organized folders in your OSF project (e.g., Materials, Data, Analysis Code, Manuscript Files) can also work well. Consider including a “readme” file or other descriptive text to help readers understand your file structure.

If a project is pre-registered, it is important that all of the plans (and hypotheses, if applicable) in the study are addressed in the main manuscript. Even results that are not statistically significant deserve discussion in the paper. If planned methods have changed, this is normal and absolutely fine. Simply disclose the change (along with accompanying rationale) in the paper, or better yet, file an addendum to your pre-registration when the change is made before proceeding. Likewise, when analysis plans change, disclose the change in the final paper. If the originally planned analysis strategy and the preferred strategy are both valid techniques, and others might disagree about which strategy is best, present results using both strategies. The details of the comparative analyses can be placed in a supplement, but discuss the analyses in the main text of the paper.

A couple of additional tools to assist with writing your open science manuscript are worth mention. First, Reference Aczel, Szaszi, Sarafoglou, Kekecs, Kucharský, Benjamin, Chambers, Fisher, Gelman, Gernsbacher, Ioannidis, Johnson, Jonas, Kousta, Lilienfeld, Lindsay, Morey, Munafò, Newell and WagenmakersAczel et al. (2020) provide a consensus-based transparency checklist that authors can complete to confirm that they have made all relevant transparency-based disclosures in their papers. The checklist can also be shared (e.g., on OSF) alongside a final manuscript to help guide readers through the disclosures. Second, R Markdown can be used to draft the entire text of your paper, not just the results section. Doing so allows you to render the final paper using a particular typesetting style more easily. More importantly, the full paper will then be reproducible. Rather than work from scratch, you may want to use the papaja package (Reference Aust and BarthAust & Barth, 2020), which provides an R Markdown template. Many researchers also like to use papaja in concert with Zotero (www.zotero.org/), an open-source reference manager.

4.2 Selecting a Journal for Your Research

Beyond questions of a journal’s topical reach and its reputation in the field, different journals have different policies when it comes to open science practices. When selecting a journal, you will want to review that journal’s submission guidelines to ensure that you understand and comply with its requirements. Another place to look for guidance on a journal’s stance on open science practices is editorial statements. These statements usually appear within the journal itself, but if the journal is owned by a society, they may also appear in society publications (e.g., American Psychological Association Monitor, Association for Psychological Science Observer).

Many journals are signatories of the TOP (Transparency and Openness Promotion) Guidelines, which specify three different levels of adoption for eight different transparency standards (Reference Nosek, Alter, Banks, Borsboom, Bowman, Breckler, Buck, Chambers, Chin, Christensen, Contestabile, Dafoe, Eich, Freese, Glennerster, Goroff, Green, Hesse, Humphreys and YarkoniNosek et al., 2015; see also https://topfactor.org). Journals with policies at level 1 require authors to disclose details about their studies in their manuscripts – for instance, whether the data associated with studies are available. At level 2, sharing of study components (e.g., materials, data, or analysis code) is required for publication, with exceptions granted for valid legal and ethical restrictions on sharing. At level 3, the journal or its designee verifies the shared components – for instance, a journal might check whether a study’s results can be reproduced from shared analysis code. Importantly, journals can adopt different levels of transparency for the different standards. For instance, a journal might adopt level 1 (disclose) for pre-registration of analysis plans, but level 3 (verify) for study materials. Again, journal submission guidelines, along with editorial statements, provide guidance as to the levels adopted for each standard.

Some journals also offer badges for adopting transparent practices. At participating journals, authors declare whether they have pre-registered a study or shared materials and/or data, and the journal then marks the resulting paper with up to three badges (pre-registration, open data, open materials) indicating the availability of the shared content.

A final consideration is the pre-printing policies of a journal. Almost certainly, you will want the freedom to share your work on a preprint repository like PsyArXiv (https://psyarxiv.com). Preprint repositories allow authors to share their research ahead of publication, either before submitting the work for peer review at a journal or after the peer-review process is complete. Some repositories deem the latter class of manuscripts “post-prints” to distinguish them from papers that have not yet been published in a journal. Sharing early copies of your work will enable to you to get valuable feedback prior to journal submission. Even if you are not ready to share a pre-publication copy of your work, sharing the final post-print increases access to the work – especially for those without access through a library including researchers in many countries, scholars without university affiliations, and the general public. Manuscripts shared on PsyArXiv are indexed on Google Scholar, increasing their discoverability.

You can check the policies of your target journal at the Sherpa Romeo database (https://v2.sherpa.ac.uk/romeo/). The journals with the most permissive policies allow sharing of the author copy of a paper (i.e., what you send to the journal, not the typeset version) immediately on disciplinary repositories like PsyArXiv. Other journals impose an embargo on sharing of perhaps one or two years. A very small number of journals will not consider manuscripts that have been shared as preprints. It’s best to understand a journal’s policy before choosing to submit there.

Importantly, sharing pre- or post-print copies of your work is free to do, and it greatly increases the reach of your work. Another option (which may even be a requirement depending on your research funder) is to publish your work in a fully open-access journal (called “gold” open access) or in a traditional journal with the option to pay for your article to be made open access (called “hybrid” open access). Gold open-access journals use the fees from articles to cover the costs of publishing, but articles are free to read for everyone without a subscription. Hybrid journals, on the other hand, charge libraries large subscription fees (as they do with traditional journals), and they charge authors who opt to have their articles made open access, effectively doubling the journal’s revenue without incurring additional costs. The fees for hybrid open access are almost never worth it, given that authors can usually make their work accessible for free using preprint repositories.

Fees to publish your work in a gold open-access journal currently vary from around US$1000 on the low end to US$3000 or more on the high end. Typically, a research funder pays these fees, but if not, there may be funds available from your university library or research support office. Some journals offer fee waivers for authors who lack access to grant or university funding for these costs. Part of open science means making the results of research as accessible as possible. Gold open-access journals are one means of achieving this goal, but preprint repositories play a critical role as well.

3.1 Poster Presentations

Poster presentations are the most common medium through which researchers disseminate findings. In this format, researchers summarize their primary aims, results, and conclusions in an easily digestible manner on a poster board. Poster sessions vary in duration, often ranging from 1 to 2 hours. Authors typically are present with their posters for the duration of the session to discuss their work with interested colleagues. Poster presentations are relatively less formal and more personal than other presentation formats, with the discussion of projects often assuming a conversational quality. That said, it is important to be prepared to answer challenging questions about the work. Typically, many posters within a particular theme (e.g., health psychology) are displayed in a large room so that audiences might walk around the room and talk one-to-one with the authors. Thus, poster sessions are particularly well-suited for facilitating networking and meeting with researchers working in similar areas.

Pragmatically, conference reviewers accept more posters for presentations than symposia, panel discussion, and workshops, and thus, the acceptance criteria are typically more lenient. Relatedly, researchers might choose posters to present findings from small projects or preliminary or pilot results studies. Symposia, panel discussions, and workshops allow for the formal presentation of more ground-breaking findings or of multiple studies. Poster presentations are an opportune time for students to become familiarized with disseminating findings and mingling with other researchers in the field.

3.2 Research Symposia

Symposia involve the aggregation of several individuals who present on a common topic. Depending on time constraints, 4–6 papers typically are featured, each lasting roughly 20 minutes, and often representing different viewpoints or facets of a broader topic. For example, a symposium on the etiology of anxiety disorders might be comprised of four separate papers representing the role of familial influences, biological risk factors, peer relationships, and emotional conditioning on the development of maladaptive anxiety. As a presenter, you might discuss one project or the findings from a few studies. Like a master of ceremonies, the symposia Chair typically organizes the entire symposia by selecting presenters, guiding the topics and style of presentation, and introducing the topic and presenters at the beginning of the symposium. In addition to these duties, the Chair often will present a body of work or a few studies at the beginning of the symposium. In addition to the Chair and presenters, a Discussant can be part of a symposium. The Discussant concludes the symposium by summarizing key findings from each paper, integrating the studies, and making more broad-based conclusions and directions for future research. Although a Discussant is privy to the presenters’ papers prior to the symposium in order to prepare the summary comments, he or she will often take notes during the presenters’ talks to augment any prepared commentary. Presenters are often researchers of varying levels of experience, while Chairs and Discussants are usually senior investigators. The formal presentation is often followed by a period for audience inquiry and discussion.

3.3 Panel Discussions

Panel discussions are similar to research symposia in that several professionals come together to discuss a common topic. Panel discussions, however, generally tend to be less formal and structured and more interactive and animated than symposia. For example, Discussants can address each other and interject comments throughout the discussion. Similar to symposia, these presentations involve the discussion of one or more important topics in the field by informed Discussants. As with symposia presentations, the Chair typically organizes these semi-formal discussions by contacting potential speakers and communicating the discussion topic and their respective roles.

3.4 Workshops

Conference workshops typically are often longer (e.g., lasting at least three hours) and provide more in-depth, specialized training than symposia and panel discussions. It is not uncommon for workshop presenters to adopt a format similar to a structured seminar, in which mini-curricula are followed. Due to the length and specialized training involved, most workshop presenters enhance their presentations by incorporating interactive (e.g., role-plays) and multimedia (e.g., video clips) components. Workshops often are organized such that the information is geared for beginner, intermediate, or advanced professionals. Often conferences are organized such that participation in workshops must be reserved in advance and there might be additional fees associated with attendance. The cost should be balanced with the opportunity of obtaining unique training in a specialized area. These are most often presented by seasoned professionals; however, more junior presenters with specialized skills/knowledge might conduct a workshop.

5.1 Choosing an Appropriate Outfit

Dress codes for conference proceedings typically are not formally stated; however, data suggest that perceptions of graduate student professionalism and competence are influenced by dress (e.g., Reference Gorham, Cohen and MorrisGorham et al., 1999). Although the appropriateness of certain attire is likely to vary, a good rule of thumb is to err on the side of professionalism. You also might consider the dress of your audience, and dress in an equivalent or more formal fashion. Although there will be people at conferences wearing unique styles of dress, students and professionals still early in their careers are best advised to dress professionally. It can be helpful to ask people who have already attended the conference what would be appropriate to wear. In addition to selecting your outfit, there are several preparatory steps you can take to help ensure a successful presentation.

5.2 Preparing for Poster Presentations

5.2.2 Tips For Poster Construction

The overriding goal for poster presentations is to summarize your study using an easily digestible, reader-friendly format (Reference GrechGrech, 2018b). As you will discover from viewing other posters, there are many different styles to do this. If you have the resources, professional printers can create large glossy posters that are well received. However, cutting large construction paper to use as a mat for laser-printed poster pages can also appear quite professional. Some companies will even print the poster on a fabric material, so it can be folded up and stored easily in a suitcase for travel to and from the conference. Regardless of the framing, it is advisable to use consistent formatting (e.g., same style and font size throughout the poster), large font sizes (e.g., at least 20-point font for text and 40-point font for headings), and alignment of graphics and text (Reference Zerwic, Grandfield, Kavanaugh, Berger, Graham and MershonZerwic et al., 2010). Another suggestion for enhancing readability and visual appeal is to use bullets, figures, and tables to illustrate important findings. Generally speaking, brief phrases (as opposed to wordy paragraphs) should be used to summarize pertinent points. It has been suggested to limit horizontal lines to 10 or fewer words and avoid using more than four colors (Reference Zerwic, Grandfield, Kavanaugh, Berger, Graham and MershonZerwic et al., 2010). In short, it is important to keep your presentation succinct and avoid overcrowding on pages. Although there are a variety of fonts available and poster boards come in all colors imaginable, it is best to keep the poster professional. In other words, Courier, Arial, or Times New Roman are probably the best fonts to use because they are easy to read and they will not distract or detract from the central message of the poster (i.e., your research). In addition, dark font (e.g., blue, black) on a light background (e.g., yellow, white) is easier to read in brightly lit rooms, which is the norm for poster sessions. Be mindful of appropriately acknowledging any funding agencies or other organizations (e.g., universities) on the poster or in the oral presentation slides. Recently, the format of posters has shifted. The “better poster” format (https://youtu.be/1RwJbhkCA58) includes the main research finding in large text in the center of the poster with extra details and figures on the sides (Figure 12.1). This allows conference attendees to quickly assess whether the research is of interest to them and if they would like more detailed information.

Figure 12.1 Poster formats.

5.3 Conducting Poster Presentations

In general, presenting a poster is straightforward – tack the poster to the board at the beginning of the session, stand next to the poster and discuss the details of the project with interested viewers, and remove the poster at the end of the session. However, we have found that a surprisingly high number of presenters do not adequately fulfill these tasks. Arriving to the poster session at least five minutes early will allow you to find your allocated space, unpack your poster, and decide where to mount it on the board. When posters consist of multiple frames, it might be easiest to lie out the boards on the floor prior to beginning to tack it up on the board.

During the poster session, remember this fundamental rule – be present. It is permissible to browse other posters in the same session; however, always arrange for a co-author or another colleague knowledgeable about the study to man the poster. Another guideline is to be available to answer questions and discuss the project with interested parties. In other words, refrain from reading, chatting with friends, or engaging in other activities that interfere with being available to discuss the study. At the conclusion of the poster session, it is important to quickly remove your poster so subsequent presenters have ample time to set up their posters. Suggestions for preparing and presenting posters are summarized in Table 12.2.

5.4 Preparing for Oral Presentations

5.5 Conducting Oral Presentations

5.5.1 Using Audiovisual Enhancements

One strategy for enhancing oral presentations is to use audio/visual stimuli, such as slides or props (e.g., Reference GrechGrech, 2019; Reference HoffHoff, 1988; Reference WilderWilder, 1994; see Table 12.4). When using visual enhancements, keep it simple, and clearly highlight important points using readable and consistent typeface (Reference Blome, Sondermann and AugustinBlome et al., 2017; Reference GrechGrech, 2018a). Information should be easily assimilated and reader-friendly, which generally means limiting text to a few phrases rather than complete sentences or paragraphs and using sufficiently large font sizes (e.g., 36- to 48-point font for titles and 24- to 36-point font for text). In addition, it is a good idea to keep titles to one line and bullets to no more than two lines of information. Additionally, color schemes should be relatively subdued and “professional” in appearance. For slide presentations, a dark background and light text might be easier to read. Utilize fonts without serifs (e.g., Arial) as opposed to fonts with serifs (e.g., Times New Roman; Reference Lefor and MaenoLefor & Maeno, 2016). Some conferences prefer a particular slide size, so it can be important to check with their requirements before designing the presentation. Additionally, depending on the room set up, it can be hard for the audience to see the bottom of slides, so it can be helpful to avoid placing text near the bottom of the slide. See Figure 12.2 for an example of a poor and good slide for an oral presentation.

Table 12.4 Using audiovisual enhancements

Examples of audiovisual aids	Slides Video clips Cartoons and comic strips
Tips for using slides	Test equipment in advance Keep it simple; use to clarify and enhance Avoid going overboard (too much might detract from presentation) Use reader-friendly format (e.g., short phrases, avoid overcrowding) Use bullet points rather than sentences Remember One × Six × Six: only ONE idea per visual; less than SIX bullets per visual; less than SIX words per bullet point Highlight important points using readable, consistent typeface Use professional color schemes (e.g., light background, dark text for overheads and dark background, light text for slides) Speak to audience, not to visual aids Stand to the side of your screen to avoid blocking audience’s view Pause as you change slides; practice for smooth transitions Be prepared to present without your overheads/slides
Tips for using videos	Test equipment in advance Pre-set volume levels and cue video in advance Introduce video clip and announce its length Dim the lights before playing Give a concluding statement following the video Use video clips to illustrate and enhance presentations

Figure 12.2 Sample poor and good slides for an oral presentation.

Using audiovisual aids, such as video clips, also can contribute substantially to the overall quality and liveliness of a presentation. When incorporating video clips, pre-set volume levels and cue up the video in advance. We also recommend announcing the length of the video, dimming lights, and giving a concluding statement following the video.

Multimedia equipment and audiovisual aids have the potential to liven up even the most uninspiring presentations; however, caution against becoming overly dependent on any medium. Rather, be fully prepared to deliver a high-quality presentation without the use of enhancements. It also might be wise to prepare a solid “back-up plan” in case your original mode of presentation must be abandoned due to equipment failure or some other unforeseen circumstance. Back-up overheads, for example, might rescue a presenter who learns of equipment failures minutes before presenting.

When using slides, it is important to avoid “going overboard” with information (Reference Blome, Sondermann and AugustinBlome et al., 2017; Reference RegulaRegula, 2020). Many of us will present research with which we are intimately familiar and invested. With projects that are particularly near and dear (e.g., theses and dissertations), it may be tempting to tell the audience as much as possible. It is not necessary, for example, to describe the intricacies of the data collection procedure and present every pre-planned and post-hoc analysis, along with a multitude of significant and non-significant F-values and coefficients. Such information overload might bore audience members, who are unlikely to care about or remember so many fine-grained details. Instead of committing this common presentation blunder, present key findings in a bulleted, easy-to-read format rather than sentences. To avoid overcrowding of slides and overheads, you might remember the One × Six × Six rule of thumb: only ONE idea per visual, less than SIX bullets per visual, and less than SIX words per bullet (Reference RegulaRegula, 2020; see Figure 12.2).

The length of your oral presentation will vary depending on time restrictions, but there are some general guidelines for how to structure your presentation. Reference Zerwic, Grandfield, Kavanaugh, Berger, Graham and MershonZerwic et al. (2010) proposed a possible structure for research presentations, which should include a title, acknowledgments, background, specific aims, methods, results, conclusions, and future directions sections. Zerwic et al. also recommended how many slides should be allocated to each section with your title, acknowledgments, background, specific aims, conclusions, and future directions sections each taking up one slide with the majority of your slides focusing on the methods and results sections.

In short, remember and hold fast to this basic dictum: Audio/visual aids should be used to clarify and enhance (Reference CohenCohen, 1990; Reference GrechGrech, 2018a; Reference WilderWilder, 1994). Aids that detract, confuse, or bore one’s audience should not be used (soliciting feedback from colleagues and peers will assist in this selection process; Reference RegulaRegula, 2020). Overly colorful and ornate visuals or excessive slide animation, for example, might detract and distract from the content of the presentation (Reference Lefor and MaenoLefor & Maeno, 2016). Likewise, visual aids containing superfluous text might encourage audience members to read your slides rather than attend to your presentation. Keeping visuals simple also might prevent another presentation faux pas: reading verbatim from slides.

1.1 Writing the Article

A central goal of scientific writing is to convey what was actually done so that the methods and procedures can be replicated. Concrete, specific, operational, objective, and precise are some of the characteristics that describe the writing style. The effort to describe research in concrete and specific ways is critically important. However, the task of the author goes well beyond description.

Preparation of the report for publication involves three interrelated tasks that I refer to as description, explanation, and contextualization. Failure to appreciate or to accomplish these tasks serves as a main source of frustration for authors, as their papers traverse the process of manuscript review toward journal publication. Description is the most straightforward task and includes providing details of the study. Even though this is an obvious requirement of the report, basic details often are omitted in published articles (e.g., sex, socioeconomic status, and race of the participants; means and standard deviations) (e.g., Reference Case and SmithCase & Smith, 2000; Reference Gerber, Arceneaux, Boudreau, Dowling, Hillygus, Palfrey, Biggers and HendryGerber et al., 2014; Reference Tate, Perdices, Rosenkoetter, McDonald, Togher, Shadish, Horner, Kratochwill, Barlow, Kazdin, Sampson, Shamseer and SampsonTate et al., 2016). Omission of basic details can hamper scientific progress. If a later study fails to replicate the findings, it could be because the sample is very different along some dimension or characteristic. Yet, we cannot surmise that without knowing at least basic details of the sample in both studies. If a study does repeat the findings, that is important, but is the new finding an extension to a new type of sample? Again, we need basic information in the studies to allow such comparisons.

Explanation is more demanding insofar as this refers to presenting the rationale of several facets of the study. The justification, decision-making process, and the connections between the decisions and the goals of the study move well beyond description. Here the reader of the manuscript has access to the author decision points. There are numerous decision points pertaining to such matters as selecting the sample, choosing among many options of how to test the idea, selecting the measures, and including various control and comparison groups. The author is obliged to explain why the specific options elected are well suited to the hypotheses or the goals of the study. There is a persuasion feature that operates here. The author of the manuscript is persuaded that the decisions are reasonable ways to address the overriding research question. Now the author must convey that to persuade the reader. In other words, explanation conveys why the procedures, measures, and so on were selected, but that explanation ought to be cogent, persuasive, and above all explicit. We do not want the reader to think, “This is an important research question, but why study it that way?” And in many cases, the related prior question of the same ilk emerges, why do we even need this study or why is study important? For the many decision points beginning with selection of the research question, these are very reasonable questions that the author ought to anticipate and pre-empt.

Finally, contextualization moves one step further away from description and addresses how the study fits in the context of other studies and in the knowledge base more generally. This latter facet of the article preparation reflects such lofty notions as scholarship and perspective, because the author places the descriptive and explanatory material into a broader context. Essentially, the author is making the case for the study based on the knowledge base. Relatively vacuous claims (e.g., this is the first study of this or the first study to include this or that control condition or measure) are rarely a strong basis for the study and often means or is interpreted as meaning that the author could not come up with something better. Without context, any “first” is not very important by itself. Indeed, it is easy to be first for a topic that is not very important and has been purposely neglected. We need a more compelling rationale.

For example, if this study is done on why people commit suicide we need the context of why this specific study ought to be done and where in the puzzle of understanding this piece fits. Perhaps prior research omitted some critical control procedure; perhaps there is a special group that has a novel characteristic that reduces (or increases) the likelihood of suicide that would inform the field in unique ways; or perhaps some new twist on a theory or intervention will have clear implications for reducing suicide attempts. These and other such comments convey three points that are wise to address: (1) there is a gap in knowledge, (2) that gap is important, and (3) that gap will be filled in whole or in part by this study.

1.2 General Comments

The three components I identified vary in difficulty. When individuals write their first project for publication, they focus heavily on the descriptive part to make sure all the material and sections are included. And this part is fundamental. Explanation and contextualization are much more difficult. Explanation requires having considered options and conveying to the reader why the one selection was a good choice. Yet one’s first study is often with or from an advisor who has made these decisions and the bases of these decisions might be buried in one of the advisor’s other articles but otherwise is tacit. As authors we need to be prepared for other scientists looking at our paper and doing their job by asking, “Why on earth did we [use: that population, measure, control condition, means of data evaluation, and so on]”. These are not only legitimate questions but are central to science.

Contextualization is even more difficult. Contextualization benefits from experience, scholarship, time, and knowledge of as many related areas of work as one can bring to bear. How is the study connected to the literature or topic, how does it relate to theory, to other disciplines, to a critical problem we ought to care about or that is now facing society? The puzzle analogy might help. A given study is one puzzle piece and merely showing that piece to someone is not inherently interesting. It may be inherently boring. Yet, the piece becomes more interesting as all the other pieces are shown (e.g., the outside box with a full photo of the puzzle) and even more interesting, fascinating actually, if one can paint a verbal picture of the whole puzzle and show how one or two pieces are needed and this study is that part! Explanation gives the rationales for decisions; contextualization determines whether the study is compelling or not. Authors often complain that the reviewers did not understand, “get it,” appreciate the importance of their study. The authors are usually completely right, but guess whose responsibility that is?

The extent to which description, explanation, and contextualization are accomplished increases the likelihood that the report will be viewed as a publishable article and facilitates integration of the report into the knowledge base. Guidelines are provided later in the chapter to convey these tasks more concretely in the preparation and evaluation of research reports. The guidelines focus on the logic of the study, the interrelations of the different sections, the rationale for specific procedures and analyses, the strengths and limitations, and where the study fits in the knowledge base. Consider main sections of the manuscript that are prepared for journal publication and how these components can be addressed.Footnote ¹

2.1 Title

The title of an article includes the key variables, focus, and population with an economy of words. The special features of the study are included to convey the focus immediately to potential readers. It is critical here to be direct, clear, and concise (e.g., “Memory loss and gains associated with aging” or “Predictors of drug use and abuse among adolescents”). These examples are especially concise. Ordinarily an author is encouraged to fit the title within 10–12 words. The words ought to be selected carefully. Titles occasionally are used to index articles in large databases. Words that are not needed or that say little (e.g., “preliminary findings,” “implications,” “new findings”) might be more judiciously replaced by substantive or content words (e.g., among preschool children, the elderly; consequences for sleep and stress) that permit the article to be indexed more broadly than it otherwise would have been.

Occasionally, comments about the method are included in the title or more commonly in the subtitle. Terms like “a pilot study” or “preliminary report” may have many different meanings, such as the fact that this is an initial or interim report of a larger research program. These words could also be gently preparing readers for some methodological surprises and even tell us not to expect too much from the design. These qualifying terms might be accurate, but they implicitly apologize or ask for mercy as well. Better to give a strong title and in the write up give the explanation (decision making) and contexts to convey why this study was done, where it fits in the scheme of this literature, and why this was important. No apologies needed; just let the reader know your thinking on the matter. Although I am reluctant to boast, my dissertation won a prize for the best qualifying terms in a title. (In the subtitle of my dissertation, I conveyed this as: “A pre-preliminary, tentative, exploratory pilot study©.”)

In some cases, terms are added to the study such as, “A Controlled Investigation,” which moves our expectation in the other direction, namely, that the present study is somehow well conducted and controlled, and perhaps by implication stands in contrast to other studies in the field (or in the author’s repertoire). Usually words noting that the investigation is controlled are not needed unless this is truly a novel feature of research on the topic. Some words when added can be important because they are novel. An example would be the subheading, “A replication.” That is important because replications are of interest and not too often published. They have taken on even increased importance given the concerns in science that many studies produce findings that are not replicable. Another word to add as a subtitle might be: A review or A meta-analysis. These are important to convey that the article is not an individual investigation but an evaluation of a broad literature.

Occasionally authors are wont to use titles with simple questions, “Is depression really a detriment to health?” or “Is childhood bullying among boys a predictor of domestic violence in adulthood?” In general, it is advisable to avoid “yes, no” questions in the title. Science and findings are often nuanced and findings are likely to be both yes and no but under very different circumstances or for some subgroups of people but not for others. As an example, consider a hypothetical yes–no question for the title of a study as, “Is cigarette smoking bad for one’s health?” For anyone on the planet, the answer might be a resounding yes. Yet, the yes–no nature of the question makes this a poor choice of title because the answer is likely to depend on either how smoking is defined (e.g., how much smoking – a cigarette a year, a pack after each meal) and how health is defined (e.g., mental, physical, what diseases, disorders). Very familiar is how horrible smoking is for one’s physical health in so many domains (e.g., heart disease, cancer, chronic respiratory disease), but the question in the title can be answered both yes and no. Less familiar is the fact that cigarette smoking and exposure to cigarette smoke (among nonsmokers) reduce the risk for Parkinson’s disease and there are reasonable explanations for that based on brain chemistry and neurotransmitters (Reference Ma, Liu, Neumann and GaoMa et al., 2017; Reference Miller and DasMiller & Das, 2007). Clearly, the hypothetical title is plainly simplistic and not very helpful or informative because we can show many circumstances in which yes and no are correct answers. I am not arguing in favor of cigarette smoking (although I used to be a chain smoker until I switched to cigarettes). I am advising against titles of empirical articles that have a yes–no question given that most answers involve essays. Few phenomena allow the simplistic thinking the question can reflect.

2.2 Abstract

The Abstract is likely to be read by many more people than is the full article. The Abstract will be entered into various databases and be accessible through Internet and online searches. Many journals list the tables of contents for their issues and provide free access on the Web to abstracts of the articles but charge for the full article. Consequently, the Abstract is the only information that most readers will have about the study. For reviewers of the manuscript and readers of the journal article, the Abstract conveys what the author studied and found. Ambiguity, illogic, and fuzziness here are ominous. Thus, the Title and Abstract are sometimes the only impression or first impression one may have about the study.

Obviously, the purpose of the Abstract is to provide a relatively brief but comprehensive statement of goals, methods, findings, and conclusions of the study. Critical methodological descriptors pertain to the participants and their characteristics, experimental and control groups or conditions, design, and major findings. Often space is quite limited; indeed a word limit (e.g., 150–250 words maximum) may be placed on the abstract. It is useful to deploy the words to make substantive statements about the characteristics of the study and the findings, rather than to provide general and minimally informative comments. For example, vacuous statements (“Implications of the results were discussed” or “Future directions for research were suggested”) ought to be replaced with more specific comments of what one or two implications and research directions are (e.g., “The findings suggest that the family and peers might be mobilized to prevent drug abuse among adolescents and that cultural influences play a major role.”). Also, the more specific comments can convey the study’s relevance and interest value beyond what is suggested by the manuscript title or the opening comments of the Abstract. As a reader, I am not going to read very eagerly an article with the vacuous “implications” or “future directions” sentences, but if I am interested in the specific topics mentioned as implications (brain activity, the immune system, family, peers, culture), this article is a must for me to read. As authors, we often lament the word restrictions placed on us in the Abstract, but the first task is to make sure we are using the existing allotment with maximum information.

2.3 Introduction

The Introduction is designed to convey the overall rationale and objectives. The task of the author is to convey in a crisp and concise fashion why this study is needed and the current questions or deficiencies the study is designed to address. The section should not review the literature in a study-by-study fashion, but rather convey issues and evaluative comments that set the stage for the study. Placing the study in the context of what is and is not known (contextualization) and the essential next step in research in the field requires mastery of the pertinent literatures, apart from reasonable communication skills. Ironically, mastery of the literature is needed so the author knows precisely what to omit from the Introduction. A vast amount of material one has mastered and that is very interesting will need to be omitted because it does not set the stage or convey the precise context for this study.

Saying that the study is important (without systematically establishing the context) or noting that no one else has studied this phenomenon (measure or sample) usually are feeble attempts to short-circuit the contextualization of the study. In a manuscript I reviewed, the author mentioned four times in the Introduction (and three more in the Discussion) that this was the first time this study has been done. This was not amusing. Someone had not advised or helped the author very much and a very poor case was made for the study. Among the tasks of the Introduction is to lead the reader to the conclusion that the study is important and worthwhile. Telling the reader that the study is important is an argument from authority and that is not how science works. This might even strongly suggest that the author has not done his or her contextualization homework.

It may be relevant to consider limitations of previous work and how those limitations can be overcome. These statements build the critical transition from an existing literature to the present study and the rationale for design improvements or additions in relation to those studies. It is important to emphasize that “fixing limitations” of prior work is not necessarily a strong basis for publishing a study. The author must convey that the limitations of prior work are central to a key building block in theory or the knowledge base. Convey that because of that specific limitation, we really do not know what we thought we did or that there is a new ambiguity that is important but hidden in prior studies considering what was studied and by what means. Alternatively, the study may build along new dimensions to extend the theory and constructs to a broader range of domains of performance, samples, and settings. The rationale for the specific study must be very clearly established. Theory and previous research usually are the proper springboard to convey the importance of the current study.

In general, the Introduction will move from the very general to the specific. The very general refers to the opening of the Introduction that conveys the area, general topic, and significance of a problem. For example, in studies of diagnosis, assessment, treatment, or prevention of clinical dysfunction, the Introduction invariably includes a paragraph to orient the reader about the seriousness, prevalence or incidence, and economic and social costs of the disorder. Reviewers of the manuscript are likely to be specialists in the area of the study and hence know the context very well. Yet, many potential readers would profit from a statement that conveys the significance, interest, and value of the main focus of the study.

After the initial material, the Introduction moves to the issues that underlie this specific study. Here the context that frames the specific hypotheses of the study are provided and reflect theory and research that are the impetus for the investigation. There is an introduction syllogism, as it were, a logic that will lead the reader from previous theory and research to the present study with a direct path. Extended paragraphs that are background without close connections to the hypotheses of the study serve as a common weakness of manuscripts rejected for publication.

The Introduction does not usually permit us to convey all the information we wish to present. In fact, the limit is usually 2–5 manuscript pages. A reasonable use of this space is in brief paragraphs or implicit sections that describe the nature of the problem, the current status of the literature, the extension to theory and research this study is designed to provide, and how the methods to be used are warranted. The penultimate or final paragraph of the Introduction usually includes a statement of the purpose of the study and the specific hypotheses and predictions. By the time the reader reaches this paragraph or set of paragraphs, it should be very clear that these hypotheses make sense, are important, and address a critical issue or need in the knowledge base. In short, the Introduction must establish that the study addresses a central issue. To the extent that the author conveys a grasp of the issues in the area and can identify the lacunae that the study is designed to fill greatly improves the quality of the report and the chances of acceptance for journal publication. By the time the readers arrive at the purpose of the study or hypotheses paragraph, they should be nodding enthusiastically and saying to themselves, “This study is really needed, it should have been done years ago, I am so glad this is being done now.” As authors we often believe a description of the study is all that is needed. The identical study (description of what was done) can be viewed as a weak and just another study or strong, compelling, and sorely needed. All this can be decided by how the Introduction is cast.

2.4 Method

This section of the paper encompasses several points related to who was studied, why, and how. The section not only describes critical procedures, but also provides the rationale for methodological decisions. Subject selection, recruitment, screening, and other features ought to be covered in detail. Initially, the subjects or clients are described. Why was this sample included and how is this appropriate to the substantive area and question of interest? In some cases, the sample is obviously relevant because participants have the characteristic of interest (e.g., parents accused of child abuse, siblings of children with autism spectrum disorder) or are in a setting of interest (e.g., daycare center, wilderness camp). In other cases, samples are included merely because they are available. Such samples, referred to as samples of convenience, may include college students or a clinic population recruited for some other purpose than to test the hypotheses of this study. The rationale for the sample should be provided to convey why this sample provides a good – or if not good, a reasonable – test of the hypotheses and whether any special features may be relevant to the conclusions.

The design is likely to include two or more groups that are treated in a specific fashion. The precise purpose of each group and the procedures to which they are exposed should be clarified. Control groups should not merely be labeled as such with the idea that the name is informative. The author should convey precisely what the group(s) is designed to control. The author is advised to identify the critical methodological concerns and to convey how these are controlled in the design. Plausible threats to experimental validity that are uncontrolled deserve explicit comment to arrest the reasonable concerns of the reviewers (see Reference KazdinKazdin, 2017).

Several measures are usually included in the study. Why the constructs were selected for study should have been clarified in the Introduction. Then the specific measures and why they were selected to operationalize the constructs should be presented in the Method section. Information about the psychometric characteristics of the measures is often highlighted. This information relates directly to the credibility of the results. Apart from individual assessment devices, the rationale for including or omitting areas that might be regarded as crucial (e.g., multiple measures, informants, settings) deserves comment.

Occasionally, ambiguous statements may enter into descriptions of measures. For example, measures may be referred to as “reliable” or “valid” in previous research, as part of the rationale for their use. There are, of course, many different types of reliability and validity. It is important to identify those characteristics of the measure found in prior research that are relevant to the present research. For example, high internal consistency (reliability) in a prior study may not be a strong argument for use of the measure in a longitudinal design where the author cares more about test–retest reliability. Even previous data on test–retest reliability (e.g., over 2 weeks) may not provide a sound basis for repeated testing over annual intervals. The author ought to present information to convey the suitability of the measures for the study.

It often appears that reliability and validity of assessment are not routinely taught, at least if one looks at Method sections of articles in clinical psychology. These are important concepts because they can determine what is measured by a given instrument and how well. One sees more routinely that authors report Cronbach’s alpha for a measure and then move on. Alpha is one measure of reliability (internal consistency) and can be very useful to know. However, this has little to do with validity of the measure and by itself is not a justification for using a specific measure without much more explanation. Perhaps add a couple of sentences in this section to comment specifically on reliability and validity and what types have been supported in prior research. This is not merely to convince a reader but also ourselves on the wisdom of electing this measure. It is unreasonable to expect the measures to have the ideal reliability and validity data that the investigator would like to make a flawless case for use of these measures. Yet, make the case from what psychometric data there are. If data are not available, include some analyses in the study to suggest the measure(s) behave in ways that suggest pertinent forms of reliability or validity (Reference KazdinKazdin, 2017).

2.5 Results

It is important to convey why specific statistical tests were selected and how these serve the goals of the study. A useful exercise is for the investigator to read that paragraph about hypotheses and predictions from the Introduction and then immediately start reading the Results section, i.e., for the moment completely bypass the Methods. The results ought to speak directly to and flow from that narrative statement in the Introduction.

Analyses often are reported in a rote fashion in which, for example, the main effects are presented and then interactions for each measure. The author presents the analyses in very much the same way as the software output. Similarly, if several dependent measures are available, a set of analyses is automatically run (e.g., omnibus tests of multivariate analyses of variance followed by univariate analyses of variance for individual measures). The tests may not relate to the hypotheses, predictions, or expectations outlined at the beginning of the paper. It is important that the statistical tests be seen and presented as tools to answer questions or enlighten features of those questions and to convey this to the reader. The reader should not be able to legitimately ask, “Why was that statistical test done?” Knowledge of statistics is critical for selecting the analyses to address the hypotheses and conditions met by the data. Yet, as important in the presentation is to convey precisely why a given statistical test or procedure is suitable to test the hypotheses and then again what the results of that test reveal in relation to those hypotheses.

It is often useful to begin the Results by presenting basic descriptors of the data (e.g., means, standard deviations for each group or condition), so the reader has access to the numbers themselves. The main body of the Results is to test the hypotheses or to evaluate the predictions. Organization of the Results (subheadings) or brief statements of hypotheses before the analyses are often helpful to prompt the author to clarify how the statistical test relates to the substantive questions and to draw connections for the reader.

Several additional or ancillary analyses may be presented to elaborate the primary hypotheses. For example, one might be able to reduce the plausibility that certain biases may have accounted for group differences based on supplementary or ancillary data analyses. Ancillary analyses may be more exploratory and diffuse than tests of primary hypotheses. Manifold variables can be selected for these analyses (e.g., sex, race, height differences) that are not necessarily conceptually interesting in relation to the goals of the study. The author may wish to present data, data analyses, and findings that were unexpected, were not of initial interest, and were not the focus of the study. The rationale for these excursions and the limitations of interpretation are worth noting. From the standpoint of the reviewer and reader, the results should make clear what the main hypotheses were, how the analyses provide appropriate and pointed tests, and what conclusions can be reached as a result. As in other portions of the manuscript, how the author has reached a decision (what analysis) and why are very important.

2.6 Discussion

The Introduction began with a statement of the need for this study and issues or lacunae in theory or research the study was designed to address. The Discussion continues the storyline by noting what we know now and how the findings address or fulfill the points noted previously. With the present findings, what puzzle piece has been added to the knowledge base, what new questions or ambiguities were raised, what other substantive areas might be relevant for this line of research, and what new studies are needed? From the standpoint of contextualization, the new studies referred to here are not merely those that overcome methodological limitations of the present study, but rather focus on the substantive next steps for research. Also, this is not the place for vacuous suggestions such as, “This study needs to be replicated with people who are …”. If one is suggesting an extension of the study to different subjects, settings, or other dimensions, specify exactly why this specific extension would be of special interest.

The Discussion usually includes paragraphs to provide an overview of the major findings, integration or relation of these findings to theory and prior research, limitations and ambiguities and their implications for interpretation, and future directions. These are implicit rather than formally delineated sections and the author ought to consider the balance of attention to each topic. Usually, the Discussion is completed within 3–5 manuscript pages.

Description and interpretation of the findings can raise a tension between what the author wishes to say about the findings and their meaning versus what can be said in light of how the study was designed and evaluated. It is in the Discussion that one can see the interplay of the Introduction, Methods, and Results sections. For example, the author might draw conclusions that are not quite appropriate given the method and findings. The Discussion may convey flaws, problems or questionable methodological decisions within the design that were not previously evident. That is, the reader of the manuscript can now state that if these are the statements the author wishes to make, the present study (design, measures, or sample) is not well suited. The slight mismatch of interpretative statements in the Discussion and Methods is a common, albeit tacit basis for not considering a study as well conceived and executed. A slightly different study may be required to support the specific statements the author makes in the Discussion. It is important to be precise about what can and cannot be asserted in light of the design and findings.

It is usually to the author’s credit to examine potential limitations or sources of ambiguity of the study. A candid, non-defensive appraisal of the study is very helpful. Here too, contextualization may be helpful because limitations of a study also are related to the body of prior research, what other studies have and have not accomplished, and whether a finding is robust across different methods of investigation. Although it is to the author’s credit to acknowledge the limitations of the study, there are limits on the extent to which reviewers grant a pardon for true confessions. At some point, the flaw is sufficient to preclude publication, whether or not the author acknowledges it. For example, the authors of the study might note, “A significant limitation of the present study is the absence of a suitable control group. We are aware that this might limit the strength of the conclusions.” Awareness here does not strengthen the demonstration itself. A huge limitation in the study is sufficiently damaging to preclude drawing valid inferences.

In noting the limitations of the study, there is a useful structure for the presentation. First note the limitation. Then discuss to the extent reasonable that this limitation is not likely to influence the conclusion (if this is the case). If the role of the limitation cannot be diminished or dismissed by sound reasoning or related data, note that addressing this issue is a logical if not important next step for research. All studies have limitations by their very nature so reasoning about their likely and unlikely impact on the findings is invariably relevant.

At other points, acknowledging potential limitations conveys critical understanding of the issues and guides future work. For example, in explaining the findings, the author may note that although the dependent measures are valid, there are many specific facets of the construct of interest that are not covered. Thus, the results may not extend to different facets of the construct as measured in different ways. Here too it is useful to be specific and to note precisely why other constructs and their measure might show different results. In short, be specific as to why a limitation or point might really make a difference. This latter use of acknowledgment augments the contribution of the study and suggests concrete lines of research.

4.1 Impetus for Reporting Guidelines

There have been a long-series of guidelines on how to conduct and report research and these are directly related to preparation of a study for publication. The history includes special emphasis on ethical treatment of participants. Regulations followed in response to atrocities of the Nazi regime during World War II and the resulting development of the Nuremburg Code (1940). Since then, many other codes have developed (e.g., Declaration of Helsinki of the World Medical Association, Belmont Report), beyond the scope of the present discussion (see Reference KazdinKazdin, 2017). Protection of participant rights remain as important as ever and even of greater concern considering new opportunities to obtain and combine data sources (“big data”), often with information that is public in some way (e.g., medical records, social media, tracking locations and purchases). Individuals may not be aware of the collection and use of the information. Even when participants may be anonymous, in fact often groups (e.g., by ethnic, culture or setting) can be readily singled out and identified in ways that can reflect quite negatively on them (e.g., Reference Metcalf and CrawfordMetcalf & Crawford, 2016; Reference ZimmerZimmer, 2010).

The need for guidelines for reporting research has emerged from multiple additional concerns. First, collaborative research currently is more the rule rather than the exception in science. Collaborations often involve scores of authors, from multiple disciplines, and from many different countries. There is interest across nations in reaching common standards in relation to the openness of research, access to information, the merit-review process, and ethical issues (e.g., Reference SureshSuresh, 2011). That has provided a critical context for providing guidelines for research and the reporting of research that span multiple disciplines, countries, and journals.

Second, lapses in what is reported in research have been well documented. For example, information often is omitted such as exactly who the participants are (e.g., subject and demographic variables) and how they were recruited, who administered treatment or experimental procedures, the extent of their training, whether the integrity or execution of treatment was assessed, fundamental characteristics of the data evaluation, and more, as reflected in citations noted previously.

Third, selective reporting of results and data analyses has been raised as a critical issue that introduces biases in individual studies and entire literatures. The selective reporting of results of some data analyses or some of the dependent measures can increase the likelihood of more chance findings in the literature (Reference Simmons, Nelson and SimonsohnSimmons et al., 2011). For example, in identifying evidence-based treatment, often there is a clear bias in how authors report the data by not presenting the full range of measures, some of which would not support conclusions about the impact of treatment (Reference De Los Reyes and KazdinDe Los Reyes & Kazdin, 2008). Guidelines are intended to foster consistency and clarity in how the study will be reported to minimize the biases in reporting that emerge.

Fourth, more flagrant than “mere” omission of information and selective reporting has been fraud and fabrication of data in science. Fraud is not new in science. However, both the visibility of fraud to the public, including the circulation through social media, and direct and disastrous implications from fraudulent studies have been more evident than ever before (see Reference Levelt and CommitteesLevelt, Noort, & Drenth Committees, 2012; Reference Watanabe and AokiWatanabe & Aoki, 2014).Footnote ²

Finally, there has been renewed concern about the replicability of research. Replication as a general tenet, if not practice, has always been the backbone of science. Given many of the points I have already mentioned (publication biases, statistical analyses, and selective reporting), various authors have reached the dramatic conclusion, occasionally supplemented with mathematical proofs and simulations, that that many and even most published research findings are not correct, i.e., are false (see Reference FrancisFrancis, 2012; Reference IoannidisIoannidis, 2005; Reference Moonesinghe, Khoury and JanssensMoonesinghe et al., 2007). Several calls for increased replication have been made. Psychology has taken the lead in calling for and supporting replications and underscoring the importance of transparency of procedures (Center for Open Science, https://cos.io/). Central to replication, of course, is making the procedures explicit and the materials and results available. Guidelines for conducting research to increase the likelihood that a study can be replicated are obviously important. Many journals, national and international, require providing information about a study and the data so they are freely available to others to facilitate re-evaluation of the data and replication of the entire study.

Overall, science has come under increased scrutiny both from within the sciences, government, and the public at large. Even though the assorted problems I highlighted are seemingly infrequent, the circulation of information (e.g., Web, news and social media) is more extensive than ever before and retractions (when authors and journals make some effort to “take back” and renounce what was published) are more visible and available as well. And news media more routinely comment on scientific findings and reflect skepticism about replication and replicability of effects (e.g., Reference LehrerLehrer, 2010). The points I have raised have served in part as the impetus for improving research, especially focusing on transparency and accountability of investigators. Guidelines have been helpful in fostering greater consistency in reporting of the research and in the process sensitizing researchers of what to attend to in advance of a study.

4.2 Sample Guidelines Briefly Noted

Several organizations and groups have developed standards for reporting research and in the process convey the need to address many of the issues I have highlighted previously. The scope of guidelines that are available is enormous. An international umbrella organization that collects, oversees, and promotes the use of research guidelines is the Equator Network. The network maintains a comprehensive database of guidelines, numbering over 400, as of this writing (www.equator-network.org/reporting-guidelines/). The network is nicely organized by type of paper (guidelines for empirical studies, literature reviews, meta-analyses) and by different methodologies (e.g., randomized trials, observational studies), and so on. With so many guidelines, with enormous overlap in what they cover, one can see this is not a minor movement to improve research.

Examples of such standards are the:

• Consolidated Standards of Reporting Trials (CONSORT; Reference Moher, Schulz and AltmanMoher et al., 2001);

• International Clinical Trials Registry Platform (ICTRP; World Health Organization, www.who.int/ictrp/en/);

• Transparent Reporting of Evaluations with Nonexperimental Designs (TREND; Reference Des Jarlais, Lyles and CrepazDes Jarlais et al., 2004);

• Strengthening for Reporting of Observational Studies in Epidemiology (STROBE; Reference Von Elm, Altman, Egger, Pocock, Gøtzsche and VandenbrouckeVon Elm et al., 2007);

• Journal Article Reporting Standards for Research in Psychology (JARS; Reference Appelbaum, Cooper, Kline, Mayo-Wilson, Nezu and RaoAppelbaum et al., 2018; Reference Levitt, Bamberg, Creswell, Frost, Josselson and Suárez-OrozcoLevitt et al., 2018);

• Standards for Reporting on Empirical Social Science Research in American Educational Research Association (AERA) publications (AERA, 2006);

• Transparency and Openness Promotion (TOP) Guidelines (Reference Aalbersberg, Appleyard, Brookhart, Carpenter, Clarke, Curry, Dahl, DeHaven, Eich, Franko, Freedman, Graf, Grant, Hanson, Joseph, Kiermer, Kramer, Kraut, Karn and VazireAalbersberg et al., 2018);

• Materials Design Analysis Reporting (MDAR; Reference HawkinsHawkins, 2019); and

• Meta-Analytic Reporting Standards (MARS; Reference Kepes, McDaniel, Brannick and BanksKepes et al., 2013).

Most of the guidelines include some combination of checklists, flow charts, and narrative explanations of what specific items are to be included in a report and what the information is designed to accomplish. I mention two briefly.

First, the CONSORT standards, mentioned above, are arguably the most familiar set of guidelines. They have been adopted by hundreds of professional journals encompassing many disciplines and countries (see www.consort-statement.org/). The CONSORT guidelines have been devised primarily for clinical trials in medical research but have extended well beyond that and are routinely used in clinical trials of psychosocial interventions. As noted in the most recent version, clinical trials have a history of omitting significant information such as description of who was included in the study; sample size calculation (e.g., why a specific size was included in relation to statistical power issues), descriptions of procedures; and presentation of procedures (e.g., randomization) that were not really invoked; and so on with other lapses (Reference Moher, Schulz and AltmanMoher et al., 2001). Beginning in the early 1990s, efforts began to make recommendations for reporting of studies and from that the CONSORT guidelines emerged.

The guidelines consist of a checklist of essential items that ought to be included in any randomized controlled trial of treatment. The checklist displays what is needed, but along with the checklist is a detailed explanation of the items and their rationale for inclusion (Reference Moher, Schulz and AltmanMoher et al., 2001). In addition, the website provides educational material and a database of materials related to reporting of randomized controlled trials (e.g., examples from real trials). In preparation of journal articles, the CONSORT criteria include a list of what to cover and how. This is more concrete than my general statements of ensuring there is a logical flow and underlying theme to the journal article. Yet, the details are basic and critical and hence these guidelines are valuable and widely adopted by many journals.

Second, ClinicalTrials.Gov (https://clinicaltrials.gov/) provides another model to guide research. This consists of preregistration of a study that requires authors to convey their plan for conducting research and analyzing the data. Preregistration allows for the range of participants in research (investigators, peer reviewers, journal editors, funding agencies, policy makers, the public at large) to determine whether the research, when completed, has deviated from the pre-registered plan. Pre-registration of research is now common across many funding agencies and journals (Reference Nosek, Ebersole, DeHaven and MellorNosek et al., 2018). ClinicalTrials.Gov is a large database that includes privately and publicly funded studies of investigations throughout the world. Indeed, this is the largest clinical trials database and as of this writing over 327,000 studies are registered and include studies from all 50 states in the United States and 209 countries (as of January 2020). When clinical trials are comparing interventions or an intervention against a control group, funding agencies (e.g., National Institutes of Health), organizations (e.g., World Health Organization), and a consortium of journal editors (the International Committee of Medical Journal Editors) require investigators to register their clinical trials in advance of the study.

In registration of one’s study, information covers diverse facets of the project. Indeed, there is a multi-page template that includes identification of the investigators of the study, the design, what the interventions are, what will be the outcome criteria (e.g., primary and secondary), the number of anticipated subjects, criteria for inclusion of the subjects, status of procedures to protect clients, and much more. Merely mentioning some of the domains that are included in the register does not underscore their significance in the conduct and reporting of the study. Consider three examples to convey the point.

First, the guidelines require specification of the outcome criteria and which outcomes or measure will be primary and secondary. This is a pre-commitment of the investigator to be clear about the outcome. This does not mean that investigators cannot look at all outcome measures or derive new ones based on interesting findings, pre-specified or not, as the study is completed. However, pre-specification can reduce the tendency in written reports to underscore, emphasize, and consider as primary, those measures that “come out,” i.e., support the hypotheses.

Second, in many studies there are multiple investigators whose roles vary in the design, execution, analysis and other facets of the study and these investigators are likely to be listed as authors. Specification of the investigators and their roles clarifies accountability for the final manuscript. Also, this requires that people in fact have a role in the study before being included. All the expected dynamics of human interactions (e.g., who does and does not get to be an author, where they are placed in the list of authors) and human emotions (e.g., indignation, disappointment, rage, helplessness) surround authorship. War stories here could fill volumes. The guidelines can help a little. At the end of a study, there is accountability of who is in charge, who had what role, and who was involved. If there is fraud or faked data or questionable practices or a manuscript retraction (once there is a question about practices or a finding), the team involved in the study and their roles can be delineated. This can enhance the integrity of the research progress by making clear that one is accountable for the study and its conduct. Third, the guidelines specify whether, how, and where the data will be stored and whether other materials critical to the study will be available. Occasionally journals or funding agencies require that the data are deposited and made available. This practice fostered by the guidelines increases the transparency of the research but also helps replication efforts.

A few comments in passing. To begin, pre-registration does not fix the research so that no further changes can be made. In fact, one can update changes that occur in the course of a study. “Pre”-registrations can be updated after participant enrollment or even after data collection has begun to document any changes that occur in the course of a study (Reference DeHavenDeHaven, 2017; Reference Nosek, Ebersole, DeHaven and MellorNosek et al., 2018). All that is required is to make sure the changes are clear, transparent, and explained. The registration still thwarts post-hoc decision making based on how the data come out or switching some measures and ignoring others, some of the sins of research. An additional point, the many guidelines are designed to improve reporting of research. However, so many facets need to be considered ahead of time in these guidelines that they necessarily influence and guide the design of a study. This article is about preparing a manuscript for journal publication. Consulting and following many widely adopted guidelines underscores the point that key issues about the publication of a journal article emerge before the first subject is run in the study.

6.1 Overview of the Journal Review Process

Alas, through careful deliberation and 30 minutes with your coauthor at a Ouija board, you select a journal and are ready to submit your manuscript for publication. Before you do, consult the Instructions to Authors written in the journal to make sure you submit the manuscript correctly. Usually manuscripts are submitted through a journal portal, i.e., electronically, in which the manuscript file and a letter of submission are uploaded to the journal website. In some cases, you may be required to include sentences or paragraphs in the letter you submit that say this study is not being considered elsewhere in another journal, has not been published before, has met ethical guidelines specified by university or institutional policy and various laws, and that you will give the copyright to the publisher if the manuscript is accepted. Processing of the manuscript could be delayed if your letter does not meet the guidelines provided in the journal.

Once the manuscript is submitted, the journal editor usually sends the electronic file to two or more reviewers who are selected because of their knowledge and special expertise in the area of the study or because of familiarity with selected features of the study (e.g., novel methods of data analyses). Reviewers may be selected from the names of authors whose articles you included in your Introduction. Some reviewers are consulting editors who review often for the journal and presumably have a perspective of the type and quality of papers the journal typically publishes; other reviewers are ad-hoc reviewers and are selected less regularly than consulting editors. Reviewers are asked to evaluate the manuscript critically and to examine whether or the extent to which:

• The question(s) is important for the field;

• The design and methodology are appropriate to the question;

• The results are suitably analyzed;

• The interpretations follow from the design and findings; and

• The knowledge yield contributes in an incremental way to what is known already.

(You may note that these bulleted points encompass the explanation and contextualization features I noted in relation to manuscript preparation. Each point is one that can be readily addressed by the author in preparing the manuscript.) Typically, reviewers are asked to give a summary recommendation (e.g., reject or accept the manuscript). All recommendations to an editor are advisory and not binding in any way. At the same time, the editor sought experts and usually follows their recommendations. Yet reviewers too must make the case for their comments.

Once the paper is reviewed, the editor evaluates the manuscript and the comments of the reviewers. In some cases, the editor may provide his or her own independent review of the paper; in other cases he or she may not review the paper at all but defer to the comments and recommendations of the reviewers. The editor writes the author and notes the editorial decision. Usually, one of three decisions is reached: the manuscript is accepted pending a number of revisions that address points of concern in the reviewers’ comments; the manuscript is rejected and will not be considered further by the journal; or the manuscript is rejected but the author is invited to resubmit an extensively revised version of the paper for reconsideration.

The accept decision usually means that the overall study was judged to provide important information and was well done. However, reviewers and the editor may have identified several points for further clarification and analysis. The author is asked to revise the paper to address these points. The revised paper would be accepted for publication.

The reject decision means that the reviewers and/or editor considered the paper to include flaws in conception, design, or execution or that the research problem, focus, and question did not address a very important issue. For journals with high rejection rates, papers are usually not rejected because they are flagrantly flawed in design. Rather, the importance of the study, the suitability of the methods for the questions, and specific methodological and design decisions conspire to serve as the basis for the decision.

The reject–resubmit decision may be used if several issues emerged that raise questions about the research and the design. In a sense, the study may be viewed as basically sound and important but many significant questions preclude definitive evaluation. The author may be invited to prepare an extensively revised version that includes further procedural details, additional data analyses, and clarification of many decision points pivotal to the findings and conclusions. The revised manuscript may be re-entered into the review process and evaluated again as if it were new. On other occasions, the manuscript may be resent to reviewers familiar with the prior version. Less often the editor may make an executive decision and accept or reject the manuscript without outside input.

Of the three letters, clearly a rejection letter is the most commonly received. Authors, and perhaps new authors in particular, are insufficiently prepared for this feature of the journal publication business.Footnote ⁴ Journals often publish their rejection rates, i.e., proportion of submitted manuscripts that are rejected, and this figure can be quite high (e.g., 70–90 percent). Often the prestige value of the journal is in part based on the high rejection rate. Yet, the rate is ambiguous at best because of self-screening among potential authors. For example, for very prestigious publication outlets (e.g., Psychological Review, Science) where psychological papers are published, the rejection rates cannot consider the fact that most authors are not likely to even try that outlet if they have a contribution that falls within the topic and format domain. Rejection rates across journals are not directly comparable. Even so, the rates give the would-be author the approximate odds if one enters the fray.

Although beyond our purpose, the review process deserves passing comment. The entire process of manuscript submission, review, and publication has been heavily lamented, debated, and criticized. The peer-review process has a long history as an effort of quality control over the content and standards of what is published (Reference SpierSpier, 2002). The topic is central to science broadly and continues to be assessed, commented on, and evaluated with efforts to alter or improve the processes (e.g., Reference Elson, Huff and UtzElson et al., 2020; Reference Kirman, Simon and HaysKirman et al., 2019). The alternatives to peer review (e.g., no review, judgment by one person such as the editor) have their own liabilities. Many journals invoke procedures where the identity of the authors and the reviewers is masked, i.e., names are not included on the manuscript sent to reviewers or the reviews sent to authors. The goal is to try to limit some of the human factors that can operate about responses to a person, name, or other facet and to allow reviewers to be candid in their evaluations without worrying about facing the colleague who will never speak to them again. The peer-review system is far from perfect. The imperfections and biases of peer review, the lack of agreement between reviewers of a given paper, the influence of variables (e.g., prestige value of the author’s institution, number of citations of one’s prior work within the manuscript) on decisions of reviewers, and the control that reviewers and editors exert over authors have been endlessly vigorously discussed (e.g., Reference Bailar and PattersonBailar & Patterson, 1985; Reference Benos, Bashari, Chaves, Gaggar, Kapoor, LaFrance, Mans, Mayhew, McGowan, Polter, Qadri, Sarfare, Schultz, Splittgerber, Stephenson, Tower, Walton and ZotovBenos et al., 2007; Reference CicchettiCicchetti, 1991; Reference SmithSmith, 2006; Reference Stahel and MooreStahel & Moore, 2014).

Understanding the review process can be aided by underscoring the one salient characteristic that authors, reviewers, and editors share, to wit, they are all human. This means that they (we) vary widely in skills, expertise, perspectives, sensitivities, motives, and abilities to communicate. Science is an enterprise of people and hence cannot be divorced from subjectivity and judgment. In noting subjectivity in the manuscript review and evaluation process, there is a false implication of arbitrariness and fiat. Quality research often rises to the top and opinions of quality over time are not idiosyncratic. Think of the peer-review process as the home-plate umpire in a baseball game. Any given call (e.g., strike) may be incorrect, arguable, and misguided. And any given pitcher or batter suffers unfairly as a result of that call. As reviewers (the umpires) make the call on your manuscript (rejection, you strike out), you too may have that occasional bad call. But over time, it is unlikely that all manuscripts an author submits receive a misguided call. Pitchers and batters earn their reputations by seeing how they perform over time, across many umpires, and many games. One looks for patterns to emerge, and this can be seen in the publication record of an active researcher.

6.2 You Receive the Reviews

Alas, the editorial process is completed (typically within three months after manuscript submission) and the reviews are in. You receive an email (or in olden days a printed letter) from the editor noting whether the paper is accepted for publication and if not whether it might be if suitably revised. It is possible that the letter will say the manuscript is accepted as is (no further changes) and praise you for your brilliance. The letter may comment further that the reviewers were awed by how the study was executed and how well the manuscript was written. If this occurs, it is the middle of the night and you are dreaming. Remain in this wonderfully pleasant state as long as you can. When you awake, your spouse, partner, or significant (p < .05) other reads the email and you read one of the three decisions noted previously.

If the manuscript is accepted, usually some changes are needed. These do not raise problems. More often than not, the manuscript is rejected. There are individual differences in how one reacts to this decision. Typically, one feels at least one of these: miffed, misunderstood, frustrated, or angry at the reviewers. Usually one has only the email comments and has limited avenues (e.g., scrutiny of the phrasing and language) for trying to identify who could have possibly rejected the manuscript. If a hard (printed) version of the reviews was sent to you, one can scrutinize the font style, key words, possible DNA remnants of the reviewers’ comments sheets, and molecules on the pages that might reveal pollutants associated with a specific city in the country. (I myself would never stoop to such behaviors but I have a “friend” who, over the years, was able to identify two not-so-friendly reviewers who unwittingly left clues that I – I mean my friend – was able to decipher.) To handle a rejection verdict, some authors select one of the very effective psychotherapies or medications for depression; others use coping strategies (e.g., anger management training, stress inoculation therapy) or complementary and integrative medicines (e.g., acupuncture, mineral baths, vegan enemas). (I myself use all these routinely with their order balanced in a Hyper-Graeco-Latin Square Design.)

The task is to publish one’s work. Consequently, it is useful and important to take from the reviews all one can to revise the manuscript. Maladaptive cognitions can harm the process. For example, when reading a review, the author might say, the reviewer misunderstood what I did or did not read this or that critical part. These claims may be true, but the onus is always on the author to make the study, its rationale and procedures, patently clear. A misunderstanding by a reviewer is likely to serve as a preview of the reactions of many other readers of the article. Indeed, most readers may not read with the care and scrutiny of the reviewers. If the author feels a rejected manuscript can be revised to address the key concerns, by all means write to the editor and explain this in detail and without righteous indignation and affect.

Authors often are frustrated at the reactions of reviewers. In reading the reactions of reviewers, the authors usually recognize and acknowledge the value of providing more details (e.g., further information about the participants or procedures). This is the descriptive facet of manuscript preparation I discussed previously. However, when the requests pertain to explanation and contextualization, authors are more likely to be baffled or defensive. This reaction may be reasonable because much less attention is given to these facets in graduate training and explanation and contextualization are much less straightforward. Also, reviewers’ comments and editorial decision letters may not be explicit about the need for explanation and contextualization. For example, some of the more general reactions of reviewers are often reflected in comments such as: “Nothing in the manuscript is new,” “I fail to see the importance of the study,” or “This study has already been done in a much better way by others.”Footnote ⁵ In fact, the characterizations may be true. Authors (e.g., me) often feel like they are victims of reviewers who wore sleep masks when they read the manuscript, did not grasp key points, and have had little exposure to, let alone mastery of, the pertinent literature. Occasionally two or more of these are true.

As often as not, it is the reviewers who might more appropriately give the victim speech. The comments I noted are great signs that the author has not made the connections among the extant literature and this study and integrated the substantive, methodological, and data-analytic features in a cohesive and thematic way. Reviewers’ comments and less than extravagant praise often reflect the extent to which the author has failed to contextualize the study to mitigate these reactions. The lesson for preparing and evaluating research reports is clear. Describing a study does not establish its contribution to the field, no matter how strongly the author feels that the study is a first.

Let us assume that the manuscript was rejected with an invitation to resubmit. As a rule, I try to incorporate as many of the reviewers’ and editor’s recommendations as possible. My view is that the reviewer may be idiosyncratic, but more likely represents a constituency that might read the article. If I can address several or all issues, clarify procedures that I thought were already perfectly clear, and elaborate a rationale or two, it is advisable to do so. Free advice from reviewers can and ought to be used to one’s advantage.

There are likely to be aspects of the reviews one cannot address. Perhaps reviewers provide conflicting recommendations, or a manuscript page limit precludes addressing or elaborating a specific point. Even more importantly, perhaps as an author one strongly disagrees with the point. Mention these in the letter to the editor that accompanies the revised manuscript. Explain what revisions were or were not made and why. If there are large revisions that alter the text (few sentences), methods or data analyses, help the editor by noting where the change can be found in the manuscript and even submit an extra copy of the manuscript in which the changes are tracked in some editing/word-processing system.

The investigator may receive a rejection letter and decide simply to submit the manuscript as is to another journal. I believe this is generally unwise. If there are fairly detailed reviews, it is to the author’s advantage to incorporate key and often not-so-key points, even if the manuscript is to go to another journal. I have often seen the same manuscript (not mine) rejected from two different journals in which there were no changes after the rejection from the first journal. The authors could have greatly improved the likelihood of publication in the second journal but were a bit stubborn about making any revisions. Even if the manuscript were to be accepted as is in the second journal, it is still likely the author missed an opportunity to make improvements after the first set of reviews was provided. In general, try to take all the recommendations and criticisms from the reviews and convert them to facets that can improve the manuscript. Obstacles to this process may stem from our natural defensive reactions as authors or a negativity bias and the occasional brutish way in which reviewers convey cogent points. (I remember being highly offended the first two or three times reviewers noted such comments, “the author [me] would not recognize a hypothesis if it fell on his lap” and “the design of this study raises very important issues, such as whether it is too late for the author [me] to consider a career change.” I have come to refer to all this as the pier-review process to underscore how often reviewers have made me want to jump off one.)

There is an additional reason to encourage taking advantage of the review process and trying to improve a manuscript we might think is perfect. For those researchers who remain in academia, one’s published studies occasionally are read as part of a promotion process. As an author we might feel relieved that a study or two was published and view that automatically as things are great. In some ways it does, but as this study is read later we still want to be sure the case was made in a compelling fashion and reviewer suggestions might help. My view is to incorporate as many recommendations, changes, and comments as possible. I begin with the view that reviewers are experts and their recommendations, concerns, and misunderstandings are facets of the manuscript it behooves me to address.

It is worthwhile and highly rewarding to publish one’s research. The process takes time and persistence. Also, contact with others through the review process can greatly improve one’s work. In my own case, reading the reviews occasionally has stimulated the next studies that I carried out. In one case, I befriended a person who had been a reviewer of my work earlier in my career. Over time and from following his work, it was very clear that he was behind an influential review although his identity had been masked. Years later over dinner, I mentioned his review in a distant past, the study it generated, and the very interesting results and, of course, expressed my gratitude. His suggestion actually led to a few studies. (His review of my manuscript was not entirely positive, which probably is the main reason I hid in the bathroom of the restaurant until he paid the check for dinner.) The lesson is more than getting one’s manuscript published. Reviews can be very educational and it is useful to let the comments sit for a while until the rage over rejection subsides.

The journal review process is not the only way to obtain input on one’s manuscript. Once in a while, I send a penultimate draft of a manuscript to experts in the field whom I do not know. I convey in a letter what I am trying to accomplish and ask if they would provide feedback. I have done this on several occasions and cannot recall any colleague who has refused to provide comments. The comments are usually detailed and quite constructive and have a different tone from those that emanate from the journal review process. The comments in turn can be used to devise the version that is submitted for publication.

2.1 Step One: Consider the Institution, the Curriculum, and the Students

It is helpful to take a realistic look at the academic culture of the place where you are teaching. You can build on your own observations by consulting with colleagues, and from students themselves, about habits, practices, and expectations that are most common (Reference NilsonNilson, 2010). If you are new to an institution, you may be surprised at the norms for teaching and even more surprised at how students typically approach their learning. Some norms are part of the formal institutional structure, such as the lecture/section method used at many research institutions, or the types of assignments given within a certain curriculum. Other norms arise informally, and are passed down from student to student, and from teacher to teacher.

The purpose of understanding teaching and learning norms is not to copy what has been done before, but to anticipate how your pedagogical choices will be received. If you are new to a campus, AND you are asking students to learn in a whole new way, you may want to ease them into the change and be prepared for a little resistance. It is not always easy to discern the norms and culture of an institution, and you may want to take some assertive steps to get the information you need. We suggest talking to as many colleagues and students as you can. Here are some questions you might ask:

• How much of the reading will students do?

• What kinds of lectures are students used to? PowerPoint? Stand-up comedy?

• Is there precedent for students working together in groups? Do they do group projects?

• What kinds of assignments are typical? Long, formal, end-of-term papers or shorter, more targeted essays and exercises?

• How much of the teaching is experiential? Formal? Innovative?

Once you have a better sense of the context in which you are teaching, you can consider your goals and determine ways to achieve them.

2.2 Step Two: Think about Who Your Students Are and How You Will Include All of Them

In the previous section, we discussed the importance of designing the course around student learning, focusing on cognitive elements that allow students to understand course goals and engage in activities that facilitate learning. But there is more to learning than cognition, because (a) students are more than mere thinkers – they are individuals with their own feelings and sense of themselves, (b) instructors are more than mere providers of knowledge – we bring our own identities, biases, and practices to the process of teaching, and (c) science is more than facts and theories – the psychological literature reflects the identities, biases and limitations of the scientists and the systems that created it. Unfortunately, teaching is an opportunity to recreate and promulgate ideas and practices that, even in their desire to illuminate and explain, allow some students to experience affirmation and growth while others experience disenfranchisement and isolation.

There is a robust literature that explains the process by which schooling includes some students and excludes others, and provides guidance to instructors for making their teaching more inclusive. For example, the literature on Stereotype Threat (Reference SteeleSteele, 2011) suggests the process by which the perception of cultural stereotypes results in different levels of performance among students. Building on the work of Reference MaslowMaslow (1962), researchers in recent decades have been exploring the concept of students’ sense of belonging as a variable that might determine motivation and engagement. This research suggests strategies of creating a more inclusive classroom (Reference StrayhornStrayhorn, 2012).

This research is ongoing and expanding. Consider becoming more familiar with the literature by reading one of the articles listed below. In the meantime, the following strategies will begin to help you address your students’ sense of belonging:

• Get to know your students and give them a chance to be known in class using ice breakers and other activities that allow students to reveal themselves.

• Build one-on-one or small group office hours into your class early in the term.

• Articulate clearly and regularly your desire for everyone to learn. Focus especially on classroom discussions as places of learning and respect. Set standards of respect and fairness for all discourse that takes place in your classroom.

2.3 Step Three: Focus on Student Learning to Define the Overall Purpose of the Course

As implied above, there is enormous benefit in framing or defining your class around core learning goals (Reference DiamondDiamond, 2008). Teachers experience renewed motivation in teaching when they move from a content-centered approach (i.e., getting through the material) to a learning-centered approach (i.e., helping students achieve). This renewed sense of purpose typically generates more creative and innovative approaches to classroom teaching, which, if nothing else, makes the process more interesting. In addition, teachers who successfully communicate their purpose to students may find their students have increased motivation and willingness to work and learn. In classes where the purpose seems to be defined around teacher’s lectures and interests, students are more likely to feel like and act like spectators. However, when a class and all its activities are defined around student learning, students are more likely to feel engaged and act like interested participants (Reference NilsonNilson, p. 18, 2010).

The process of defining your goals and objectives begins by asking yourself how you want your students to be different by the end of the course. This holistic approach to thinking about change in students is the first step in Backward Design of your course, a design strategy that begins with the most important aspect of your course, your goals for your students (Reference Wiggins and McTigheWiggins and McTighe, 2005). The difficulty in accomplishing this stems from the challenge of figuring out which goals are most important, and how some goals are prerequisite to other goals. In addition, teachers typically have objectives of different types. For example, there may be facts we want students to know, or skills we want them to acquire or feelings and appreciations we hope they develop. We may also have goals around how students experience us, our course, and our field, and all of these might reflect our own values and beliefs about undergraduate education. All of these goals should be part of your inventory, but then, as mentioned above, the challenge is making sense of them, giving certain goals priority, and turning them into a plan of action. One way to do this is to define a large, terminal goal – the main objective you want your students to be able to achieve by the end of the course – and then work backward, identifying as many of the subordinate skills students will need as you can. Here is an example:

2.4 Step Four: Develop a Course Plan that Pulls Everything Together

Every course has multiple elements: purposes and goals, motivation and incentives, content, activities, and assessments and grading are just a sampling. The end product of a course plan is a course syllabus in which you and your students should be able to identify all of these elements. As a course designer, you should consider how each element of the course fits together or aligns, and how subsequent elements of the course build on previous elements (Reference WulffWulff, 2005). The degree to which your pedagogical plan is transparent to students is an issue we will discuss in a later section. For now, consider using the questions below to develop your plan for how you will teach each aspect of the course.

• What is the purpose of this section or chapter (develop a skill, practice a technique, master an area of knowledge)?

• How will I motivate students to learn this? Is the material or skill innately interesting or valuable? What does it teach students to do? Will students benefit from a demonstration or model? Will I use a graded test or assignment to increase motivation?

• What new information will students need (theories, studies, examples, etc.) and how will they get it (lecture, reading, video, observation, etc.)?

• What action will students perform on that new information (write about it, discuss it with peers, experiment on it, reflect on it, etc.)?

• How will I assess what students are learning (graded paper, ungraded written assignment, observation)?

2.5 Step Five: Write a Course Syllabus that Establishes a Contract between You and Your Students

The final step in designing a course is the presentation of the syllabus. The syllabus accomplishes one essential goal: it supplies students with all the information they need in order to understand and complete the course in a way that helps them set their expectations and guide their behavior. It can be useful to think of the syllabus as an informal contract between you and your students.

There are many styles of syllabi. Some faculty members choose to put everything in writing, including the purpose of the course and the rationale for its design, while others include just bare-bones logistical information about due dates, grade requirements, and texts. To help you decide how much detail to include, think about what is important to you and to your students. Also, use the syllabus as a reference or teaching tool throughout the semester. Tone and style are both personal choices; however, be aware that the tone of the syllabus does communicate something to your students. You’ve seen hundreds of syllabi in your lifetime, so we don’t have to describe one. Still, you might want to consider these suggestions for writing a good one.

• Do not take for granted that your students know more about your institution than you do. Remember, many students in any classroom are just as new as you are. Avoid abbreviations and lingo. Remember, first-year students and part-time students may not be familiar with nicknames and other local jargon. Stick to the facts and include as many as you can.

• Highlight the most important ideas or processes of the course. Don’t be afraid to include some big ideas in the syllabus, especially if they provide a context or purpose for the course.

• Build your weekly calendar around questions rather than topics.

• Edit carefully the calendar information you include. Be aware of holidays and other campus activities. Remember, students will use this syllabus to plan their semester.

• Make the document as useful as possible, so that students will keep it and look to it often. Whether on paper or on the web, the syllabus should be a useful document that you and your students refer to because it has good, reliable information.

• Build the weekly items in your syllabus around questions to be answered rather than topics to be covered.

3.1 Choose a Textbook, Readings, and Resources that Help You Teach

The trick here is to resist the temptation to let the tail wag the dog by allowing the content and structure of the text to structure the goals of your course. Few faculty members ever find the perfect textbook until they break down and author their own, and even then, there always seems to be something out of place, missing, or overemphasized. Most instructors use supplemental readings and digital resources to add emphasis and to provide students with more varied ways of learning material.

Some easy ways to find a respectable pool of good textbooks to consider are: ask your colleagues to make recommendations (they will know the level and type of textbook students at your institution are used to, and they may even have direct experience teaching with that book); the core collection or reserve room of the library will have copies of all the textbooks currently in use; write to publishers for review copies of books you have heard of or seen advertised; consult an online resource, e.g., A Compendium of Introductory Psychology Texts at http://teachpsych.org/otrp/resources.

Your first consideration is the content. You cannot teach effectively from a book that you neither respect nor agree with, unless you design the entire course around debunking the text, which many students find confusing. This does not mean that you have to agree with everything the authors present. Allowing students to see you display a little healthy disagreement with authority of the text is probably good for most students, but it should not be a daily ritual. Find a book that provides intelligent and scholarly treatment of most topics, and that does so in language that you and your students can understand and appreciate. If the book organizes material in a way that advances your understanding of things, then you have an additional advantage. Quality of content is the most important consideration, but textbooks contain many other qualities that can help you teach more effectively. Some of these include: illustrative examples that explain concepts in various ways; exercises and activities that you can use during class or as out-of-class assignments; side bars and special inserts that discuss related topics like teaching students about the field, real-life or policy applications, personal biographies of researchers or historically important research. Most books these days also include student study aids, such as review questions or self-test. A textbook today may include an online supplement that can help you develop lectures or add vibrant visual material to the class. Where cost is an important consideration, consider using one of the increasing number of “open-source” or royalty-free textbooks available in electronic form. Textbooks come with myriad bells and whistles, not all of which will be helpful. Remember, the book is just a tool to help you teach better. It is not the entire class, nor is it a script that you have to follow. On the other hand, students are used to focusing on “the book” and looking to it for answers and guidance, so you are smart to get one that really complements your approach.

3.2 Be Smart and Creative in Your Use of Class Time

There are many things you can do with class time other than lecture. Class time is your most valuable teaching commodity, but to make the most of it, we need to design it in the context of what students will do in other settings such as read, do homework, or work with other students. With this in mind, class time is probably not the best time for students to encounter new material for the first time. Research in science education, for example, suggests that class time is a good opportunity to let students work together, and for you to observe students at work and give them timely appropriate feedback (Reference Deslauriers, Schelew and WiemanDeslauriers et al., 2011). If you are a stimulating lecturer who can motivate, stimulate, and inspire students to greater heights of academic achievement, then some amount of lecture will likely serve you and your students well. But uninspired lectures that simply cover material, particularly material that can be learned by reading or watching a video, are a poor use of valuable time with students. If you need to introduce or review material, do it quickly – within 15 or 20 minutes. Use the remainder of the time to:

• organize small-group tasks that allow students to engage or question material; or

• all-class discussions about interesting controversial topics. These can be organized as debates, or extended role-play exercises that ask students to take the perspective of a point of view or theoretical orientation; or

• demonstrations with discussion and analysis.

The design of class time is even more important if your class is longer than 50 minutes or only meets once a week. In these cases, it is important to break the class into clear segments with clear goals.

Class time with students is a valuable resource that you must steward with advanced planning. Design your class so that students arrive with questions or insights they have gleaned on their own. Use class time to give students a chance to practice analyzing theories, applying concepts, building models, or simply answering questions. Some good examples appear below.

3.3 Design Assignments that Allow Students to Make Better Use of Class Time

Students spend more time completing assignments than any other aspect of school, so it is vital that assignments require students to do significant, targeted, academic work. Students’ performance on assignments can be improved by connecting some aspect of the assignment to the work students will do in class the next day. For example, if students are writing reports on research articles, have them use some aspect of those reports to do an in-class analysis. Motivation can be further increased by setting up in-class peer groups that require individuals to come to class prepared. As you begin to develop your first assignments, look back on some of the assignments you were given, and ask colleagues for their ideas. Consider the assignments that are typically used in psychology classes – the research report, the case analysis, compare and contrast, journal article review, lab report – because these are forms that may be familiar to students. Then focus on the specific goals and objectives you have created for that section of that course, and modify the assignment in the following ways.

• Consider designing a series of assignments that build students’ skills over the course of term.

• Align what students are doing in their assignments with what they are doing in class. Students can practice modes of analysis in class that they can apply to assignments. Conversely, students build expertise in assignments that they use to participate in class. Make these connections clear to students.

• Match the length, difficulty, and scope of the task to the skills you want students to demonstrate. Shorter, focused assignments typically offer more stimulating educational experiences than longer, more complex works. At some point, it may become necessary for undergraduates to demonstrate their ability to sustain an analysis or project for over 40 pages, but such work is often done as an undergraduate thesis or capstone project.

• Communicate the purpose of the assignment in clear terms of high academic standards. Focus on what students are accomplishing for themselves.

• Effective assignments are clearly defined and have well-established standards. It does no good to wait until you grade a paper to tell students what you were looking for.

• Effective assignments are no larger than the skills they are designed to teach. Don’t ask students to produce huge products and long papers to demonstrate small skills over and over again.

• Effective assignments produce real products, with form and structure. Rather than asking students to write a 5-page paper on X, ask them to write up a case analysis or grant proposal, mock legal brief, committee report, letter to the editor, or a publishable book review.

• Effective assignments may make use of imagination and perspective, ask students to take on a role and write from that perspective, e.g., take the role of a patient, or speculate on a hypothetical situation.

• Effective assignments combine the demonstration of well-defined skills and abilities with opportunities for creativity, uniqueness and personal expression.

• Effective assignments ask students to demonstrate skills that are directly related to the core goals of the course. That is, students should have to rely on what they learned in class to successfully complete an assignment.

• Effective assignments often include students working in pairs or teams, although students should be individually accountable for their own work and their own grades.

3.4 Use Assessment and Grading to Review Students’ Work and Give them Necessary Feedback

Grading students’ work effectively is a critically important part of teaching, and not easily done. First of all, let’s define our practice. Assessment is the practice of critically reviewing students’ performance. It can be formal or informal. It can result in constructive feedback, or simply a shift in our perception. Assessments, such as ungraded quizzes or “clicker” questions, can also be used to help students assess their own understanding. Assessments are powerful teaching tools. They keep teachers and students connected to learning and they provide both with valuable guidelines for how to succeed.

Sometimes, as with formal tests, quizzes, and papers, our assessments result in grades. Grades are fraught because they are typically associated with formal, institutional records. In other words, they have lasting consequences. All assessments should be accurate, fair, constructive, and timely, but grades need to be especially so. Like it not, they are a big part of what motivates students to work. Because of that, the achievement of grades should be based on your central values and objectives for whatever course you teach. Here are some suggestions.

• Grading begins with your very first thoughts about the course. Once you identify the skills and abilities you want students to demonstrate, you must assign value to the achievement of those skills and to the partial achievement of those skills, and then translate that value into what every grading scheme your institution requires. Listed below are a set of considerations that you can apply to every assignment or test you grade, as well as to the overall grade.

• Establish and communicate specific standards for everything you grade. Inform students upfront what will be graded and how. Reaffirm those standards in the comments that accompany your grades. Remember, the primary purpose of grades is to give students useful feedback about their progress.

• Begin grading short assignments and in-class work early in the term. This will help students become familiar with your standards and their level of preparation.

• Establish ground rules to achieve fairness in grading. Inconsistency in rules and procedures will communicate favoritism and capriciousness. It is not necessary to establish rigid practices to achieve a sense of fairness; however, your rules must apply to all your students and in the same way.

• Grade a variety of student work. Make sure your grading structure reflects all of the objectives you have identified for the course. Naturally, you will want to give greater weight to the core objectives. However, you can keep students working and learning at a steady pace throughout the term if your grading scheme gives them continuous feedback about how well they are doing along the way.

• The grading of participation in class should, like all other grades, include clearly defined standards.

Remember that, under the best of circumstances, grading is difficult. Grading brings to the forefront a fundamental conflict inherent in our work as teachers. We are helpful guides, mentors, and coaches who work compassionately and tirelessly to help students master a new terrain; but we are also gatekeepers, charged with setting and enforcing standards for participating in a profession (Reference ElbowElbow, 1986). The tension between those two roles is enough to give all of us a knot in our gut when faced with a difficult grading task. The best way to mediate this conflict, fortunately, is relatively straightforward: set out clear standards that students must meet at the outset; then enjoy your role as helpful guide.

4.1 How Do I Get Started Using Instructional Technology in my Teaching?

Psychology researchers are often proud of being Jills-of-all trades. Many of our research projects call on a broad spectrum of skills, and we may often have to switch hats from manager, to programmer, to carpenter in a space of an afternoon. It’s tempting to bring some of those skills to bear on developing technological solutions to teaching problems. However, the costs of developing teaching technologies from scratch are often prohibitive (in terms of your time). Activities like website design may be fun for some of us, but they compete for scarce time with syllabus design, lesson planning, and student contact. Keeping it simple should be a paramount consideration for implementing any technological innovation.

At most universities, the shift to remote teaching has necessitated a vast expansion of technology services to instructors. It is likely that your psychology department will have its own staff of technologists and instructional designers who can assist you as needed. Get to know these people. In the meantime, consider that modern information technology has enabled some discipline-agnostic approaches for engaging students and facilitating reaching learning goals. For example:

• Novel approaches and tools for collaborative knowledge construction, like shared virtual whiteboards, allow small groups to engage in real-time interactions around a shared artifact to create collaborative mind-maps and other diagrammatic or narrative representations. Two widely available examples of such tools are gSlides and LucidChart.

• The availability of volunteer-driven knowledge creation and curation communities provides students an unprecedented chance to engage with the content as contributors. Consider, for example, class assignments that ask student teams to create or improve Wikipedia articles, create and manage new Reddit threads, or ask and answer Quora questions.

Both the discipline-specific and the discipline-agnostic tools have evolved at a phenomenal rate in the last decade, and make today an especially exciting time to use technology to accelerate the teaching of psychology.

3.1 K Awards

There are a number of types of K awards (for more details see: https://researchtraining.nih.gov/programs/career-development). The most relevant for early career psychologists are the K01 (Mentored Research Scientist Development Award for career development in a new area of research or for a minority candidate), K08 (Mentored Clinical Scientist Development Award for development of the independent clinical research scientist), and K23 (Mentored Patient-Oriented Research Career Development Award for development of the independent research scientist in the clinical arena). There also are mid-career and even later career development awards that provide resources for investigators to develop new areas of expertise – and provide mentorship to junior investigators. The K award usually requires that at least 75 percent of your effort (9 calendar months in NIH terms) be devoted to the research project and to career development for 3–5 years. These awards are evaluated as training mechanisms. Applications require not only a research plan but also a training plan for career development activities under the guidance of a research mentor, local collaborators, and external consultants. The university must usually agree to release the PI from most teaching, clinical, and administrative duties. In return, NIH will pay the PI’s salary, up to certain limits. There is a great deal of variation among the different NIH institutes as to which Career Awards are available, what PI qualifications they expect, the dollar limits for salary and research expenses that they will award, their application deadlines, and their supplemental proposal instructions. It is best to contact the relevant institute prior to preparing your proposal to be sure you understand that institute’s guidelines for a K award.

3.2 R Grants

The R grants most relevant to the early academic include the R03, R21, R34, and R01 (for more details see: http://grants.nih.gov/grants/funding/funding_program.htm#RSeries). The R03, small grant program, provides limited funding for a short period of time. Funding is available for two years with a budget up to $50,000 per year. Some institutes (e.g., NIDA) also offer rapid transition awards called a B/Start (i.e., Behavioral Science Track Award for Rapid Transition), which consists of one year of funding for $75,000. Because reviewers submit reviews without a full review meeting, this mechanism often includes a shorter lag time to completion of the review process (i.e., funding occurs within approximately 6 months of the date of receipt of the application).

The R21 is considered to be an exploratory/developmental research grant used to support the early stages of project development (e.g., pilot or feasibility studies). Funding is available for two years and the budget cannot exceed $275,000. Extensive preliminary data are not expected, but applications must make clear that the proposed research is sound and that the investigators and available resources are appropriate to the task. While the R21 mechanism can sometimes be considered to be most relevant for previously funded senior investigators undertaking high-risk/high-reward research and/or a new area of research, it is our experience that first-time investigators can be successful seeking R21s if their idea is novel and has potential for transformative impact in their field of study.

The R34 is a clinical trial planning grant intended to support development of a clinical trial. This program may support: establishment of the research team, development of tools for data management, development of a trial design, finalization of the protocol, and preparation of a manual. For example, NIDA offers this mechanism exclusively for treatment development and some initial testing. The R34 lasts for 3 years with a budget of $450,000, with no more than $225,000 in direct costs allowed in any single year.

The R01 is NIH’s most commonly used grant program which is generally awarded for 3–5 years. There is no specific budget limit, but budgets under a particular amount can be submitted with less detail than more expensive R01s (called modular and typically $250,000 direct costs each year). Budgets over a particular amount (typically $500,000 direct costs each year) must obtain institute approval before being submitted. Although you should request the budget you need to conduct your project, an extremely large scope and budget in an application from a new investigator may raise red flags for reviewers. Interestingly, it is our experience that some early career investigators avoid the R01 mechanism because of their junior status; however, as outlined below, NIH has taken some steps to encourage early career investigators with a “big” idea and adequate pilot data to consider an R01.

3.3 K/R Hybrids

Of note, there is an additional mechanism that serves as a bridge between a K award and an R grant called a Pathway to Independence Award (K99/R00, nicknamed kangaroo). This mechanism provides up to five years of support consisting of two phases. The first phase provides 1–2 years of mentored support as a postdoctoral fellow. The second phase is up to 3 years of independent support (contingent on securing an independent research position). Recipients are expected to compete for independent R01 support during the second phase to allow for continued funding once the K99/R00 support has ended. Eligible principal investigators must have no more than 5 years of postdoctoral research training.

3.4 Advantages and Disadvantages of K Awards and R Grants

K awards and R grant mechanisms each have a number of advantages and disadvantages. A K award can provide 50–100 percent of your salary (depending on the type of K and branch of NIH) for up to 5 years. This allows for a more highly stable period of funding than the typical R01, which usually funds only 20–40 percent of the principal investigator’s (PI) salary for a period of 3–5 years. This allows investigators to concentrate on their specified research efforts without the concerns or distractions of needing to constantly be pursuing additional sources of support or fulfilling extensive clinical or teaching responsibilities at their university. Other advantages of the K award are the opportunities for mentorship, training, and thoughtful development of a programmatic line of research in the PI’s chosen area. The K will provide funding (typically $50,000 in addition to salary support) specifically to support these critical opportunities, which include: time and funds for focused coursework, study materials, access to consultants and mentors – and funds to travel to meet with off-site mentors at their research labs or attend professional conferences. These resources are paired with a highly personalized training plan that is developed as a part of the grant application. Because career development and training is a central aspect of K awards, the expectation of research is different and more modest than that for an R grant that will have a much more highly specified research project (and no training component).

For all of those reasons, the K award is very well suited for the needs of junior investigators who may have only limited pilot data of their own and require additional training experiences before attempting the larger-scale R grant projects. Also, the fact that a K award covers most if not all of one’s salary can be very helpful in environments that require a large percentage or even all of one’s salary to be covered on grants, which often include most medical school positions. It is unusual for more than 33 percent of one’s salary to be covered by an R01, even though the latter is a larger grant because it is less focused on training/support of a junior investigator and more focused on supporting the research project. More recently, even psychology departments and other similar environments that are more associated with hard salary funding have begun creating positions or providing greater flexibility in existing positions for those with a K Award, expanding the scope of environments in which they have great value. Nevertheless, the K award is not necessarily the best mechanism for some junior investigators. Some are discouraged by the prospect of an ongoing role as “trainee.” Others are deterred by the lack of flexibility in the mechanism itself. For example, a K can be transferred to other institutions, but it can take some time and specific evidence that the new environment can support the research and that relevant and willing mentors are on-site. They also do not provide sufficient funding to implement large-scale research projects (e.g., a randomized clinical trial). Moreover, they require significant institutional support documented within the application that is not always proffered or feasible for budgetary reasons or instructional needs. Mentors on Ks do not receive financial support from the grant, which can create challenges getting engagement and sufficient time devotion. K awards also pay a vastly lower indirect cost rate (8 percent) than R grants (typically in the 50–65 percent range). Indirect costs are funds provided to the applicant’s institution to cover the costs of administering and supporting the applicant’s research. This amount is above and beyond the funds provided to the applicant for the research (called direct costs), but is calculated as a percentage of the direct costs. Although this should not lead you to apply for an R grant over a K award if the latter is a better choice for you and your research, you should be aware that the disparity in indirect costs of a K award may leave junior faculty investigators at a disadvantage in terms of obtaining additional institutional support once the application is funded and the research begins. Finally, there has been considerable chatter in recent years that while a K award can really jump-start a career for some, they might counterintuitively slow progress to an R grant for others. Because K awards cover so much of one’s salary it is actually challenging to show one has the available effort to devote to other projects, particularly as the PI, and there is always some confusion about when someone with a K is “allowed” to start submitting R-level grants. As such, K awardees may be slower to write their first R than those without a K award. We should be clear, this is not an argument against a K award per se, as they are indeed great for one’s career, but this unexpected potential impediment to future development of an independently funded research portfolio is important to be aware of for those who pursue a K. These timing concerns need to be balanced with preliminary evidence that researchers who have received a K award compared to those who did not have great success getting their first NIH award (https://nexus.od.nih.gov/all/2019/04/02/association-between-receiving-an-individual-mentored-career-development-k-award-and-subsequent-research-support/).

A major advantage of the conventional R01 award (and to a lesser extent other R grants) is the significantly larger project budgets, dictated by the specific requirements of the scientific protocol. However, new investigators applying for any R grant must be prepared to demonstrate to the review committee that they have the appropriate background, expertise, and skills to implement and complete an independent research project. There are a number of ways to successfully demonstrate these qualities. They include the availability of relevant scientific pilot data, a “track record” of publications in your area of research, and a thorough, well-conceived, and convincingly argued research plan (i.e., scientific protocol). Applications for R funding are evaluated almost exclusively on their scientific merit, significance, and innovation. R01 grants are quite competitive, but there is a tangible advantage in the evaluation process if you are a new investigator defined as not previously or currently holding R01 support (previous R01 submissions do not affect this status until one is funded). Specifically, in many cases your application will be considered in a separate pool of applications devoted to only new investigators. This “levels the playing field” and prevents your application from competing directly with applications from more seasoned investigators. While not a significant disadvantage per se, as noted above, even large-scale R grants rarely cover all or even most of one’s salary. This is unlikely to be a problem in environments where other “hard” or “soft” funding is available, but should be a consideration where one is expected to cover a large portion of their salary as an R grant alone will not be sufficient in these cases.

3.5 Application Types

A large percentage of applications are investigator-initiated (often called “unsolicited”). Investigator-initiated applications can be submitted according to published submission deadlines, most often in February, June, and October. Applications that fall under special interest areas such as HIV/AIDS have different deadlines that accommodate a faster review, so you are encouraged to check these deadlines closely (see https://grants.nih.gov/grants/how-to-apply-application-guide/due-dates-and-submission-policies/due-dates.htm).

Another option is to submit in response to a Request for Applications (RFA). RFAs are meant to stimulate research activity to address NIH-identified high priority issues and areas. They do not utilize regular deadlines and are announced with a specified deadline (often less than 4 months from the announcement). As such, researchers most interested and immersed in these areas of research have a decided advantage because they are likely to have already thought through some of the key issues and in some cases already have available pilot data that could serve as the base for the RFA submission. Of note, these applications typically are reviewed by specially convened panels that are selected based on the specific RFA and are therefore likely to have significant relevant expertise. As one might guess this can be an advantage in that one is getting a review from individuals who are most qualified to evaluate that application. However, an expert also may have particular expectations about how things should be done and may be more likely to focus on esoteric aspects of the application that might go unnoticed by reviewers with less expertise in that area.

One source of confusion can be Program Announcements. PAs are similar to RFAs in that they are issued by one or many Institutes and outline topics that are of particular interest. Like an RFA, PAs provide a level of assurance that the type of research you are proposing will be of interest to the institute that issued the PA. More recently, NIH has begun phasing out PAs and opting to implement Notice of Special Interest (NOSI). A NOSI is a new format for NIH Institutes/Centers to share and update their research priorities. NOSIs are intended to replace PAs that do not have special review criteria or set-aside funds. Each NOSI describes aims in a specific scientific area(s) and points to Funding Opportunity Announcements (FOAs) through which investigators can apply for support.

4.1 Project Summary

The project summary is a two-part overview of your proposed project. The first part is the abstract in which you have 30 lines to describe succinctly every major aspect of the proposed project, including a brief background, specific aims, objectives, and/or hypotheses, public health significance, innovative aspects, methodology proposed, expected results, and implications. The second component of the Project Summary is the Project Narrative, which provides a plain-language 2–3 sentence description of your application.

4.2 Aims

Aims provide a one-page statement of your goal, objectives, and expected outcomes and implications. The aims should start with a brief statement of the problem and its public health impact, followed by what is known, and then the gap between what is known and how your project will address this gap. The most important part is the statement of your specific aims and the hypotheses you have for each aim. These statements should be concise and include clear, testable hypotheses. Occasionally, you may include an exploratory aim that addresses an important question but for which enough information is not available to draw a hypothesis; however, these should be used sparingly. You then should conclude with a summary paragraph that also suggests the research directions and implications that this work will spawn. NIH wants long-term not short-term relationships with its applicants. As such, your ability to discuss how this work will not be a single effort but the start of an effective line of research is crucial. A handy template of possible steps to follow in arranging your aim statement is provided in Table 15.1 and we provide guidance on things to consider doing and to avoid doing at the end of this section in Table 15.2.

4.3 Research Strategy

4.4 Human Subjects

Although it is not placed in the body of the research plan, the section on the protection of human subjects and the inclusion of both genders, children, and underserved members of minority groups is an important part of your application. It should carefully describe aspects of the grant related to the risk–benefit ratio and demonstrate that all necessary precautions are in place to protect the rights and safety of human subject participants. In most R grants this section includes virtually all of the information expected in an application for Institutional Review Board (IRB) approval. This should include strategies to ensure adequate recruitment of underserved groups and a clear statement for why certain groups aren’t included, especially if for methodological reason (which also should be noted in the section on Approach). This section also should include a data and safety monitoring plan as it is now required for all clinical trials (phases I, II, or III) and a monitoring board for larger-scale trials, multi-site trials, and those including vulnerable populations (e.g., prisoner populations).

4.5 Additional Sections

The following sections also need to be included in your grant application: Appendix Materials, Bibliography & References Cited, Care and Use of Vertebrate Animals in Research, Consortium/Contractual Arrangements, Consultants, Facilities & Other Resources, Resource Sharing Plan(s), Select Agents, Multiple PD/PI, and Use of Internet Sites. See http://grants.nih.gov/grants/writing_application.htm for additional details. Additional content sections specific to K award applications include the Candidate’s Background, Career Goals and Objectives, Career Development/Training Activities during Award Period, Training in the Responsible Conduct of Research, Statements by Mentor, Co-mentor(s), Consultants, Contributors, Description of Institutional Environment, and Institutional Commitment to the Candidate’s Research Career Development.

6.1 Core Review Criteria

Your application is evaluated on the following five core review criteria: (1) Significance, (2) Investigator(s), (3) Innovation, (4) Approach, and (5) Environment. For a detailed outline of these criteria with a comparison with previous criteria, see http://grants.nih.gov/grants/peer/guidelines_general/comparison_of_review_criteria.pdf. As we have covered Innovation and Approach thoroughly in “Preparing Your Application,” we focus only on the key features of the other three criteria here.

• Significance. Is this work addressing an important question and will have an impact on the field in terms of knowledge, application, or in the best case scenario both? It is not crucial that the application be immediately addressed in a submission (especially in more basic research projects), but reviewers will want to see evidence of how this work ultimately could have such impact.

• Investigator(s). Are you qualified to conduct this project and how well does your team of collaborators (or mentors for career awards) provide specific support in areas where your experience and expertise could be supplemented? For evaluating your credentials, reviewers often will focus on training and specific research productivity. Also, evidence that there is a specific role for the collaborators/mentors is crucial, as is some evidence of past work together or future plans to ensure their participation. This can be best represented in a letter of support and clearly articulated in “personnel justification,” which is an additional administrative section of the grant not covered here.

• Environment. Can the work be carried out with adequate institutional support and resources? Additionally, are there unique features of the scientific environment, subject populations, or collaborative arrangements that are evident at the research site? These strengths should be clearly articulated in “facilities,” which is an additional administrative section of the grant not covered here.

6.2 Overall Impact/Priority Score

For each of the five core review criteria, reviewers evaluate your application and provide a score from 1 (exceptional) to 9 (poor). Each reviewer then also provides an overall score, also from 1 to 9. There are no clear guidelines to reviewers in how to develop the overall score from the scores for the core areas, and it is not meant to be an average or median score. Moreover, your score can be influenced by several other additional criteria including human (or animal) subject issues. Reviewers can make recommendations about your budget, but these recommendations should not affect your score.

From the overall scores of each reviewer as well as the other committee members, a normalized average is calculated and multiplied by 10 to provide a final priority score from 10 to 90, where 10 is the best score possible. As much as we’d like to indicate a range of likely fundable scores, there just simply aren’t hard rules that apply in all cases across all institutes (but for some guidance see: www.nlm.nih.gov/ep/FAQScores.html). With that said, many PIs would be quite pleased with a score under 30.

8.1 Respect Deadlines

For many individuals deadlines are crucial to setting goals, staying on task, and not losing motivation. Be aware of the deadlines and what goes into getting things done in a timely manner. Be more conservative with things that rely on others, such as letters of support or analytic sections prepared by a statistician. With that said, deadlines can have their drawbacks, because they can lead to procrastination and a burst of work near the deadline, without ample time to run ideas past others and have a sufficient pre-review of the application from collaborators and potentially helpful colleagues. For this reason it can be useful to utilize a timeline for each step along the way to submitting your application. As can be seen in Table 15.3 illustrating a mock timeline, it can take nearly two years from the start of idea development for a grant to actual funding should resubmission be needed (as it most often is). This timeline illustrates the previously stated notion that grant funding is more of a marathon than a sprint.

8.2 Ensure Feasibility

While you want your application to be methodologically rigorous and have high impact on your field, you cannot lose sight of feasibility. The reviewer code word used when there are doubts about feasibility is “over-ambitious,” and it is a clear kiss of death when this term is used to describe an application. Therefore, keep your specific aims focused, and make hypotheses that you can clearly tie back to theory and/or pre-existing data. Consider the necessity of multiple studies within a single grant. Although these can be quite elegant, the connection between studies can provide many pitfalls, especially if subsequent studies rely on particular results from initial studies. Remember, although your passion for your research area may be strong and your intellectual curiosity high, each grant application represents only one small step in a research career that may last for several decades. Try not to be ruled by emotions (especially when receiving and responding to critical feedback) and keep a clear eye on your long-term goals. Persistence, patience, and creative problem solving are usually critical ingredients in the career of a successful independently funded investigator.

8.3 Be Clear

NIH clearly states that you cannot have any contact with reviewers before, during, or after your review. Therefore, the only way you can get your point across is the extent to which you communicate with them in the application. Within the section on Approach, the subsections on Preliminary Studies as well as Potential Problems, Alternative Strategies, and Benchmarks for Success provide a great opportunity for this. For the subsection on Preliminary Studies, you can make your case that you have sufficient background (and pilot data especially for an R01) to conduct this work and that it marks a logical next step in this line of research, both for you and for the field in general. For the subsections on Potential Problems and Alternative Strategies (previously referred to as design considerations), this is your chance to walk reviewers through the highly complex discussions you and your collaborators had when you determined the best decisions for the application. This is an interesting section and presents a real opportunity because some applicants largely ignore it and at best tell the reviewers essentially “don’t worry we know what we are doing” or “we’ve got it covered.” As a new investigator, it is up to you to ensure that the reviewers understand the decisions you made. This section also increases the odds that the primary reviewer can best present your application and that others reading can quickly understand some of the key features of your application. Think of it as giving reviewers access to all the critical thought that went into the strategies you ultimately chose (as well as those you didn’t choose). Finally, your Benchmarks for Success show a level of sophistication and often can help ameliorate any fears about feasibility. This section would benefit greatly from a table that outlines the planned activities of the grant and the deliverables at each time point.

8.4 Show You Know the Literature and Your Work is Adding to it

Especially as a young investigator, your research team is crucial and it is important for you to clearly highlight their role in your application. For K awards, mentors are especially key elements of the successful application. It is critical to tell a clear story of each person’s role in your training, with as much detail as possible. Explicitly, it is not enough to simply list the “right” people. It is necessary to explain who they are and why they were chosen, show that you will have the right training experience with them, and describe how each mentor will contribute to your career development.

For R grants and the young investigator, the role of collaborators can be a bit more ambiguous. In some academic settings, you may experience a tension between the traditional value placed on independence and the emerging growth of team-based or multidisciplinary science where it’s no longer expected (or even possible) for one individual to master all elements of a complex research project. In fact, at NIH it is usually expected that applications will include a team of experts representing different domains. For example, in applications related to mental health and addictions it is common to see psychologists, psychiatrists, statisticians, anthropologists, epidemiologists, neuroscientists, economists, etc. collaborating together. A true research team will involve well-selected experts that can work well together, each contributing unique and relevant expertise to the proposed project. It is crucial to clearly articulate the key parts of the application and the role that each collaborator plays in those parts.

8.5 Trust the System and Put Your Best Work Forward

As mentioned above, there are strategies to increase the odds of funding such as trying to steer your application to the most appropriate committee, “guessing” what likely reviewers might want, and talking to program staff to avoid making mistakes or proceeding in a negative direction. However, you should be careful about these efforts becoming more about gaming the system than developing the best application for you. It is important to note that for every great game player, there is a straight-shooting scientist who has a strong sense of their interests, is willing to find a mechanism in NIH that accommodates that interest, makes efforts to align their interests with that of NIH including RFAs and PAs but does not let this betray their own actual interests, and simply allows the process to play out. This is not to say that some strategizing is not warranted, but when the strategies approach more of a game-like level, they hold as much likelihood of backfiring (or simply being irrelevant) than actually helping.

4.1 Academic Pipeline

First and foremost, let’s address the fact that fewer women than men apply for tenure-track academic positions. This is almost surely a product of a “leaky pipeline.” A “leaky pipeline” describes a systematic exit of certain people from the career path. Gender-related pipeline leaks are well-known in science (Reference AlperAlper, 1993), and it is possible that Psychology is “leakier” or worse than other fields when it comes to our pipeline because we start with so many women interested in our undergraduate major. Yet at each stage from undergraduate, to PhD, to postdoc, to faculty positions, women drop out of the field at rates disproportionate to men (Reference Ceci, Ginther, Kahn and WilliamsCeci et al., 2014; Reference Gruber, Mendle, Lindquist, Schmader, Clark, Bliss-Moreau, Akinola, Atlas, Barch, Barrett, Borelli, Brannon, Bunge, Campos, Cantlon, Carter, Carter-Sowell, Chen, Craske and WilliamsGruber et al., 2021). Meanwhile, women are over-represented in adjunct professor positions, university administration, and in fields outside of research such as education and healthcare (APA, 2017; NCES, 2013), suggesting that women are systematically “opting-out” of tenure-track academic Psychology. As we note above, when women do apply for tenure-track jobs, they are more likely to get them than men (both in large observational studies and in experiments; see Reference Gruber, Mendle, Lindquist, Schmader, Clark, Bliss-Moreau, Akinola, Atlas, Barch, Barrett, Borelli, Brannon, Bunge, Campos, Cantlon, Carter, Carter-Sowell, Chen, Craske and WilliamsGruber et al., 2021), so that’s good news. As we mention above, another source of good news is the fact that women who go up for tenure are now as likely as men to get it. Yet, women lag behind men in the rate of being named full professor, which is the highest (non-administrative) academic rank post-tenure. It’s not clear based on the data whether this is just a time lag, and we’ll see more women fill the ranks of full professor in the years to come, or whether women are getting “stuck” at the mid-career associate level rank and are not moving to more senior level positions and recognition (Reference Gruber, Mendle, Lindquist, Schmader, Clark, Bliss-Moreau, Akinola, Atlas, Barch, Barrett, Borelli, Brannon, Bunge, Campos, Cantlon, Carter, Carter-Sowell, Chen, Craske and WilliamsGruber et al., 2021).

Why is it that women leak from the pipeline at a greater rate than men, and why are there not more women in full professor roles if women have been filling the pipeline in large numbers since the 1980s? One possibility is that women do not want academic Psychology jobs and never did – in this scenario, women are leaving the field at each juncture because they are choosing careers that they prefer more. That is of course a reasonable interpretation, especially at the undergraduate level, where many may see Psychology as a great generalist major that will prepare them for other careers. Alternatively, you could argue that this means that we’re missing out on the opportunity to convince more women that they might like to go on to become scientists in our field. Either way, this doesn’t address why the pipeline leaks following a PhD, or especially a postdoc, when trainees have gotten further along the career path toward becoming professors. Don’t get us wrong: it would be great if women were actively choosing careers that they most prefer. But we suspect that women are, at least in some proportion, being forced to “opt out” of the pipeline due to a combination of factors at various levels. We review these pressures in full in Reference Gruber, Mendle, Lindquist, Schmader, Clark, Bliss-Moreau, Akinola, Atlas, Barch, Barrett, Borelli, Brannon, Bunge, Campos, Cantlon, Carter, Carter-Sowell, Chen, Craske and WilliamsGruber et al. (2021) and point to a couple of especially important ones here.

Let’s start with the systemic factors that might be at play in the pipeline. Systemic factors are those related to the values, norms, and institutions that our society creates that in turn impact interpersonal and intrapersonal behavior. One major systemic set of values and norms – that in turn shape our institutions – are gender role expectations. Gender roles are prevalent cultural stereotypes about the behaviors, personalities, and occupations that women and men should engage in and hold (e.g., Reference Wood, Eagly, Olson and ZannaWood & Eagly, 2012). Gender role stereotypes have a major impact on the institutions in which we work and live. The fact of the matter is, our academic institutions (and work institutions, and political systems, more generally) were not created with caregiving in mind. Yet due to gender-based stereotypes related to caregiving and the biological practicalities of childbirth and early child rearing, women are expected to be – and frequently are – the primary caregivers of others in our society. Childcare, eldercare, and care for extended family and community members in need often falls to women. Our society expects, and frankly, benefits from, the largely free caregiving labor expended by women. For instance, when it comes to childcare, American mothers spend roughly 75 percent more hours per week on childcare than fathers (14.0 vs. 8.0 hours; Reference Geiger, Livingston and BialikGeiger et al., 2019). Women are likely aware of these realities – in fact, they have been implicitly or explicitly faced with them since early childhood and both women and men see academia as relatively incompatible with raising children (Reference Mason, Wolfinger and GouldenMason et al., 2013). The difference may be that (some) men expect that they will have a spouse who can pick up the slack with regards to childcare while they focus on their academic career. As most academic women are part of dual-career partnerships, they are much less likely to have this luxury than their male counterparts (Reference Gruber, Mendle, Lindquist, Schmader, Clark, Bliss-Moreau, Akinola, Atlas, Barch, Barrett, Borelli, Brannon, Bunge, Campos, Cantlon, Carter, Carter-Sowell, Chen, Craske and WilliamsGruber et al., 2021). What seems key to women opting into academic careers is seeing other women navigate both a career and kids. When women PhD students interact with women faculty who have children, they are more likely to pursue academic careers (Reference Mason, Wolfinger and GouldenMason et al., 2013), underscoring the importance of representation in science and having access to women mentors (see Reference Lindquist, Gruber, Schleider, Beer, Bliss-Moreau and WeinstockLindquist et al., 2020).

These realities are not unfamiliar to us at all. We have all struggled in some way with trying to combine our career trajectory with our preferred geographical location, dual careers, having children, and/or taking care of elderly parents. These are realities that at least some of our male colleagues don’t face in the same magnitude.

Of course, the gender stereotypes that influence systems also impact interpersonal behavior and a person’s own beliefs. These in turn influence who sees themselves in certain types of careers and can impact whether women opt into academic careers. In America, gender stereotypes include ideas that women are warm caregivers who focus on communal goals whereas men are assertive breadwinners who focus on self-achievement (e.g., Reference Wood, Eagly, Olson and ZannaWood & Eagly, 2012). Women may thus initially be drawn to Psychology majors in unequal numbers because it is a field that – at least stereotypically – is seen as high on stereotypically female qualities, such as caregiving and communion. Psychology is also low on stereotypically male qualities, such as requiring “brilliance” to succeed (Reference Leslie, Cimpian, Meyer and FreelandLeslie et al., 2015). Yet, academic Psychology may prove to have qualities that run against these stereotypes – it can be competitive, requires self-promotion and assertiveness, and does not always have immediate application to communal goals. Gender role congruity theory suggests that these systemic, societal norms interact with others’ and a person’s own view of themselves to predict how interpersonal processes unfold for women as they climb the career ladder.

For instance, gender role congruity shapes how others behave toward women. As women ascend the career ladder, they can get pushback from others for embodying behaviors that are necessary for a scientific career (e.g., being agentic, being a leader, promoting one’s work) but are seen as male-typed. As we review in Reference Gruber, Mendle, Lindquist, Schmader, Clark, Bliss-Moreau, Akinola, Atlas, Barch, Barrett, Borelli, Brannon, Bunge, Campos, Cantlon, Carter, Carter-Sowell, Chen, Craske and WilliamsGruber et al. (2021) there is indeed evidence that women get pushback for engaging in gender-incongruent behaviour – women who try to hold positions of power and who assert themselves often receive blowback from others. Women who identify with other under-represented identities can experience the additive effect of multiple stereotypes (e.g., the angry black woman, the overly emotional Latina; see Reference Gruber, Mendle, Lindquist, Schmader, Clark, Bliss-Moreau, Akinola, Atlas, Barch, Barrett, Borelli, Brannon, Bunge, Campos, Cantlon, Carter, Carter-Sowell, Chen, Craske and WilliamsGruber et al., 2021).

Again, we have collectively experienced our fair share of this, ranging from microaggressions to full on aggression.

Gender role congruity also shapes how women see themselves and what roles they are comfortable embodying. Women who have spent their whole life being implicitly or explicitly taught to be submissive, communal, and not overly assertive may feel personally uncomfortable embodying these counter-normative behaviors. Thus, as women ascend the career ladder, these systemic factors may increasingly make them feel like academic Psychology is not for them. This fact is almost certainly exacerbated by the very visible demographic shift that occurs as women go from being surrounded by predominantly women to being the only woman in the room.

4.2 Productivity

It’s clear from these data that women are likely opting out of academic Psychology at greater rates than men. But, in cases where women opt into academic Psychology, what accounts for their lesser productivity? We suspect that a similar confluence of systemic, interpersonal, and intrapersonal factors play a role. Take publishing and grant submission, for instance. Across all of academia, women publish fewer papers and submit fewer grants than men (see Reference Gruber, Mendle, Lindquist, Schmader, Clark, Bliss-Moreau, Akinola, Atlas, Barch, Barrett, Borelli, Brannon, Bunge, Campos, Cantlon, Carter, Carter-Sowell, Chen, Craske and WilliamsGruber et al., 2021). At the systems level, this may be because women are spending their time elsewhere. Childcare is likely one big factor, as we discussed above. However, women also report spending more time on other things besides research when at work, perhaps as a product of systems-level pressures. For instance, in Reference Gruber, Mendle, Lindquist, Schmader, Clark, Bliss-Moreau, Akinola, Atlas, Barch, Barrett, Borelli, Brannon, Bunge, Campos, Cantlon, Carter, Carter-Sowell, Chen, Craske and WilliamsGruber et al. (2021) we review the fact that women are likely spending more time on unpaid service within their departments, insofar as service is seen as a “communal” activity that is more expected from women. Women also report that they spend far more time on unquantified sorts of service activities such as mentoring undergraduates, discussing careers with graduate students, and generally “taking care of the academic family” (Reference Guarino and BordenGuarino & Borden, 2017). These stereotypes are exacerbated for BIWOC, who may be the “token” member of their under-represented group and seen as the one in charge of diversity efforts in the department. BIWOC and other under-represented scholars may also be expected to, or feel compelled to, mentor students who are under-represented in academia (see Reference Gruber, Mendle, Lindquist, Schmader, Clark, Bliss-Moreau, Akinola, Atlas, Barch, Barrett, Borelli, Brannon, Bunge, Campos, Cantlon, Carter, Carter-Sowell, Chen, Craske and WilliamsGruber et al., 2021 for a discussion).

It is possible that interpersonal processes such as bias also contribute to differences in publishing. There is some evidence that editors serve as biased gatekeepers of the science that gets published in their journals. For instance, evidence shows that from 1974 to 2018, male editors at top social, cognitive, and developmental journals were significantly less likely to accept papers that were authored by women versus men; female editors at this time did not show any bias (Reference Bareket-Shavit, Goldie, Mortenson and RobertsBareket-Shavit et al., in preparation). This effect mirrors evidence that white editors are less likely to accept the papers of BIPOC scholars in Psychology (Reference Roberts, Bareket-Shavit, Dollins, Goldie and MortensonRoberts et al., 2020) and suggests that BIWOC may experience an additive effect when it comes to publishing. An unpublished paper in economics suggests that women authors may face a much longer review process than men, during which their papers are held to higher standards (Reference HengleHengle, 2020).

Fortunately, the data on grant review do not seem to show the same degree of bias. Although men hold more grants overall, this appears to be because they submit more grants overall (see Reference Gruber, Mendle, Lindquist, Schmader, Clark, Bliss-Moreau, Akinola, Atlas, Barch, Barrett, Borelli, Brannon, Bunge, Campos, Cantlon, Carter, Carter-Sowell, Chen, Craske and WilliamsGruber et al., 2021). Studies do not find that women’s grants are reviewed more poorly and, in some cases, women’s grants may even fare better than men’s in review (Reference Gruber, Mendle, Lindquist, Schmader, Clark, Bliss-Moreau, Akinola, Atlas, Barch, Barrett, Borelli, Brannon, Bunge, Campos, Cantlon, Carter, Carter-Sowell, Chen, Craske and WilliamsGruber et al., 2021). In many grant review processes, there is an evaluation process for the person carrying out the science (“the researcher”) in addition to a review process for the science itself. There is some evidence for bias when the decision architecture encourages reviewers to focus on or foreground the “researcher” versus the “science” (Reference Witteman, Hendricks, Straus and TannenbaumWitteman et al., 2019). Women’s grants are rated as worse when the ratings of the “researcher” are more heavily weighted than the ratings of the proposed science itself, consistent with other evidence showing that women are less likely to be described as “leaders” and “pioneers” in reviews (Reference Magua, Zhu, Bhattacharya, Filut, Potvien, Leatherberry, Lee, Jens, Malikireddy, Carnes and KaatzMagua et al., 2017).

Intrapersonal processes also play a role in women’s productivity. The biggest take-away from the data is that women submit fewer products (be they publications or grants) than men and this may be a product of their own beliefs or preferences. Women may believe that they should spend more time on communal tasks such as service and doing so may take away precious research time (note that this might happen because they feel pressure or because they truly get reward in engaging in these other tasks). Women may also have internalized bias and expect to get more pushback on their work. This can create a cycle of perfectionism in which women take much longer to produce publications or grants. The evidence is perhaps consistent with this interpretation insofar as women’s grants are (at least in some data sets) rated as stronger than men’s but men consistently submit more grants over all (see Reference Gruber, Mendle, Lindquist, Schmader, Clark, Bliss-Moreau, Akinola, Atlas, Barch, Barrett, Borelli, Brannon, Bunge, Campos, Cantlon, Carter, Carter-Sowell, Chen, Craske and WilliamsGruber et al., 2021). This suggests that women may be taking a different approach to men by placing all their metaphorical eggs in a single, perfectly crafted basket, whereas men are placing eggs in multiple … er … less well-constructed baskets. Finally, as we review in Reference Gruber, Mendle, Lindquist, Schmader, Clark, Bliss-Moreau, Akinola, Atlas, Barch, Barrett, Borelli, Brannon, Bunge, Campos, Cantlon, Carter, Carter-Sowell, Chen, Craske and WilliamsGruber et al. (2021), there are small but consistent differences in assertiveness and self-assuredness, which may mean that women are just less comfortable than men in “getting their ideas out there,” an internalized gender stereotype that could ultimately hurt women’s productivity.

Again, we certainly have personal experiences of what has seemed like bias in the publication and grant-receipt sphere.

4.3 Impact and Financial Remuneration

Finally, let’s deal with scientific impact and financial remuneration. Even controlling for productivity, women have less impact in Psychology in terms of citations and are paid less (Reference Gruber, Mendle, Lindquist, Schmader, Clark, Bliss-Moreau, Akinola, Atlas, Barch, Barrett, Borelli, Brannon, Bunge, Campos, Cantlon, Carter, Carter-Sowell, Chen, Craske and WilliamsGruber et al., 2021). As we review in Reference Gruber, Mendle, Lindquist, Schmader, Clark, Bliss-Moreau, Akinola, Atlas, Barch, Barrett, Borelli, Brannon, Bunge, Campos, Cantlon, Carter, Carter-Sowell, Chen, Craske and WilliamsGruber et al. (2021) and Reference Lindquist, Gruber, Schleider, Beer, Bliss-Moreau and WeinstockLindquist et al. (2020), there are longstanding and consistent biases in citation rates that cause men’s papers to be more highly cited than women’s. Some of this is driven by men’s relative greater tendency to self-cite (Reference King, Bergstrom, Correll, Jacquet and WestKing et al., 2017). Other recent evidence finds that in neuroscience, not only do men self-cite more, but they also cite other men more than they do women (meanwhile, women cite both men and women equally; Reference Dworkin, Linn, Teich, Zurn, Shinohara and BassettDworkin et al., 2020).

Other evidence again points to systemic, interpersonal, and intrapersonal processes that impact both women’s and men’s behaviors around publishing in the journals most likely to be impactful. In Psychology, as the impact of a journal increases, the prevalence of women authors on the papers published there decreases linearly (Reference Odic and WojcikOdic & Wojcik, 2020). This may be due to systemic factors: for instance, if women are spending relatively more of their time on other tasks like service, teaching, or childcare at home, they might have less time to take a risk and “aim high” by submitting first to a top-tier journal and then revising their manuscript afresh for each new submission if it doesn’t get in.

Women might also be less likely to submit to high-impact journals because they might have had bad experiences there in the past. The findings showing that editor gender is predictive of the gender of published authors (Reference Bareket-Shavit, Goldie, Mortenson and RobertsBareket-Shavit et al., in preparation) is suggestive that women might have systematic difficulty publishing at certain journals. Note that to our knowledge, research has not addressed how journal impact, editorial gender, and women’s publication rates interact. That said, stereotypes that women are less “brilliant” than men (Reference Leslie, Cimpian, Meyer and FreelandLeslie et al., 2015) could contribute to implicit bias against women’s findings at top journals. At least in Psychology, many editors are not blind to author names (and presumed gender), even if reviewers are. Finally, women may not publish in the top-tier journals that are most impactful because of small but stable gender differences in intrapersonal processes such as self-esteem or self-promotion (see Reference Gruber, Mendle, Lindquist, Schmader, Clark, Bliss-Moreau, Akinola, Atlas, Barch, Barrett, Borelli, Brannon, Bunge, Campos, Cantlon, Carter, Carter-Sowell, Chen, Craske and WilliamsGruber et al., 2021). Women may simply feel that their work is not important enough to be considered at top-tier journals due to internalized stereotypes that women are less “brilliant” than men or that women should not self-promote.

While the findings on impact are disconcerting, what is particularly concerning is that women are also less recognized financially for their work. As we review in Reference Gruber, Mendle, Lindquist, Schmader, Clark, Bliss-Moreau, Akinola, Atlas, Barch, Barrett, Borelli, Brannon, Bunge, Campos, Cantlon, Carter, Carter-Sowell, Chen, Craske and WilliamsGruber et al. (2021), women receive on average, 68–99 percent of what men receive in salary as professors. These discrepancies are starkest for full professors, where women at R1 universities make 81 percent of what men make. It is possible to argue that women make less because they are less productive on average – and salaries and raises are based at least in part on merit – but a recent study of New Zealand academics found gender disparities in financial compensation even when comparing equally productive men and women scholars (Reference Brower and JamesBrower & James, 2020). For women on 9-month salaries (typical of US tenure-track jobs), differences in base pay may become exacerbated by differences in grant funding success when grant funds are used to pay 3 months of “summer salary.”

The gender pay gap, as it’s called, is certainly not unique to academia and its mechanisms are hotly debated. Some of the gender pay gap may be linked to systemic factors, such as women’s mobility when applying for and accepting jobs in a wide range of geographical locations. As we discuss above, being in a dual-career partnership may influence women’s desire to apply for tenure-track jobs; even if they do apply for those jobs, it may limit women’s ability to apply broadly/accept any job because heterosexual women are more likely than heterosexual men to put their spouse’s career first in the case of a “two-body problem” (Reference Mason, Wolfinger and GouldenMason et al., 2013). This fact alone could hinder women’s negotiation abilities and ability to seek out the best-paying position.

In addition to systemic factors, there are well-known interpersonal and intrapersonal factors that contribute to the gender pay gap, such as women’s ability to successfully negotiate for themselves during initial offers and to attain retention offers. Again, due to stereotypes of women as communal, negotiation partners such as chairs and deans may be less likely to expect women to negotiate for pay raises either during hiring or retention and may be biased against them if they do engage in negotiation (Reference Amanatullah and MorrisAmanatullah & Morris, 2010). Meta-analyses show that women negotiate less than men do (Reference Kugler, Reif, Kaschner and BrodbeckKugler et al., 2018), either because they are aware of the potential backlash associated with doing so (an interpersonal explanation) or because they are not comfortable advocating for themselves (an intrapersonal explanation). Other studies find that women negotiate just as much as men, but are less likely to have their requests granted (Reference Artz, Goodall and OswaldArtz et al., 2018), suggesting that women may receive feedback over time that negotiation efforts are not worth it. Women are also less likely to receive outside job offers than men, which can result in fewer “retention offers” from their home university that increase salaries over time. This may be because other schools are less likely to seek out women to “poach,” because women are less likely to apply for these jobs, or because their home universities are less likely to put up the money to retain them. It’s worth noting that women are also granted less financial support outside of salaries when compared to men. In the biomedical sciences, men receive 2.5 times more in start-up funds than do women (Reference Sege, Nykiel-Bub and SelkSege et al., 2015), which could alone lead to the discrepancies in productivity and grant receipt that may exacerbate the gender pay gap over time.

Our personal experiences of gender differences in impact and finances are varied. On the one hand, we feel lucky to be well-remunerated for the work that we love doing. On the other hand, we are aware of ways in which there has not been gender equality in our pay and we have experienced “equity adjustments” to our salaries.