Note: Submitting authors should also consult the general guidance provided in the Language & Cognition 'Preparing your materials' page.

Author and reviewer guidelines for Registered Reports

Registered Reports are a form of empirical article that involves a two-stage review process. At Stage 1, the article will be reviewed prior to the collection of data. At Stage 2, the article will be reviewed after the results are in. The following infographic gives an overview of the process.

Registered Reports allow a results-blind review process. The cornerstone of this submission format is that a large part of the manuscript will be assessed prior to data collection. The purpose of this is to reduce publication bias (only significant results get published), with reviewers and editors unable to reject an article on the basis of having obtained null results, or on the basis of the results deviating from predictions.

In the context of Language & Cognition, Registered Reports will most likely be suitable for experimental studies, including formal replications, as opposed to observational studies, case studies, or exploratory research. We recommend that corpus-related work is submitted as a Regular Submission. 

The Open Science Framework provides a helpful ten-point check list for submitting a Stage 1 Registered Report.

Stage 1

Submissions to Stage 1 will include the description of the key research question and background literature to motivate this research question, the specific hypotheses to be investigated, a detailed description of the planned experimental procedures, an analysis pipeline, a specification of stopping rules, and a justification of the planned sample size.

Importantly, we will review the Stage 1 submission based on the standard criteria that we also apply to regular submissions at Language & Cognition, including the proposed study’s fit with the theme of the journal, its theoretical importance and novelty, its methodological rigor, and appropriateness of the proposed analyses.

Word limit: For submissions at Stage 1, articles should not exceed the word limit of 4,000 words, which we will enforce strictly. General Editors have the option of desk-rejecting a submission if the submission exceeds this word limit or lacks any of the other basic requirements of a Registered Report outlined here.

Stage 2

At Stage 1, reviewers assess study proposals before data are collected. Upon successful completion of this stage an “in principle acceptance” commitment by the journal is given to the authors. At Stage 2, reviewers consider the full study, including results and interpretation. At this stage, acceptance leads to publication.

Preparing your Stage 1 manuscript

A Stage 1 manuscript submission should contain the following:

• Abstract
• Introduction
• Methods section: A detailed methods section including a description of experimental procedures in sufficient detail to allow another researcher to repeat the methodology exactly, without requiring further information. These procedures must be adhered to exactly or any Stage 2 submission can be rejected. The methods section should include specification of proposed sample characteristics, including criteria for data exclusion. In addition, stopping rules and planned sample size should be specified and justified (e.g., via power analysis).
• Data analysis plan: A detailed data analysis plan that includes information about data pre-processing and where appropriate, annotation procedures. Ideally, this data analysis plan should include a specification of the planned statistical model, including a note on contingencies (such as measures that will be undertaken when models do not converge). The data analysis plan should specify whether null hypothesis significance testing or Bayesian methods will be used. For significance testing, the alpha level needs to be specified in advance, anticipating the potential of multiple comparisons problems. For Bayesian methods, priors and inference criteria need to be specified in advance.
• Potential results section: A potential results and implications section, where authors outline all possible results to the best of their ability and carefully interpret what each would say about the research question(s).

Pre-registration

If an article gets accepted at Stage 1, we require that the experiment and analysis are pre-registered (with the feedback by reviewers incorporated), ideally on the Open Science Framework.

Manuscript withdrawal

It is possible that authors with in-principle acceptance following Stage 1 may wish to withdraw their manuscript following or during data collection.

Possible reasons could include major technical error, or an inability to complete the study due to other unforeseen circumstances. In all such cases, manuscripts can of course be withdrawn at the authors’ discretion. However, the journal will publicly record each case in that the original abstract will be published on a dedicated website. 

Partial withdrawals are not possible; i.e. authors cannot publish part of a registered study by selectively withdrawing one of the planned experiments. Such cases must lead to withdrawal of the entire paper. Studies that are not completed by the agreed Stage 2 submission deadline (which can be extended in negotiation with the editorial office) will be considered withdrawn and will be subject to a Withdrawn Registration.

Frequently Asked Questions (FAQs)

What are the criteria for evaluating registered reports?

Reviewers will be asked to review Registered Reports along the following five dimensions at Stage 1:

• Fit with the core themes relevant to Language & Cognition (see website)
• Importance of the research question
• Quality of the literature review
• Soundness of the proposed hypotheses
• Soundness and feasibility of the proposed design, including statistical power
• Whether the proposed experiment offers an adequate and appropriate test of the hypotheses
• Whether proposed methods are appropriate and sufficiently detailed.

Reviewers will be asked to review Registered Reports along the following three dimensions at Stage 2:

• Adherence to the rationale and plan laid out in the Stage 1 submission
• Whether conclusions are justified by the data
• The overall quality of the discussion section, and whether any claims made in this section follow from the results.

Reviewers are asked not to consider the outcome of the empirical results in their review. 

How does a Registered Report differ from a traditional research paper?

A Registered Report differs from the 'regular submission' category only in that the proposed experiment has not yet been conducted, and in that the review process is split into two stages to allow a results-blind review at Stage 1.

What is the word limit and can authors have an appendix?

The word limit for Registered Reports is 4,000 words (excluding abstract and references). Authors are free to use an online appendix (such as via an Open Science Framework repository) to provide details that reviewers may find helpful in evaluating the design.

Can authors conduct the proposed experiments before or during Stage 1 review?

No. However, although this is not a requirement, authors are allowed to have undertaken pilot testing to validate the proposed analysis pipeline.

If a Registered Report is 'accepted in principle', how long do authors have to conduct the study?

One year, but based on a solid justification authors can negotiate longer time lines if necessary.

When and where do I pre-register my experiments?

After receiving reviews from Stage 1, the experiment(s) need(s) to be pre-registered. The pre-registration has to address or incorporate the feedback provided by the reviewers. We recommend the Open Science Framework for pre-registration.

What if the results do not support the hypotheses?

That is fine.

What if a 'null result' is obtained?

That is fine. 

Can reviewers recommend rejection based on the experiments having obtained null results?

No.

Can reviewers recommend rejection based on the experiments having obtained results that differ from the original predictions?

No.

Is it possible to conduct analyses that were not mentioned in the first stage of the review process?

Yes, if these are explicitly flagged as novel post-hoc analyses that were not part of the original research plan, and if the analyses are clearly justified.

What if some of my analyses or future experiments depend on the preliminary results?

If your follow-up experiments rely on the outcome of prior ones, the logic for follow-up experiments needs to be carefully outlined. It is possible to perform unregistered post hoc analyses if they are clearly flagged as deviating from the original analysis plan, and if they are justified, methodologically sound, and informative. It is possible and indeed recommended to lay out contingencies in the Stage 1 submission (“if the outcome of Experiment 1 is X, we will …”).

Will the same reviewers review both Stage 1 and Stage 2?

Where possible, the same reviewers will be asked to review Stage 1 and Stage 2; the original reviewers will be re-contacted after the experiments have been conducted and submitted at Stage 2.

However, General Editors have the option of desk-rejecting submissions at Stage 2 when there are obvious and unmotivated deviations from the plan set out in Stage 1. In some cases that are particularly straightforward, reviewers will not be asked to review at Stage 2 if the General Editor decides to accept the article without sending it out to re-review.

Last updated: 22 March 2021