As organizational scientists, we should be striving to produce useful and replicable research. Recently, how we (and really all sciences) conduct our research has drawn critique. From our relative disuse of inductive research methods, to unethical practices during the review process, to publication bias, to HARKING, to P-hacking, and so on, we must take meaningful steps so that we (and the consumers of our research) have confidence that our findings are meaningful, replicable, and honest. To that end, Journal of Business and Psychology will be launching a special initiative — results-masked review submission option.
In this alternative path, authors submit the intro, methods, measurement info, and analysis plan of a completed study (no results or discussion in the first round). This abbreviated paper then undergoes a peer review and is evaluated on the merits, rigor, and quality of the project rather than what was actually found.
Our goal is to encourage authors to propose conceptually sound, interesting, and methodologically rigorous research without concern for whether the results will be statistically significant. Instead, we want the focus to be on the importance of the research question and the rigor of the research design. We should welcome the results from sound research no matter if they support proposed hypotheses, yield ‘null’ results, or replicate (or fail to replicate) previous work. Simply speaking, well conceived, designed, and conducted research should form the corpus of knowledge. We believe this special initiative provides an opportunity to do just that.
The results-masked review approach is appropriate for inductive, deductive, mixed methods, and papers involving multiple studies. In the case of the latter type, the author is encouraged to contact the editor to decide on the best approach for submission. There are multiple options as the results-masked review approach is highly flexible. At times, it may make sense to include the first study with results, and then the follow-up studies with out results (this is sometimes ideal for research that starts inductive and then moves to deductive approaches). Other times it is useful to include all studies with out results. Overall, an initial conversation with the editor can clarify an approach that makes sense for the project in question – ultimately it is not useful to assume a one size fits all approach given the eclectic nature of research.
Note, we also welcome traditional registered reports submissions following the same process.
Guidelines for authors and reviewers
Registered Reports are a form of empirical article in which the introduction, methods, and proposed analyses are pre-registered and reviewed prior to research being conducted. This format of article seeks to neutralize a variety of questionable research practices, including submitting research based on inadequate statistical power, selective reporting of results, P-Hacking, HARKING, and publication bias (Chambers, 2013). Besides traditional registratered reports, we allow results-masked submission. In the latter model, researchers submit work that has been completed, but only the Introduction, Method, and Proposed Analyses are submitted for review. Thus the work will be judged on the merits of the research question and methodology, not the findings.
Initial submissions will be reviewed by the Editor using standard screening criteria (e.g., comprehensiveness, fit and clarity). Those that pass this review will then be sent for in-depth masked peer review (Stage 1). Following review, the article will then be either rejected, receive an R&R, or accepted in principle for publication. Following in-principle acceptance (IPA), the authors will then produce a more traditionally formatted manuscript that includes Results and Discussion sections (Stage 2). Assuming that the stage 2 submission is consistent with what was committed to in stage one, and a sensible interpretation and discussion of the findings, the manuscript will be published regardless of the results. It is critical to note that this process is designed to incent the authors to have compelling research questions, excellent methodology, and then search for “truth” in their data. It is recognized that the search for “truth” can be messy, may produce non-significant results, and may lead to more questions than answers.
The registered reports approach is appropriate for inductive, deductive, mixed methods, and papers involving multiple studies. In the case of the latter type, the author is encouraged to contact the editor to decide on the best approach for submission. At times, it may make sense to include the first study with results, and then the follow-up studies with out results (this is sometimes ideal for research that starts inductive and then moves to deductive approaches). Other times it is useful to include all studies with out results. Overall, an initial conversation with the editor can clarify an approach that makes sense for the project in question – ultimately it is not useful to assume a one size fits all approach given the eclectic nature of research.
Stage 1: Initial manuscript submission and review
Before sending for in-depth peer review, the editorial team will assess manuscripts for adherence to basic requirements. These Stage 1 submissions should include the manuscript (details below) and a brief cover letter.
The cover letter should include:
- A statement identifying any conflicts of interests.
- A statement identifying whether data used have been used in any other research study, and if so, how.
Manuscript preparation guidelines – Stage 1
Initial Stage 1 submissions should include the following sections:
- A review of the relevant literature that motivates the research question and a full description of the research aims and hypotheses if applicable. The work proposed can be inductive or deductive in nature. A strong theoretical (especially for inductive work) and/or conceptual rationale (especially for deductive work) is provided.
- Full description of proposed sample characteristics, including criteria for participant inclusion and exclusion, and description of procedures/practices for defining outliers or excluding data.
- A detailed description of research procedures. These procedures must be consistently reported in the Stage 2 manuscript or the paper will be summarily rejected (unless approved in advance).
- Proposed analysis pipeline, including a precise description of all planned analyses. Any covariates or regressors must be stated including a rationale for their inclusion. Often, proposed analyses involving covariates should be reported with and without the covariate(s) included. Where analysis decisions are contingent on the outcome of prior analyses, these contingencies should be specified. Only pre-planned analyses can be reported in the main Results section of Stage 2 submissions. However, unplanned post hoc analyses will be admissible in a separate section of the Results.
- Studies involving null hypothesis significance testing must include a priori statistical power analysis. Estimated effect sizes for power analysis should be justified with reference to the existing literature. For studies involving Bayesian hypothesis testing, the predictions of the theory must be specified so that a Bayes factor can be calculated. Authors should indicate what distribution will be used to represent the predictions of the theory and how its parameters will be specified.
- Discussion of the sample and sampling methods to assure that the research questions can be answered in a fair and appropriate manner (e.g., no floor or ceiling effects).
Key note: ALTHOUGH STAGE ONE SUBMISSIONS RESEMBLE THESIS AND DISSERTATION PROPOSALS IN FORM, PAPERS SUBMITED TO JBP NEED TO HAVE THE TIGHTNESS AND COHERENCE OF WELL-CONSTRUCTED JOURNAL MANUSCRIPTS.
In considering stage 1 papers, reviewers will be asked to assess:
- The significance/importance of the research and research question(s).
- The logic, rationale, and plausibility of the proposed hypotheses and/or research questions.
- The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis where applicable).
- Measurement/psychometric information where applicable.
- Whether the authors have used measures and procedures that are of sufficient methodological rigor for ensuring that the results obtained are able to test the stated hypotheses.
- Whether the sample used provides a good test of the stated hypotheses (e.g., no major floor or ceiling effects, no systematic bias or distortion).
Following Stage 1 peer review, manuscripts will be rejected outright, offered the opportunity to revise, or receive in-principle acceptances (IPA) indicating that the article will be published pending accurate and complete analyses, as well as a defensible and evidence-bound interpretation of the results. Note, IPA decisions are time-bound. Time-limits will be noted if an IPA is granted (e.g., typically 2-3 months). Time extensions will be considered on a case-by-case basis.
Please note that any deviation from the stated research procedures or inconsistencies between methods from Stage 1 to Stage 2, regardless of how minor it may seem to the authors, could lead to rejection of the manuscript. Note that registered analyses must be undertaken, but additional unregistered analyses can also be included in a final manuscript.
Stage 2: Full manuscript review
Authors prepare and resubmit their manuscript for full review. The cover letter should acknowledge any deviations or new analyses from Stage 1. This is a full and traditional manuscript, with the following additions:
- Background and Rationale
- Please note that the Introduction should not be altered from the approved Stage 1 submission, and the stated hypotheses should not be amended or appended. Depending on the timeframe, new relevant literature may have appeared between Stage 1 and Stage 2. Any such literature should be covered in the Discussion.
- Results & Discussion
- The outcome of all registered analyses must be reported in the manuscript, except in rare instances where a registered and approved analysis is subsequently shown to be logically flawed or unfounded. In such cases, the authors, reviewers, and editor must agree that a collective error of judgment was made and that the analysis is inappropriate. In such cases the analysis would still be mentioned in the Methods but omitted with justification from the Results.
- It is reasonable that authors may wish to include additional analyses that were not included in the registered submission. For instance, a new analytic approach might become available between IPA and full review, or a particularly interesting and unexpected finding may emerge. Such analyses are admissible but must be reported in a separate section of the Results titled “Exploratory analyses” for deductive research or “additional analyses” for inductive work. Where the interpretation depends on inferential statistical analysis, authors should be careful not to base their conclusions entirely on the outcome of statistically significant exploratory tests.
- Exact p values and effect sizes must be reported for all null hypothesis significance tests.
- Any other pertinent information. It is better to provide too much information than too little information.
The resubmission will ideally be considered by the same reviewers as in the registration stage. In considering papers at Stage 2, reviewers will be asked to decide:
- Whether the Introduction and methods are the same as the approved Stage 1 submission (analysis plan is omitted).
- Whether results are reported clearly, appropriately and consistently with what was registered.
- Whether any unregistered exploratory or additional statistical analyses are justified, methodologically sound, and informative.
- Whether the authors’ conclusions are justified given the data (e.g., absent meaningful floor and/or ceiling effects).
- Whether the discussion section effectively contextualizes the results theoretically, conceptually and practically.
Basically, Stage 2 reviews are consistent with current practices for reviewing results and discussion sections.
Crucially, reviewers will be informed that editorial decisions will not be based on the support or lack of support of the hypotheses.
There is very little risk for the author in Stage 2 reviews. The author may be asked to revise and resubmit, however, based on reviewer/editor feedback.
We also would like to thank the Center for Open Science for their support and dedication towards improving science.