Preregistration

What is preregistration?

A preregistration is a timestamped document that includes details about a study, such as research questions, hypotheses, methods, and analytic strategies. Preregistration differs from other open science practices, such as registered reports and preprints, in that it is written prior to data collection and is not peer reviewed. The amount of detail included in a preregistration can vary. In the simplest case, a preregistration can comprise merely a hypothesis or perhaps a brief description of the methods. On the other extreme, a detailed preregistration can include code, power analyses, participant exclusion criteria, etc., in addition to the information described in the simple case. In the following sections, we will provide more information about the different components that can be included in a preregistration with a specific focus on how preregistration can benefit linguistic research. More concretely, we will cover the who, why, what, and how for preregistering your study in linguistics.

Who can benefit from preregistration?

Everybody! Preregistration is an important practice for conducting open science and anyone interested in reducing the likelihood of committing questionable research practices (QRPs) and making their research more transparent should consider using them. This tutorial focuses specifically on preregistration for researchers in any area of linguistics.

Why preregister your study?

Researchers have to make critical choices throughout the process of designing and conducting research. To give a concrete example, consider a phonotician interested in investigating lexical stress. In this case, the researcher would likely focus on some acoustic correlates associated with stress, such as pitch, duration, and/or intensity. Aside from choosing which acoustic correlate(s) to measure, she would also need to specify the domain in which these measurements are taken. Is it the nucleus of the stressed/unstressed syllables? Or perhaps the entire syllable? Where exactly is the measurement taken? The mid-point, perhaps? Or maybe an average over the entire syllable? All of these choices can have crucial consequences on downstream analyses. This type of inherent flexibility is referred to as researcher degrees of freedom. The purpose of preregisteration is to document all of these critical aspects of the study. Furthermore, preregistration can deter individuals from QRPs, such as HARKing or p-hacking, by requiring them to explicitly state critical decisions in the research pipeline prior to ever collecting data.

For more information about QRPs and how they relate to the replicability crisis see What is open science?.

There are many additional benefits to preregistering studies. Preregistration can serve as proof one is engaging in confirmatory data analysis (as opposed to exploratory data analysis), as it establishes a timeline of decisions made before and after data collection. Moreover, by including a greater level of detail in the preregistration, the researcher is forced to think about aspects of their study that might typically be left for a later stage, e.g., statistical analyses. This necessarily requires the researcher to invest more time upfront, but often has the benefit of increasing the likelihood of discovering critical errors in the study design.

What should you pregister?

In short, you can preregister anything that you think warrants being timestamped prior to conducting your study. What exactly this entails will vary widely depending on your area of linguistics and the type of study you are conducting. To illustrate, we’ll consider a concrete example.

Let’s imagine you are a psycholinguist preparing an experiment using self-paced reading. You might want to focus on the research questions, hypotheses, and critical details related to the methodology, such as who the participants are, how you plan to recruit them, how many you plan to recruit, etc. You also might consider explaining what the critical variables are that you will manipulate with your design, what transformations you plan to perform on the data, and what modeling strategies you will use for statistical inferences. All of these decisions, while important, might not be applicaple in every situation. To summarize, the type of information included in a preregistration will likely be related to one of the following:

1. RQ/Hypotheses
2. Method
3. Analysis

Importantly, it may be overwhelming to incorporate all of them in your preregistration if you are just getting started in Open Science. That’s fine. Feel free to pick what you think might be most relevant to your study to begin, and, with time and practice, you will get better at deciding what is relevant for your research.

Level of details

A lot of concerns of putting Open Science into practice or doing preregistration are about the possible overload of “extra work”. On the contrary, preregistration should be helping us work in a more efficient manner both in the long run and in short terms. Besides, you have complete control of how much detail you want to include. The more detailed the preregistration is, the more effort you need to put into, and as a result the less work down the road. Here we only propose some of the levels of details that may help you determine how much details you can incorporate in your preregistration (Figure 2). The darker the shade, the more detailed it is (thus more work). Again, you can always incorporate different levels of details in different parts.

Figure 1: Levels of detail

How to preregister your study

There are websites dedicated to Open Science practices. OSF is one and it provide platform and/or template we can use.

OSF

• OSF
  • Start from scratch
  • Start from an existing OSF project or component

Github

Pre registration involves making a plan for your research project before collecting and analyzing data. This plan can be documented and shared on Github, which is a popular web-based hosting service for version control and collaborative software development.

Here are the steps to set up a pre registration for your research on Github:

1. Create a Github account: If you do not already have a Github account, create one by visiting the Github homepage and following the instructions.
2. Create a new repository: Once you are logged in to Github, you can create a new repository to store your pre registration documents. Click the “New” button in the top left corner of the screen and follow the prompts to create a new repository.
3. Choose a template: Github provides several templates for different types of repositories. You can choose a template that is specifically designed for pre registration documents, such as the Open Science Framework (OSF) Preregistration template.
4. Add your pre registration documents: Once you have chosen a template, you can add your pre registration documents to the repository. These documents should include a detailed description of your research question, hypotheses, methods, and analysis plan. You can use Markdown, a lightweight markup language, to format your documents on Github.
5. Share your repository: Once your pre registration documents are complete, you can share your repository with other researchers or members of the public. You can do this by sharing the link to your repository or by inviting collaborators to contribute to the project.
6. Update your repository: As you collect and analyze data, you may need to update your pre registration documents. You can do this by making changes to your documents on Github and committing the changes to the repository.

By setting up a preregistration on Github, you can make your research more transparent and reproducible. Other researchers can review your plan and provide feedback, and you can use the repository to document any changes you make to your research plan over time.

Aspredicted.org

1. One author creates the pre-registration.
2. Participating authors are emailed, requesting approval.
3. If all approve, it is saved but remains private until an author makes it public; or remains private forever.(Why?)
4. Authors may share an anonymous version of the pre-registration with reviewers.
5. If made public, the final .pdf (sample) is automatically stored in the web-archive.

By hand

Write, timestamp + freeze, and then check yourself

To set up a pre registration for your research from scratch on paper, you can follow these steps:

1. Define your research question and hypotheses: Start by identifying the research question you are trying to answer and any hypotheses you have about the expected results.
2. Select your variables and measures: Identify the independent and dependent variables in your study and describe the measures you will use to operationalize these variables.
3. Determine your sample size and recruitment strategy: Specify the number of participants you plan to recruit and describe how you will recruit them.
4. Outline your research design: Describe the research design you will use to address your research question, such as a between-subjects or within-subjects design.
5. Specify your data analysis plan: Describe the statistical analyses you plan to use to test your hypotheses, including any planned exploratory analyses and methods to handle missing data.
6. Plan for ethical considerations: Address any ethical considerations in your study, such as obtaining informed consent from participants, ensuring confidentiality and privacy, and handling any potential risks or harms.
7. Consider any potential limitations: Anticipate any potential limitations of your study, such as sampling bias or measurement error.
8. Write up your pre registration plan: Once you have completed these steps, write up your pre registration plan in a clear and concise manner on paper. Be sure to include all relevant information, such as your research question, hypotheses, variables and measures, sample size and recruitment strategy, research design, data analysis plan, ethical considerations, and potential limitations.

To read more about preregistration in linguisticis: https://www.degruyter.com/document/doi/10.1515/ling-2019-0048/html?lang=en

Reference Kathawalla, U. K., Silverstein, P., & Syed, M. (2021). Easing into open science: A guide for graduate students and their advisors. Collabra: Psychology, 7(1)