last update: 16/07/2020
While ManyBabies studies are by definition cross-lab projects, an individual lab may wish to use ManyBabies data for secondary analyses, honors projects, or other purposes. Below we articulate philosophical considerations and concrete policies for ensuring that such projects maintain the open, collaborative spirit of ManyBabies.
Philosophy and considerations underlying the policy
ManyBabies projects are unique for a number of reasons:
1. Preregistration. Preregistration has only recently begun to be widely adopted as part of the scientific method, and for many researchers their contribution to ManyBabies may be their first exposure to this approach. Pregistration brings with it certain commitments about how the data will be treated. It is important that there be no confusion between the primary, preregistered analyses that test clearly defined a priori hypotheses, and any secondary or exploratory analyses that may be performed, whether on the full dataset or on preliminary or partial datasets. In addition, it is important that any such analyses not influence the decision-making of data collectors during the collection period - particularly regarding sample sizes and “stopping rules.” 2. Open Science. ManyBabies’ commitment to Open Science means that all data and analysis scripts that are formally part of ManyBabies are made available to the research community for scrutiny (assuming appropriate participant consent has been obtained in the case of non-anonymous data, videos, etc., otherwise secondary, anonymized data will be shared instead). Thus, data and materials will typically be open during data collection, even before any initial report is finalized. For projects that are not formally part of ManyBabies but that build on ManyBabies studies in some way (for example testing different populations), we strongly encourage the adoption of the same open science practices. 3. Collaborative Science. Data for ManyBabies projects are being collected by a large number of laboratories around the world. Each laboratory has its own resources (funding, students, infrastructure), works within the policies and constraints of its own governing bodies (institutional, granting agencies, ethics, oversight, laws) and has expectations to be appropriately rewarded for its contribution. At the same time, each laboratory is making use of the shared resources (stimuli, design and protocols, shared data, etc.) and our hope is that no laboratory has to worry that its contributions to the project were “scooped” in any way. For that reason and to ensure productive collaboration, we make specific recommendations regarding data analysis and presentation (see below) that takes place outside the scope of the preregistered confirmatory analyses. 4. Pedagogical and student authorship considerations. Most contributing laboratories depend on the work of students and trainees. Because of the very large number of contributing laboratories and the scientific needs of the ManyBabies projects, there is relatively little room for students (particularly undergraduate students) to make a meaningful intellectual contribution to the initial design of the project, including materials and the primary preregistered analyses. It is therefore necessary to allow some room for students to create their own intellectually-meaningful projects based on their contributions to the larger ManyBabies study(ies), for student participation in the project to be viable and meaningful. Most student honours programs, internship funding programs, etc., have very specific requirements for this kind of intellectual contribution. ManyBabies needs to be as flexible as possible within the reasonable constraints stemming from points 1-3 to allow for such student participation to take place.
Presentation of the project
Professional presentations. We ask as a norm that participants refrain from presenting their own data on the project in a professional capacity (e.g., scientific posters and presentations at conferences, online dissemination in blogs or other venues, journal articles, book chapters, etc. – when in doubt please contact the Governing Board) until data collection is complete and the final analyses have been conducted and published as a preprint. If you need an exception to this policy, e.g. because of requirements from a funder, please contact the Governing Board.
Once data collection is complete, the rules for secondary analyses and follow-up projects will apply (see below). There may also be other types of opportunities to present the ManyBabies project at conferences and other venues, that do not involve sharing data from in-progress studies. In smaller venues, you are free to make such presentations (an example would be presenting the ManyBabies study design, highlighting your own participation, at a departmental or university event). For opportunities at international conferences or similar higher-profile venues, please also consult the Governing Board prior to presenting yourself as a representative of ManyBabies.
Pedagogical presentations. We encourage trainees to participate in ManyBabies projects. As part of participation, there may be opportunities or requirements for presentation (e.g., of an honors thesis, to a departmental or lab group, etc.). We ask that groups and trainees respect the following guidelines:
- If you wish to use any data provided by other laboratories than your own in your analysis, you must ask permission from that laboratory.
- Please include an explicit statement in your presentation: “This presentation contains interim results from a larger, ongoing project as part of the ManyBabies collaborative. Analyses are not definitive and are subject to revision.”
- Please take all possible precautions to avoid biasing ongoing data collection both within and outside your lab, e.g. by precisely finalizing and confirming your lab’s stopping rule for data collection before doing analysis, and by avoiding biasing terms like “success” and “failure” when describing your lab’s results.
- If possible, upload your final dataset before the presentation to further prevent biases.
- The audience for these presentations should typically be local and small-scale and not be the same professional conferences that will constitute the audience for presentation of the final study results (e.g., a departmental, university, or undergraduate conference is fine, but SRCD or ICIS would be inappropriate).
- Unless there is an explicit pedagogical or funding requirement you are fulfilling, please avoid sharing your presentation publicly (e.g. on a public-facing university website) until the data collection period is complete. At that point, public sharing is encouraged (with the included statement outlined in Point 2).
- Please share your presentation, posters, and reports with the ManyBabies group, e.g. by emailing it to the organizers of your particular project! We are excited to collect these visible accomplishments of our trainee members.
Outreach presentations and demonstrations: An important aspect of open science is communicating with the general public and such presentations are thus in principle welcome. When presenting the project to the general public, the considerations above apply (e.g., don’t use biasing terms, highlight the collaborative, large-scale nature of the overall project). When demonstrating an ongoing study set-up, we ask only that you use alternate stimuli. The governing board will do its best to assist you in finding suitable replacements.
Participant/lab newsletter: Here the same considerations as for pedagogical presentations apply.
Secondary analyses and follow-up projects
Since we are collecting a rich dataset we strongly encourage follow-up projects and exploratory analyses of the original data. Follow-up projects can include conceptual replications of the main experiment with, for example, new populations or stimuli, or subsequent (identical or different) studies with the same participants. Exploratory analyses can be used to generate new testable predictions or compare analysis methods. For example, a researcher might be interested in investigating initial versus final trial performance or comparing different data transformations.
If you are considering a side project, as a norm we ask that you write to the relevant project listserv announcing your project and give other contributors ample opportunity – at least a week – to join in. We hope that this policy reflects the collaborative spirit of the group.
While exploratory analyses are encouraged, we ask that ManyBabies contributors engaging in follow-up analyses conform with the same high standards of research practice as the primary analysis. Where possible, all analyses (including secondary and exploratory analyses) should be preregistered before conducting any data exploration, and datasets and scripts involved in the analyses should be made publicly available.