16  During data collection

16.1 Restarts, recalibration (eye-tracker) and pauses

Restarts

Any pauses in the procedure can potentially lead to an erasure of the weak memory trace for the familiarization stimuli (Gerken, p.c. noticed a loss of effect if there is a 30+ second pause between familiarization and test phase). Therefore, we ask: NO RESTARTS. Do not restart any trials, regardless of short looking, fussiness, experimental error, etc. Very short trials will be excluded centrally during the analysis process, but should be fully reported and not repeated.

Recalibration (eye-trackers)

DO NOT RECALIBRATE once the session has started.

Pauses

If babies are fussy, it can be necessary to attempt to get them settled. You may use the natural pause of an attention getter to let the parent non-verbally resettle their baby for a very short time (a couple of seconds, according to your practices). However, longer pauses should result in the cessation of the study.

Suggestion: if the baby is fussy and you would like to try to continue the experiment after the baby calms down just so that the parent gets the full experience, you are welcome to do so, but this data must be excluded. For these babies, you would report any data collected before the actual experiment terminated from fuss-out/inattention, not the repeat or continued demonstration.

16.2 Exclusions and reporting errors and infant “fuss-outs”

Decisions regarding discards for experimental reasons will be made centrally and need to be reported consistently, hence it is important to retain all data from all infants that entered the testing room. Specifically, the data you send in for central analysis should use the “toe in the testing room” standard: if the infant made it so far as the testing room, the Pparticipant Data files dataset should include their data. In ManyBabies studies, we do not discard “partial data.” It is important for both reporting purposes and potential secondary analyses to have full information about both “usable” and “unusable” trials, even for infants who did not complete or provide usable data for all familiarization and test trials. Trial-level exclusions may not result in the exclusion of the participant from the dataset. These instructions are counter to many labs’ normal internal practices, so please ensure that RAs are fully trained on this element. There are specific columns in the data templates to report the reasons that some of this data should not be included in the main analysis; every lab is expected to report at least some ‘unusable’ data!

Session-level and trial-level errors

There are two possible kinds of exclusion: session-level (exclude all data from this participant from analysis) and trial-level (exclude just this trial, e.g. because there was interference or the baby had already fussed out). Session-level errors are reported in the Participant Data file and trial-level errors are reported in the Trial Data file. Most lab-reported errors will be “trial-level” errors. If the infant completed any trials, please do NOT report the infant as a session-level error EVEN IF YOU LATER REALIZE THE CHILD DOES NOT MEET ALL INCLUSION CRITERIA. Medical issues and developmental concerns. Please do not exclude based on medical issues / developmental concerns. Instead, provide sufficient detail in the ‘medical_issue’and/or ’developmental_concern’ fields for the analysis team to review and categorize centrally. As with all exclusions, it is important to include these in your data if the infant entered the testing room.

For clear-cut judgements such as age, please just use the appropriate columns to report the values that would result in exclusion (e.g. “age = 3 years”). For subjective judgements such as whether the baby was too fussy, the central decision will be made based on the original coder/RA’s judgment, since it would be impractical to review comments/videos and make the decision centrally. Therefore, please be very clear (but succinct) in the ‘trial_error_info’ and ‘session_error_info’ fields about whether a given trial should be discarded and why. Please classify as one of the listed ‘Value example’ options in the Data Dictionary where possible. Use the “other” category sparingly and only if absolutely necessary. In the ‘notes’ column, please provide further succinct but sufficient information so that it is clear why a given trial or infant was excluded (e.g. “mom pointed at the screen”). Remember that this will be the only information that will be available to the analysis team as they compile and analyze the dataset; ambiguous entries will be problematic for data analysis!

Session-level errors

Session level errors are those which result in discarding the data for a participant completely. Session level errors include: Fuss-out after entering the testing room but prior to starting the experiment Parental interference affecting ALL trials (e.g. caregiver refuses to wear the eye covering properly through all trials, or chronically failing to comply with experimenter instructions such that the validity of the entire session is in question). Experimenter error that makes the entire session unuseable (e.g. ran the wrong test session). Equipment failure that makes the entire session unuseable (e.g. video was not playing properly) Failure to calibrate (eyetrackers only) Infant was tested but was found to not meet the inclusion criteria (e.g. vision issues reported on the demographics questionnaire). Only in cases where infants did not start the experiment (i.e., did not see any stimuli) due to fuss-out or equipment/calibration failure, labs are welcome to reschedule the appointment to another occasion; please indicate this in the participant data form. With regards to session-level errors: Please make sure to include participants excluded for session-level errors in your Participant Data file. You do not need to report them in the Trial Data file (see below for details about the required formats of these files). Data from infants where session-level errors are reported do NOT count towards the minimum sample size. Please do NOT mark a participant as a “session-level” error if an infant entered the testing room and completed the first familiarization phase even if they provided no usable test trial data.

Trial-level errors:

Trial-level errors are those which result in discarding the data for a single trial. Each of the 12 test trials should be marked as either ‘no error’ (usable) or ‘error’ (to be discarded). Trials with no usable data should have ‘NA’ marked in their data columns. Your lab will make the decision and report whether a trial should be excluded, as well as the reason why. These can include temporary instances of issues such as: parental interference in the middle of the experiment (e.g. mother speaks to infant during a trial and gets their attention) experimenter error or equipment failure that affects one or more trials but not the entire session transitory infant fussiness involving only a very brief pause in testing (see below for further details on reporting), or fussiness where the infant “fusses out” after the initiation of one or more test trials With regards to trial-level errors note: For transitory fussiness, ONLY mark the trials on which the infant was actually fussy - further exclusions may be implemented centrally, but it is important that any potentially usable trials after a transitory fussiness period be included in the submitted data. For example, if the infant was fussy on trials 8 and 10 but not trials 1-7,9,11 and 12, only trials 8 and 10 should be marked as an exclusion. If the infant provides some usable trials and then “fusses out” so that the study is terminated prematurely, please make this clear by marking ‘NA’ on trials that were not given.

Trial level errors resulting in further exclusion:

In most cases, trial level exclusions should result in exclusion ONLY of the trial immediately impacted by the exclusion (e.g. transitory fussiness, parent distracting baby by scratching their nose). However, in some cases, it may be necessary to exclude all subsequent trials, because of concerns that the infant is no longer “on task”. If this is the case, please mark all subsequent trials as trial level errors and explain the reasoning in the notes. Some examples of this type of situation: Any time (due to fussiness or other reasons) there is a break between trials of more than a few seconds. Equipment failure that raises concerns about the validity of subsequent trials. Significant parental interference/engagement or external noise that diverts the infant’s attention to the extent that it raises concerns about the infant being distracted across all trials. Note that we have included a test in the robustness checking section of the manuscript about the effect of discarding trials after an initial excluded trial.