Annex C. Meta-analysis method

Description of studies included in the meta-analysis

The full coding of 44 studies has been completed. Twenty-one of the studies reported research from the United States. Of the remaining studies, 16 reported research from different European countries (Germany [6], Netherlands [2], Portugal [3], and one study each from Flemish Community of Belgium, Finland, Ireland, Spain, and the United Kingdom) and 7 research from other countries (Australia [3], Chile [2], China [1], and Tanzania [1]).

Five of the studies from the United States came from the NCEDL and SWEEP databases (Brock & Curby, 2014; Buckrop, Roberts, & LoCasale-Crouch, 2016; M. Burchinal, Vandergrift, Pianta, & Mashburn, 2010; Chang et al., 2007; Howes et al., 2008), and seven studies from the NICHD SECCYD (National Institute of Child Health and Human Development Study of Early Child Care and Youth Development) database (Bub, 2009; M. Burchinal et al., 2011; M. R. Burchinal, Lowe Vandell and Belsky, 2014; McCartney et al., 2010; NICHD ECCRN, 2007; O’Connor, Scott, McCormick and Weinberg, 2014; Vandell et al., 2010).

The majority of studies from Germany [5] came from the BiKS database (Kuger, Kluczniok, Kaplan, & Rossbach, 2015; Lehrl, Smidt, Grosse, & Richter, 2013; Richter, Lehrl, & Weinert, 2015; Smidt, Lehrl, Anders, Pohlmann-Rother, & Kluczniok, 2012).

Two of the studies from Australia came from the E4Kids (Effective Early Educational Experiences) database (Niklas, Tayler, & Cohrssen, 2017; Tayler, Ishimine, Cloney, Cleveland, & Thorpe, 2013).

Additional databases from large-scale national cohort studies included the First Steps database in Finland (Lerkkanen et al., 2012), the Pre-COOL database in the Netherlands (Slot, Boom, Verhagen, & Leseman, 2017), the Un Buen Comienzo database in Chile (Bowne et al., 2016a), and the EPPE (Effective Provision of Pre-School Education) database in the United Kingdom (Sylva et al., 2013).

The primary goal of the studies was to describe quality in ECEC settings (n=13), to explore associations between structural and process quality (n=11) and/or associations between ECEC quality and children’s development and learning (n=33). Fifteen studies reported cross-sectional data (i.e. all data were collected at one measurement point); the remaining studies were longitudinal studies with at least two waves of data collection.

Of the independent samples, (i.e. excluding overlapping datasets by retaining only the largest sample size), the total sample size of the studies coded was 3 110 ECEC staff (lead teachers) and 16 386 children, together from 1,977 ECEC centres. Sample sizes ranged from 92 to 2 938 children, and from 16 to 694 ECEC staff. When reported (n=26), the samples were representative at the state/regional level where the data was collected for most studies (n=22). Additional studies reported that the study population was representative at the national level (n=1: (Slot, Boom, et al., 2017) or at the local level (n=3; (Bowne et al., 2016a; Cadima et al., 2016; Phillips et al., 2009). All of the studies that included child data had similar numbers of boys and girls (43% to 53% girls). Children’s age was, on average, 66 months (ranging from 37 months to 180 months; SD=36.9). When reported, the majority of ECEC staff was female (95% to 100%) and, on average, 37 years old (SD=4.80).

The majority of studies (n=37) recruited participants in the ECEC setting. With regard to the type of ECEC setting (public and/or private), information was available for 15 studies of which five studies recruited participants from both, public and private ECEC settings. The remaining studies recruited participants from public ECEC settings. The primary population of children served in the ECEC centres was diverse in terms of background and included children from low-income backgrounds, immigrant children, ethnic minority children, and children whose home language is different from the language used in the ECEC centre.

Description of Measures: Structural and process quality

Most studies described structural features of the ECEC setting as part of the sample description. This resulted in vastly different operationalisations of structural quality indicators across studies, thus limiting the comparability of results.

Process quality was operationalised as staff-child interactions in all but 7 studies and most commonly assessed through observational measures. Measures were the Early Childhood Environment Rating Scale (ECERS; (Harms, Clifford, & Cryer, 1998; Sylva et al., 2006); n=13), the Classroom Assessment Scoring System (CLASS; (R. C Pianta, La Paro, & Hamre, 2008); n=16), the Observational Record of the Caregiving Environment (ORCE; NICHD ECCRN, 1996; n=7), and the Classroom Observation System (COS; NICHD ECCRN, 2004; n=2). Nine studies used a self-report measure (in all cases the Student-Teacher Relationship Scale, STRS, was used; (Robert C. Pianta, 2001). Of these studies, the self-report was used in addition to the observational measure in 7 studies. In 23 studies, a global score of the measure was used to describe the quality of staff-child interactions. In addition, 18 studies focused on positive interactions (i.e. warmth, responsiveness, emotional climate), 4 studies on negative interactions (i.e. conflict, permissiveness, negative climate), 10 studies on instructional interactions (i.e. general instructional support and cognitive stimulation but not content-specific instruction), and 8 studies on organisation/management of routines/chaos.

Fifteen studies focused on the quality or more exposure to developmental and educational activity indicators of process quality which were assessed using observational (n=13) and self-report measures (n=2). Different measures were used across studies, including, for example, the ECERS-E (Harms et al., 1998); n=3), the Early Literacy and Language Classroom Observation (ELLCO; (Smith, Dickinson, & Sangeorge, 2002); n=1), the Emerging Academics Snapshot (Ritchie, Howes, Kraft-Sayre, & Weiser, 2001; n=1), or the Target Child Observation (Kuger, Pflieger, & Roßbach, 2006); n=1). Despite the variety of measures, all studies focused on educational activities, i.e. early literacy or early numeracy activities.

None of the studies, however, investigated peer interactions as an indicator of process quality.

Evaluating the quality of the measures used to assess process quality

Most of the studies reported information regarding the reliability of process quality measures (n=37 for the measure of staff-child interactions and n=11 for the measure of developmental and educational activities). The majority of studies reported alpha reliability coefficients, ranging between 0.68 and 0.95 for the measure of staff-child interactions and between 0.72 and 0.91 for the measure of developmental and educational activities. Some studies reported interrater reliability coefficients, ranging between 0.69 and 0.90 for the measure of staff-child interactions and between 0.71 and 0.90 for the measure of developmental and educational activities.

Description of measures: Child data

Data on children’s development and learning was most commonly assessed using standardised performance tests (n=21) and ECEC staff/parent ratings (n=8). Two studies used self-report measures. Available child data were grouped into emerging academic skills (early numeracy and literacy) and social and behavioural skills (behaviour regulation, executive function, behavioural problems, social competence). Common measures of emerging academic skills included the Peabody Picture Vocabulary Test (PPVT), the Woodcock-Johnson Tests of Cognitive Ability, and ECEC staff or parent rating scales to assess social and behavioural competencies.

Evaluating the quality of the measures used to assess children’s development and learning

Information regarding the measures’ reliability was limited. Only 17 studies reported reliability information, with alpha coefficients ranging between 0.72 and 0.98. However, in the majority of cases where reliability was not reported, studies used well-established and validated measures.

Description of effect sizes and preparation of data for estimation of overall effects

One important goal of meta-analysis was to compare multiple studies to estimate the overall, or combined, measure of effect size. For the current analysis, effect size is the degree and direction of association, or correlation, between two variables (e.g. between indicators of structural and process quality for Research Question 1, and between indicators of structural/process quality and child development and learning for Research Question 2).

Most of the studies reported correlation indices in the form of either zero-order correlation coefficient or regression coefficient. The former simply reflects the association between two variables without controlling for additionally related variables, while the latter is mostly produced as a result of multivariate regression where additional control variables are included. Both zero-order correlation coefficient and regression coefficient are standardised, such that a coefficient value ranges between -1 to 1 and being closer to 0 means little association between two variables. A coefficient of negative value would mean that an increase in the measure of one variable is associated with a decrease in the measure of the other variable, while a coefficient of positive value would mean that both variables increase or decrease in same direction. Only standardised coefficients were used in the analyses. Unstandardised coefficients of an association between two variables were standardised by multiplying the unstandardised coefficient with the coefficient of the standard deviation of variable 1 divided by the standard deviation variable 2.

The unit of analysis in this meta-analysis is at the study level – that is, each study that provides at least one effect size measure. Of the 44 studies included in the coding, a total of 34 studies (n=17 reporting research from the United States and n=17 reporting research from outside the Unites States) contributed at least one usable effect size in the analysis. Note that at least two unique studies are needed to run a meta-analysis and to obtain an estimate of combined effect size measure. For this reason, any studies that reported an effect size for an association that no other studies reported could not be used. Four studies that reported only unstandardised correlation coefficients could not be used in the analysis, because three did not report standard deviations for variables (Melhuish et al., 2013; NICHD ECCRN, 2007; Robert C Pianta, Mashburn, Downer, Hamre, & Justice, 2008) and the other appeared to report erroneous standard deviation values (M. Burchinal et al., 2010). Without a reported standard deviation for variables, unstandardised correlation coefficients cannot be standardised. Lastly, studies that used special regression methods that did not allow for effect size standardisation (Bowne et al., 2016b; Smidt et al., 2012; Sylva et al., 2013) were also omitted from the analysis.

Of the 34 studies that provided at least one effect size measure in our analysis, 14 studies adjusted their effect sizes by statistically accounting for nesting among subjects. Nested structure within data exists when a group of children are nested within an ECEC staff, and a group of ECEC staff are in turn nested within a centre/program. Adjusting for such hierarchical nesting may improve the accuracy of effect size, by accounting for the similarity among the subjects within a group compared to the subjects nested within other groups. In the current meta-analysis, we did not differentiate effect sizes based on whether nesting was adjusted in original studies.

Aggregation of effect sizes

To increase the number of unique studies for a test, we aggregated variables within a study for the following indicators: for staff-child interactions we combined positive interactions, instructional interactions and organisation/management of routines/chaos into one indicator; for developmental and educational activity, we combined all indicators into one; and for child data we combined behavioural and social-emotional indicators into one. Variables were combined within each study, by first converting effect size measures to z-score via Fisher’s z-transformation method and then averaging these z-scores and back-converting the average into Pearson’s r correlation coefficient using an equation suggested by (Corey, Dunlap, & Burke, 1998).

Statistical analysis

All statistical analysis was conducted using Comprehensive Meta Analysis software V.3. We used a two-tailed significance level of 0.05 throughout the analysis. Two types of meta-analysis were used in the current study: (general) meta-analysis and sub-group meta-analysis.

First, to estimate the combined measure of association between two variables, we ran a meta-analysis for each association of interest. Twenty such meta-analyses were conducted. An analysis was conducted if there were at least two studies contributing effect size measures; in other words, the sample size of each analysis was at minimum 2 (median=3).

The result of each analysis is presented in a forest plot. In these, original effect sizes reported from each study are depicted as black squares, and a summary effect size depicted as a blue diamond. Summary effect size is the estimated combined association between two variables. A summary effect size not significantly different from 0 (dotted line in a forest plot) would suggest little association, while a positive summary effect size closer to 1 would suggest stronger positive association and the opposite for a negative summary effect size closer to -1.

To obtain a summary effect size, original effect sizes are aggregated by assigning different ‘weights’ to each effect size (rather than simply averaging all effect sizes), hence the different sizes of black squares. The weighting is done because studies often differ in the precision of reported effect sizes.

There are two models used in meta-analysis, fixed-effect model and random-effects model, which make different assumptions about how to evaluate which effect sizes may be more precise and thus should receive higher weights. A fixed-effect model assumes that there is one true effect size that underlies all the studies in an analysis. As such, observed variability in effect sizes would be only due to sampling error. Under fixed-effect model, an effect size provided from a study with larger sample size receives higher weight because a larger sample size increases the precision of an estimate.

In contrast, a random-effects model allows a possibility that there are multiple true effect sizes and that studies may differ because they reflect meaningful differences among the studies due to the characteristics of their populations. Because each study is assumed to represent a distinct population, weighting is more balanced across studies but is more sensitive to the number of available effect sizes.

A random-effects model is more appropriate when considerable variation is expected among studies, such as in the current meta-analysis where we aimed to compare studies conducted on possibly quite heterogeneous populations from around the world. However, because of the low sample size per analysis, as well as over-representation of research from the United States with large sample sizes, we flexibly chose between fixed-effect and random-effects model assumptions following these criteria: Whenever there were more than two of US studies plus two non-US studies represented in the sample of unique studies, we used a random-effects model. Whenever there were less than two of either US or non-US studies represented in the sample of unique studies, we used both fixed-effect and random-effects model. Finally, when only US studies were available, we used a fixed-effect model.

Second, we conducted sub-group meta-analysis to test whether effect sizes would vary according to 1) countries and 2) children’s socio-economic background. Country comparisons were only possible to conduct across US and non-US studies (i.e. combining all other countries represented in the sample of 34 studies), due to the availability of data. Socio-economic background comparisons were conducted only among studies from the United States.

Because all the nine available studies conducted on disadvantaged children (i.e. children from low-income populations) were from the United States, we restricted the comparison within the United States to reduce potential addition of heterogeneity other than the difference due to children’s socio-economic background. In a sub-group meta-analysis, variability of effect sizes across sub-groups is compared to the overall variability of effect sizes. Whether the former is significantly larger than the latter is tested by Cochran’s Q-test. We assumed random-effects model for every sub-group meta-analysis, to account for the possibility that two sub-groups being compared may come from distinct populations.