Annex B. Technical notes on analyses in this report
The statistics presented in this report were derived from data obtained through samples of centres, centre leaders and staff (see Annex A). For these statistics to be meaningful for a country, they need to reflect the whole population from which they were drawn and not merely the sample used to collect them. Thus, survey weights must be used in order to obtain design-unbiased estimates of population or model parameters.
Final weights allow the production of country-level estimates from the observed sample data. The estimation weight indicates how many population units are represented by a sampled unit. The final weight is the combination of many factors, reflecting the probabilities of selection at the various stages of sampling and the response obtained at each stage. Other factors may also come into play as dictated by special conditions to maintain the unbiasedness of the estimates (e.g. adjustment for staff working in more than one centre). A detailed description of the sampling and weighting procedures can be found in the TALIS Starting Strong 2018 Technical Report (OECD, 2019[1]).
Statistics in this report that are based on the responses of centre leaders and that contribute to estimates related to centre leaders were estimated using centre weights (CNTRWGT). Results based only on responses of staff or on responses of staff and leaders (i.e. responses from centre leaders were merged with staff responses) were weighted by staff weights (STAFFWGT).
Standard errors
The statistics in this report represent estimates based on samples of staff and centres, rather than values that could be calculated if every staff member and leader in every country had answered every question. Consequently, it is important to measure the degree of uncertainty of the estimates. In TALIS Starting Strong, each estimate has an associated degree of uncertainty that is expressed through a standard error. The use of confidence intervals provides a way to make inferences about the population statistics in a manner that reflects the uncertainty associated with the sample estimates. From an observed sample statistic and assuming a normal distribution, it can be inferred that the corresponding population result would lie within the confidence interval in 95 out of 100 replications of the measurement on different samples drawn from the same population. The reported standard errors were computed with a balanced repeated replication methodology.
Differences between sub-groups
Differences between sub-groups among staff (e.g. teachers and assistants) and centre characteristics (e.g. centres with a high concentration of children from socio-economically disadvantaged homes and centres with a low concentration of children from socio-economically disadvantaged homes) were tested for statistical significance. All differences marked in bold in the data tables of this report are statistically significantly different from 0 at the 95% confidence level. In the case of differences between sub-groups, the standard error is calculated by taking into account that the two sub-samples are not independent.
Scales
In this report, several scales are used in descriptive and regression analyses. All these scales are the result of extensive construction and evaluation processes using guidelines and experience from TALIS 2018 and prior cycles.
The scaling construction process summarises responses from multiple items related to the same dimension into a single indicator. Because TALIS Starting Strong measures the self-reported practices and beliefs of staff and leaders from countries with different cultural backgrounds and in different settings (i.e. pre-primary education and centres for children under age 3), building the scales entails a number of methodological issues. In particular, individual and cultural factors affect the interpretation of questions. This may produce differences in levels of endorsement or frequency in survey responses and it may also affect the item correlation structure used to summarise the information and thus limit the comparability of the resulting indicators. In order to effectively use these scales for further analysis, it is important to consider the specific scale properties, such as their reliability and validity in a cross-cultural context.
To understand whether scales from TALIS Starting Strong could be considered comparable across countries and levels of ECEC, measurement invariance was tested. The most restrictive level of measurement invariance, scalar invariance, is reached once the indicator satisfies three properties:
1. The structure of the scale is the same across groups, meaning that the scale is built using the same set of items across groups.
2. The strength of the associations between the scale and the items (factor loadings) are equivalent. This property makes it possible to claim that one unit of change in the scale will lead to the same amount of average change in the items that constitute the construct across different groups.
3. The intercepts/thresholds for all items across groups are equivalent. If the intercepts of the items for all groups are equivalent, then the expected value of the items becomes the same across groups when the value of the indicator is zero and means can be compared across groups.
If only properties (1) and (2) are satisfied, then the scale reaches metric invariance. This means that these scales can be used for comparison within countries and comparisons across countries of the strength of the association between the scales and other factors. However, the means of these scales cannot be compared across countries. If only property (1) is satisfied, the scale reaches configural invariance. This means that results using these indicators are meaningful within countries, but cannot be compared across countries.
By design, all scales have a midpoint of 10 and a standard deviation of 2. This means that scales with values above 12 can be considered high. The fact that all scales have the same midpoint helps interpret the level of prevalence of a specific practice or attitude, regardless of the frequency with which it is expected to occur in the centre (or target group) or be held by staff. Additional information on the construction and validation of the scales included in this report can be found in Chapter 11 of the TALIS Starting Strong 2018 Technical Report (OECD, 2019[1]).
A number of scales from TALIS Starting Strong assess different dimensions of process quality; that is, the quality of the various interactions between the ECEC workforce, children and parents. These scales are described in detail in Chapter 2 and Annex C of the first volume of TALIS Starting Strong (OECD, 2019[2]). Scales used for the first time in the current volume are described below.
Staff engagement in collaborative professional practices
The scale of staff engagement in collaborative practices (S1COLL) summarises staff reports on the frequency with which they engage at their centre in several activities involving collaboration with other staff (SS1G23). It reached metric invariance for target populations at both the pre-primary level (ISCED Level 02) and in settings for children under age 3. For more details, see Chapter 11 of the TALIS Starting Strong 2018 Technical Report (OECD, 2019[1]).
Staff sense of self-efficacy in supporting children’s development, learning and well-being
The staff scale of self-efficacy for supporting children’s development, learning and well-being (S1SECD) summarises staff reports on the extent to which they feel they can do different activities with children at their centre (SS1G24). It reached metric invariance for target populations at both the pre-primary level (ISCED Level 02) and in settings for children under age 3. The scale was not part of the first data release in 2019, but was added to the database in 2020. The construction and validation processes followed the same procedures as the main scaling for the TALIS Starting Strong Survey 2018. For more details, see Chapter 11 of the TALIS Starting Strong 2018 Technical Report (OECD, 2019[1]) and Annex on the scale.
Centre leader support for pedagogical learning
The scale of leader support for pedagogical learning (S1LEADS) summarises centre leaders’ reports about areas of pedagogical leadership within their ECEC setting (SL1G32). It reached metric invariance for target populations at both the pre-primary level (ISCED Level 02) and in settings for children under age 3. For more details, see Chapter 11 of the TALIS Starting Strong 2018 Technical Report (OECD, 2019[1]).
Table A B.1 lists all the scales used in this publication and their levels of measurement invariance.
Complex variables
Number of staff and children in the centre
TALIS Starting Strong asks leaders to indicate the number of staff in different categories working in their ECEC centres (leaders, teachers, assistants, staff for individual children, staff for special tasks, interns and other staff) (SL1G17) and the number of children enrolled in the centre (SL1G19).
This information is used to derive several indicators describing the staff and children in the centre: the share of different types of staff working at the centre (i.e. leaders, teachers, assistants and other staff); the number of teachers and leaders compared to the total number of staff at the centre; the number of staff per child at the centre. If the centre covers pre-primary education (ISCED Level 02) and provision for children under age 3, children and staff at both levels are considered in those numbers.
The number of staff per child at the centre refers to the total number of staff working in a centre, regardless of their role, divided by the total number of children enrolled. Because the number of staff per individual child is very low, when specific examples are cited for comparative purposes, they are presented as “number of staff per ten children”, which is obtained by multiplying the number of staff per child by ten.
These indicators differ from administrative data capturing similar constructs, for instance because TALIS Starting Strong data do not allow differentiation between part-time and full-time employment at the centre level. Furthermore, regulations often refer to staffing requirements at the group or classroom/playgroup/group level, rather than for the centre as a whole.
Number of staff and children in the target group
A similar set of variables is also built at the level of the target group. TALIS Starting Strong asks staff to take the example of the target group (the first group of children they were working with on the last working day before the day of the survey). Respondents indicate the category that best represents their role when working with this group of children (leader, teacher, assistant, staff for individual children, staff for special tasks, interns and other staff) (SS1G36), as well as the number of girls and boys who made up the group (SS1G37).
This information is used to derive three indicators: 1) the number of children per target group; 2) the number of staff working with the same target group on the same day; and 3) the number of staff per child working with the same target group on the same day.
The number of staff per child with the same target group on the same day refers to the number of staff working with the same target group, regardless of their role, divided by the number of children in the target group. Because the number of staff per individual child is very low, when specific examples are cited for comparative purposes, they are presented as “number of staff per ten children”, which is obtained by multiplying the number of staff per child by ten.
The number of staff per child working with the same target group on the same day reflects a specific situation and is, therefore, different from the number of staff per child at the centre level. Staff may be working with the same target group at different moments of the day and not together, or may work part time. Children in the same group may also change over the day into different group compositions, and children’s attendance hours can differ. This concept also differs from the regulated maximum numbers of children per staff member, as that could include some restrictions on the staff to be included (depending on their qualifications or role) and can be specific to the age group of children.
As there is no indicator clarifying which target group each staff member referred to, several staff members may have referred to the same target group. This can result in a bias, as some target groups may be overrepresented in the data.
Share of staff who left their ECEC centre in the previous year
Leaders participating in TALIS Starting Strong reported on the number of staff who left the ECEC centre in the previous year (SL1G18B). The share of staff who left their ECEC centre in the previous year is obtained by dividing this variable by the total number of staff at the centre at the time leaders responded to the survey.
Novice and experienced staff and leaders
The novice and experienced staff variables were calculated by using staff reports about their total years working as an ECEC staff (SS1G06B). Respondents were considered novice staff if they had worked for three years or less in the ECEC sector in staff roles (i.e. teachers, assistants, staff for individual children, staff for special tasks and interns), and experienced staff if they had worked for more than three years in the sector in staff roles.
The novice and experienced centre leader variables were calculated by using centre leader reports about their total years working as an ECEC centre leader (SL1G05B). Respondents were considered novice centre leaders if they had worked for three years or less in the ECEC sector in a centre leader role, and experienced centre leaders if they had worked for more than three years in the sector in a centre leader role.
Staff’s weekly hours spent without children
The number of weekly working hours without children was calculated by using staff reports to questions about their weekly total weekly working hours (SS1G18) and about their weekly hours spent with children (SS1G19). To calculate this indicator, the number of hours spent with children (SS1G19) was subtracted from the total number of hours worked (SS1G18). Negative values, resulting from staff who reported more hours spent with children than overall working hours, were excluded.
Percentage of working hours spent without children
The percentage of working hours without children variable was calculated by multiplying the number of weekly working hours spent without children (see above) by 100 and divided by the total number of weekly working hours using staff reports about their total weekly working hours (SS1G18). Some analysis required this continuous variable to be split into national quarters (see below).
Percentage of working hours spent with children
The percentage of working hours without children variable was calculated by using staff reports about their total weekly working hours (SS1G18) and about their weekly working hours spent with children (SS1G19). To calculate this indicator, the number of weekly working hours spent with children was multiplied by 100 and divided by the total number of weekly working hours. All values that exceeded 100%, resulting from staff who reported more hours spent with children than overall working hours, were excluded. Some analysis required this continuous variable to be split into national quarters (see below).
The number of weekly working hours of ECEC centre leaders was calculated by using centre leader reports about their usual working hours per week in this centre (SL1G06). To calculate this indicator, all values of 10 weekly hours or below (equivalent to 2 hours per day, on average, for a 5-day work week) and above 60 weekly hours (equivalent to 12 hours per day, on average, for a 5-day work week) were excluded to reduce the impact of outliers on the analysis of national averages and percentiles.
Level of ECEC centre autonomy for tasks
This set of indicators was derived from ECEC centre leader reports about who has significant responsibilities for eight different tasks related to curriculum, policies, staffing and budgeting (SL1G21): “deciding which activities for children are offered” and “choosing which materials/toys are used” (curriculum-related tasks); “establishing monitoring plans for children’s development, well-being and learning” and “approving children for admission to the centre” (policies-related tasks); “appointing or hiring staff” and “dismissing or suspending staff from employment” (staffing-related tasks); and “establishing staff salaries” and “deciding on budget allocations within the centre” (budgeting-related tasks).
For each task, a new indicator was calculated by recoding the original responses into three categories: 1) “full autonomy”, which corresponds to original responses that only indicate that “me and/or other members of staff” have significant responsibility; 2) “partial autonomy”, which corresponds to the responses that indicate that “me and/or other members of staff”, plus the “centre governing board” and/or the “local municipality/regional, state or national/federal authority” have significant responsibility; 3) “no autonomy”, which corresponds to responses that indicate that only the “centre governing board” and/or “the local municipality/regional, state or national authority/federal authority” have significant responsibility.
For Japan, the indicator was calculated separately since the response categories to the questions about who has significant responsibilities were adapted to the national context. For each task, the indicator was created by recoding the original responses into the following three categories: 1) “full autonomy”, which corresponds to original responses that only indicate that “me and/or other members of staff” have significant responsibility; 2) “partial autonomy”, which corresponds to the responses that indicate that “me and/or other members of staff”, plus the "Incorporated/social institutes'' and/or the “local, municipality/regional, state, or national/federal authority” have significant responsibility; 3) “no autonomy”, which corresponds to responses that indicate that only the “incorporated/social institutes”' or the “local municipality/regional, state or national /federal authority” have significant responsibility.
Scope of ECEC centre autonomy for resources and pedagogy
To describe the scope of ECEC centre autonomy for resources and pedagogy, two indicators were derived from centre leader reports about who has significant responsibility for such tasks (SL1G21).
Centre autonomy for resources refers to these four items: 1) “appointing or hiring staff”; 2) “dismissing or suspending staff from employment”; 3) “establishing staff's salaries”; and 4) “deciding on budget allocations within the centre”. Centre autonomy for pedagogy refers to the following four items: 1) “establishing monitoring plans for children's development, well-being and learning”; 2) “approving children for admission to the centre”; 3) “choosing which materials/toys are used”; and 4) “deciding which activities for children are offered”.
For each specific task, centre leaders’ responses were recoded into the category “full autonomy”, where centre leaders responded only that “me and/or other members of staff” have significant responsibility.
For each group of tasks (i.e. resources and pedagogy), an indicator “full autonomy in the majority of tasks” was then created based on centre leaders reporting only that “me and/or other members of staff" have significant responsibility in at least three of the four items.
Centre-level indices of human and material resources
TALIS Starting Strong 2018 asked ECEC centre leaders about the extent to which the lack of various human and material resources hindered their centres’ capacity to provide a quality environment for children’s development, learning and well-being (SL1G34).
Two centre-level indicators of resources were based on these reports. The index of shortage of human resources was derived from the following four items: 1) “shortage of qualified staff”; 2) “shortage of staff for the number of enrolled children”; 3) “shortage of staff with competence in working with children from socio-economically disadvantaged homes”; and 4) “shortage of staff with competence in working with children with special needs”. The index of shortage of material resources was derived from the following four items: 1) “shortage or inadequacy of indoor space”; 2) “shortage or inadequacy of outdoor play space”; 3) “shortage or inadequacy of play or learning materials (e.g. books, picture books, building blocks, clay, paint)”; and 4) “shortage or inadequacy of digital technology for play and learning (e.g. computers, tablets, smart boards)”.
Both indices were recoded to have three values: “not a problem of shortage” (value=1) if leaders responded “not at all” to 3 or 4 items (allowing “to some extent” for 1 item); “problematic shortage” (value=3) if leaders responded “quite a bit” or “a lot” to 3 or 4 items; and “minor problem” (value=2), an intermediate category for all other combinations of leader responses.
Higher values on these indices therefore mean that ECEC centre leaders view the amount and/or quality of resources in their centres as an obstacle to providing quality environments for children.
Some analysis using complex continuous variables required these variables to be transformed into interval categories. To accommodate for this need, the report makes use of national quarters. In each country, the weighted distribution of the continuous variable is split into equally sized categories, following the rank order. For instance, the cut-off point between the first quarter and the second quarter of the scale of staff engagement in collaborative professional practices is the 25th percentile of the distribution of the scale in a specific country. As a result, the range of these intervals will differ across countries and vary with the properties of the distribution in each country.
Country-specific regression analyses were performed to examine the associations between different variables. Multiple linear regression was used in those cases where the dependent (or outcome) variable was considered continuous, for example with the process quality indicators. Binary logistic regression was employed when the dependent (or outcome) variable was a binary categorical variable, for example a high versus a low level of stress (see Chapter 3).
When interpreting results from regressions, it is important to keep in mind that each regression coefficient represents the additional effect of adding that variable to the model, if the effects of all other variables in the model are already accounted for. It is also important to note that, because cross-sectional survey data were used in these analyses, no causal conclusions can be drawn.
The centre (or staff, or target group) characteristics of interest can relate to one another and with other characteristics of the staff member or centre leader who is reporting. Thus, the regression analyses were performed through an estimation of the associations of interest, holding all other characteristics constant. In the models, the associations between a specific centre (or staff, or target group) feature and the outcome variable were examined after accounting for a set of centre and staff characteristics, described below. Control variables included in the regression models were selected based on theoretical reasoning and to ensure comparability of the model across countries. For some countries, the number of staff or centres in a particular category was too low to draw conclusions. Results are presented only when they are based on a minimum of 30 staff or 10 centres.
Staff and centre characteristics used in the models
The typical regression model used in this report includes the following set of variables as independent variables. In some cases, additional variables of interest are added depending on the purpose of the analysis, while in other cases fewer variables are used as controls. Footnotes to tables for all models presented in the report provide specific information on the variables included in respective models (see Annex C). Table 3.5 in Chapter 3 provides specific information on the variables included in models analysing sources of stress.
Staff education level is aggregated into three categories: 1) secondary education or below (ISCED Level 3 or below); 2) post-secondary non-tertiary education or short-cycle tertiary education (ISCED Level 4 or 5); and 3) bachelor degree or equivalent or more (ISCED Level 6 or more), which is set as the reference.
Staff specifically trained to work with children versus staff without specific training (without specific training as the reference).
Staff experience refers to the number of years of experience in any ECEC centre, in three categories: 1) less than five years; 2) between five and nine years; and 3) more than nine years, which is set as the reference.
Permanent employment versus fixed-term contracts/self-employment (two categories with fixed-term contracts as the reference).
Working full time versus part time (part time as the reference).
Leader/teacher: the respondent is either a leader or a teacher in the target group. All other categories, including assistants, are grouped and taken as the reference.
Centre in city: the centre is in a municipality with more than 15 000 people, with a location with fewer people taken as the reference.
Public management versus private management (private management as the reference).
Number of children in the centre (or target group), in quarters. In each country, the distribution of answers from leaders on the number of children can be divided into four equal quarters with increasing numbers of children per centre. The first quarter is set as the reference: the respondent works in a centre (or target group) with a number of children among the 25% lowest of the country distribution.
Number of staff per child, in quarters: the total number of staff working in the centre (or target group), regardless of their role, divided by the number of children in the centre (or target group). The first quarter is set as the reference: the respondent works in a centre (or target group) with a number of staff per child among the 25% lowest of the country distribution.
Concentration of children from socially disadvantaged homes: the proportion of children from socio-economically disadvantaged homes in the centre (target group) is greater than or equal to 11%, with a proportion of 10% or less set as the reference.
Strength of association from multiple linear regression models
Multiple linear regression analysis provides insights into how the value of a continuous dependent (or outcome) variable changes when any one of the independent (or explanatory) variables varies while all other independent variables are held constant.
In tables and figures in this report, the strength of association between two variables relates to the magnitude of the unstandardised coefficient corresponding to the independent variable of interest, when predicting the dependent variable in a multiple linear regression model, while all other independent variables are held constant.
Regression coefficients in bold in the data tables presenting the results of regression analysis are statistically significantly different from 0 at the 95% confidence level.
Odds ratios from binary logistic regression models
Binary logistic regression analysis enables the estimation of the relationship between one or more independent (or explanatory) variables and the dependent (or outcome) variable with two categories. The regression coefficient of a logistic regression is the estimated increase in the log odds of the outcome per unit increase in the value of the predictor variable.
In tables and figures in this report, these coefficients are transformed into odds ratios by being exponentiated to make results more interpretable in terms of probability. The odds ratio is a measure of the relative likelihood of a particular outcome across two groups. An odds ratio indicates the degree to which an explanatory variable is associated with a categorical outcome variable, while all other independent variables are held constant. An odds ratio below 1 denotes a negative association, and thus a lower likelihood of observing the outcome; an odds ratio above 1 indicates a positive association, and thus a higher likelihood of observing the outcome; and an odds ratio of 1 means that there is no association.
The odds ratios in bold in the data tables indicate that the odds ratio is statistically significantly different from 1 at the 95% confidence level. To compute statistical significance around the value of 1 (the null hypothesis), the odds ratio statistic is assumed to follow a log-normal distribution, rather than a normal distribution.
Correlation coefficients measure the strength and direction of the statistical association between two variables. Correlation coefficients vary between -1 and 1; values around 0 indicate a weak association, while the extreme values indicate the strongest possible negative or positive association. The Pearson correlation coefficient measures the strength and direction of the linear relationship between two variables.
Cross-country averages are provided for pre-primary (ISCED Level 02) settings throughout the report. These averages correspond to the arithmetic mean of the nine country estimates. Comparisons between a single country and the international average are not used because the averages reflect only nine countries. Each country makes a substantial contribution to the overall average and therefore a comparison between the averages and a single country may overestimate the similarity of that country’s results with those from the other countries.
References
[2] OECD (2019), Providing Quality Early Childhood Education and Care: Results from the Starting Strong Survey 2018, TALIS, OECD Publishing, Paris, https://dx.doi.org/10.1787/301005d1-en.
[1] OECD (2019), TALIS Starting Strong 2018 Technical Report, OECD Publishing, Paris, http://www.oecd.org/education/talis/TALIS-Starting-Strong-2018-Technical-Report.pdf.