Annex A. Technical notes on sampling procedures, response rates and adjudication for TALIS Starting Strong 2018
The objective of the Starting Strong Teaching and Learning International Survey (TALIS Starting Strong) 2018 was to obtain a representative sample in each participating country of staff and leaders providing early childhood education and care (ECEC) for each level of ECEC in which the country participated. The international sampling plan for TALIS Starting Strong used a stratified two-stage probability sampling design. This means that staff (second-stage units, or secondary sampling units) were randomly selected from the list of in-scope staff in each of the randomly selected ECEC centres (first-stage units, or primary sampling units). The leader at each centre (i.e. the person with the most responsibility for administrative, managerial and/or pedagogical leadership) was automatically selected for participation as well. For countries with integrated ECEC systems that participated in data collection for both pre-primary education and settings for children under age 3, programmes serving both age groups were split between the two samples so that each programme could be selected for participation in only one level of ECEC. A more detailed description of the survey design and its implementation can be found in the TALIS Starting Strong 2018 Technical Report (OECD, 2019[1]).
Staff in pre-primary education settings (ISCED Level 02) are those who, as part of their regular duties in their centre, provide learning opportunities in programmes at ISCED Level 02. Staff for children under age 3 are those who, as part of their regular duties in their centre, provide learning opportunities for children in this age group. Staff who provide learning opportunities for both target populations in their centres are included in the TALIS Starting Strong universe. There is no minimum cut-off for how much time staff need to be engaged at either level of ECEC.
The international target population of TALIS Starting Strong restricts the survey to those staff and leaders who work in officially registered settings providing ECEC. Centres exclusively for children with special educational needs are deemed out of scope. Also considered out of scope are: short-term substitute educators (to replace staff on sick leave); nannies and other people involved in informal arrangements; volunteers who occasionally came in to provide a special activity; auxiliary staff (e.g. cleaners, cooking staff) who did not interact regularly in a pedagogical manner with the children; and medical and therapeutic staff (e.g. speech therapists, occupational therapists) whose work was primarily non-pedagogical.
For national reasons, participating countries could choose to restrict the coverage of their national implementation of TALIS Starting Strong. For example, all participating countries decided to exclude home-based settings (within the homes of the respective staff) from the pre-primary education universe, while some countries included these settings in the universe of settings providing services for children under age 3. This report excludes home-based programmes for children under age 3, focusing only on centre-based ECEC to enhance comparability across the two levels of ECEC provision. Participating countries were invited to keep sample exclusions to a minimum by keeping the national survey population to at least 95% of ECEC staff. The national project manager for each country was required to document the reasons for any exclusions.
For both levels of ECEC, the same requirements for sample size and precision of estimates were established. To allow for reliable estimation and modelling while permitting some amount of non-response, TALIS Starting Strong 2018 set the minimum number of centres per country for each population of interest (pre-primary education and settings for children under age of 3) at 180. Within each centre, the minimum number of staff members selected was eight. If there were fewer than eight staff members in a centre, then all staff members were selected. Participating countries could choose to augment their national sample by selecting more centres, or by selecting more staff within each selected centre, or by increasing both. In some cases, because the average number of staff in the centres was lower than the number expected in the international plan, the number of centres sampled was increased.
The basic principle that guided the adjudication was to determine, for each participating country and for each level of ECEC, whether the data released to the countries are fit to provide policy-relevant, robust international indicators and analysis on staff and leaders. To establish fitness for use, a number of quality assurance processes were designed and activated throughout the survey process. Some processes relied on expert advice and opinion, some on qualitative information and learned judgement, some on quantitative information. More detailed information is available in the TALIS Starting Strong 2018 Technical Report (OECD, 2019[1]).
During the adjudication session, each individual data set (i.e. the combination of participating countries and levels of ECEC) was submitted to the same examination. In addition, both staff participation and leader participation were adjudicated for each combination of participating countries and levels of ECEC.
The issues evaluated concerned the questionnaire’s adaptation to national context, translation and verification, quality of the sampling frame, handling of out-of-scope and refusal units (i.e. staff and/or centres), within-centre sampling, data collection, data cleaning, the reports of quality observers, participation rates, and overall compliance with the technical standards (see the TALIS Starting Strong 2018 Technical Report (OECD, 2019[1])). Once each survey process was assessed, a recommended rating was formulated, accounting for the participation rates and for any unresolved issues. The adjudication rules, based on participation rates for leaders and staff, are shown in Tables A.1 and A.2.
The following is a guide to help data users appreciate the limitations on use or quality:
Good: The participating country’s data can be used for all reporting and analytical purposes and should be included in international comparisons.
Fair (A): National and subnational estimates can be produced. Some staff characteristics may suffer from larger standard errors (s.e.), hence the warning “Fair”. No additional warnings to users appear necessary.
Fair (B, only for staff data adjudication): National and subnational estimates can be produced. Some subnational estimates may be of lower precision (larger s.e.) if sample size is locally low, hence the warning “Fair”. No additional warnings to users appear necessary.
Some subnational estimates may be of lower precision (larger s.e.) if sample size is locally low, hence the warning “Fair”. But a note on data quality could appear pointing to the outcome of the non-response bias analysis.
Since centre participation is somewhat lower than under (B), comparing subnational estimates should be done with care, as some of those results are based on few centres.
Comparing small subnational estimates with similar groups from other participating countries is unlikely to uncover any statistically meaningful differences, as s.e. are likely too large.
In addition to the warnings issued for the previous category, a note should warn users of indications of non-response biases in some estimates.
Comparisons of subnational estimates should be limited to groups with larger sample sizes.
At this point, the sample represents between 37% and 56% of the workforce, from a rather small sample of centres.
Comparisons with similar groups in other participating countries would not be encouraged.
Poor (E, only for staff data adjudication): Subnational estimates would not be recommended. There should be a note pointing out the difficulty of obtaining a representative sample of centres.
Poor (F, only for staff data adjudication): Limitations similar to those of line E, but there should be a note pointing out the difficulty of obtaining at least 50% participation of the selected sample of centres. There are risks of having a non-representative sample of centres.
Insufficient: Weights should not be calculated for any official tabulations. Hence, data should not be incorporated in international tables, models, averages, etc.1
The participation rates and the adjudication rating per participating country and level of ECEC are presented in Tables A.3-A.6. These tables display the participation rate estimates that were the most favourable for the adjudication rating. The most favourable estimates could have been weighted or unweighted depending on the characteristics of the country, the staff and leader population, and the level of ECEC.
This section lists issues to be noted regarding the sampling or field operations that should be considered when interpreting the data reported for the following countries:
The data collection period was reopened for a few weeks to encourage additional centres and staff to participate.
Non-response bias analysis failed to show that there is not a high risk of centre and staff non-response bias.
In two centres serving children under age 3, staff listings were found to be incorrect; these centres were considered as “non-participant”.
In Iceland, a shortage of certified ECEC teachers means that staff without this credential (i.e. assistants) may be serving as teachers in some settings. Although data were collected on these role divisions, the distinction was unclear in many instances. Therefore, this overall role distinction is not used in TALIS Starting Strong because it is not meaningful for Iceland.
For pre-primary education settings (ISCED Level 02), ultra-Orthodox centres were excluded after the survey because of the low participation rates in this sector. The exclusion rate therefore exceeded 5%.
The data collection period was extended for centres serving children under age 3 to accommodate the split system in Israel.
References
[1] OECD (2019), TALIS Starting Strong 2018 Technical Report, OECD Publishing, Paris, http://www.oecd.org/education/talis/TALIS-Starting-Strong-2018-Technical-Report.pdf.
Note
← 1. At its November 2018 meeting in Paris, the TALIS Starting Strong 2018 Technical Advisory Group recommended that data from participating countries that had not reached 50% participation should nonetheless be weighted and displayed in tables.