4. Ensuring cross-cultural reliability and validity

Performance tasks and scoring

In a Performance Task (PT), students are asked to generate their own answers to questions as the evidence of skill attainment (Hyytinen and Toom, 2019[8]; Shavelson, 2010[9]). PTs are typically based on the criterion-sampling approach (Hyytinen and Toom, 2019[8]; McClelland, 1973[10]; Shavelson, 2010[9]; Shavelson, Zlatkin-Troitschanskaia and Mariño, 2018[11]). In that sense, their aim is to elicit what students know and can do (McClelland, 1973[10]; Shavelson, 2010[9]; Zlatkin-Troitschanskaia et al., 2019[12]). That is, PTs engage students in applying their skills in genuine or “authentic” contexts, not simply memorising a body of factual knowledge or recognising the correct answer from a list of options. A PT, with its open-ended questions, requires the integration of several skills, for example, analysing, evaluating, and synthesising information and justifying conclusions by utilising the available evidence (Hyytinen et al., 2015[13]; Shavelson, 2010[9]).

Recently, several challenges relating to the use, reliability, and interpretation of PTs have been reported (Attali, 2014[14]; Shavelson, Zlatkin-Troitschanskaia and Mariño, 2018[11]; Zlatkin-Troitschanskaia et al., 2019[12]). Answering a PT takes time and effort from students. Thus, relatively few constructs can be observed and assessed within one PT compared to multiple-choice tests (Zlatkin-Troitschanskaia et al., 2019[12]). Another challenge relates to scoring. The way in which students’ responses are scored plays a crucial role in assessment validity (Solano-Flores, 2012[15]). The scoring criteria of PT responses need to be developed and defined so that they are in line with the construct measured (Zlatkin-Troitschanskaia et al., 2019[12]). Moreover, students’ written responses are typically scored by using human evaluation. Therefore, it has been suggested that the scoring of PTs is open to bias (Hyytinen et al., 2015[13]; Popham, 2003[16]).

In scoring, the scorers analyse the quality of the response based on a scoring rubric that describes the characteristics of typical features at each score level. It has been assumed that it is very difficult for scorers to converge on a single scoring standard (Attali, 2014[14]; Braun et al., 2020[17]). Disagreements about the quality of the response across the scorers result in inconsistencies in scoring. However, most of the inconsistencies stem from incorrect interpretation of the scoring rubric (Borowiec and Castle, 2019[18]). Extensive scorer training and rubric development via cognitive interviews have been proposed as solutions for ensuring consistent scoring (Borowiec and Castle, 2019[18]; Shavelson, Baxter and Gao, 1993[19]); see also Zlatkin-Troitschanskaia et al (2019[12]). Consequently, the use of PTs is considered time-consuming and expensive, as a large amount of time and effort is needed to train scorers and to score the responses consistently (Attali, 2014[14]; Braun, 2019[20]).

Cross-cultural challenges in PT scoring

Cross-cultural assessments may have challenges due to cultural and linguistic differences or technical and methodological issues (Hambleton, 2004[4]; Solano-Flores, 2012[15]). This case study focuses on cross-cultural challenges in the scoring of PTs. To our knowledge, cross-cultural scoring presents two types of possible challenges. First, the scoring rubric may not meet the conventions of the second culture, in this case Finnish. Second, scorers may interpret responses and the scoring rubric in a different way than originally intended due to cultural differences (Braun et al., 2020[17]). For instance, in Finland there is no substantive assessment culture: there are hardly any high-stakes assessments in schools or higher education institutions (e.g., (Sahlberg, 2011[21])), and therefore, scorers may have limited experience with such evaluation.

Furthermore, Finnish is inherently different from English, which is the original language of the present assessment. Finnish belongs to the Finno-Ugric language group, distinct from English and Indo-European languages. This means the languages are structurally very different. It also seems that linguistic conventions are different between the two languages. For instance, it has been found that Finnish has different rhetoric practices compared to English (Mauranen, 1993[22]). These differences between languages may influence how scorers from each country interpret responses and scoring principles.

It has been noted that often in cross-cultural assessments, the same individuals do not establish the content to be assessed, write and develop the task, create the scoring rubric, and score students’ answers. Therefore, it is important that there is a shared understanding of the elements of the assessment among different contributors in order to make sure that the constructs are measured and scored in the same manner across cultures (Solano-Flores, 2012[15]).

United States context

In the United States, most participating institutions of higher education have traditionally used CLA+ for value-added purposes. To this end, they assess a sample of entering first-year students typically during the fall semester and compare those students to a sample of exiting fourth-year students typically assessed during the spring semester. The demographic characteristics of the student population in the USA are presented in Table 4.1.

Finnish context

The main feature of evaluation in Finnish higher education is that it is enhancement-oriented so that the focus is on providing support and information to further enhance the quality of the programmes and institutions. Consequently, the evaluation of Finnish higher education favours more formative ways of evaluation as opposed to summative evaluation focusing on the achievement of specified targets (Ursin, 2020[24]). Hence, the use of standardised tests, such as CLA+, is not typical in Finnish higher education institutions, and this case study was the first of this calibre to utilise a standardised test to investigate generic skills of Finnish undergraduate students. Nonetheless, similar to the United States, the 18 participating higher education institutions from Finland tested entering first-year students typically in the fall semester and exiting third-year students typically during the spring semester. The Finnish sample consisted of 2 402 students (1 538 entering and 864 exiting students) from the 2019-2020 academic year (Table 4.2).

PTs

For the PT, students are given an engaging, real-world scenario along with a set of documents such as research articles, newspaper and magazine articles, maps, graphs, and opinion pieces pertaining to the scenario. They are asked to make a decision or recommendation after analysing these documents and to write a response justifying their decision/recommendation by providing reasons and evidence both for their argument and against the opposing argument(s). There is no one correct answer for any PT. Rather, students are scored on how well they support and justify their decision with the information provided in the documents. No prior knowledge of any particular domain is necessary to do well, nor is there an interaction between the topic of the PT and the students’ field of study (Steedle and Bradley, 2012[25]).

The CLA+ scoring rubric (Chapter 2) for the PT has three subscores: Analysis and Problem Solving (APS), Writing Effectiveness (WE), and Writing Mechanics (WM).

The APS subscore measures students’ ability to interpret, analyse, and evaluate the quality of the information that is provided in the document library. This entails identifying information that is relevant to a problem, highlighting connected and conflicting information, detecting flaws in logic and questionable assumptions, and explaining why information is credible, unreliable, or limited. It also evaluates how well students consider and weigh information from discrete sources to make decisions, draw conclusions, or propose a course of action that logically follows from valid arguments, evidence, and examples. Students are also expected to consider the implications of their decisions and suggest additional research when appropriate.

The WE subscore evaluates how well students construct an organised and logically cohesive argument by providing elaboration on facts or ideas. For example, students can explain how evidence bears on the problem by providing examples from the documents and emphasising especially convincing evidence.

The WM subscore measures how well students follow the grammatical and writing conventions of the native language. This includes factors such as vocabulary, diction, punctuation, transitions, spelling, and phrasing.

Each student receives a raw score from 1-6 on each of the subscores, so, in this study, student total scores ranged from 3-18. If a student failed to respond to the task or gave a response that was off topic, a score of 0 (the equivalent of N/A) was given for all three subscores, and these student responses were eliminated from any subsequent analyses.

The rubric was also translated from English into Finnish. The scoring rubric was originally developed to be specific to each PT (Klein et al., 2007[26]; Shavelson, 2008[27]). However, this method of using unique rubrics for individual tasks created an issue with the standardisation, and thus validity, of the assessment. In 2008, a standardised version of the scoring rubric was introduced to address this threat to validity. The standardised version was designed to measure the underlying constructs assessed (i.e., analytic reasoning and evaluation, problem solving, writing effectiveness, and writing mechanics) that are applicable to all versions of the PT. This method of using a common rubric for PTs was found to be useful for improving students’ skills (with feedback) across multiple fields of study (Cargas, Williams and Rosenberg, 2017[28]).

In fact, having a common rubric for a performance-based assessment of domain-agnostic skills was one of the reasons the CLA was selected to be the anchor for the “generic skills strand” of the Assessment of Higher Education Learning Outcomes (AHELO) project, an international feasibility study sponsored by the OECD (AHELO, 2012[29]; 2014[30]; Tremblay, Lalancette and Roseveare, 2012[31]) to measure higher education students’ “generic” skills using PTs in a global context. The results from this feasibility study indicated that it was indeed possible to validly and reliably assess students’ critical thinking and written communication skills with a common, translated and culturally adapted PT and rubric (Zahner‬ and Steedle, 2014[1])‬‬‬‬‬‬‬‬‬‬‬‬‬‬. This result was replicated in a subsequent international comparative study between American and Italian higher education students (Zahner and Ciolfi, 2018[2]).

SRQs

CLA+ includes a set of 25 SRQs that are also document-based and designed to measure the same construct as the APS subscore of the PT. Ten measure Data Literacy (DL) (e.g. making an inference); ten, critical reading and evaluation (CRE) (e.g. identifying assumptions); and five measure critiquing arguments (CA) (e.g. detecting logical fallacies). This section of the assessment was not analysed for this project as it was automatically machine-scored.

Translation and adaptation

CLA+ is translated and culturally adapted using the internationally accepted five-step translation process that is in compliance with International Test Commission (Bartram et al., 2018[32]) guidelines. The Council for Aid to Education (CAE) follows the guidelines used for the localisation process of major international studies such as PISA, Trends in International Mathematics and Science Study (TIMSS), Progress in International Reading Literacy Study (PIRLS), PIAAC, and AHELO. The process includes a translatability review, double translation and reconciliation, client review, focused verification, and cognitive labs.

During the translatability review, source material is reviewed to confirm that the text will adapt well to the native language and culture. Particular attention is paid to disambiguation of source, respecting key correspondences between stimuli and questions, and deciding what should or should not be adapted to local context. Two independent translators then review the text and provide translations. The translations are reconciled and sent to the lead project manager for review and an opportunity to provide minor suggestions. The translated CLA+ items are sent for a focused verification. Cognitive labs are then carried out with the assistance of participating institutions and the institutional teams to ensure the translation and adaptation process was effective.

The CLA+, including the rubric used in this study, was translated into Finnish and Swedish, which are the two main official languages of Finland. The adaptation and translation of both language versions of the task included several phases, as described above. Firstly, the test instruments were translated from English into the target language. Then, two translators (who had knowledge of English-speaking cultures but whose native language was the primary language of the target culture) independently checked and confirmed the translations. Subsequently, the research team in Finland reconciled and verified the revisions. The translations were then pretested in cognitive labs among 20 Finnish undergraduate students, with final modifications incorporated, as necessary. The use of cognitive labs with think-aloud protocols and interviews made it possible to ensure that the translation and adaptation process had not altered the meaning or difficulty of the task (Hyytinen et al., 2021[33]; Leighton, 2017[34]).

This case study only investigated a subset of Finnish student responses. Finnish was specifically selected due to its differences from English as well as the significantly larger number of student responses in Finnish compared to Finnish-Swedish.

CLA+ was administered online via a secure testing platform during an assigned testing window ranging from August 2019 into March 2020. In September 2019, a CAE measurement science team member went to Finland to conduct a two-day in-person scorer training. The first day consisted of an introduction to the PT, an overview of the online scoring interface, a thorough review of the PT and all of the associated documents, and a review of the scoring handbook, followed by initial scoring and calibration of the training papers. As part of scoring the calibration papers, each scorer independently scored a student’s response to the PT. Each scorer then shared their three subscores (APS, WE, and WM) with the group. The CAE colleague then revealed the CAE-verified score for the training paper. A discussion of the CAE score compared to the Finnish scores followed each paper. On the second day, the group completed the scoring of the preselected calibration papers as well as discussed a plan for completing the scoring process.

Following this in-person meeting, the two lead scorers in Finland double-scored the first batch of student responses and selected their calibration papers based on their agreement and the distribution of scores. Fifty calibration papers were selected to be inputted into the scoring queue in order to check for consistency of scoring. A scorer would receive one of these calibration papers every 15 or 20 responses. If they did not score within the appropriate range of the previously scored paper, they would enter into a separate training queue of more previously scored validity papers. If they failed again, they would need remediation, which is one-on-one consultation with the lead scorer. Following remediation, the scorer would once again need to pass a set of training papers before being allowed back into the scoring queue. Remediation occurred once for the Finnish scoring team. As a point of reference, the American scorers, in any given administration of this PT, require remediation an average of 1.7 times per semester, although the volume of student responses is much larger for the American scorers.

Once scoring commenced, there was an internal calibration meeting for the Finnish team as well as a meeting with CAE as many months had passed between the training and scoring.

Scoring equivalency case study

In order to assess the equivalency of scoring across countries, we sought to answer two research questions. The first was how well does a translated and culturally adapted performance-based assessment requiring students to generate a written response get scored across countries and languages? The second was whether equivalence can be established across the two forms of the assessment.

As part of this case study, two sets of scoring equivalency papers were selected and scored. Both sets were selected from a larger pool of responses that had perfect agreement between the initial two scorers in the system. Since mid-level scores were overrepresented in this reduced response pool, the final sample was selected by randomly selecting 1-4 responses from each score level.

Sample A consisted of 20 papers initially written in English that had previously been scored by American scorers. For this case study, these papers were scored by Finnish scorers who had mastery of the English language (Table 4.3). Sample B consisted of 20 papers that were initially written in Finnish and scored by Finnish scorers that were subsequently translated into English and scored by American scorers. In the translation of these Sample B papers from Finnish into American English, all the typos and other errors in the original student response were included as much as possible, given the differences between the two languages. These student responses were scored by American scorers.

For this equivalency case study, scorers from both countries who scored the cross-country responses were blind to the scores from the other pair of scorers. Scorer agreement within countries was examined by calculating correlations between scorers. The scoring equivalence data were analysed by comparing mean scores on common sets of translated responses across countries.

Scoring equivalency case study

This case study investigated whether PT results could be reliably scored in a standardised international testing environment. Analyses were conducted to investigate whether student responses received the same scores regardless of language or country. “Sameness” was examined in two ways: relative and absolute. The first refers to whether the relative standings of the responses were consistent (i.e. highly correlated) regardless of language or country. The second reflects whether the mean scores of the responses were equal.

Relative Quality of Scores: How well does a translated and culturally adapted performance-based assessment requiring students to generate a written response get scored across countries and languages?

To determine whether the scorers agreed with one another about the relative quality of the responses, correlation coefficients among pairs of scorers, both within and across countries, were calculated. The within-country analyses showed that student responses could be reliably scored within each country. The data presented in Table 4.4 and Table 4.5 show the inter-rater correlations for double-scoring U.S. students by U.S. scorers and Finnish students by Finnish scorers, respectively. These data are presented to show the results of the operational testing.

For the scoring equivalency case study, each student response was scored by a total of four scorers: two from the United States and two from Finland. As shown in Table 4.6, the inter-country correlations of the total scores, for two scorers, in each country were high, with r = .95 for Sample A and r = .93 for Sample B. The correlation of .95 indicates that the two Finnish scorers that scored the 20 U.S. PTs had a strong linear relationship. The correlation of .93 indicates that the two U.S. scorers that scored the 20 Finnish PTs that were translated into English also had a strong linear relationship.

Correlations between the two teams for the subscores were not as high as they were for total score, ranging from r = .88–.92 for Sample A and r = .89–.91 for Sample B (Table 4.7).

WM may need to be discussed in more detail because the Finnish scorers were not native English speakers and the Finnish papers that were back-translated into English may not have captured all of the language nuances of Finnish.

Absolute Quality of Scores: Can equivalency be established across the two forms of the assessment?

To determine whether the scorers agreed with one another on the absolute quality of the scores, the mean scores across countries were analysed. Figure 4.1 illustrates the mean total scores for each country for Sample A and Sample B.

Figure 4.1. Mean total scores for each sample

Note: A = English; B = Finnish to English.

There was no significant difference in the mean total scores for both sets of 20 papers in Sample A (t₁₉ = -1.70; p = .11) and Sample B (t₁₉ = -0.14; p = .89). The pattern of mean scores suggests that translating responses did not affect the perceived response quality or that there was no difference in scorer leniency across countries.

The individual subscores were also analysed. For Sample A (Figure 4.2), the difference between the two groups of scorers was not significant for APS and WE, but for WM, there was a significant difference between the two groups (MUSA = 3.55, MFin = 4.03; p = .004). One possible explanation is that the Finnish scorers, although familiar with American English, were scoring student responses in a language that was not their native language.

Figure 4.2. Mean subscores for sample A (English)

Note: APS = Analysis and Problem Solving; WE = Writing Effectiveness; WM = Writing Mechanics.

There were no significant differences in the average subscores across the two teams for Sample B (Figure 4.3), where the Finnish student responses were translated and culturally adapted into English. The observed difference in means for Sample A for WM was not seen in Sample B.

Figure 4.3. Mean subscores for Sample B (Finnish to English)

Note: APS = Analysis and Problem Solving; WE = Writing Effectiveness; WM = Writing Mechanics.

The results suggest that translated responses are scored the same as responses originally composed in the native language of the scorer.