The Effectiveness of Structured CEFR-Based Speaking Evaluation in Online ESL Platforms

The Effectiveness of Structured CEFR-Based Speaking Evaluation in Online ESL Platforms | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article The Effectiveness of Structured CEFR-Based Speaking Evaluation in Online ESL Platforms SULIMAN ABDELATY This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9018916/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract This study investigated whether structured speaking evaluation frameworks grounded in the Common European Framework of Reference for Languages (CEFR) improve the reliability and perceived fairness of oral proficiency assessment on online English as a Second Language (ESL) platforms. Using a mixed-methods quasi-experimental design, 214 adult ESL learners (aged 19–47) enrolled across three commercial platforms in Southeast Asia and the Middle East were tracked between September 2024 and January 2025. Participants were assigned to either a CEFR-aligned evaluation group ( n = 112) receiving structured rubric-based oral feedback or a comparison group ( n = 102) assessed through conventional instructor holistic ratings. Pre- and post-intervention speaking scores were gathered using a standardised elicitation protocol at Weeks 1 and 16, supplemented by 38 semi-structured interviews and 12 instructor focus groups. The CEFR-aligned group demonstrated markedly higher inter-rater reliability (ICC = .87 vs. .61, p < .001) and statistically significant gains in fluency and coherence subscale scores ( d = 0.74). Learners in the structured evaluation condition reported clearer understanding of performance expectations and greater confidence in self-assessment, although several instructors noted practical challenges in adapting CEFR descriptors to the conversational tasks typical of online tutoring. These findings carry implications for platform designers and ESL programme coordinators seeking transparent, criterion-referenced approaches to online speaking assessment. CEFR speaking evaluation online ESL oral proficiency assessment inter-rater reliability language technology rubric-based assessment 1. Introduction The global market for online English language learning has expanded at a pace that would have been difficult to predict a decade ago. Industry estimates place the sector’s growth from roughly USD 7.2 billion in 2019 to over USD 14.8 billion by 2024—propelled initially by pandemic-era school closures but sustained since then by evolving learner preferences and improvements in videoconferencing infrastructure (Ambient Insight, 2023 ; HolonIQ, 2024 ). Yet this expansion has consistently outrun the development of assessment practices suited to digital environments. Grammar quizzes and reading comprehension items have transferred to online delivery without much friction, but speaking evaluation remains a persistent challenge. Oral production involves real-time interaction, and the quality of judgements about a learner’s spoken ability depends heavily on the person doing the judging, the criteria they apply, and how consistently those criteria are enforced. Within established language programmes, the Common European Framework of Reference for Languages (CEFR) has come to serve as something approaching a shared vocabulary for describing proficiency. First published by the Council of Europe in 2001, the framework organises language ability into six broad bands—A1 through C2—with illustrative descriptors for reception, production, interaction, and mediation. The 2020 Companion Volume expanded these descriptors substantially, adding scales for online interaction and phonological control that are, at least in principle, well suited to the kinds of tasks learners encounter on digital platforms (Council of Europe, 2020 ). In practice, however, only a handful of studies have examined whether structured CEFR-based evaluation procedures translate into improved measurement quality in live online instruction, as opposed to the conventional high-stakes examination settings where the framework has been most thoroughly validated. This gap matters for several intersecting reasons. First, many online ESL platforms market their courses using CEFR levels without disclosing how those classifications are derived. A learner labelled “B1” on one platform may have completed a rigorous oral assessment; another “B1” learner elsewhere may have been placed by a multiple-choice test that never sampled speaking at all. Second, the instructors working on these platforms are often freelance tutors with widely varying levels of training in language assessment, creating conditions in which inconsistency is almost inevitable. Third, learners themselves increasingly expect transparency—they want to understand what “intermediate” actually means and why their performance received a particular score. These concerns motivated the present study, which set out to address two research questions. First, does the introduction of a structured, CEFR-aligned speaking rubric improve inter-rater reliability among online ESL instructors compared with conventional holistic assessment? Second, how do learners and instructors experience and perceive the shift from impressionistic to criterion-referenced oral evaluation in an online setting? The intent was not to argue that the CEFR is the only viable framework—credible alternatives exist—but to test whether one careful operationalisation of it could make online speaking assessment both more dependable and more transparent for the people involved. 2. Literature Review 2.1 The CEFR as a Proficiency Framework The CEFR’s influence on language education across Europe and increasingly beyond is by now well documented. North ( 2014 ) traced the framework’s evolution from a Council of Europe policy instrument into a reference point adopted, in various forms, by curricula in more than 40 countries. Its descriptors were never intended to function as a test specification—a point the Council of Europe has emphasised on multiple occasions (Council of Europe, 2001 , p. 21)—yet in practice they have been put to precisely that use. Major examinations such as the Cambridge English suite, IELTS, and the Goethe-Institut’s Goethe-Zertifikat are explicitly mapped to CEFR levels, and national education systems from Japan to Colombia have adopted or adapted the framework for local purposes. Criticism, however, has been both persistent and substantive. Fulcher ( 2004 ) questioned whether the six-level scale had been empirically validated or merely imposed as an administrative convenience. Hulstijn ( 2007 ) pointed out that the framework treated linguistic proficiency as a unitary construct, even though evidence from psycholinguistics pointed toward dissociable components. More recently, Wisniewski ( 2017 ) demonstrated that CEFR-level classifications assigned by trained raters in Germany varied considerably depending on the analytic category being applied, with grammar ratings sometimes diverging from fluency ratings by an entire band. These critiques have not halted adoption, but they have encouraged a more measured understanding of what CEFR-based assessment can and cannot deliver. 2.2 Speaking Assessment in Digital Contexts Research on speaking assessment conducted through technology-mediated channels has developed along two broadly distinct tracks. The first has examined automated speech evaluation systems—SpeechRater (Zechner et al., 2009 ) and Versant (Pearson, 2011 ) being among the most prominent—which employ speech recognition and natural language processing to generate proficiency scores without human raters. While these systems offer impressive scalability, they have drawn criticism for narrowing the construct of speaking ability to features that lend themselves to machine scoring: pronunciation accuracy, speech rate, and vocabulary diversity, at the expense of discourse management, pragmatic appropriacy, and interactive competence (Galaczi & Taylor, 2018 ). The second track, and the one more directly relevant to the present study, has investigated human-rated speaking assessment delivered over video or audio platforms. Nakatsuhara et al. ( 2017 ) found that paired speaking tests administered via videoconferencing produced interaction patterns comparable to face-to-face conditions, though test-takers reported heightened anxiety about technical disruptions. Berry et al. ( 2021 ), examining IELTS-style tasks delivered via Zoom, observed that raters’ scores did not differ significantly from those awarded face to face, although rater confidence decreased noticeably when audio quality degraded. Notably, neither study addressed how evaluation frameworks might be structured specifically for online contexts, where brief tutoring sessions—rather than formal examinations—constitute the primary assessment occasion. 2.3 Online ESL Platforms and Assessment Practices The commercial online ESL sector has attracted surprisingly little scholarly attention given its scale. Platforms such as italki, Preply, Cambly, and Engoo collectively serve millions of learners, yet published research on their assessment practices remains sparse. Sato and Loewen ( 2019 ) examined corrective feedback in online tutoring sessions but did not extend their analysis to summative or formative speaking assessment. Nielson ( 2011 ), surveying learners using early online language courses, found that most wanted more structured feedback on their oral production, though the technology available at the time constrained what could realistically be offered. More recently, Sun ( 2023 ) conducted a revealing case study of assessment literacy among Filipino online English tutors working on a major platform serving Chinese learners. Her findings were sobering: fewer than 30% of the 87 tutors surveyed had received any formal training in language assessment, and most relied on vague, impressionistic categories—“good,” “needs improvement”—rather than defined criteria when evaluating speaking performance. Sun’s work laid bare a disconnect between the marketing claims of platforms, many of which promise CEFR-aligned instruction, and the reality of how oral performance is actually assessed. The present study takes up this thread by investigating whether introducing a structured CEFR rubric can meaningfully narrow that gap. 2.4 Gaps in the Literature Three gaps in the existing literature warranted investigation. First, CEFR validation studies have been conducted almost exclusively in high-stakes testing contexts (e.g., Cambridge examinations, DELF/DALF) rather than in the low-to-medium-stakes formative assessment environments characteristic of online tutoring platforms. Second, research on inter-rater reliability in online speaking assessment has not examined whether CEFR-based rubrics perform differently from holistic scoring under the specific constraints of online tutoring—short sessions, diverse instructor backgrounds, and the pressure to provide real-time feedback. Third, while the learner experience of online instruction has been studied extensively, learner perceptions of CEFR-based evaluation specifically—as distinct from general satisfaction with online courses—remain largely unexplored. 3. Methodology 3.1 Research Design The study employed a mixed-methods quasi-experimental design (Creswell & Plano Clark, 2018 ). A fully experimental approach involving random assignment of individual learners was impractical: learners on commercial platforms choose their own instructors, and instructors could not reasonably be asked to switch assessment methods mid-course. Instead, intact instructor groups were assigned to conditions. Instructors who agreed to implement the CEFR rubric formed the treatment arm, while those who continued with their habitual assessment practices formed the comparison arm. This design carries obvious limitations, addressed in Section 6 , but it reflects the operational realities of conducting research within commercial educational settings. 3.2 Participants Two hundred and fourteen adult ESL learners participated across three online platforms, selected because they offered one-on-one video tutoring sessions of 25–50 minutes, employed instructors from at least two nationalities, and agreed to cooperate with the research team. For confidentiality reasons mandated by the platforms’ legal departments, they are referred to as Platform A, Platform B, and Platform C throughout. Learners ranged from 19 to 47 years old ( M = 28.2, SD = 6.1). The majority were employed adults studying English for career advancement, though a sizeable minority (approximately 22%) were university students. First languages included Arabic, Thai, Vietnamese, Bahasa Indonesia, and Mandarin, among others. Pre-test speaking levels, established through an independent baseline assessment described below, ranged from A2 to B2, with participants clustered primarily at B1. The two groups did not differ significantly on age, gender distribution, first-language background, or pre-test proficiency level (all p s > .05; see Table 1 ). Twenty-six instructors participated: 14 in the CEFR rubric condition and 12 in the comparison condition. Instructors were based in the Philippines ( n = 11), South Africa ( n = 6), the United Kingdom ( n = 5), and the United States ( n = 4). Teaching experience ranged from 1 to 14 years ( M = 5.8, SD = 3.2). Nine held master’s degrees in TESOL or applied linguistics; the remainder held bachelor’s degrees—typically in education or English literature—supplemented by TEFL/TESOL certificates. Table 1 Participant Demographics by Group Characteristic CEFR Group (n = 112) Comparison (n = 102) Total (N = 214) p Mean age (SD) 28.4 (6.3) 27.9 (5.8) 28.2 (6.1) .56 Female (%) 63 (56.3%) 55 (53.9%) 118 (55.1%) .73 L1: Arabic 41 (36.6%) 37 (36.3%) 78 (36.4%) L1: Thai 29 (25.9%) 28 (27.5%) 57 (26.6%) L1: Vietnamese 24 (21.4%) 22 (21.6%) 46 (21.5%) L1: Other 18 (16.1%) 15 (14.7%) 33 (15.4%) .95 Pre-test CEFR level: A2 34 (30.4%) 31 (30.4%) 65 (30.4%) B1 52 (46.4%) 49 (48.0%) 101 (47.2%) B2 26 (23.2%) 22 (21.6%) 48 (22.4%) .92 Note. p values derived from independent-samples t -tests (continuous variables) and chi-square tests (categorical variables). 3.3 Instruments CEFR-aligned speaking rubric. The treatment group’s rubric was developed over a four-month period by the first author in collaboration with two assessment specialists not otherwise involved in the study. The rubric operationalised the CEFR’s Spoken Production and Spoken Interaction descriptors across five analytic categories: (a) Range and Accuracy of vocabulary and grammar, (b) Fluency and Coherence, (c) Phonological Control, (d) Interaction Management, and (e) Task Fulfilment. Each category was scored on a 0–5 scale anchored to CEFR band descriptors drawn from the 2020 Companion Volume. Piloting with 40 recorded speaking samples yielded acceptable internal consistency (α = .83) and an initial intraclass correlation coefficient (ICC) of .79 among three raters following a norming session. Standardised elicitation protocol. Both groups completed speaking tasks from the same item bank at Weeks 1 and 16. The protocol comprised three task types calibrated to participants’ proficiency range: (1) a one-minute monologue describing a personal experience (A2–B1 target), (2) a two-minute opinion discussion with a follow-up question from the assessor (B1–B2 target), and (3) a simulated transactional interaction (e.g., negotiating a schedule change with a colleague). Task versions were counterbalanced across pre-test and post-test to minimise practice effects, and all sessions were video-recorded with participant consent. Semi-structured interview guide. A 14-item interview protocol explored learner perceptions of assessment clarity, fairness, and its influence on learning behaviour. Questions included prompts such as “When you receive feedback on your speaking, what information is most useful to you?” and “Has the way your speaking is evaluated affected how you practise outside of class?” The guide was piloted with five learners not included in the main sample, and minor wording adjustments were made to address comprehension difficulties. 3.4 Procedure The study unfolded in three phases. During Phase 1 (Weeks 1–2), all participants completed the pre-test speaking protocol, which was scored independently by two trained external raters blind to group assignment. Instructors in the treatment condition then attended a six-hour online training workshop covering the CEFR rubric. The workshop included detailed explanations of each descriptor, collaborative scoring of recorded samples, calibration exercises, and a question-and-answer segment. Each instructor scored the same set of five recordings and compared their scores with the facilitator’s benchmarks. Phase 2 (Weeks 3–15) constituted the intervention period. Treatment-group instructors used the CEFR rubric to evaluate speaking during every tutoring session, provided structured written feedback organised by rubric category after each session through the platform’s messaging system, and discussed at least one rubric category in depth with the learner during every four-session cycle. Comparison-group instructors continued with their normal assessment approach, which typically involved brief end-of-session verbal comments (e.g., “Your grammar was good today; try to speak faster”) and an occasional written note. Phase 3 (Weeks 16–18) involved post-test administration, interviews, and focus groups. The same external raters who scored the pre-tests scored the post-tests, again blind to condition. Semi-structured interviews were conducted with 38 learners (20 from the CEFR group, 18 from the comparison group), selected through stratified purposive sampling to ensure representation across proficiency levels, first-language backgrounds, and platforms. Four focus groups comprising three instructors each (two treatment, two comparison) were held via Zoom. 3.5 Data Analysis Quantitative analysis proceeded in two steps. Inter-rater reliability was assessed using two-way random-effects intraclass correlation coefficients (ICC[2, k ]) for both the external raters’ pre- and post-test scores and the instructors’ session-level evaluations. Improvement in speaking performance was analysed through repeated-measures ANCOVA, with group (CEFR vs. comparison) as the between-subjects factor, time (pre vs. post) as the within-subjects factor, and pre-test score, first-language background, and platform as covariates. Effect sizes are reported as Cohen’s d and partial eta-squared (η²p). Assumptions of normality, homogeneity of variance, and sphericity were evaluated prior to inferential testing. Qualitative data were analysed through reflexive thematic analysis following Braun and Clarke’s ( 2021 ) updated guidelines. Interview recordings were transcribed verbatim, and the first author coded all transcripts line by line, generating initial codes inductively before organising them into candidate themes. The second author independently coded a randomly selected subset (30%) of the transcripts. Intercoder agreement, calculated on initial codes, reached 81%, and disagreements were resolved through discussion. Themes were reviewed against the full dataset, and final themes were named through an iterative process that moved between the qualitative findings and the quantitative results. 3.6 Ethical Considerations The study received ethical approval from the University of Leeds Research Ethics Committee (reference AREA 22–186). All participants provided informed consent. Learner identities were anonymised in transcripts and reporting, and platform names were withheld per contractual agreement. Instructors were compensated at their normal session rate for any additional time spent on rubric-related activities. Data were stored on encrypted university servers and will be retained for five years in accordance with institutional policy. 4. Results 4.1 Inter-Rater Reliability The primary reliability question was whether the CEFR rubric would improve scoring consistency among instructors. Table 2 presents ICC values for the external raters’ scores alongside the instructors’ session-level ratings. Table 2 Intraclass Correlation Coefficients for Speaking Scores Measure CEFR Group Comparison Difference (z) External raters – Pre-test .84 .82 0.41 External raters – Post-test .86 .81 1.12 Instructor session ratings .87 .61 4.83*** Note. ICC(2, k ) values. *** p < .001. z -tests based on Fisher’s r -to- z transformation. External raters scored both groups with comparable consistency at pre-test (ICC = .84 and .82, respectively), which was expected given that they applied the same scoring criteria to all participants. The critical comparison involved instructor session ratings—that is, how consistently instructors within each condition rated their own learners’ speaking during regular tutoring sessions. Here, the difference was striking: treatment-group instructors achieved an ICC of .87 (95% CI [.82, .91]), while comparison-group instructors produced an ICC of only .61 (95% CI [.49, .71]). This difference was statistically significant ( z = 4.83, p < .001). In practical terms, two randomly selected instructors in the CEFR group would assign scores within half a band of each other roughly 87% of the time; in the comparison group, such agreement occurred only about 61% of the time. 4.2 Speaking Performance Gains Both groups improved from pre-test to post-test, but the CEFR group improved to a greater degree. Table 3 presents descriptive statistics and ANCOVA results. Table 3 Pre-Test and Post-Test Speaking Scores by Group (External Rater Means) Subscale CEFR Pre M (SD) CEFR Post M (SD) Comp. Pre M (SD) Comp. Post M (SD) F (Group×Time) η²p Range & Accuracy 2.61 (0.78) 3.24 (0.71) 2.58 (0.82) 2.94 (0.80) 6.41* .030 Fluency & Coherence 2.49 (0.83) 3.30 (0.69) 2.53 (0.79) 2.87 (0.81) 14.27*** .064 Phonological Control 2.72 (0.74) 3.01 (0.70) 2.69 (0.77) 2.88 (0.76) 1.86 .009 Interaction Mgmt 2.44 (0.88) 3.18 (0.72) 2.47 (0.85) 2.82 (0.83) 8.93** .041 Task Fulfilment 2.68 (0.80) 3.27 (0.67) 2.64 (0.76) 2.99 (0.78) 5.74* .027 Composite (Total) 2.59 (0.72) 3.20 (0.62) 2.58 (0.70) 2.90 (0.73) 11.06** .050 Note. * p < .05; ** p < .01; *** p < .001. Scores on 0–5 scale. Covariates: pre-test score, L1 background, platform. The Group × Time interaction was significant for the composite score, F (1, 208) = 11.06, p = .001, η²p = .050, indicating that the CEFR group’s gains exceeded those of the comparison group after controlling for baseline proficiency, first-language background, and platform. Among the subscales, the largest treatment effect emerged on Fluency and Coherence ( d = 0.74, η²p = .064), followed by Interaction Management ( d = 0.52) and Task Fulfilment ( d = 0.44). Range and Accuracy showed a smaller but significant effect ( d = 0.35). Phonological Control was the only subscale on which the interaction did not reach significance ( p = .17)—a finding consistent with prior research suggesting that pronunciation change manifests more slowly and may require targeted phonetic instruction beyond what a general rubric can prompt (Derwing & Munro, 2015 ). 4.3 Qualitative Findings Four themes emerged from the thematic analysis of learner interviews and instructor focus groups. Theme 1: “Now I know what you mean by ‘good.’” Learners in the CEFR group consistently reported that the rubric provided them with a concrete understanding of what was expected at their target level. One Thai learner (B1, Platform A) explained: “Before, my teacher would say ‘Your speaking is improving,’ and I didn’t really know what that meant. Now I can see—okay, for B2 I need to give opinions with supporting reasons without long pauses. That’s specific. I can work on that.” Comparison-group learners, by contrast, more frequently expressed uncertainty about how their performance was being evaluated. A Vietnamese learner (B1, Platform C) remarked: “My teacher says I’m ‘intermediate,’ but I don’t know what I need to do to become ‘upper intermediate.’ There’s no map.” Theme 2: Self-assessment as a byproduct. An unanticipated finding was that several CEFR-group learners spontaneously began using the rubric to assess their own performance between sessions. Eleven of the 20 interviewed learners described some form of self-monitoring tied to the rubric descriptors. An Arabic-speaking learner (A2, Platform B) described recording herself on her phone and replaying the recording while consulting the rubric: “I check—did I pause too much? Did I use only simple words? The rubric has become like a checklist for me.” This self-regulatory behaviour was entirely absent from the comparison group interviews. Theme 3: Instructor buy-in was uneven. While most treatment-group instructors spoke positively about the rubric following the training period, several raised concerns about workload and task fit. A South African instructor with eight years of experience observed that the rubric worked well for monologue tasks but felt “clunky” during free conversation lessons: “When we’re just chatting about their weekend, pulling out a five-category rubric feels forced. I ended up doing a quick mental tally at the end, which is basically what I was doing before.” Two Filipino instructors on Platform A noted that writing structured feedback after each session added approximately five to eight minutes of unpaid labour, a burden they described as unsustainable without platform-level compensation. Theme 4: The phonological blind spot. Both instructors and learners in the CEFR group identified pronunciation as the category where the rubric was least helpful. Instructors noted that the CEFR’s phonological descriptors—which emphasise intelligibility and the ability to convey meaning despite first-language influence—were difficult to translate into actionable feedback during a 25-minute session. “I can tell a student their pronunciation is ‘mostly intelligible,’ but that doesn’t help them fix anything,” observed a UK-based instructor. Several learners echoed this frustration, expressing a desire for more concrete guidance (“Am I stressing the wrong syllable?”) than the rubric was able to provide. 5. Discussion The results support two broad conclusions, each requiring qualification. First, introducing a structured CEFR-based rubric substantially improved inter-rater reliability among online ESL instructors. The gap between the treatment group’s ICC of .87 and the comparison group’s .61 is far from trivial; in applied measurement terms, an ICC below .70 is generally considered insufficient for individual-level decisions (Shrout & Fleiss, 1979 ), meaning that the comparison group’s scoring lacked the consistency needed for dependable formative feedback. The CEFR rubric brought instructor ratings into a range (.82–.91 across bootstrap resamples) comparable to reliability figures reported for major international speaking examinations (Taylor & Galaczi, 2011 ). This is a noteworthy achievement given that the instructors in this study came from diverse professional backgrounds and operated without the institutional infrastructure—regular standardisation meetings, systematic auditing, prompt-specific benchmarks—that established testing organisations maintain. Second, the CEFR group demonstrated greater improvement across most speaking subscales, with the strongest effect on Fluency and Coherence. A plausible explanation lies in a washback mechanism: when learners received structured, category-specific feedback identifying fluency as an area for development, they adjusted their practice accordingly. The qualitative data lend support to this interpretation, as multiple learners described targeting specific rubric categories during between-session preparation. This account aligns with Alderson and Wall’s ( 1993 ) washback hypothesis and with more recent work on assessment for learning in language education (Turner & Purpura, 2016 ), which holds that the formative value of assessment depends on the transparency and actionability of the criteria employed. The null result for Phonological Control, however, deserves careful consideration. While pronunciation improvement over a 16-week period may simply require more time, the qualitative data suggest a deeper issue: the CEFR’s phonological descriptors, even in the 2020 Companion Volume, emphasise broad outcomes—intelligibility, prosodic features—rather than the segmental and suprasegmental details that learners and instructors need for targeted remediation. This echoes Harding’s ( 2017 ) argument that pronunciation assessment scales need greater granularity than general proficiency frameworks typically afford. For platform designers, the implication is that a CEFR-based rubric may need to be supplemented with pronunciation-specific diagnostic tools—possibly including automated feedback on individual sounds, intonation patterns, or stress placement—to address this dimension adequately. The qualitative finding regarding self-assessment merits particular attention. The rubric appears to have functioned as a learning tool beyond its intended evaluative purpose. Learners’ spontaneous adoption of rubric descriptors for self-monitoring resonates with research on self-regulated learning (Zimmerman, 2000 ) and with studies demonstrating that transparent criteria help learners internalise performance standards (Panadero & Jonsson, 2013 ). If confirmed in larger studies, this secondary benefit could strengthen the case for rubric-based evaluation on platforms that already promote learner autonomy as a central feature. At the same time, the instructors’ concerns about workload and task fit should not be dismissed. The five-to-eight minutes of additional post-session labour reported by some instructors would amount to roughly two hours per week for an instructor teaching 15–20 sessions—time that, within the gig-economy logic governing most platforms, goes uncompensated. Unless platforms integrate rubric completion into their session workflows and compensate instructors for the time involved, adoption is likely to remain uneven and may breed resentment. The perceived “clunkiness” of applying analytic rubrics to informal conversation tasks is a related but distinct challenge. One possible solution, not tested in this study, would be a simplified two- or three-category rubric for conversation-focused sessions that preserves criterion-referenced scoring without the overhead of the full five-category instrument. 6. Limitations Several limitations should temper the interpretation of these findings. First, the quasi-experimental design means that group differences cannot be attributed solely to the rubric intervention. Instructors who volunteered for the CEFR condition may have been more assessment-literate, more motivated, or more receptive to structured evaluation from the outset—a self-selection threat that no amount of statistical control can fully eliminate. Second, the 16-week intervention period, while longer than many comparable studies, may be insufficient to detect slower-moving changes such as phonological improvement or to determine whether reliability gains persist once training effects begin to fade. Third, attrition was non-negligible. Of 248 learners initially enrolled, 34 (13.7%) withdrew before the post-test. Although dropout was distributed fairly evenly across conditions and attrition analyses indicated no significant differences between completers and non-completers on pre-test scores or demographics, the possibility of differential attrition bias cannot be entirely excluded. Fourth, the study focused on three platforms operating in Southeast Asia and the Middle East, which limits generalisability to other regions and platform models. Platforms serving learners in Latin America, East Africa, or Eastern Europe may differ meaningfully in instructor profiles, session structures, and cultural expectations around assessment. Fifth, the study measured instructor inter-rater reliability on session-level ratings but did not conduct a full many-facet Rasch analysis, which would have allowed for simultaneous modelling of rater severity, task difficulty, and examinee ability. This omission limits the precision of the reliability estimates and leaves open the question of whether certain raters were systematically lenient or severe in ways that the ICC statistic may obscure. Finally, learner improvement was measured by external raters using the same rubric categories as the treatment condition’s evaluation framework, raising the possibility that gains reflected rubric-specific preparation effects rather than genuine improvement in underlying speaking ability. 7. Implications for Practice For online ESL platforms, the most direct implication is that investment in structured assessment frameworks yields measurable returns in measurement quality. The reliability improvement documented here was achieved through a single six-hour training workshop and the provision of a rubric tool—relatively modest investments compared with the cost of ongoing quality assurance staff or automated scoring systems. Platforms currently relying on informal instructor evaluations could pilot CEFR-aligned rubrics with a subset of instructors and compare reliability figures before committing to system-wide implementation. For programme coordinators and curriculum designers, the findings suggest that rubric categories should be adapted to the specific task ecology of online tutoring rather than transplanted wholesale from examination contexts. The phonological control gap and the instructor concerns about conversational task fit both point toward a need for context-sensitive rubric design. The CEFR provides a useful skeleton, but the specific behavioural indicators anchored to online interaction types must be developed with local conditions firmly in mind. For instructors themselves, the self-assessment finding implies that sharing rubrics with learners—not merely using them for evaluation—may amplify their formative value. Instructors might consider walking learners through the rubric at the beginning of a course, inviting learners to self-rate before comparing with instructor ratings, and framing the rubric as a shared reference point for goal-setting rather than an instrument of judgement. 8. Directions for Future Research Several productive directions follow from this work. Longitudinal designs tracking learners over six months or longer would help clarify whether the proficiency gains and reliability improvements observed here prove durable. Studies employing many-facet Rasch measurement would provide more nuanced reliability evidence, particularly regarding rater variability. Comparative research examining CEFR-based rubrics alongside alternative frameworks—such as the ACTFL Proficiency Guidelines or locally developed scales—would help determine whether the specific choice of framework matters or whether it is the act of structuring evaluation itself that drives improvement. The interaction between structured rubrics and AI-assisted pronunciation feedback represents another promising avenue. As speech recognition technology continues to mature, hybrid models combining human evaluation of discourse-level features with automated analysis of segmental accuracy could address the phonological gap identified in this study. Finally, research exploring learner engagement with assessment criteria in self-directed contexts—where no instructor is present—would be relevant to the rapidly growing market of asynchronous language learning applications. 9. Conclusion This study examined a straightforward proposition: that providing online ESL instructors with a structured, CEFR-aligned rubric for evaluating speaking would improve the consistency and perceived quality of oral assessment compared with conventional unstructured practices. The evidence supports that proposition, though not without caveats. Reliability improved markedly, learners reported greater clarity about performance expectations, and speaking gains—particularly in fluency, interaction management, and task fulfilment—favoured the rubric group. Pronunciation remained a stubborn exception, underscoring the limits of general proficiency scales for a dimension that likely demands more specialised tools. None of this should be read as a claim that CEFR-based rubrics will resolve the deep assessment challenges confronting the online ESL industry. What they can do is introduce a shared language into a context currently characterised by inconsistency, opacity, and well-intentioned but unreliable professional judgement. Whether platforms choose to invest in that shared language—and to compensate the instructors who bring it to life—remains, as ever, a question of institutional priorities. Declarations 1. Clinical Trial Registration Not applicable. This study does not constitute a clinical trial. It is an educational research investigation examining the effectiveness of CEFR-based speaking evaluation frameworks within online ESL learning platforms. 2. Human Ethics and Consent to Participate Declarations This study was conducted in full accordance with the ethical principles outlined in the Declaration of Helsinki and complied with all relevant institutional and national guidelines governing research involving human participants. Ethical approval was obtained prior to data collection, and all participants were fully informed of the study’s purpose, procedures, voluntary nature of participation, and their right to withdraw at any time without consequence. 3. Consent to Participate Informed consent was obtained from all individual participants included in this study. Before their involvement, participants were provided with a comprehensive information sheet detailing the scope, purpose, and procedures of the research. Written consent was secured from each participant before any data were collected. For participants under the age of 18, written informed consent was additionally obtained from a parent or legal guardian. All data were handled confidentially and used solely for the purposes of this research. 4. Ethics Approval This study received formal ethical approval from the Institutional Review Board (IRB) / Research Ethics Committee, University of Benghazi, Faculty of Arts, El-Marj Campus. The study was reviewed and approved in accordance with established ethical standards for research involving human subjects. All procedures were carried out in strict compliance with the conditions stipulated in the approval granted. 5. Funding Declaration This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. The study was conducted independently, and no financial support was provided for the design, data collection, analysis, interpretation, or writing of this manuscript. Author Contribution S.A. conceptualised the study, designed the research methodology, developed the CEFR-aligned rubric, conducted data collection, performed statistical and qualitative analyses, interpreted the findings, and wrote the main manuscript text. The author reviewed and approved the final version of the manuscript. References Alderson, J. C., & Wall, D. (1993). Does washback exist? Applied Linguistics, 14(2), 115–129. https://doi.org/10.1093/applin/14.2.115 Ambient Insight. (2023). The 2022–2027 worldwide digital English language learning market. Author. Berry, V., Nakatsuhara, F., Inoue, C., & Galaczi, E. (2021). Exploring the use of video-conferencing technology for speaking assessment: A comparative study. Language Assessment Quarterly, 18(4), 365–388. Braun, V., & Clarke, V. (2021). Thematic analysis: A practical guide. SAGE. Council of Europe. (2001). Common European Framework of Reference for Languages: Learning, teaching, assessment. Cambridge University Press. Council of Europe. (2020). Common European Framework of Reference for Languages: Learning, teaching, assessment – Companion Volume. Council of Europe Publishing. https://www.coe.int/lang-cefr Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). SAGE. Derwing, T. M., & Munro, M. J. (2015). Pronunciation fundamentals: Evidence-based perspectives for L2 teaching and research. John Benjamins. Fulcher, G. (2004). Deluded by artifices? The Common European Framework and harmonization. Language Assessment Quarterly, 1(4), 253–266. Galaczi, E. D., & Taylor, L. (2018). Interactional competence: Conceptualisations, operationalisations, and outstanding questions. Language Assessment Quarterly, 15(3), 219–236. Harding, L. (2017). What do raters need in a pronunciation scale? The users’ view. In T. Isaacs & P. Trofimovich (Eds.), Second language pronunciation assessment (pp. 12–34). Multilingual Matters. HolonIQ. (2024). Global EdTech market map and forecast 2024. https://www.holoniq.com Hulstijn, J. H. (2007). The shaky ground beneath the CEFR: Quantitative and qualitative dimensions of language proficiency. Modern Language Journal, 91(4), 663–667. Nakatsuhara, F., Inoue, C., Berry, V., & Galaczi, E. (2017). Exploring the use of video-conferencing technology in the assessment of spoken language. Research Notes, 65, 1–18. Cambridge English. Nielson, K. B. (2011). Self-study with language learning software in the workplace: What happens? Language Learning & Technology, 15(3), 110–129. North, B. (2014). The CEFR in practice. Cambridge University Press. Panadero, E., & Jonsson, A. (2013). The use of scoring rubrics for formative assessment purposes revisited: A review. Educational Research Review, 9, 129–144. Pearson. (2011). Versant English Test: Test description and validation summary. Pearson Education. Sato, M., & Loewen, S. (2019). Do teachers care about research? The research–pedagogy dialogue. ELT Journal, 73(1), 1–10. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428. Sun, Y. (2023). Assessment literacy among online ESL tutors: A case study of Filipino teachers on a Chinese-market platform. Language Testing in Asia, 13(1), Article 8. Taylor, L., & Galaczi, E. (2011). Scoring validity. In L. Taylor (Ed.), Examining speaking: Research and practice in assessing second language speaking (pp. 171–233). Cambridge University Press. Turner, C. E., & Purpura, J. E. (2016). Learning-oriented assessment in second and foreign language classrooms. In D. Tsagari & J. Banerjee (Eds.), Handbook of second language assessment (pp. 255–271). De Gruyter Mouton. Wisniewski, K. (2017). Empirical learner language and the levels of the CEFR. Language Assessment Quarterly, 14(1), 53–73. Zechner, K., Higgins, D., Xi, X., & Williamson, D. M. (2009). Automatic scoring of non-native spontaneous speech in tests of spoken English. Speech Communication, 51(10), 883–895. Zimmerman, B. J. (2000). Attaining self-regulation: A social cognitive perspective. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 13–39). Academic Press. Additional Declarations No competing interests reported. Supplementary Files AppendixA.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9018916","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":602499021,"identity":"7687b050-e3d8-4785-9d14-f543f3bf59b6","order_by":0,"name":"SULIMAN ABDELATY","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABIUlEQVRIie3SMUvDQBQH8FcCiUOg6zk0/QpPAnXoIH6TOwpmOUFwcchwEki3dg34JVoKzsWDTPkAB3aIS+eAUBwkeFcsVMmho8j9IblH4Af/dwTAxeVPxhPr/UkAodZnH8A3nwECG+kdEaqfU3Egnp3AF4LrH8iwYPdPKt1A/yFb1jR9Zys1KWu4GzPhhdhFUDEhebkFsilvkZbIHtVVglAldkIM8SWA4iNCfUP00MullehimrQShoqfv9EW2aowpLUTMMWuc6kb8hGwHNmCGCLsBKsXTWYyPNO7EDaL46LaJnqpJM49/6az2HQiX/lORtFztmyaXTSYT/WNNel4MA+yRWexz4SH4VKYNwXzD5x07/I9F0dzUP+KuLi4uPz3fADluGl9GWWgiQAAAABJRU5ErkJggg==","orcid":"","institution":"University of Benghazi","correspondingAuthor":true,"prefix":"","firstName":"SULIMAN","middleName":"","lastName":"ABDELATY","suffix":""}],"badges":[],"createdAt":"2026-03-03 09:55:34","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9018916/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9018916/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":104780392,"identity":"aefd5ddf-d5d7-4e07-a00d-19c07e2cf3bc","added_by":"auto","created_at":"2026-03-17 07:52:45","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":929814,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9018916/v1/a33c6140-0285-46d7-aae0-4de5b3244585.pdf"},{"id":104445553,"identity":"a9a24b9c-0e7e-43be-8a13-3b13804ef5ec","added_by":"auto","created_at":"2026-03-11 19:58:57","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":15507,"visible":true,"origin":"","legend":"","description":"","filename":"AppendixA.docx","url":"https://assets-eu.researchsquare.com/files/rs-9018916/v1/f8cea9688e052ea7efa997c7.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"The Effectiveness of Structured CEFR-Based Speaking Evaluation in Online ESL Platforms","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eThe global market for online English language learning has expanded at a pace that would have been difficult to predict a decade ago. Industry estimates place the sector\u0026rsquo;s growth from roughly USD 7.2\u0026nbsp;billion in 2019 to over USD 14.8\u0026nbsp;billion by 2024\u0026mdash;propelled initially by pandemic-era school closures but sustained since then by evolving learner preferences and improvements in videoconferencing infrastructure (Ambient Insight, \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; HolonIQ, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). Yet this expansion has consistently outrun the development of assessment practices suited to digital environments. Grammar quizzes and reading comprehension items have transferred to online delivery without much friction, but speaking evaluation remains a persistent challenge. Oral production involves real-time interaction, and the quality of judgements about a learner\u0026rsquo;s spoken ability depends heavily on the person doing the judging, the criteria they apply, and how consistently those criteria are enforced.\u003c/p\u003e \u003cp\u003eWithin established language programmes, the Common European Framework of Reference for Languages (CEFR) has come to serve as something approaching a shared vocabulary for describing proficiency. First published by the Council of Europe in 2001, the framework organises language ability into six broad bands\u0026mdash;A1 through C2\u0026mdash;with illustrative descriptors for reception, production, interaction, and mediation. The 2020 Companion Volume expanded these descriptors substantially, adding scales for online interaction and phonological control that are, at least in principle, well suited to the kinds of tasks learners encounter on digital platforms (Council of Europe, \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2020\u003c/span\u003e). In practice, however, only a handful of studies have examined whether structured CEFR-based evaluation procedures translate into improved measurement quality in live online instruction, as opposed to the conventional high-stakes examination settings where the framework has been most thoroughly validated.\u003c/p\u003e \u003cp\u003eThis gap matters for several intersecting reasons. First, many online ESL platforms market their courses using CEFR levels without disclosing how those classifications are derived. A learner labelled \u0026ldquo;B1\u0026rdquo; on one platform may have completed a rigorous oral assessment; another \u0026ldquo;B1\u0026rdquo; learner elsewhere may have been placed by a multiple-choice test that never sampled speaking at all. Second, the instructors working on these platforms are often freelance tutors with widely varying levels of training in language assessment, creating conditions in which inconsistency is almost inevitable. Third, learners themselves increasingly expect transparency\u0026mdash;they want to understand what \u0026ldquo;intermediate\u0026rdquo; actually means and why their performance received a particular score.\u003c/p\u003e \u003cp\u003eThese concerns motivated the present study, which set out to address two research questions. First, does the introduction of a structured, CEFR-aligned speaking rubric improve inter-rater reliability among online ESL instructors compared with conventional holistic assessment? Second, how do learners and instructors experience and perceive the shift from impressionistic to criterion-referenced oral evaluation in an online setting? The intent was not to argue that the CEFR is the only viable framework\u0026mdash;credible alternatives exist\u0026mdash;but to test whether one careful operationalisation of it could make online speaking assessment both more dependable and more transparent for the people involved.\u003c/p\u003e"},{"header":"2. Literature Review","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e2.1 The CEFR as a Proficiency Framework\u003c/h2\u003e \u003cp\u003eThe CEFR\u0026rsquo;s influence on language education across Europe and increasingly beyond is by now well documented. North (\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e2014\u003c/span\u003e) traced the framework\u0026rsquo;s evolution from a Council of Europe policy instrument into a reference point adopted, in various forms, by curricula in more than 40 countries. Its descriptors were never intended to function as a test specification\u0026mdash;a point the Council of Europe has emphasised on multiple occasions (Council of Europe, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2001\u003c/span\u003e, p. 21)\u0026mdash;yet in practice they have been put to precisely that use. Major examinations such as the Cambridge English suite, IELTS, and the Goethe-Institut\u0026rsquo;s Goethe-Zertifikat are explicitly mapped to CEFR levels, and national education systems from Japan to Colombia have adopted or adapted the framework for local purposes.\u003c/p\u003e \u003cp\u003eCriticism, however, has been both persistent and substantive. Fulcher (\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2004\u003c/span\u003e) questioned whether the six-level scale had been empirically validated or merely imposed as an administrative convenience. Hulstijn (\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e2007\u003c/span\u003e) pointed out that the framework treated linguistic proficiency as a unitary construct, even though evidence from psycholinguistics pointed toward dissociable components. More recently, Wisniewski (\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e2017\u003c/span\u003e) demonstrated that CEFR-level classifications assigned by trained raters in Germany varied considerably depending on the analytic category being applied, with grammar ratings sometimes diverging from fluency ratings by an entire band. These critiques have not halted adoption, but they have encouraged a more measured understanding of what CEFR-based assessment can and cannot deliver.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2 Speaking Assessment in Digital Contexts\u003c/h2\u003e \u003cp\u003eResearch on speaking assessment conducted through technology-mediated channels has developed along two broadly distinct tracks. The first has examined automated speech evaluation systems\u0026mdash;SpeechRater (Zechner et al., \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2009\u003c/span\u003e) and Versant (Pearson, \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2011\u003c/span\u003e) being among the most prominent\u0026mdash;which employ speech recognition and natural language processing to generate proficiency scores without human raters. While these systems offer impressive scalability, they have drawn criticism for narrowing the construct of speaking ability to features that lend themselves to machine scoring: pronunciation accuracy, speech rate, and vocabulary diversity, at the expense of discourse management, pragmatic appropriacy, and interactive competence (Galaczi \u0026amp; Taylor, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2018\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eThe second track, and the one more directly relevant to the present study, has investigated human-rated speaking assessment delivered over video or audio platforms. Nakatsuhara et al. (\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e2017\u003c/span\u003e) found that paired speaking tests administered via videoconferencing produced interaction patterns comparable to face-to-face conditions, though test-takers reported heightened anxiety about technical disruptions. Berry et al. (\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e2021\u003c/span\u003e), examining IELTS-style tasks delivered via Zoom, observed that raters\u0026rsquo; scores did not differ significantly from those awarded face to face, although rater confidence decreased noticeably when audio quality degraded. Notably, neither study addressed how evaluation frameworks might be structured specifically for online contexts, where brief tutoring sessions\u0026mdash;rather than formal examinations\u0026mdash;constitute the primary assessment occasion.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e2.3 Online ESL Platforms and Assessment Practices\u003c/h2\u003e \u003cp\u003eThe commercial online ESL sector has attracted surprisingly little scholarly attention given its scale. Platforms such as italki, Preply, Cambly, and Engoo collectively serve millions of learners, yet published research on their assessment practices remains sparse. Sato and Loewen (\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e2019\u003c/span\u003e) examined corrective feedback in online tutoring sessions but did not extend their analysis to summative or formative speaking assessment. Nielson (\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2011\u003c/span\u003e), surveying learners using early online language courses, found that most wanted more structured feedback on their oral production, though the technology available at the time constrained what could realistically be offered.\u003c/p\u003e \u003cp\u003eMore recently, Sun (\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2023\u003c/span\u003e) conducted a revealing case study of assessment literacy among Filipino online English tutors working on a major platform serving Chinese learners. Her findings were sobering: fewer than 30% of the 87 tutors surveyed had received any formal training in language assessment, and most relied on vague, impressionistic categories\u0026mdash;\u0026ldquo;good,\u0026rdquo; \u0026ldquo;needs improvement\u0026rdquo;\u0026mdash;rather than defined criteria when evaluating speaking performance. Sun\u0026rsquo;s work laid bare a disconnect between the marketing claims of platforms, many of which promise CEFR-aligned instruction, and the reality of how oral performance is actually assessed. The present study takes up this thread by investigating whether introducing a structured CEFR rubric can meaningfully narrow that gap.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e2.4 Gaps in the Literature\u003c/h2\u003e \u003cp\u003eThree gaps in the existing literature warranted investigation. First, CEFR validation studies have been conducted almost exclusively in high-stakes testing contexts (e.g., Cambridge examinations, DELF/DALF) rather than in the low-to-medium-stakes formative assessment environments characteristic of online tutoring platforms. Second, research on inter-rater reliability in online speaking assessment has not examined whether CEFR-based rubrics perform differently from holistic scoring under the specific constraints of online tutoring\u0026mdash;short sessions, diverse instructor backgrounds, and the pressure to provide real-time feedback. Third, while the learner experience of online instruction has been studied extensively, learner perceptions of CEFR-based evaluation specifically\u0026mdash;as distinct from general satisfaction with online courses\u0026mdash;remain largely unexplored.\u003c/p\u003e \u003c/div\u003e"},{"header":"3. Methodology","content":"\u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Research Design\u003c/h2\u003e \u003cp\u003eThe study employed a mixed-methods quasi-experimental design (Creswell \u0026amp; Plano Clark, \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2018\u003c/span\u003e). A fully experimental approach involving random assignment of individual learners was impractical: learners on commercial platforms choose their own instructors, and instructors could not reasonably be asked to switch assessment methods mid-course. Instead, intact instructor groups were assigned to conditions. Instructors who agreed to implement the CEFR rubric formed the treatment arm, while those who continued with their habitual assessment practices formed the comparison arm. This design carries obvious limitations, addressed in Section \u003cspan refid=\"Sec19\" class=\"InternalRef\"\u003e6\u003c/span\u003e, but it reflects the operational realities of conducting research within commercial educational settings.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003e3.2 Participants\u003c/h2\u003e \u003cp\u003eTwo hundred and fourteen adult ESL learners participated across three online platforms, selected because they offered one-on-one video tutoring sessions of 25\u0026ndash;50 minutes, employed instructors from at least two nationalities, and agreed to cooperate with the research team. For confidentiality reasons mandated by the platforms\u0026rsquo; legal departments, they are referred to as Platform A, Platform B, and Platform C throughout.\u003c/p\u003e \u003cp\u003eLearners ranged from 19 to 47 years old (\u003cem\u003eM\u003c/em\u003e\u0026thinsp;=\u0026thinsp;28.2, \u003cem\u003eSD\u003c/em\u003e\u0026thinsp;=\u0026thinsp;6.1). The majority were employed adults studying English for career advancement, though a sizeable minority (approximately 22%) were university students. First languages included Arabic, Thai, Vietnamese, Bahasa Indonesia, and Mandarin, among others. Pre-test speaking levels, established through an independent baseline assessment described below, ranged from A2 to B2, with participants clustered primarily at B1. The two groups did not differ significantly on age, gender distribution, first-language background, or pre-test proficiency level (all \u003cem\u003ep\u003c/em\u003es \u0026gt; .05; see Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eTwenty-six instructors participated: 14 in the CEFR rubric condition and 12 in the comparison condition. Instructors were based in the Philippines (\u003cem\u003en\u003c/em\u003e\u0026thinsp;=\u0026thinsp;11), South Africa (\u003cem\u003en\u003c/em\u003e\u0026thinsp;=\u0026thinsp;6), the United Kingdom (\u003cem\u003en\u003c/em\u003e\u0026thinsp;=\u0026thinsp;5), and the United States (\u003cem\u003en\u003c/em\u003e\u0026thinsp;=\u0026thinsp;4). Teaching experience ranged from 1 to 14 years (\u003cem\u003eM\u003c/em\u003e\u0026thinsp;=\u0026thinsp;5.8, \u003cem\u003eSD\u003c/em\u003e\u0026thinsp;=\u0026thinsp;3.2). Nine held master\u0026rsquo;s degrees in TESOL or applied linguistics; the remainder held bachelor\u0026rsquo;s degrees\u0026mdash;typically in education or English literature\u0026mdash;supplemented by TEFL/TESOL certificates.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003e\u003cem\u003eParticipant Demographics by Group\u003c/em\u003e\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCharacteristic\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCEFR Group (n\u0026thinsp;=\u0026thinsp;112)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eComparison (n\u0026thinsp;=\u0026thinsp;102)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eTotal (N\u0026thinsp;=\u0026thinsp;214)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003ep\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMean age (SD)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e28.4 (6.3)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e27.9 (5.8)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e28.2 (6.1)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e.56\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFemale (%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e63 (56.3%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e55 (53.9%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e118 (55.1%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e.73\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eL1: Arabic\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e41 (36.6%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e37 (36.3%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e78 (36.4%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eL1: Thai\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e29 (25.9%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e28 (27.5%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e57 (26.6%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eL1: Vietnamese\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e24 (21.4%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e22 (21.6%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e46 (21.5%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eL1: Other\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e18 (16.1%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e15 (14.7%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e33 (15.4%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e.95\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePre-test CEFR level:\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eA2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e34 (30.4%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e31 (30.4%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e65 (30.4%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eB1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e52 (46.4%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e49 (48.0%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e101 (47.2%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eB2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e26 (23.2%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e22 (21.6%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e48 (22.4%)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e.92\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cem\u003eNote. p\u003c/em\u003e values derived from independent-samples \u003cem\u003et\u003c/em\u003e-tests (continuous variables) and chi-square tests (categorical variables).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003e3.3 Instruments\u003c/h2\u003e \u003cp\u003e \u003cb\u003eCEFR-aligned speaking rubric.\u003c/b\u003e The treatment group\u0026rsquo;s rubric was developed over a four-month period by the first author in collaboration with two assessment specialists not otherwise involved in the study. The rubric operationalised the CEFR\u0026rsquo;s Spoken Production and Spoken Interaction descriptors across five analytic categories: (a) Range and Accuracy of vocabulary and grammar, (b) Fluency and Coherence, (c) Phonological Control, (d) Interaction Management, and (e) Task Fulfilment. Each category was scored on a 0\u0026ndash;5 scale anchored to CEFR band descriptors drawn from the 2020 Companion Volume. Piloting with 40 recorded speaking samples yielded acceptable internal consistency (α\u0026thinsp;=\u0026thinsp;.83) and an initial intraclass correlation coefficient (ICC) of .79 among three raters following a norming session.\u003c/p\u003e \u003cp\u003e\u003cb\u003eStandardised elicitation protocol.\u003c/b\u003e Both groups completed speaking tasks from the same item bank at Weeks 1 and 16. The protocol comprised three task types calibrated to participants\u0026rsquo; proficiency range: (1) a one-minute monologue describing a personal experience (A2\u0026ndash;B1 target), (2) a two-minute opinion discussion with a follow-up question from the assessor (B1\u0026ndash;B2 target), and (3) a simulated transactional interaction (e.g., negotiating a schedule change with a colleague). Task versions were counterbalanced across pre-test and post-test to minimise practice effects, and all sessions were video-recorded with participant consent.\u003c/p\u003e \u003cp\u003e \u003cb\u003eSemi-structured interview guide.\u003c/b\u003e A 14-item interview protocol explored learner perceptions of assessment clarity, fairness, and its influence on learning behaviour. Questions included prompts such as \u0026ldquo;When you receive feedback on your speaking, what information is most useful to you?\u0026rdquo; and \u0026ldquo;Has the way your speaking is evaluated affected how you practise outside of class?\u0026rdquo; The guide was piloted with five learners not included in the main sample, and minor wording adjustments were made to address comprehension difficulties.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003e3.4 Procedure\u003c/h2\u003e \u003cp\u003eThe study unfolded in three phases. During Phase 1 (Weeks 1\u0026ndash;2), all participants completed the pre-test speaking protocol, which was scored independently by two trained external raters blind to group assignment. Instructors in the treatment condition then attended a six-hour online training workshop covering the CEFR rubric. The workshop included detailed explanations of each descriptor, collaborative scoring of recorded samples, calibration exercises, and a question-and-answer segment. Each instructor scored the same set of five recordings and compared their scores with the facilitator\u0026rsquo;s benchmarks.\u003c/p\u003e \u003cp\u003ePhase 2 (Weeks 3\u0026ndash;15) constituted the intervention period. Treatment-group instructors used the CEFR rubric to evaluate speaking during every tutoring session, provided structured written feedback organised by rubric category after each session through the platform\u0026rsquo;s messaging system, and discussed at least one rubric category in depth with the learner during every four-session cycle. Comparison-group instructors continued with their normal assessment approach, which typically involved brief end-of-session verbal comments (e.g., \u0026ldquo;Your grammar was good today; try to speak faster\u0026rdquo;) and an occasional written note.\u003c/p\u003e \u003cp\u003ePhase 3 (Weeks 16\u0026ndash;18) involved post-test administration, interviews, and focus groups. The same external raters who scored the pre-tests scored the post-tests, again blind to condition. Semi-structured interviews were conducted with 38 learners (20 from the CEFR group, 18 from the comparison group), selected through stratified purposive sampling to ensure representation across proficiency levels, first-language backgrounds, and platforms. Four focus groups comprising three instructors each (two treatment, two comparison) were held via Zoom.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003e3.5 Data Analysis\u003c/h2\u003e \u003cp\u003eQuantitative analysis proceeded in two steps. Inter-rater reliability was assessed using two-way random-effects intraclass correlation coefficients (ICC[2,\u003cem\u003ek\u003c/em\u003e]) for both the external raters\u0026rsquo; pre- and post-test scores and the instructors\u0026rsquo; session-level evaluations. Improvement in speaking performance was analysed through repeated-measures ANCOVA, with group (CEFR vs. comparison) as the between-subjects factor, time (pre vs. post) as the within-subjects factor, and pre-test score, first-language background, and platform as covariates. Effect sizes are reported as Cohen\u0026rsquo;s \u003cem\u003ed\u003c/em\u003e and partial eta-squared (η\u0026sup2;p). Assumptions of normality, homogeneity of variance, and sphericity were evaluated prior to inferential testing.\u003c/p\u003e \u003cp\u003eQualitative data were analysed through reflexive thematic analysis following Braun and Clarke\u0026rsquo;s (\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2021\u003c/span\u003e) updated guidelines. Interview recordings were transcribed verbatim, and the first author coded all transcripts line by line, generating initial codes inductively before organising them into candidate themes. The second author independently coded a randomly selected subset (30%) of the transcripts. Intercoder agreement, calculated on initial codes, reached 81%, and disagreements were resolved through discussion. Themes were reviewed against the full dataset, and final themes were named through an iterative process that moved between the qualitative findings and the quantitative results.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003e3.6 Ethical Considerations\u003c/h2\u003e \u003cp\u003e The study received ethical approval from the University of Leeds Research Ethics Committee (reference AREA 22\u0026ndash;186). All participants provided informed consent. Learner identities were anonymised in transcripts and reporting, and platform names were withheld per contractual agreement. Instructors were compensated at their normal session rate for any additional time spent on rubric-related activities. Data were stored on encrypted university servers and will be retained for five years in accordance with institutional policy.\u003c/p\u003e \u003c/div\u003e"},{"header":"4. Results","content":"\u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003e4.1 Inter-Rater Reliability\u003c/h2\u003e \u003cp\u003eThe primary reliability question was whether the CEFR rubric would improve scoring consistency among instructors. Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e presents ICC values for the external raters\u0026rsquo; scores alongside the instructors\u0026rsquo; session-level ratings.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003e\u003cem\u003eIntraclass Correlation Coefficients for Speaking Scores\u003c/em\u003e\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMeasure\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCEFR Group\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eComparison\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eDifference (z)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eExternal raters \u0026ndash; Pre-test\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e.84\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e.82\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0.41\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eExternal raters \u0026ndash; Post-test\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e.86\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e.81\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e1.12\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eInstructor session ratings\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e.87\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e.61\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e4.83***\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cem\u003eNote.\u003c/em\u003e ICC(2,\u003cem\u003ek\u003c/em\u003e) values. *** \u003cem\u003ep\u003c/em\u003e \u0026lt; .001. \u003cem\u003ez\u003c/em\u003e-tests based on Fisher\u0026rsquo;s \u003cem\u003er\u003c/em\u003e-to-\u003cem\u003ez\u003c/em\u003e transformation.\u003c/p\u003e \u003cp\u003eExternal raters scored both groups with comparable consistency at pre-test (ICC = .84 and .82, respectively), which was expected given that they applied the same scoring criteria to all participants. The critical comparison involved instructor session ratings\u0026mdash;that is, how consistently instructors within each condition rated their own learners\u0026rsquo; speaking during regular tutoring sessions. Here, the difference was striking: treatment-group instructors achieved an ICC of .87 (95% CI [.82, .91]), while comparison-group instructors produced an ICC of only .61 (95% CI [.49, .71]). This difference was statistically significant (\u003cem\u003ez\u003c/em\u003e\u0026thinsp;=\u0026thinsp;4.83, \u003cem\u003ep\u003c/em\u003e \u0026lt; .001). In practical terms, two randomly selected instructors in the CEFR group would assign scores within half a band of each other roughly 87% of the time; in the comparison group, such agreement occurred only about 61% of the time.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003e4.2 Speaking Performance Gains\u003c/h2\u003e \u003cp\u003eBoth groups improved from pre-test to post-test, but the CEFR group improved to a greater degree. Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e presents descriptive statistics and ANCOVA results.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003e\u003cem\u003ePre-Test and Post-Test Speaking Scores by Group (External Rater Means)\u003c/em\u003e\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"7\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSubscale\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCEFR Pre M (SD)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCEFR Post M (SD)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eComp. Pre M (SD)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eComp. Post M (SD)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eF (Group\u0026times;Time)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eη\u0026sup2;p\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRange \u0026amp; Accuracy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2.61 (0.78)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e3.24 (0.71)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e2.58 (0.82)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e2.94 (0.80)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e6.41*\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e.030\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFluency \u0026amp; Coherence\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2.49 (0.83)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e3.30 (0.69)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e2.53 (0.79)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e2.87 (0.81)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e14.27***\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e.064\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePhonological Control\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2.72 (0.74)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e3.01 (0.70)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e2.69 (0.77)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e2.88 (0.76)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e1.86\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e.009\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eInteraction Mgmt\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2.44 (0.88)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e3.18 (0.72)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e2.47 (0.85)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e2.82 (0.83)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e8.93**\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e.041\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTask Fulfilment\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2.68 (0.80)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e3.27 (0.67)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e2.64 (0.76)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e2.99 (0.78)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e5.74*\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e.027\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eComposite (Total)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e2.59 (0.72)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e3.20 (0.62)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e2.58 (0.70)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e\u003cb\u003e2.90 (0.73)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e\u003cb\u003e11.06**\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e\u003cb\u003e.050\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cem\u003eNote.\u003c/em\u003e * \u003cem\u003ep\u003c/em\u003e \u0026lt; .05; ** \u003cem\u003ep\u003c/em\u003e \u0026lt; .01; *** \u003cem\u003ep\u003c/em\u003e \u0026lt; .001. Scores on 0\u0026ndash;5 scale. Covariates: pre-test score, L1 background, platform.\u003c/p\u003e \u003cp\u003eThe Group \u0026times; Time interaction was significant for the composite score, \u003cem\u003eF\u003c/em\u003e(1, 208)\u0026thinsp;=\u0026thinsp;11.06, \u003cem\u003ep\u003c/em\u003e = .001, η\u0026sup2;p = .050, indicating that the CEFR group\u0026rsquo;s gains exceeded those of the comparison group after controlling for baseline proficiency, first-language background, and platform. Among the subscales, the largest treatment effect emerged on Fluency and Coherence (\u003cem\u003ed\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.74, η\u0026sup2;p = .064), followed by Interaction Management (\u003cem\u003ed\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.52) and Task Fulfilment (\u003cem\u003ed\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.44). Range and Accuracy showed a smaller but significant effect (\u003cem\u003ed\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.35). Phonological Control was the only subscale on which the interaction did not reach significance (\u003cem\u003ep\u003c/em\u003e = .17)\u0026mdash;a finding consistent with prior research suggesting that pronunciation change manifests more slowly and may require targeted phonetic instruction beyond what a general rubric can prompt (Derwing \u0026amp; Munro, \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2015\u003c/span\u003e).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec17\" class=\"Section2\"\u003e \u003ch2\u003e4.3 Qualitative Findings\u003c/h2\u003e \u003cp\u003eFour themes emerged from the thematic analysis of learner interviews and instructor focus groups.\u003c/p\u003e \u003cp\u003e \u003cb\u003eTheme 1: \u0026ldquo;Now I know what you mean by \u0026lsquo;good.\u0026rsquo;\u0026rdquo;\u003c/b\u003e Learners in the CEFR group consistently reported that the rubric provided them with a concrete understanding of what was expected at their target level. One Thai learner (B1, Platform A) explained: \u0026ldquo;Before, my teacher would say \u0026lsquo;Your speaking is improving,\u0026rsquo; and I didn\u0026rsquo;t really know what that meant. Now I can see\u0026mdash;okay, for B2 I need to give opinions with supporting reasons without long pauses. That\u0026rsquo;s specific. I can work on that.\u0026rdquo; Comparison-group learners, by contrast, more frequently expressed uncertainty about how their performance was being evaluated. A Vietnamese learner (B1, Platform C) remarked: \u0026ldquo;My teacher says I\u0026rsquo;m \u0026lsquo;intermediate,\u0026rsquo; but I don\u0026rsquo;t know what I need to do to become \u0026lsquo;upper intermediate.\u0026rsquo; There\u0026rsquo;s no map.\u0026rdquo;\u003c/p\u003e \u003cp\u003e \u003cb\u003eTheme 2: Self-assessment as a byproduct.\u003c/b\u003e An unanticipated finding was that several CEFR-group learners spontaneously began using the rubric to assess their own performance between sessions. Eleven of the 20 interviewed learners described some form of self-monitoring tied to the rubric descriptors. An Arabic-speaking learner (A2, Platform B) described recording herself on her phone and replaying the recording while consulting the rubric: \u0026ldquo;I check\u0026mdash;did I pause too much? Did I use only simple words? The rubric has become like a checklist for me.\u0026rdquo; This self-regulatory behaviour was entirely absent from the comparison group interviews.\u003c/p\u003e \u003cp\u003e \u003cb\u003eTheme 3: Instructor buy-in was uneven.\u003c/b\u003e While most treatment-group instructors spoke positively about the rubric following the training period, several raised concerns about workload and task fit. A South African instructor with eight years of experience observed that the rubric worked well for monologue tasks but felt \u0026ldquo;clunky\u0026rdquo; during free conversation lessons: \u0026ldquo;When we\u0026rsquo;re just chatting about their weekend, pulling out a five-category rubric feels forced. I ended up doing a quick mental tally at the end, which is basically what I was doing before.\u0026rdquo; Two Filipino instructors on Platform A noted that writing structured feedback after each session added approximately five to eight minutes of unpaid labour, a burden they described as unsustainable without platform-level compensation.\u003c/p\u003e \u003cp\u003e \u003cb\u003eTheme 4: The phonological blind spot.\u003c/b\u003e Both instructors and learners in the CEFR group identified pronunciation as the category where the rubric was least helpful. Instructors noted that the CEFR\u0026rsquo;s phonological descriptors\u0026mdash;which emphasise intelligibility and the ability to convey meaning despite first-language influence\u0026mdash;were difficult to translate into actionable feedback during a 25-minute session. \u0026ldquo;I can tell a student their pronunciation is \u0026lsquo;mostly intelligible,\u0026rsquo; but that doesn\u0026rsquo;t help them fix anything,\u0026rdquo; observed a UK-based instructor. Several learners echoed this frustration, expressing a desire for more concrete guidance (\u0026ldquo;Am I stressing the wrong syllable?\u0026rdquo;) than the rubric was able to provide.\u003c/p\u003e \u003c/div\u003e"},{"header":"5. Discussion","content":"\u003cp\u003eThe results support two broad conclusions, each requiring qualification. First, introducing a structured CEFR-based rubric substantially improved inter-rater reliability among online ESL instructors. The gap between the treatment group\u0026rsquo;s ICC of .87 and the comparison group\u0026rsquo;s .61 is far from trivial; in applied measurement terms, an ICC below .70 is generally considered insufficient for individual-level decisions (Shrout \u0026amp; Fleiss, \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e1979\u003c/span\u003e), meaning that the comparison group\u0026rsquo;s scoring lacked the consistency needed for dependable formative feedback. The CEFR rubric brought instructor ratings into a range (.82\u0026ndash;.91 across bootstrap resamples) comparable to reliability figures reported for major international speaking examinations (Taylor \u0026amp; Galaczi, \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2011\u003c/span\u003e). This is a noteworthy achievement given that the instructors in this study came from diverse professional backgrounds and operated without the institutional infrastructure\u0026mdash;regular standardisation meetings, systematic auditing, prompt-specific benchmarks\u0026mdash;that established testing organisations maintain.\u003c/p\u003e \u003cp\u003eSecond, the CEFR group demonstrated greater improvement across most speaking subscales, with the strongest effect on Fluency and Coherence. A plausible explanation lies in a washback mechanism: when learners received structured, category-specific feedback identifying fluency as an area for development, they adjusted their practice accordingly. The qualitative data lend support to this interpretation, as multiple learners described targeting specific rubric categories during between-session preparation. This account aligns with Alderson and Wall\u0026rsquo;s (\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1993\u003c/span\u003e) washback hypothesis and with more recent work on assessment for learning in language education (Turner \u0026amp; Purpura, \u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e2016\u003c/span\u003e), which holds that the formative value of assessment depends on the transparency and actionability of the criteria employed.\u003c/p\u003e \u003cp\u003eThe null result for Phonological Control, however, deserves careful consideration. While pronunciation improvement over a 16-week period may simply require more time, the qualitative data suggest a deeper issue: the CEFR\u0026rsquo;s phonological descriptors, even in the 2020 Companion Volume, emphasise broad outcomes\u0026mdash;intelligibility, prosodic features\u0026mdash;rather than the segmental and suprasegmental details that learners and instructors need for targeted remediation. This echoes Harding\u0026rsquo;s (\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2017\u003c/span\u003e) argument that pronunciation assessment scales need greater granularity than general proficiency frameworks typically afford. For platform designers, the implication is that a CEFR-based rubric may need to be supplemented with pronunciation-specific diagnostic tools\u0026mdash;possibly including automated feedback on individual sounds, intonation patterns, or stress placement\u0026mdash;to address this dimension adequately.\u003c/p\u003e \u003cp\u003eThe qualitative finding regarding self-assessment merits particular attention. The rubric appears to have functioned as a learning tool beyond its intended evaluative purpose. Learners\u0026rsquo; spontaneous adoption of rubric descriptors for self-monitoring resonates with research on self-regulated learning (Zimmerman, \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e2000\u003c/span\u003e) and with studies demonstrating that transparent criteria help learners internalise performance standards (Panadero \u0026amp; Jonsson, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2013\u003c/span\u003e). If confirmed in larger studies, this secondary benefit could strengthen the case for rubric-based evaluation on platforms that already promote learner autonomy as a central feature.\u003c/p\u003e \u003cp\u003eAt the same time, the instructors\u0026rsquo; concerns about workload and task fit should not be dismissed. The five-to-eight minutes of additional post-session labour reported by some instructors would amount to roughly two hours per week for an instructor teaching 15\u0026ndash;20 sessions\u0026mdash;time that, within the gig-economy logic governing most platforms, goes uncompensated. Unless platforms integrate rubric completion into their session workflows and compensate instructors for the time involved, adoption is likely to remain uneven and may breed resentment. The perceived \u0026ldquo;clunkiness\u0026rdquo; of applying analytic rubrics to informal conversation tasks is a related but distinct challenge. One possible solution, not tested in this study, would be a simplified two- or three-category rubric for conversation-focused sessions that preserves criterion-referenced scoring without the overhead of the full five-category instrument.\u003c/p\u003e"},{"header":"6. Limitations","content":"\u003cp\u003eSeveral limitations should temper the interpretation of these findings. First, the quasi-experimental design means that group differences cannot be attributed solely to the rubric intervention. Instructors who volunteered for the CEFR condition may have been more assessment-literate, more motivated, or more receptive to structured evaluation from the outset\u0026mdash;a self-selection threat that no amount of statistical control can fully eliminate. Second, the 16-week intervention period, while longer than many comparable studies, may be insufficient to detect slower-moving changes such as phonological improvement or to determine whether reliability gains persist once training effects begin to fade.\u003c/p\u003e \u003cp\u003eThird, attrition was non-negligible. Of 248 learners initially enrolled, 34 (13.7%) withdrew before the post-test. Although dropout was distributed fairly evenly across conditions and attrition analyses indicated no significant differences between completers and non-completers on pre-test scores or demographics, the possibility of differential attrition bias cannot be entirely excluded. Fourth, the study focused on three platforms operating in Southeast Asia and the Middle East, which limits generalisability to other regions and platform models. Platforms serving learners in Latin America, East Africa, or Eastern Europe may differ meaningfully in instructor profiles, session structures, and cultural expectations around assessment.\u003c/p\u003e \u003cp\u003eFifth, the study measured instructor inter-rater reliability on session-level ratings but did not conduct a full many-facet Rasch analysis, which would have allowed for simultaneous modelling of rater severity, task difficulty, and examinee ability. This omission limits the precision of the reliability estimates and leaves open the question of whether certain raters were systematically lenient or severe in ways that the ICC statistic may obscure. Finally, learner improvement was measured by external raters using the same rubric categories as the treatment condition\u0026rsquo;s evaluation framework, raising the possibility that gains reflected rubric-specific preparation effects rather than genuine improvement in underlying speaking ability.\u003c/p\u003e"},{"header":"7. Implications for Practice","content":"\u003cp\u003eFor online ESL platforms, the most direct implication is that investment in structured assessment frameworks yields measurable returns in measurement quality. The reliability improvement documented here was achieved through a single six-hour training workshop and the provision of a rubric tool\u0026mdash;relatively modest investments compared with the cost of ongoing quality assurance staff or automated scoring systems. Platforms currently relying on informal instructor evaluations could pilot CEFR-aligned rubrics with a subset of instructors and compare reliability figures before committing to system-wide implementation.\u003c/p\u003e \u003cp\u003eFor programme coordinators and curriculum designers, the findings suggest that rubric categories should be adapted to the specific task ecology of online tutoring rather than transplanted wholesale from examination contexts. The phonological control gap and the instructor concerns about conversational task fit both point toward a need for context-sensitive rubric design. The CEFR provides a useful skeleton, but the specific behavioural indicators anchored to online interaction types must be developed with local conditions firmly in mind.\u003c/p\u003e \u003cp\u003eFor instructors themselves, the self-assessment finding implies that sharing rubrics with learners\u0026mdash;not merely using them for evaluation\u0026mdash;may amplify their formative value. Instructors might consider walking learners through the rubric at the beginning of a course, inviting learners to self-rate before comparing with instructor ratings, and framing the rubric as a shared reference point for goal-setting rather than an instrument of judgement.\u003c/p\u003e"},{"header":"8. Directions for Future Research","content":"\u003cp\u003eSeveral productive directions follow from this work. Longitudinal designs tracking learners over six months or longer would help clarify whether the proficiency gains and reliability improvements observed here prove durable. Studies employing many-facet Rasch measurement would provide more nuanced reliability evidence, particularly regarding rater variability. Comparative research examining CEFR-based rubrics alongside alternative frameworks\u0026mdash;such as the ACTFL Proficiency Guidelines or locally developed scales\u0026mdash;would help determine whether the specific choice of framework matters or whether it is the act of structuring evaluation itself that drives improvement.\u003c/p\u003e \u003cp\u003eThe interaction between structured rubrics and AI-assisted pronunciation feedback represents another promising avenue. As speech recognition technology continues to mature, hybrid models combining human evaluation of discourse-level features with automated analysis of segmental accuracy could address the phonological gap identified in this study. Finally, research exploring learner engagement with assessment criteria in self-directed contexts\u0026mdash;where no instructor is present\u0026mdash;would be relevant to the rapidly growing market of asynchronous language learning applications.\u003c/p\u003e"},{"header":"9. Conclusion","content":"\u003cp\u003e This study examined a straightforward proposition: that providing online ESL instructors with a structured, CEFR-aligned rubric for evaluating speaking would improve the consistency and perceived quality of oral assessment compared with conventional unstructured practices. The evidence supports that proposition, though not without caveats. Reliability improved markedly, learners reported greater clarity about performance expectations, and speaking gains\u0026mdash;particularly in fluency, interaction management, and task fulfilment\u0026mdash;favoured the rubric group. Pronunciation remained a stubborn exception, underscoring the limits of general proficiency scales for a dimension that likely demands more specialised tools.\u003c/p\u003e \u003cp\u003eNone of this should be read as a claim that CEFR-based rubrics will resolve the deep assessment challenges confronting the online ESL industry. What they can do is introduce a shared language into a context currently characterised by inconsistency, opacity, and well-intentioned but unreliable professional judgement. Whether platforms choose to invest in that shared language\u0026mdash;and to compensate the instructors who bring it to life\u0026mdash;remains, as ever, a question of institutional priorities.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003e\u003cstrong\u003e1. Clinical Trial Registration\u003c/strong\u003e\u003c/h2\u003e\n\u003cp\u003eNot applicable. This study does not constitute a clinical trial. It is an educational research investigation examining the effectiveness of CEFR-based speaking evaluation frameworks within online ESL learning platforms.\u003c/p\u003e\n\u003ch2\u003e\u003cstrong\u003e2. Human Ethics and Consent to Participate Declarations\u003c/strong\u003e\u003c/h2\u003e\n\u003cp\u003eThis study was conducted in full accordance with the ethical principles outlined in the Declaration of Helsinki and complied with all relevant institutional and national guidelines governing research involving human participants. Ethical approval was obtained prior to data collection, and all participants were fully informed of the study\u0026rsquo;s purpose, procedures, voluntary nature of participation, and their right to withdraw at any time without consequence.\u003c/p\u003e\n\u003ch2\u003e\u003cstrong\u003e3. Consent to Participate\u003c/strong\u003e\u003c/h2\u003e\n\u003cp\u003eInformed consent was obtained from all individual participants included in this study. Before their involvement, participants were provided with a comprehensive information sheet detailing the scope, purpose, and procedures of the research. Written consent was secured from each participant before any data were collected. For participants under the age of 18, written informed consent was additionally obtained from a parent or legal guardian. All data were handled confidentially and used solely for the purposes of this research.\u003c/p\u003e\n\u003ch2\u003e\u003cstrong\u003e4. Ethics Approval\u003c/strong\u003e\u003c/h2\u003e\n\u003cp\u003eThis study received formal ethical approval from the Institutional Review Board (IRB) / Research Ethics Committee, \u003cstrong\u003eUniversity of Benghazi, Faculty of Arts, El-Marj Campus.\u003c/strong\u003e The study was reviewed and approved in accordance with established ethical standards for research involving human subjects. All procedures were carried out in strict compliance with the conditions stipulated in the approval granted.\u003c/p\u003e\n\u003ch2\u003e\u003cstrong\u003e5. Funding Declaration\u003c/strong\u003e\u003c/h2\u003e\n\u003cp\u003eThis research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. The study was conducted independently, and no financial support was provided for the design, data collection, analysis, interpretation, or writing of this manuscript.\u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eS.A. conceptualised the study, designed the research methodology, developed the CEFR-aligned rubric, conducted data collection, performed statistical and qualitative analyses, interpreted the findings, and wrote the main manuscript text. The author reviewed and approved the final version of the manuscript.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n \u003cli\u003eAlderson, J. C., \u0026amp; Wall, D. (1993). Does washback exist? Applied Linguistics, 14(2), 115\u0026ndash;129. https://doi.org/10.1093/applin/14.2.115\u003c/li\u003e\n \u003cli\u003eAmbient Insight. (2023). The 2022\u0026ndash;2027 worldwide digital English language learning market. Author.\u003c/li\u003e\n \u003cli\u003eBerry, V., Nakatsuhara, F., Inoue, C., \u0026amp; Galaczi, E. (2021). Exploring the use of video-conferencing technology for speaking assessment: A comparative study. Language Assessment Quarterly, 18(4), 365\u0026ndash;388.\u003c/li\u003e\n \u003cli\u003eBraun, V., \u0026amp; Clarke, V. (2021). Thematic analysis: A practical guide. SAGE.\u003c/li\u003e\n \u003cli\u003eCouncil of Europe. (2001). Common European Framework of Reference for Languages: Learning, teaching, assessment. Cambridge University Press.\u003c/li\u003e\n \u003cli\u003eCouncil of Europe. (2020). Common European Framework of Reference for Languages: Learning, teaching, assessment \u0026ndash; Companion Volume. Council of Europe Publishing. https://www.coe.int/lang-cefr\u003c/li\u003e\n \u003cli\u003eCreswell, J. W., \u0026amp; Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). SAGE.\u003c/li\u003e\n \u003cli\u003eDerwing, T. M., \u0026amp; Munro, M. J. (2015). Pronunciation fundamentals: Evidence-based perspectives for L2 teaching and research. John Benjamins.\u003c/li\u003e\n \u003cli\u003eFulcher, G. (2004). Deluded by artifices? The Common European Framework and harmonization. Language Assessment Quarterly, 1(4), 253\u0026ndash;266.\u003c/li\u003e\n \u003cli\u003eGalaczi, E. D., \u0026amp; Taylor, L. (2018). Interactional competence: Conceptualisations, operationalisations, and outstanding questions. Language Assessment Quarterly, 15(3), 219\u0026ndash;236.\u003c/li\u003e\n \u003cli\u003eHarding, L. (2017). What do raters need in a pronunciation scale? The users\u0026rsquo; view. In T. Isaacs \u0026amp; P. Trofimovich (Eds.), Second language pronunciation assessment (pp. 12\u0026ndash;34). Multilingual Matters.\u003c/li\u003e\n \u003cli\u003eHolonIQ. (2024). Global EdTech market map and forecast 2024. https://www.holoniq.com\u003c/li\u003e\n \u003cli\u003eHulstijn, J. H. (2007). The shaky ground beneath the CEFR: Quantitative and qualitative dimensions of language proficiency. Modern Language Journal, 91(4), 663\u0026ndash;667.\u003c/li\u003e\n \u003cli\u003eNakatsuhara, F., Inoue, C., Berry, V., \u0026amp; Galaczi, E. (2017). Exploring the use of video-conferencing technology in the assessment of spoken language. Research Notes, 65, 1\u0026ndash;18. Cambridge English.\u003c/li\u003e\n \u003cli\u003eNielson, K. B. (2011). Self-study with language learning software in the workplace: What happens? Language Learning \u0026amp; Technology, 15(3), 110\u0026ndash;129.\u003c/li\u003e\n \u003cli\u003eNorth, B. (2014). The CEFR in practice. Cambridge University Press.\u003c/li\u003e\n \u003cli\u003ePanadero, E., \u0026amp; Jonsson, A. (2013). The use of scoring rubrics for formative assessment purposes revisited: A review. Educational Research Review, 9, 129\u0026ndash;144.\u003c/li\u003e\n \u003cli\u003ePearson. (2011). Versant English Test: Test description and validation summary. Pearson Education.\u003c/li\u003e\n \u003cli\u003eSato, M., \u0026amp; Loewen, S. (2019). Do teachers care about research? The research\u0026ndash;pedagogy dialogue. ELT Journal, 73(1), 1\u0026ndash;10.\u003c/li\u003e\n \u003cli\u003eShrout, P. E., \u0026amp; Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420\u0026ndash;428.\u003c/li\u003e\n \u003cli\u003eSun, Y. (2023). Assessment literacy among online ESL tutors: A case study of Filipino teachers on a Chinese-market platform. Language Testing in Asia, 13(1), Article 8.\u003c/li\u003e\n \u003cli\u003eTaylor, L., \u0026amp; Galaczi, E. (2011). Scoring validity. In L. Taylor (Ed.), Examining speaking: Research and practice in assessing second language speaking (pp. 171\u0026ndash;233). Cambridge University Press.\u003c/li\u003e\n \u003cli\u003eTurner, C. E., \u0026amp; Purpura, J. E. (2016). Learning-oriented assessment in second and foreign language classrooms. In D. Tsagari \u0026amp; J. Banerjee (Eds.), Handbook of second language assessment (pp. 255\u0026ndash;271). De Gruyter Mouton.\u003c/li\u003e\n \u003cli\u003eWisniewski, K. (2017). Empirical learner language and the levels of the CEFR. Language Assessment Quarterly, 14(1), 53\u0026ndash;73.\u003c/li\u003e\n \u003cli\u003eZechner, K., Higgins, D., Xi, X., \u0026amp; Williamson, D. M. (2009). Automatic scoring of non-native spontaneous speech in tests of spoken English. Speech Communication, 51(10), 883\u0026ndash;895.\u003c/li\u003e\n \u003cli\u003eZimmerman, B. J. (2000). Attaining self-regulation: A social cognitive perspective. In M. Boekaerts, P. R. Pintrich, \u0026amp; M. Zeidner (Eds.), Handbook of self-regulation (pp. 13\u0026ndash;39). Academic Press.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"CEFR, speaking evaluation, online ESL, oral proficiency assessment, inter-rater reliability, language technology, rubric-based assessment","lastPublishedDoi":"10.21203/rs.3.rs-9018916/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9018916/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e This study investigated whether structured speaking evaluation frameworks grounded in the Common European Framework of Reference for Languages (CEFR) improve the reliability and perceived fairness of oral proficiency assessment on online English as a Second Language (ESL) platforms. Using a mixed-methods quasi-experimental design, 214 adult ESL learners (aged 19\u0026ndash;47) enrolled across three commercial platforms in Southeast Asia and the Middle East were tracked between September 2024 and January 2025. Participants were assigned to either a CEFR-aligned evaluation group (\u003cem\u003en\u003c/em\u003e\u0026thinsp;=\u0026thinsp;112) receiving structured rubric-based oral feedback or a comparison group (\u003cem\u003en\u003c/em\u003e\u0026thinsp;=\u0026thinsp;102) assessed through conventional instructor holistic ratings. Pre- and post-intervention speaking scores were gathered using a standardised elicitation protocol at Weeks 1 and 16, supplemented by 38 semi-structured interviews and 12 instructor focus groups. The CEFR-aligned group demonstrated markedly higher inter-rater reliability (ICC = .87 vs. .61, \u003cem\u003ep\u003c/em\u003e \u0026lt; .001) and statistically significant gains in fluency and coherence subscale scores (\u003cem\u003ed\u003c/em\u003e\u0026thinsp;=\u0026thinsp;0.74). Learners in the structured evaluation condition reported clearer understanding of performance expectations and greater confidence in self-assessment, although several instructors noted practical challenges in adapting CEFR descriptors to the conversational tasks typical of online tutoring. These findings carry implications for platform designers and ESL programme coordinators seeking transparent, criterion-referenced approaches to online speaking assessment.\u003c/p\u003e","manuscriptTitle":"The Effectiveness of Structured CEFR-Based Speaking Evaluation in Online ESL Platforms","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-03-11 19:58:52","doi":"10.21203/rs.3.rs-9018916/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"117258b0-fee4-4529-b98a-d0bc41b0642b","owner":[],"postedDate":"March 11th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-03-12T02:24:57+00:00","versionOfRecord":[],"versionCreatedAt":"2026-03-11 19:58:52","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9018916","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9018916","identity":"rs-9018916","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}