Standardized Assessment of LLM English Proficiency

Standardized Assessment of LLM English Proficiency | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Standardized Assessment of LLM English Proficiency Shangchao Min, Shaonan Wang, Xinyu Gao, Hui Wang, Zhiling Jin, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-8820245/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Large language models (LLMs) are increasingly used in language learning and assessment, yet their English proficiency is seldom reported against interpretable proficiency standards. We introduce the China’s Standards of English Language Ability (CSE) framework to assess the proficiency levels and subskills of LLMs. The test is referred to as the CSEBench and comprises 624 expert-annotated multiple-choice items across CSE Levels 2–7. Each item is accompanied by metadata, including difficulty level and subskill labels covering vocabulary, syntax, phonology, and cohesion/discourse. Critically, the dataset includes test responses from 2,050 middle school and and sophomore college students who are learning English as a second language. We evaluate closed-source models, open-source baselines, and enhanced open-source variants incorporating additional supervision and external knowledge. Results show a clear proficiency divide: after mapping model scores to CSE levels, closed‑source models consistently reach CSE Level 6, whereas most open‑source baselines cluster around CSE Levels 3–4. A follow‑up cognitive diagnostic analysis reveals that while closed‑source LLMs exhibit broad competence across subskills, open‑source models display persistent deficits—most pronounced in phonology. Crucially, these weaknesses are shown to be substantially reducible through targeted enhancements. CSEBench thus offers a proficiency-interpretable testbed for reporting LLM English ability and diagnosing subskill gaps. Physical sciences/Mathematics and computing Biological sciences/Psychology Social science/Psychology proficiency assessment CSE item response theory cognitive diagnosis benchmark evaluation large language models 1 Introduction Large language models (LLMs) are now routinely integrated into English learning products, such as tutoring chatbots, writing feedback systems, and test preparation tools (Lin & Crosthwaite, 2024 ; Wan & Moorhouse, 2025 ). As their role expands toward high-stakes language assessment, a critical evaluation gap remains: there is no standardized, interpretable way to measure and report their own English proficiency. Existing studies typically characterize LLM “language ability” using aggregate accuracy on heterogeneous benchmarks or broad comparative claims (e.g., “near‑human level”). While informative for tracking model progress, such reports offer limited guidance for educational decision‑making. Educators and policymakers instead require proficiency‑oriented answers: What level of English can a model reliably handle, and which specific linguistic subskills remain problematic? A central obstacle to answering these questions is that most existing Natural language processing (NLP) benchmarks are not psychometrically anchored to an external proficiency scale. Benchmark construction has long been central to evaluation in NLP and, more recently, in LLM research. Early efforts focused on single‑task datasets targeting core competencies such as reading comprehension and inference, including SQuAD for extractive question answering (Rajpurkar et al., 2016), SNLI for natural language inference (Bowman et al., 2015), and FEVER for fact verification (Thorne et al., 2018). With the rise of general‑purpose pretrained models (Devlin et al., 2019), evaluation shifted toward multi‑task and general capability suites, most notably GLUE and SuperGLUE (Wang et al., 2018, 2019 ), alongside probing and task‑collection frameworks such as SentEval and DecaNLP (Conneau & Kiela, 2018; McCann et al., 2018 ), and broader language modeling tests emphasizing discourse‑level context (Paperno et al., 2016). Parallel work introduced benchmarks targeting social harms and bias, such as BOLD and ToxiGen (Dhamala et al., 2021 ; Hartvigsen et al., 2022). As LLMs reached frontier scale, evaluation increasingly emphasized higher‑difficulty and human‑centric benchmarks. MMLU assesses broad knowledge and reasoning across academic disciplines (Hendrycks et al., 2021 ), while BIG‑bench provides a large and heterogeneous suite designed to surface capability breadth and systematic failure modes, with BBH focusing on especially challenging tasks (Srivastava et al., 2023 ; Suzgun et al., 2023). A complementary trend uses standardized examinations as evaluation material, motivated by their alignment with human curricula and expert judgment. AGIEval compiles real exam questions from educational and professional testing contexts (Zhong et al., 2024), while MMMU and M3Exam extend exam‑based evaluation to multimodal, multilingual, and multi‑level settings (Yue et al., 2024 ; Zhang et al., 2023 ). Collectively, these efforts move evaluation beyond narrowly constructed tasks toward assessments with clearer human reference points. However, benchmarking LLMs on exam-style questions does not, by itself, enable meaningful proficiency-level reporting. Most benchmarks still rely on aggregate metrics such as overall accuracy or task-level averages. These scores are difficult to interpret as a specific proficiency level (e.g., “CSE Level 5”) because they conflate differences in item difficulty, content coverage, and test composition. As a result, two models with similar accuracy may in fact demonstrate markedly different underlying abilities. This limitation motivates the integration of established educational measurement practices to move beyond raw scores and toward interpretable, scale-referenced proficiency estimates. Even assessments derived from authentic exams often combine items of varying difficulty without a principled scaling mechanism, leaving model performance ambiguous and resistant to level‑based interpretation (e.g., “advanced” or “Level 5”). In contrast, China’s Standards of English Language Ability (CSE) provides a nationally recognized, descriptor‑based proficiency framework that organizes English ability into nine levels, from beginner (Level 1) to expert (Level 9) (National Education Examinations Authority [NEEA], 2018). Each level is defined by detailed, context‑specific “can‑do” descriptors. For instance, CSE Level 7 involves comprehension of complex academic texts and participation in nuanced discussions, whereas Level 5—commonly expected of university graduates—corresponds to effective communication in general academic and professional contexts. Importantly, the CSE framework also supports fine‑grained analysis of linguistic subskills, such as grammatical control or discourse cohesion, at each level. Mapping LLM performance onto the CSE therefore enables proficiency to be reported in educationally meaningful terms rather than as abstract leaderboard scores. Such mapping supports direct comparison across models on a shared scale, facilitates tracking of developmental progress across model versions, and clarifies the suitability of models for specific educational uses. Moreover, once the proficiency levels of LLMs are well established, models could potentially be used as simulated test takers for items targeting different CSE levels. This opens the possibility of reducing reliance on large human samples for initial item calibration, yielding substantial savings in time, cost, and human resources. Educational measurement provides well‑established methodologies for achieving these goals. Item Response Theory (IRT) models the probabilistic relationship between latent ability and item responses, estimating item difficulty and discrimination so that both items and examinees can be located on a common scale (Hambleton et al., 1991 ). When assessments involve multiple forms, linking and equating procedures—often implemented via common‑item anchor designs—ensure score comparability across forms and administrations (Kolen & Brennan, 2014 ). To translate continuous ability estimates into categorical proficiency levels, standard‑setting methods such as Bookmark or Angoff rely on expert judgment to define cut scores representing minimally competent performance at each level (Cizek & Bunch, 2007 ). Compared with heuristic mappings used in some LLM evaluations, these techniques provide a far stronger validity argument for claims such as “model performance corresponds to Level X.” Beyond overall proficiency levels, diagnostic assessment seeks to infer mastery of underlying subskills. Cognitive diagnosis models (CDMs) use a Q‑matrix to specify which attributes each item measures and infer attribute mastery probabilities from observed response patterns (Rupp et al., 2010 ). The GDINA framework, in particular, allows flexible interactions among attributes and can incorporate higher‑order structures to model correlations among subskills via an overarching ability factor. In current LLM evaluation, phenomenon tagging and error taxonomies offer useful descriptive insights; however, CDM‑based approaches go further by producing model‑based mastery estimates tied directly to expert‑defined skill specifications. This supports more interpretable and actionable claims about where a model succeeds or fails linguistically. Against this background, we introduce CSEBench, a benchmark designed to support standardized, interpretable proficiency reporting for LLMs under the CSE framework. CSEBench comprises 624 expert‑annotated multiple‑choice questions spanning CSE Levels 2–7. Each item is annotated with (i) a CSE difficulty level, (ii) one of four target subskills: vocabulary, syntax, phonology, or cohesion/discourse, (iii) metadata including student level, detailed subskills, average accuracy. The items are drawn from two large‑scale diagnostic forms (junior‑high and college versions) connected by 13 anchor items, enabling a psychometrically principled ordering of item difficulty. Our evaluation pipeline follows established language‑testing practice rather than ad‑hoc heuristics. First, we estimate item difficulty using a two‑parameter logistic (2PL) item response theory (IRT) model fitted to responses from 2,050 Chinese EFL learners (1,102 eighth‑grade students and 948 college sophomores). Second, we conduct a Bookmark standard‑setting study with trained experts to establish cut scores that define minimally qualified performance for each CSE level. This yields a defensible mapping from an LLM’s item responses to an interpretable CSE level. To move beyond an overall label, we then perform cognitive diagnosis analysis using an expert‑defined Q‑matrix and higher‑order cognitive diagnosis models (selected by model fit), which produce subskill‑level mastery probabilities that pinpoint where a model is likely to succeed or fail. Using this framework, we evaluate a diverse set of systems: closed‑source models (GPT‑4o (Hurst et al., 2024 ), Claude‑4 (Anthropic et al., 2025), DeepSeek‑R1 (Guo et al., 2025 ), Gemini‑2.5‑Pro (Comanici et al., 2025 ), open‑source baselines (Mistral‑7B‑Instruct (Jiang et al., 2023 ), LLaMA‑3‑8B‑Instruct (Grattafiori et al., 2024 ), Qwen2.5‑7B‑Instruct (Qwen et al., 2025), RWKV‑v6‑7B (Peng et al., 2024 ), and several enhanced Qwen variants (Gao et al., 2026) that incorporate additional supervision or external knowledge. Empirically, closed‑source models consistently attain CSE Level 6 on our forms, whereas most open‑source baselines cluster around CSE Levels 3–4. Qwen2.5‑7B‑Instruct emerges as the strongest open‑source baseline, reaching CSE Level 5. Diagnostic analysis reveals that the largest and most persistent gap for open‑source models lies in phonology‑related items, and we demonstrate that targeted enhancements can substantially improve mastery on this subskill. 2 CSEBench 2.1 Participants Standard-setting panel . Two domain experts participated in the standard setting process. Both were core members of the test development team who had been directly involved in item refinement, test construction, subskill tagging, and difficulty marking. Moreover, as specialists in language testing, they were familiar with CSE proficiency level descriptors for general language ability and corresponding subskills. Subskill coders. Five content experts coded the language subskills of each diagnostic test item. We selected coders who: (a) participated in the development of these two diagnostic test forms, and (b) were familiar with CDA approaches. All coders were faculty members or Ph.D. candidates in Applied Linguistics programs with extensive knowledge in language testing. Test takers. A total of 1,102 Grade 8 students from public middle schools and 948 college sophomores from universities in central and eastern China participated in the tests. All participants were Chinese EFL learners with Mandarin as their native language. The Grade 8 cohort was generally situated within CSE levels 2–4, while the sophomore cohort typically fell within CSE levels 4–6. 2.2 Instrument We assembled two large-scale diagnostic forms by selecting high-quality items from an existing item bank. This approach was intended to (i) obtain sufficiently large response samples for stable parameter estimation, (ii) limit student testing time, and (iii) cover a broad range of CSE levels. The junior-high form contains 39 multiple-choice items and the college form contains 35 items, with 13 anchor items shared across forms to support linking. Both forms target four attributes: phonology, cohesion, vocabulary, and syntax. 2.3 Data samples Each item record in CSEBench includes the item text and multiple annotations/empirical statistics. Answer is the keyed option. Student level indicates the intended examinee group/ability band for the item (as used in the original diagnostic test design). Subskill is the primary linguistic attribute targeted by the item (phonology, cohesion, vocabulary, or syntax). Sublabel provides a finer-grained label under the subskill (e.g., Fixed Collocations under vocabulary). CSE Difficulty Level is the targeted/annotated CSE level for the item. Average Accuracy is the proportion of examinees who answered the item correctly in our sample. Discrimination is the item discrimination index estimated from response data (higher values indicate better separation between higher- and lower-ability test takers; negative values may indicate a potentially problematic item). I’ve got an “A” in the examination. That’s a good _____. You’ll surely win a second. A. result B. start C. news D. idea Answer Student level Subskill Sublabel CSE Difficulty Level Average Accuracy Discrimination B High vocabulary Noun 2 0.239 0.308 Hi, Mary, ________ beautiful day! Yes, it is. A. How B. What a C. What D. How a Answer Student level Subskill Sublabel CSE Difficulty Level Average Accuracy Discrimination B Low syntax Exclamatory sentence 3 0.508 0.556 We can’t trust him. He always________ some excuses for doing something wrong. A. makes up B. sets up C. takes up D. puts up Answer Student level Subskill Sublabel CSE Difficulty Level Average Accuracy Discrimination A Low vocabulary Fixed Collocations 4 0.467 0.517 Which three of the six words below have the stress on the first syllable? ① picture, ② maintain, ③ complete, ④ contact, ⑤ language, ⑥ defeat. A. ① ④ ⑤ B. ① ② ⑥ C. ② ③ ⑤ D. ③ ④ ⑥ Answer Student level Subskill Sublabel CSE Difficulty Level Average Accuracy Discrimination A High phonology Stress 5 0.416 0.309 ____ disobeys the class rules will be punished. A. No matter who B. Whoever C. Who D. All Answer Student level Subskill Sublabel CSE Difficulty Level Average Accuracy Discrimination B High syntax Subject clause 6 0.419 0.500 If it _____ for his invitation the other day, I should not be here now. A. had not been B. should not be C. were not to be D. should not have been Answer Student level Subskill Sublabel CSE Difficulty Level Average Accuracy Discrimination A High syntax Subjunctive Mood 7 0.750 0.462 2.4 Item quality analysis To evaluate the psychometric quality of CSEBench, we calculated item-level indices (item difficulty and discrimination index) as well as test-level reliability. For the whole item bank (CSEBench), the mean item difficulty was 0.486 and the mean discrimination index was 0.324. According to Ebel and Frisbie ( 1991 ), item difficulty values around 0.50 are generally considered desirable, and discrimination indices above 0.30 are typically interpreted as indicating adequate item functioning. These results indicate that the item bank demonstrates appropriate difficulty level and satisfactory discriminative power. For the two large-scale diagnostic tests, the college-level form showed a mean difficulty of 0.480 and a mean discrimination index of 0.309, while the junior high form showed a mean difficulty of 0.394 and a mean discrimination index of 0.377. Test reliability, as measured by Cronbach’s α , was 0.74 for the college section and 0.82 for the junior high section. Both values exceed the commonly accepted threshold of 0.70 for educational assessments (Nunnally & Bernstein, 1994 ), indicating satisfactory reliability for both forms. 3. Models 3.1 Model suite Closed-source models. We evaluated commercially hosted models accessed via API, including GPT-4o, Claude 4, DeepSeek-R1, and Gemini 2.5 Pro, which are commonly reported to deliver strong general performance across a wide range of tasks. Open-source models. We evaluated publicly available instruction-tuned models, including Mistral-7B-Instruct, LLaMA-3-8B-Instruct, Qwen2.5-7B-Instruct, and RWKV-v6-7B. These models differ in parameterization, training data, and instruction-tuning procedures, providing a representative set of open models for comparison. Enhanced Qwen2.5 variants. In addition to the base Qwen2.5 instruction model, we evaluated several enhanced variants: Qwen2.5-Instruct CrossAttn : Qwen2.5-Instruct augmented with a cross-attention mechanism using recursive structure (Gao et al., 2026). Qwen2.5-Instruct CrossAttn (Syntax) : continued fine-tuning of the CrossAttn variant on CoNLL-2000 to strengthen syntactic knowledge. Qwen2.5-Instruct CrossAttn (RAG) : CrossAttn variant augmented with retrieval over an external knowledge graph. Qwen2.5-VL CrossAttn : a CrossAttn variant augmented with a visual-language (VL) component to incorporate visual knowledge. 3.2 Implementation details Enhanced variant construction. Our CrossAttn variants follow the checkpoint-compatible gated tree cross-attention (GTCA) design (Gao et al., 2026), which attaches a gated cross-attention side branch to the decoder and allows token representations to attend to a cached constituency “chunk memory” while leaving the backbone architecture unchanged. Constituency parse trees are computed offline and aligned to token spans (Berkeley Neural Parser; Kitaev & Klein, 2018) and then cached to ensure deterministic structural inputs and to avoid parsing overhead at training and inference time. For MCQA-style prompts, constituency parsing is applied within each field rather than over the entire prompt: the question stem and each candidate option are parsed independently, after which the resulting span-aligned trees are concatenated back in the original prompt order. Concretely, consider a prompt formatted as “Question: … Options: A. coat B. road C. broad D. goal”. Parsing this whole string as a single sentence can cause the parser to treat the option marker (e.g., “A.”) and the option text as part of one global syntactic structure, yielding cross-field constituents that connect the stem with option fragments. Field-wise parsing avoids such artificial attachments and produces cleaner trees whose constituents respect the intended boundaries between the stem and options. Qwen2.5-Instruct CrossAttn. Starting from Qwen2.5-7B-Instruct, a checkpoint-compatible GTCA branch is attached to the language decoder so that token representations can attend to hierarchical chunk memory derived from constituency parses. The constituency parses are computed offline and cached, which yields deterministic structure signals and avoids parsing overhead during training. Continued training is then conducted on the MMLU auxiliary split (99,842 examples) using a unified multiple-choice QA (MCQA) format (Hendrycks et al., 2021), and the optimization follows the three-stage schedule in Gao et al. (2026): LoRA adapters are first trained to adapt the model to the MCQA interface with the structural pathway disabled; next, the backbone and LoRA adapters are frozen while only the GTCA branch are trained; finally, LoRA adapters and GTCA branch are jointly refined with the backbone kept frozen. Throughout this process, the learning signal is defined by the MCQA objective, i.e., maximizing the likelihood of the correct answer under the unified multiple-choice format. Qwen2.5-Instruct CrossAttn (Syntax). Building on Qwen2.5-Instruct CrossAttn , the Syntax variant introduces an additional three-class token-level head on top of the final hidden states to provide explicit supervision over shallow syntactic structure. Training proceeds via continued finetuning on the CoNLL-2000 chunking dataset (Tjong Kim Sang & Buchholz, 2000), the training split contains 8,937 sentences and the test split contains 2,013 sentences. The original chunk tags are mapped to BIO labels {B, I, O}, after which the token-level cross-entropy loss is optimized to predict chunk boundaries, while the head itself remains negligible in size relative to the backbone and the GTCA branch. As a result, the training target is explicitly defined at the token level (BIO classification), while the underlying MCQA interface of the base CrossAttn model remains unchanged. Qwen2.5-Instruct CrossAttn (RAG). The RAG setting reuses the trained weights of Qwen2.5-Instruct CrossAttn and introduces no additional training, instead, only the inference pipeline is modified. Given an item question , DeepSeek-R1 generates a knowledge snippet intended to summarize rules or background facts that can support solving the item, and is concatenated with to form a knowledge-augmented input. In practice, the generated snippets are particularly effective for knowledge intensive item types (e.g., phonology, vocabulary, and syntax), and typically provide compact, option grounded cues. See Appendix for the knowledge-generation prompt and representative cases of the knowledge generated by DeepSeek-R1. This design is best viewed as a proxy for evidence-augmented inference: it bypasses retrieval and indexing rather than implementing a full retriever, since no external knowledge graph is constructed and no retrieval corpus is searched in this variant. The prediction target remains identical to the base setting, namely producing the correct option under the MCQA interface, but with the additional generated context available at inference time. Qwen2.5-VL CrossAttn. The multimodal variant is derived from Qwen2.5-VL-7B-Instruct by attaching the same GTCA branch to the language decoder, thereby extending the CrossAttn mechanism to a visual-language backbone. Continued training uses UAIT, a synthetic dataset of 400 images depicting uncommon action scenes (Ling & Ding, 2026), where each image is paired with a two-option VQA-style multiple-choice item: one option corresponds to the common-text generated before role replacement, and the other corresponds to the uncommon-text obtained after swapping agent and patient roles. The model is finetuned as a visual multiple-choice task by maximizing the likelihood of the correct option letter, which encourages visual grounding for semantic role assignments in counter-common-sense scenes. For evaluation on CSEBench, the resulting model is subsequently queried using the standard text-only inputs to keep the benchmark interface consistent across model families. Evaluation settings. Open-source models are evaluated with their official chat templates and default tokenizers, while closed-source models are queried through public APIs. Decoding is deterministic (temperature = 0, top-p = 1) with a maximum of 8 generated tokens. We run each item three times under each prompting condition and take majority vote when the outputs differ (ties broken by the first run). Options are not shuffled. The predicted option is extracted as the first valid letter in {A, B, C, D}, if no valid letter is found, the item is marked incorrect. 3.3 Prompting conditions We evaluate all models on CSEBench under two prompting conditions: Zero-shot : the model selects the correct option and outputs only the option letter. Few-shot : the model is given two demonstration items followed by the target item, and outputs only the option letter. Zero-shot prompt Instruction: Choose the correct option based on the question. Output only the corresponding letter of the option without providing a reason. Input: {input} Response: Few-shot prompt Instruction: Choose the correct option based on the question. Output only the corresponding letter of the option without providing a reason. Input: They said the eighteenth and last lesson ___ quite easy. A. is B. was C. are D. were Response: B Input: Marx left his homeland for some ___ reasons. A. politically B. politics C. political D. politician Response: C Input: {input} Response: 4. Methods: Mapping LLMs to Proficiency Levels 4.1 Procedure RQ1: What CSE proficiency levels do different models achieve? We addressed RQ1 via a standard-setting study to derive cut scores corresponding to CSE Levels 3–6 on the two diagnostic test forms. We adopted the bookmark method, where panelists place a bookmark in an ordered item booklet to indicate the point at which a just qualified candidate (JQC) at a given CSE level is expected to answer items below the bookmark correctly (i.e., items the JQC should master), but not items above it. To support consistent judgments, items were ordered from easiest to most difficult using IRT-estimated difficulty parameters. This ordering allowed panelists to interpret each item relative to the overall difficulty continuum before placing level-specific bookmarks. The standard-setting session consisted of (i) panelist training on the bookmark procedure and calibration on CSE performance-level descriptors, followed by (ii) two rounds of judgments. In Round 1, panelists placed bookmarks independently without discussion. In Round 2, they revisited their placements after discussing Round 1 rationales and reviewing summary classification information (e.g., the projected percentage of test takers classified into each level under Round 1 cuts). We use the Round 2 placements as the final cut scores for CSE Levels 3–6. RQ2: How do LLMs perform across subskills (phonology, cohesion, vocabulary, and syntax)? We addressed RQ2 using cognitive diagnostic assessment (CDA) to estimate LLM mastery of fine-grained language attributes from item-level responses and the Q-matrix (item–attribute mapping). The diagnostic framework included four attributes defined during test development: A1 phonology, A2 cohesion, A3 vocabulary, and A4 syntax. Content experts assigned each item to one or more relevant attributes: they first coded independently and then resolved disagreements through discussion to produce the final Q-matrix used in CDA. 4.2 Statistical analysis IRT for item calibration and standard setting (RQ1) To obtain item parameters for the ordered item booklet, we fit a unidimensional two-parameter logistic (2PL) IRT model using the mirt package in R (Chalmers, 2012). The resulting difficulty estimates were used to rank items for the bookmark procedure and to ensure that cut-score decisions were made with respect to a calibrated difficulty continuum. CDA for subskill mastery estimation (RQ2) To estimate attribute mastery patterns, we conducted CDA using the GDINA package in R (v2.7.8; Ma et al., 2020). We fit five cognitive diagnostic models: the saturated G-DINA model; two compensatory models (DINO, ACDM); and two non-compensatory models (DINA, RRUM). Each model was specified with a higher-order attribute structure parameterized as 2PL (i.e., a higher-order CDM). Model selection was performed using relative fit indices. After selecting the best-fitting model, we computed attribute mastery probabilities for each LLM and compared performance across the four language attributes to identify subskill-specific strengths and weaknesses. 5. Results 5.1 CSE level alignment The standard-setting results (see Table 1) indicate that all closed-source models in our study reach at least CSE-6 under both zero-shot and few-shot prompting. Because the underlying item bank was designed for Grade 8 and second-year university EFL learners, it contains few CSE-7 items , which limits our ability to make reliable claims about proficiency above CSE-6. In contrast, most open-source baselines cluster at CSE-3 to CSE-4 , with Qwen2.5-7B-Instruct as a notable exception, reaching CSE-5 in both prompting conditions. Among the enhanced Qwen2.5 variants , performance is heterogeneous: several variants remain at CSE-5, while two models show clear gains and achieve CSE-6 , approaching the closed-source systems. Table 1. Alignment of LLMs to CSE Levels. Models Zero-shot Few-shot Open-source model RWKV-v6-7b CSE-3 CSE-3 Meta-Llama-3-8B-Instruct CSE-3 CSE-4 Qwen2.5-7B-Instruct CSE-5 CSE-5 Mistral-7B-Instruct-v0.3 CSE-3 CSE-3 Closed-source baseline GPT-4o CSE-6 CSE-6 Claude4 CSE-6 CSE-6 Deepseek-R1 CSE-6 CSE-6 Gemini2.5-Pro CSE-6 CSE-6 Enhanced model Qwen2.5-Instruct-Cross-Attn CSE-5 CSE-5 Qwen2.5-Instruct-Cross-Attn-syntax CSE-5 CSE-5 Qwen2.5-Instruct-Cross-Attn-rag CSE-6 CSE-6 Qwen2.5-vl-Cross-Attn CSE-6 CSE-6 5.2 Model selection and subskill mastery Table 2 reports the relative fit statistics for five higher-order cognitive diagnostic models (HO-CDMs). For −2LL , AIC , and BIC , smaller values indicate better relative fit. The HO-GDINA model achieved the best overall fit (lowest AIC and −2LL) and was selected for subsequent mastery analysis. Table 2. Relative model fit of CDMs. Model AIC BIC -2LL HO-DINA 94568.34 95301.18 94308.34 HO-DINO 94428.75 95161.59 94168.75 HO-ACDM 94207.82 95064.68 93903.82 HO-RRUM 94205.19 95062.05 93901.19 HO-GDINA 94187.38 95269.73 93803.38 Using HO-GDINA, we estimated each model’s general ability and attribute mastery probabilities for the four subskills. As shown in Table 3, closed-source models consistently exhibit near-ceiling mastery across cohesion, vocabulary, and syntax, and also perform strongly on phonology, with only minor variation. Gemini 2.5 Pro yields the highest (near-saturated) mastery estimates among the closed-source systems. Among open-source baselines, Qwen2.5-7B-Instruct is the strongest overall: it exceeds the mastery threshold (probability > 0.5) for cohesion, vocabulary, and syntax, but remains markedly weak on phonology (0.0352 in both prompting settings). The remaining open-source baselines do not reach mastery on any attribute. The enhanced models show that targeted augmentation can substantially improve subskill performance. In particular, Qwen2.5-Instruct-CrossAttn (RAG) achieves near-ceiling mastery across all attributes, including phonology (0.9936), and is competitive with the closed-source models. Qwen2.5-VL CrossAttn also performs strongly, but its phonology mastery drops from 0.9679 (zero-shot) to 0.9132 (few-shot), suggesting some sensitivity to the few-shot prompt format. One plausible reason for the improvement observed with the RAG model is that this setting adds short evidence that states key facts directly. This reduces the need for the model to recall phonological and lexical facts from memory. The CrossAttn model also uses cached constituency chunks, which can help it link the stem to each option. The Qwen2.5-VL CrossAttn model improve because its training encourages careful option comparison. This objective can strengthen fine grained discrimination and role reasoning. These gains can transfer to better text only performance on CSEBench. The lower phonology mastery in few-shot prompts may come from interference from the demonstrations. Longer contexts can pull attention toward the exemplar patterns, which may weaken attention to short spelling to sound cues. Table 3. General ability and mastery probabilities of LLMs across four language subskills. General ability A1_Cohesion A2_Phonology A3_Vocabulary A4_Syntax zero -shot few -shot zero -shot few -shot zero-shot few-shot zero-shot few-shot zero-shot few-shot RWKV-v6-7b -0.6271 -0.7494 0.1689 0 0.0131 0.0232 0 0 0 0 Meta-Llama-3-8B-Instruct -0.7647 0.1074 0 0.6875 0.0053 0.4523 0 0.1616 0 0 Qwen2.5-7B-Instruct 1.1456 1.1456 1 1 0.0352 0.0352 1 1 1 1 Mistral-7B-Instruct-v0.3 -0.7362 -0.6840 0 0 0.0387 0.0997 0 0 0 0 GPT-4o 1.6630 1.6642 1 1 0.9966 0.9987 1 1 1 1 Claude4 1.6642 1.6644 1 1 0.9987 0.9993 1 1 1 1 Deepseek-R1 1.6642 1.6644 1 1 0.9987 0.9993 1 1 1 1 Gemini2.5-Pro 1.6644 1.6644 1 1 0.9993 0.9993 1 1 1 1 Qwen2.5-Instruct-Cross-Attn 1.2119 1.3639 1 1 0.1583 0.4409 1 1 1 1 Qwen2.5-Instruct-Cross-Attn-syntax 1.2119 1.3639 1 1 0.1583 0.4409 1 1 1 1 Qwen2.5-Instruct-Cross-Attn-rag 1.6614 1.6614 1 1 0.9936 0.9936 1 1 1 1 Qwen2.5-vl-Cross-Attn 1.6476 1.6181 1 1 0.9679 0.9132 1 1 1 1 6. Discussion Across widely used LLM evaluations, a consistent pattern has emerged: models often appear highly capable in aggregate yet exhibit substantial unevenness across task types, difficulty levels, and evaluation protocols. On broad benchmark suites such as MMLU and BIG‑bench, strong overall performance can mask sharp drops on harder reasoning subsets or compositional tasks (Hendrycks et al., 2021 ; Srivastava et al., 2023 ; Suzgun et al., 2023). Exam‑based evaluations similarly show that frontier models can perform competitively on portions of standardized tests while remaining brittle on multi‑step reasoning or specialized knowledge (Zhong et al., 2024). Reliability‑focused benchmarks further reveal systematic weaknesses that are obscured by surface‑level fluency: TruthfulQA demonstrates that models may produce confident but consistently false responses aligned with common misconceptions (Lin et al., 2022), while HumanEval shows that gains from scale and instruction tuning coexist with persistent functional failures on non‑trivial programming tasks (Chen et al., 2021 ). Preference‑based and dialogue evaluations likewise indicate that perceived conversational quality can diverge substantially from task accuracy and that rankings are sensitive to judging protocols (Zheng et al., 2023 ; Chiang et al., 2024 ). Together with broader critiques of contamination, dataset artifacts, and narrow metrics (Liang et al., 2022 ), these findings underscore the limitations of accuracy‑centric benchmarking. These limitations become particularly consequential in educational applications, where LLM performance must be interpreted in terms of language ability rather than aggregate task success. However, most existing NLP benchmarks are not psychometrically anchored to an external proficiency scale. As a result, LLM “language ability” is typically inferred from overall accuracy on heterogeneous benchmarks or from broad comparative claims, which are poorly aligned with the proficiency-oriented judgments required in educational contexts—namely, what level of English a model can reliably handle and which linguistic subskills remain fragile. Rather than treating uneven performance as a collection of task‑specific anomalies, we mapped LLM responses onto a calibrated CSE scale reveals systematic differences in underlying language proficiency. Under this lens, closed‑source models consistently operate above the proficiency range targeted by the learner populations used to calibrate CSEBench, whereas most open‑source baselines cluster in substantially lower CSE bands. This distinction is not easily recoverable from raw accuracy alone, but becomes salient once item difficulty, linking, and standard setting are taken into account. In this sense, CSE‑aligned reporting reframes benchmark results from how many items the model answered correctly to what level of English ability this performance credibly represents. This proficiency‑oriented perspective connects our work to a growing line of research that evaluates LLMs using human language proficiency frameworks, most prominently the Common European Framework of Reference for Languages (CEFR). Recent studies examine whether models can apply CEFR descriptors, generate text at targeted CEFR levels, or control linguistic difficulty for language learners (Benedetto et al., 2024; Benedetto et al., 2025 ; Malik et al., 2024). Notably, Benedetto et al. conceptualize CEFR not merely as a post‑hoc labeling scheme but as a behavioral target for evaluation, while Malik et al. show that combining fine‑tuning and reinforcement learning can substantially improve CEFR‑aligned generation, in some cases allowing smaller models to outperform GPT‑4 at lower cost. Related systems also assess learner writing against CEFR variants using hybrid feature‑based and model‑based approaches (Uchida & Negishi, 2025 ). While complementary, most CEFR‑oriented LLM studies rely on relatively coarse validation signals, such as global level judgments, readability metrics, or indirect complexity proxies. They typically do not provide item‑level linking arguments or psychometric calibration comparable to those used in educational measurement. As a result, proficiency claims are often difficult to compare across test forms, datasets, or populations. By contrast, CSEBench integrates calibrated item difficulties, anchor‑based linking, and expert standard setting, enabling defensible and comparable proficiency‑level interpretations. Our findings therefore suggest that proficiency frameworks such as CEFR or CSE can be substantially strengthened when paired with established measurement models, rather than applied solely as descriptive labels or generation targets. Beyond overall proficiency levels, our diagnostic analyses reveal systematic subskill‑level differences across models. Phonology emerges as the most discriminative and challenging subskill: even the strongest open‑source baseline shows near‑zero mastery on phonology‑related items despite high mastery in vocabulary, syntax, and cohesion/discourse. This pattern suggests that standard instruction tuning and large‑scale text‑only corpora are insufficient for developing sensitivity to stress patterns and phonological contrasts, which are only weakly encoded in orthographic text. Models augmented with external knowledge or multimodal signals substantially reduce this gap, indicating a promising direction for improvement. The robustness of proficiency estimates also warrants attention. Few‑shot prompting does not uniformly improve performance and, in some cases, slightly degrades subskill mastery, most notably for phonology. This effect is particularly visible in enhanced models such as Qwen2.5‑VL CrossAttn, suggesting sensitivity to exemplar choice, ordering, and prompt structure. Despite these robustness concerns, mapping model performance to CSE levels yields interpretations that are directly actionable for educational contexts. In our setting, closed‑source systems consistently reach CSE Level 6, exceeding the proficiency range of the student populations used for calibration, whereas most open‑source baselines fall within Levels 3–4, with Qwen2.5‑7B‑Instruct reaching Level 5. This stratification suggests differentiated roles for LLMs in educational applications: higher‑proficiency systems may function as effective scaffolding agents or adaptive interlocutors for upper‑intermediate learners, while lower‑proficiency models may be more appropriate for beginner‑level practice or constrained instructional settings. Importantly, these conclusions rely on proficiency‑level interpretation rather than absolute accuracy, highlighting the added value of scale‑referenced evaluation. Taken together, these findings support the central claim of this work: evaluating LLMs as language users benefits from the same psychometric rigor applied to human language assessment. Proficiency‑grounded benchmarks such as CSEBench make it possible to express model capabilities in terms that are interpretable, comparable, and educationally meaningful. Beyond evaluation, such mappings open the door to new applications, including the principled use of LLMs as simulated test takers, diagnostic tools, or adaptive instructional agents—provided their proficiency limits are clearly understood. More broadly, our results suggest that future LLM evaluation efforts may benefit from closer integration with educational measurement theory, especially when models are deployed in learning‑ and assessment‑critical contexts. 7. Limitation and Future Work Despite the strengths of CSEBench and the psychometrically grounded evaluation pipeline, several limitations should be acknowledged, which also point to important directions for future work. Coverage of proficiency range and ceiling effects. The current version of CSEBench spans CSE Levels 2–7 and is calibrated using junior-high and college EFL learners. While this range captures a broad spectrum of practical proficiency, it introduces ceiling effects for frontier-scale closed-source models, which consistently exceed the upper end of the calibrated ability range. As a result, distinctions among high-performing models at advanced levels cannot be fully resolved. Future work will expand item coverage to CSE Levels 8–9, incorporating more demanding academic and professional language tasks. Extending the scale upward will enable finer-grained differentiation among advanced models and support more precise claims about near-expert or expert-level proficiency. Prompting effects and robustness of proficiency estimates. As observed in our results, few-shot prompting can introduce instability in subskill mastery estimates, particularly for phonology. This sensitivity raises an important limitation: proficiency estimates may vary with prompt design, exemplar selection, or formatting, even when the underlying model remains unchanged. Future work should conduct controlled robustness studies that vary exemplar number, type, and order, as well as randomize prompts, to quantify variance attributable to prompting. Developing standardized prompting protocols—or prompt-invariant evaluation methods—will be crucial for reliable proficiency reporting. Scope of linguistic skills assessed. CSEBench currently targets four subskills: vocabulary, syntax, phonology, and cohesion/discourse, operationalized through multiple-choice items. While this design supports psychometric calibration and diagnostic modeling, it does not capture productive skills such as speaking and writing, nor does it assess interactional competence or pragmatic appropriateness in open-ended settings. Future extensions could incorporate constructed-response items, spoken or multimodal tasks, and automated scoring models that are themselves calibrated within the same proficiency framework. This would move evaluation closer to the full construct coverage envisioned in the CSE. Validation in operational educational settings. Finally, although CSEBench is grounded in established testing practice, it has not yet been validated in real-world educational deployments. Future work will evaluate the benchmark’s utility in operational contexts, including adaptive tutoring systems, placement decisions (e.g., assigning learners to appropriate courses or proficiency bands), and the generation of diagnostic feedback. Such studies are essential for establishing consequential validity—demonstrating not only that proficiency estimates are interpretable, but also that their use leads to improved educational outcomes. Taken together, these limitations highlight both the current boundaries of CSEBench and the broader research agenda it enables. Addressing them will further strengthen the case for proficiency-grounded, psychometrically principled evaluation of LLMs in language learning and assessment contexts. 8. Conclusion We presented CSEBench , a CSE-aligned benchmark for standardized assessment of LLM English proficiency across levels and subskills. Using IRT and expert standard setting, we mapped LLMs to CSE levels and showed that closed-source models reach CSE-6, while most open-source baselines cluster around CSE-3–4; Qwen2.5-7B-Instruct reaches CSE-5, and enhanced Qwen variants approach closed-source performance. A CDM analysis highlights subskill gaps—especially phonology—and demonstrates that targeted enhancements can substantially improve mastery. Declarations Ethical approval: The experimental procedures were approved by the Ethics Committee of the College of Biomedical Engineering and Instrument Science, Zhejiang University (Approval No. 2024–12). Consent to participate: Due to the minimal-risk nature of this behavioral study, a waiver of documented consent (opt-out procedure) was granted by the Ethics Committee. Prior to data collection, an information letter describing the study's purpose, procedures, and participants' rights was distributed to the parents or legal guardians of all potential participants. The letter explained that participation was entirely voluntary, that all data would be kept strictly confidential, and that no identifying information would appear in any reports or publications arising from the research. Parents or guardians who did not wish their child to participate were asked to complete and return an opt-out form. In addition, verbal assent was obtained from each child participant before the study commenced, and children were informed that they could withdraw from the study at any time without consequence. Failure to return the opt-out form, combined with the child's verbal assent, was taken as consent to participate. Ethics statement Competing interests: The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. Data availability: The dataset is publicly available at https://osf.io/6x9sv/overview?view_only=58ace849d6e1402fbc4fd5188747d17c Ethical approval: The experimental procedures were approved by the Ethics Committee of the College of Biomedical Engineering and Instrument Science, Zhejiang University (Approval No. 2024–12). Consent to participate: Due to the minimal-risk nature of this behavioral study, a waiver of documented consent (opt-out procedure) was granted by the Ethics Committee. Prior to data collection, an information letter describing the study's purpose, procedures, and participants' rights was distributed to the parents or legal guardians of all potential participants. The letter explained that participation was entirely voluntary, that all data would be kept strictly confidential, and that no identifying information would appear in any reports or publications arising from the research. Parents or guardians who did not wish their child to participate were asked to complete and return an opt-out form. In addition, verbal assent was obtained from each child participant before the study commenced, and children were informed that they could withdraw from the study at any time without consequence. Failure to return the opt-out form, combined with the child's verbal assent, was taken as consent to participate. Author contributions: First Author: Conceptualization, Resources, Methodology, Software, Writing – original draft; Second Author: Conceptualization, Resources, Investigation, Writing – original draft, Review & Editing, Supervision, Project administration; Third Author: Modeling, Writing – review & editing, Visualization; Fourth Author: Formal analysis, Writing – review & editing; Fifth Author: Modeling; Sixth Author: Modeling; Seventh Author: Conceptualization, Resources, Writing – review & editing, Supervision, Project administration. References Anthropic. (2025). System Card: Claude Opus 4 & Claude Sonnet 4 . Anthropic. https://www-cdn.anthropic.com/6be99a52cb68eb70eb9572b4cafad13df32ed995.pdf Benedetto, L., Gaudeau, G., Caines, A., & Buttery, P. (2025). Assessing how accurately large language models encode and apply the common European framework of reference for languages. Computers and Education: Artificial Intelligence, 8, 100353. https://doi.org/10.1016/j.caeai.2024.100353 Bowman, G. and P., Samuel R. and Angeli. (2015). A large annotated corpus for learning natural language inference. In C. and S. Màrquez Lluís and Callison-Burch (Ed.), Proceedings of the 2015 conference on empirical methods in natural language processing (pp. 632–642). Association for Computational Linguistics. https://doi.org/10.18653/v1/D15-1075 Chalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1-29. https://doi.org/10.18637/jss.v048.i06 Chen, M., Tworek, J., Jun, H., Yuan, Q., Ponde de Oliveira Pinto, H., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., & others. (2021). Evaluating large language models trained on code. arXiv E-Prints, arXiv-2107. https://arxiv.org/abs/2107.03374 Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhu, B., Zhang, H., Jordan, M. I., Gonzalez, J. E., & Stoica, I. (2024). Chatbot arena: An open platform for evaluating LLMs by human preference. Proceedings of the 41st International Conference on Machine Learning, ICML’ 24. https://proceedings.mlr.press/v235/chiang24b.html Cizek, G. J., & Bunch, M. B. (2007). Standard Setting: A Guide to Establishing and Evaluating Performance Standards on Tests. SAGE Publications. https://doi.org/10.4135/9781412985918 Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., ... & Velic, M. (2025). Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. https://arxiv.org/abs/2507.06261 Conneau, A., & Kiela, D. (2018). SentEval: An Evaluation Toolkit for Universal Sentence Representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). https://aclanthology.org/L18-1269/ Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (pp. 4171-4186). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423 Dhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W., & Gupta, R. (2021). BOLD: Dataset and metrics for measuring biases in open-ended language generation. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT’ 21, 862-872. https://doi.org/10.1145/3442188.3445924 Ebel, R. L., & Frisbie, D. A. (1991). Essentials of educational measurement (5 th ed.) Pearson College Div. ISBN 978-0132846134. Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., ... & Vasic, P. (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783. https://arxiv.org/abs/2407.21783 Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., ... & He, Y. (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. https://arxiv.org/abs/2501.12948 Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of Item Response Theory. SAGE Publications. ISBN 9780803936478. Hartvigsen, S. and P., Thomas and Gabriel. (2022). ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. In P. and V. Muresan Smaranda and Nakov (Ed.), Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: Long papers) (pp. 3309–3326). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.acl-long.234 Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., & Steinhardt, J. (2021). Measuring massive multitask language understanding. https://arxiv.org/abs/2009.03300 Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., ... & Kivlichan, I. (2024). Gpt-4o system card. arXiv preprint arXiv:2410.21276. https://arxiv.org/abs/2410.21276 Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., ... & El Sayed, W. (2023). Mistral 7B. arXiv preprint arXiv:2310.06825. https://arxiv.org/abs/2310.06825 Kitaev, D., Nikita and Klein. (2018). Constituency parsing with a self-attentive encoder. In Y. Gurevych Iryna and Miyao (Ed.), Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: Long papers) (pp. 2676–2686). Association for Computational Linguistics. https://doi.org/10.18653/v1/P18-1249 Kolen, M. J., & Brennan, R. L. (2014). Test Equating, Scaling, and Linking: Methods and Practices (3rd ed.). Springer. https://doi.org/10.1007/978-1-4939-0317-7 Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., & others. (2022). Holistic evaluation of language models. arXiv Preprint arXiv:2211.09110. https://arxiv.org/abs/2211.09110 Lin, J. and E., Stephanie and Hilton. (2022). TruthfulQA: Measuring how models mimic human falsehoods. In P. and V. Muresan Smaranda and Nakov (Ed.), Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: Long papers) (pp. 3214–3252). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.acl-long.229 Ling, C., & Ding, N. (2026). Evaluating the encoding competence of visual language models using uncommon actions. https://arxiv.org/abs/2601.07737 Lin, S., & Crosthwaite, P. (2024). The grass is not always greener: Teacher vs. GPT-assisted written corrective feedback. System , 127 , 1–19. https://doi.org/10.1016/j.system.2024.103529 Malik, S. and P., Ali and Mayhew. (2024). From tarzan to Tolkien: Controlling the language proficiency level of LLMs for content generation. In A. and S. Ku Lun-Wei and Martins (Ed.), Findings of the association for computational linguistics: ACL 2024 (pp. 15670–15693). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.findings-acl.926 McCann, B., Keskar, N. S., Xiong, C., & Socher, R. (2018). The natural language decathlon: Multitask learning as question answering. https://arxiv.org/abs/1806.08730 National Education Examinations Authority (NEEA). (2018). China’s standards of English language ability . Beijing, China: Higher Education Press & Shanghai Foreign Language Education Press. https://cse.neea.edu.cn/html1/report/18112/9627-1.htm Nunnally, J.C. & Bernstein, I. H. (1994). Psychometric theory. 3rd ed. McGraw-Hill Companies. ISBN 9780070478497. Paperno, G. and L., Denis and Kruszewski. (2016). The LAMBADA dataset: Word prediction requiring a broad discourse context. In N. A. Erk Katrin and Smith (Ed.), Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: Long papers) (pp. 1525–1534). Association for Computational Linguistics. https://doi.org/10.18653/v1/P16-1144 Peng, B., Goldstein, D., Anthony, Q., Albalak, A., Alcaide, E., Biderman, S., ... & Zhu, R. J. (2024). Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence. arXiv preprint arXiv:2404.05892. https://arxiv.org/abs/2404.05892 Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, Jian, Tu, J., Zhang, J., Yang, Jianxin, Yang, Jiaxi, Zhou, J., ... Qiu, Z. (2025). Qwen2.5 technical report. https://arxiv.org/abs/2412.15115 Rajpurkar, J. and L., Pranav and Zhang. (2016). SQuAD: 100,000+ questions for machine comprehension of text. In K. and C. Su Jian and Duh (Ed.), Proceedings of the 2016 conference on empirical methods in natural language processing (pp. 2383–2392). Association for Computational Linguistics. https://doi.org/10.18653/v1/D16-1264 Rupp, A. A., Templin, J. L., & Henson, R. A. (2010). Diagnostic Measurement: Theory, Methods, and Applications. Guilford Press. ISBN 9781606235287. Sang, E. F. T. K., & Buchholz, S. (2000). Introduction to the CoNLL-2000 shared task: Chunking. https://arxiv.org/abs/cs/0009008 Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, Aditya, Garriga-Alonso, A., Kluska, A., Lewkowycz, A., Agarwal, A., Power, A., Ray, A., Warstadt, A., Kocurek, A. W., Safaya, A., Tazarv, A., ... Wu, Z. (2023). Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research. https://openreview.net/forum?id=uyTL5Bvosj Suzgun, N. and S., Mirac and Scales. (2023). Challenging BIG-bench tasks and whether chain-of-thought can solve them. In J. and O. Rogers Anna and Boyd-Graber (Ed.), Findings of the association for computational linguistics: ACL 2023 (pp. 13003–13051). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.findings-acl.824 Thorne, A. and C., James and Vlachos. (2018). FEVER: a large-scale dataset for fact extraction and VERification. In H. and S. Walker Marilyn and Ji (Ed.), Proceedings of the 2018 conference of the north American chapter of the association for computational linguistics: Human language technologies, volume 1 (long papers) (pp. 809–819). Association for Computational Linguistics. https://doi.org/10.18653/v1/N18-1074 Uchida, S., & Negishi, M. (2025). Assigning CEFR-J levels to English learners’s writing: An approach using lexical metrics and generative AI. Research Methods in Applied Linguistics, 4(2), 100199. https://doi.org/10.1016/j.rmal.2025.100199 Wan, Y., & Moorhouse, B. L. (2025). Using call annie as a generative artificial intelligence speaking partner for language learners. RELC Journal, 56(2), 489–498. https://doi.org/10.1177/00336882231224813 Wang, A. and M., Alex and Singh. (2018). GLUE: a multi-task benchmark and analysis platform for natural language understanding. In G. and A. Linzen Tal and Chrupała (Ed.), Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP (pp. 353–355). Association for Computational Linguistics. https://doi.org/10.18653/v1/W18-5446 Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019). SuperGLUE: a stickier benchmark for general-purpose language understanding systems. In Proceedings of the 33rd international conference on neural information processing systems. Curran Associates Inc. https://dl.acm.org/doi/10.5555/3454287.3454581 Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang, W., ... Chen, W. (2024). MMMU: a massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 9556–9567. https://doi.org/10.1109/CVPR52733.2024.00913 Zhang, W., Aljunied, S. M., Gao, C., Chia, Y. K., & Bing, L. (2023). M3Exam: A multilingual, multimodal, multilevel benchmark for examining large language models. Proceedings of the 37th International Conference on Neural Information Processing Systems, Nips’ 23. https://arxiv.org/abs/2306.05179 Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). Judging LLM-as-a-judge with MT-bench and chatbot arena. Proceedings of the 37th International Conference on Neural Information Processing Systems, Nips’ 23. https://arxiv.org/abs/2306.05685 Zhong, R. and G., Wanjun and Cui. (2024). AGIEval: A human-centric benchmark for evaluating foundation models. In H. and B. Duh Kevin and Gomez (Ed.), Findings of the association for computational linguistics: NAACL 2024 (pp. 2299–2314). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.findings-naacl.149 Additional Declarations No competing interests reported. Supplementary Files Supplementaryinformation20260208.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-8820245","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":592942007,"identity":"f397b78c-9484-46da-bf08-3e3d6d16ecd7","order_by":0,"name":"Shangchao Min","email":"","orcid":"","institution":"Zhejiang University","correspondingAuthor":false,"prefix":"","firstName":"Shangchao","middleName":"","lastName":"Min","suffix":""},{"id":592942009,"identity":"7cd41b83-cf10-4821-9967-ae7e1e41d613","order_by":1,"name":"Shaonan Wang","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABEklEQVRIiWNgGAWjYDACCWTOBwYGGTCDh4EhQQKrejQtjDPAiknRwsxDjBb52c0PHxfU3GEwOH728GvbHTY8BjcSGB+8bWPIk2zArsXgzjFj4xnHnjEYnMlLs849kwbSwmw4t42hWBqHLQYSCWbSPGyHGcwO5JgZ57YdBmlhk+ZtY0ich8thM9K/SfP8A2o5/8bM2BKihf03Pi0MN3LMgGYCtdzIMX7MCLWFGaRlNi6H3cgpNubtO8xjf+ONGWMv0C+SZx42S845J5E4E4f3gQ7b+Jjn22E5yf4c4w8/d9jI8R1PPvjhTZlN4owDuFwGAaDoYJNgBBqscABEMuCMSBTA/AGkWB6He0bBKBgFo2DkAgCMolu3XCiyOwAAAABJRU5ErkJggg==","orcid":"","institution":"Hong Kong Polytechnic University","correspondingAuthor":true,"prefix":"","firstName":"Shaonan","middleName":"","lastName":"Wang","suffix":""},{"id":592942011,"identity":"49c0a21b-7ed9-4e62-bd60-35e30bb3b3fd","order_by":2,"name":"Xinyu Gao","email":"","orcid":"","institution":"Zhejiang University","correspondingAuthor":false,"prefix":"","firstName":"Xinyu","middleName":"","lastName":"Gao","suffix":""},{"id":592942013,"identity":"9494806c-256d-453c-a052-3df290ba0f6d","order_by":3,"name":"Hui Wang","email":"","orcid":"","institution":"Zhejiang University","correspondingAuthor":false,"prefix":"","firstName":"Hui","middleName":"","lastName":"Wang","suffix":""},{"id":592942015,"identity":"f6b0fd9e-c928-4d7a-bcf2-b62e782b6d77","order_by":4,"name":"Zhiling Jin","email":"","orcid":"","institution":"Zhejiang University","correspondingAuthor":false,"prefix":"","firstName":"Zhiling","middleName":"","lastName":"Jin","suffix":""},{"id":592942017,"identity":"dc2100c4-c65a-41fd-8b8c-00a35a1d9866","order_by":5,"name":"Chen Ling","email":"","orcid":"","institution":"Zhejiang University","correspondingAuthor":false,"prefix":"","firstName":"Chen","middleName":"","lastName":"Ling","suffix":""},{"id":592942018,"identity":"764c1bc8-f22b-45ed-946e-2bbe259d19be","order_by":6,"name":"Nai Ding","email":"","orcid":"","institution":"Zhejiang University","correspondingAuthor":false,"prefix":"","firstName":"Nai","middleName":"","lastName":"Ding","suffix":""}],"badges":[],"createdAt":"2026-02-08 08:53:16","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-8820245/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-8820245/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":108491122,"identity":"33f08859-cefa-4750-8fed-cc5a4f220f36","added_by":"auto","created_at":"2026-05-05 09:52:24","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":494308,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-8820245/v1/62076a3a-5938-4935-a163-6b88df86b873.pdf"},{"id":102982491,"identity":"6bc3707b-1493-4a53-822c-772d3d51a2c5","added_by":"auto","created_at":"2026-02-19 09:22:01","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":25220,"visible":true,"origin":"","legend":"","description":"","filename":"Supplementaryinformation20260208.docx","url":"https://assets-eu.researchsquare.com/files/rs-8820245/v1/4b0f630fc55cfa9f36d949ca.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Standardized Assessment of LLM English Proficiency","fulltext":[{"header":"1 Introduction","content":"\u003cp\u003eLarge language models (LLMs) are now routinely integrated into English learning products, such as tutoring chatbots, writing feedback systems, and test preparation tools (Lin \u0026amp; Crosthwaite, \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Wan \u0026amp; Moorhouse, \u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e2025\u003c/span\u003e). As their role expands toward high-stakes language assessment, a critical evaluation gap remains: there is no standardized, interpretable way to measure and report their own English proficiency. Existing studies typically characterize LLM \u0026ldquo;language ability\u0026rdquo; using aggregate accuracy on heterogeneous benchmarks or broad comparative claims (e.g., \u0026ldquo;near‑human level\u0026rdquo;). While informative for tracking model progress, such reports offer limited guidance for educational decision‑making. Educators and policymakers instead require proficiency‑oriented answers: What level of English can a model reliably handle, and which specific linguistic subskills remain problematic?\u003c/p\u003e \u003cp\u003eA central obstacle to answering these questions is that most existing Natural language processing (NLP) benchmarks are not psychometrically anchored to an external proficiency scale. Benchmark construction has long been central to evaluation in NLP and, more recently, in LLM research. Early efforts focused on single‑task datasets targeting core competencies such as reading comprehension and inference, including SQuAD for extractive question answering (Rajpurkar et al., 2016), SNLI for natural language inference (Bowman et al., 2015), and FEVER for fact verification (Thorne et al., 2018). With the rise of general‑purpose pretrained models (Devlin et al., 2019), evaluation shifted toward multi‑task and general capability suites, most notably GLUE and SuperGLUE (Wang et al., 2018, \u003cspan citationid=\"CR42\" class=\"CitationRef\"\u003e2019\u003c/span\u003e), alongside probing and task‑collection frameworks such as SentEval and DecaNLP (Conneau \u0026amp; Kiela, 2018; McCann et al., \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e2018\u003c/span\u003e), and broader language modeling tests emphasizing discourse‑level context (Paperno et al., 2016). Parallel work introduced benchmarks targeting social harms and bias, such as BOLD and ToxiGen (Dhamala et al., \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2021\u003c/span\u003e; Hartvigsen et al., 2022).\u003c/p\u003e \u003cp\u003eAs LLMs reached frontier scale, evaluation increasingly emphasized higher‑difficulty and human‑centric benchmarks. MMLU assesses broad knowledge and reasoning across academic disciplines (Hendrycks et al., \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2021\u003c/span\u003e), while BIG‑bench provides a large and heterogeneous suite designed to surface capability breadth and systematic failure modes, with BBH focusing on especially challenging tasks (Srivastava et al., \u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Suzgun et al., 2023). A complementary trend uses standardized examinations as evaluation material, motivated by their alignment with human curricula and expert judgment. AGIEval compiles real exam questions from educational and professional testing contexts (Zhong et al., 2024), while MMMU and M3Exam extend exam‑based evaluation to multimodal, multilingual, and multi‑level settings (Yue et al., \u003cspan citationid=\"CR43\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Zhang et al., \u003cspan citationid=\"CR44\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). Collectively, these efforts move evaluation beyond narrowly constructed tasks toward assessments with clearer human reference points.\u003c/p\u003e \u003cp\u003eHowever, benchmarking LLMs on exam-style questions does not, by itself, enable meaningful proficiency-level reporting. Most benchmarks still rely on aggregate metrics such as overall accuracy or task-level averages. These scores are difficult to interpret as a specific proficiency level (e.g., \u0026ldquo;CSE Level 5\u0026rdquo;) because they conflate differences in item difficulty, content coverage, and test composition. As a result, two models with similar accuracy may in fact demonstrate markedly different underlying abilities. This limitation motivates the integration of established educational measurement practices to move beyond raw scores and toward interpretable, scale-referenced proficiency estimates.\u003c/p\u003e \u003cp\u003eEven assessments derived from authentic exams often combine items of varying difficulty without a principled scaling mechanism, leaving model performance ambiguous and resistant to level‑based interpretation (e.g., \u0026ldquo;advanced\u0026rdquo; or \u0026ldquo;Level 5\u0026rdquo;). In contrast, China\u0026rsquo;s Standards of English Language Ability (CSE) provides a nationally recognized, descriptor‑based proficiency framework that organizes English ability into nine levels, from beginner (Level 1) to expert (Level 9) (National Education Examinations Authority [NEEA], 2018). Each level is defined by detailed, context‑specific \u0026ldquo;can‑do\u0026rdquo; descriptors. For instance, CSE Level 7 involves comprehension of complex academic texts and participation in nuanced discussions, whereas Level 5\u0026mdash;commonly expected of university graduates\u0026mdash;corresponds to effective communication in general academic and professional contexts. Importantly, the CSE framework also supports fine‑grained analysis of linguistic subskills, such as grammatical control or discourse cohesion, at each level.\u003c/p\u003e \u003cp\u003eMapping LLM performance onto the CSE therefore enables proficiency to be reported in educationally meaningful terms rather than as abstract leaderboard scores. Such mapping supports direct comparison across models on a shared scale, facilitates tracking of developmental progress across model versions, and clarifies the suitability of models for specific educational uses. Moreover, once the proficiency levels of LLMs are well established, models could potentially be used as simulated test takers for items targeting different CSE levels. This opens the possibility of reducing reliance on large human samples for initial item calibration, yielding substantial savings in time, cost, and human resources.\u003c/p\u003e \u003cp\u003eEducational measurement provides well‑established methodologies for achieving these goals. Item Response Theory (IRT) models the probabilistic relationship between latent ability and item responses, estimating item difficulty and discrimination so that both items and examinees can be located on a common scale (Hambleton et al., \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e1991\u003c/span\u003e). When assessments involve multiple forms, linking and equating procedures\u0026mdash;often implemented via common‑item anchor designs\u0026mdash;ensure score comparability across forms and administrations (Kolen \u0026amp; Brennan, \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2014\u003c/span\u003e). To translate continuous ability estimates into categorical proficiency levels, standard‑setting methods such as Bookmark or Angoff rely on expert judgment to define cut scores representing minimally competent performance at each level (Cizek \u0026amp; Bunch, \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2007\u003c/span\u003e). Compared with heuristic mappings used in some LLM evaluations, these techniques provide a far stronger validity argument for claims such as \u0026ldquo;model performance corresponds to Level X.\u0026rdquo;\u003c/p\u003e \u003cp\u003eBeyond overall proficiency levels, diagnostic assessment seeks to infer mastery of underlying subskills. Cognitive diagnosis models (CDMs) use a Q‑matrix to specify which attributes each item measures and infer attribute mastery probabilities from observed response patterns (Rupp et al., \u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e2010\u003c/span\u003e). The GDINA framework, in particular, allows flexible interactions among attributes and can incorporate higher‑order structures to model correlations among subskills via an overarching ability factor. In current LLM evaluation, phenomenon tagging and error taxonomies offer useful descriptive insights; however, CDM‑based approaches go further by producing model‑based mastery estimates tied directly to expert‑defined skill specifications. This supports more interpretable and actionable claims about where a model succeeds or fails linguistically.\u003c/p\u003e \u003cp\u003eAgainst this background, we introduce CSEBench, a benchmark designed to support standardized, interpretable proficiency reporting for LLMs under the CSE framework. CSEBench comprises 624 expert‑annotated multiple‑choice questions spanning CSE Levels 2\u0026ndash;7. Each item is annotated with (i) a CSE difficulty level, (ii) one of four target subskills: vocabulary, syntax, phonology, or cohesion/discourse, (iii) metadata including student level, detailed subskills, average accuracy. The items are drawn from two large‑scale diagnostic forms (junior‑high and college versions) connected by 13 anchor items, enabling a psychometrically principled ordering of item difficulty.\u003c/p\u003e \u003cp\u003eOur evaluation pipeline follows established language‑testing practice rather than ad‑hoc heuristics. First, we estimate item difficulty using a two‑parameter logistic (2PL) item response theory (IRT) model fitted to responses from 2,050 Chinese EFL learners (1,102 eighth‑grade students and 948 college sophomores). Second, we conduct a Bookmark standard‑setting study with trained experts to establish cut scores that define minimally qualified performance for each CSE level. This yields a defensible mapping from an LLM\u0026rsquo;s item responses to an interpretable CSE level. To move beyond an overall label, we then perform cognitive diagnosis analysis using an expert‑defined Q‑matrix and higher‑order cognitive diagnosis models (selected by model fit), which produce subskill‑level mastery probabilities that pinpoint where a model is likely to succeed or fail.\u003c/p\u003e \u003cp\u003eUsing this framework, we evaluate a diverse set of systems: closed‑source models (GPT‑4o (Hurst et al., \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2024\u003c/span\u003e), Claude‑4 (Anthropic et al., 2025), DeepSeek‑R1 (Guo et al., \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e2025\u003c/span\u003e), Gemini‑2.5‑Pro (Comanici et al., \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2025\u003c/span\u003e), open‑source baselines (Mistral‑7B‑Instruct (Jiang et al., \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e2023\u003c/span\u003e), LLaMA‑3‑8B‑Instruct (Grattafiori et al., \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e2024\u003c/span\u003e), Qwen2.5‑7B‑Instruct (Qwen et al., 2025), RWKV‑v6‑7B (Peng et al., \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e2024\u003c/span\u003e), and several enhanced Qwen variants (Gao et al., 2026) that incorporate additional supervision or external knowledge. Empirically, closed‑source models consistently attain CSE Level 6 on our forms, whereas most open‑source baselines cluster around CSE Levels 3\u0026ndash;4. Qwen2.5‑7B‑Instruct emerges as the strongest open‑source baseline, reaching CSE Level 5. Diagnostic analysis reveals that the largest and most persistent gap for open‑source models lies in phonology‑related items, and we demonstrate that targeted enhancements can substantially improve mastery on this subskill.\u003c/p\u003e"},{"header":"2 CSEBench","content":"\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e\u003cb\u003e2.1 Participants\u003c/b\u003e\u003c/h2\u003e \u003cp\u003e \u003cem\u003eStandard-setting panel\u003c/em\u003e. Two domain experts participated in the standard setting process. Both were core members of the test development team who had been directly involved in item refinement, test construction, subskill tagging, and difficulty marking. Moreover, as specialists in language testing, they were familiar with CSE proficiency level descriptors for general language ability and corresponding subskills.\u003c/p\u003e \u003cp\u003e\u003cem\u003eSubskill coders.\u003c/em\u003e Five content experts coded the language subskills of each diagnostic test item. We selected coders who: (a) participated in the development of these two diagnostic test forms, and (b) were familiar with CDA approaches. All coders were faculty members or Ph.D. candidates in Applied Linguistics programs with extensive knowledge in language testing.\u003c/p\u003e \u003cp\u003e \u003cem\u003eTest takers.\u003c/em\u003e A total of 1,102 Grade 8 students from public middle schools and 948 college sophomores from universities in central and eastern China participated in the tests. All participants were Chinese EFL learners with Mandarin as their native language. The Grade 8 cohort was generally situated within CSE levels 2\u0026ndash;4, while the sophomore cohort typically fell within CSE levels 4\u0026ndash;6.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003e2.2 Instrument\u003c/h2\u003e \u003cp\u003eWe assembled two large-scale diagnostic forms by selecting high-quality items from an existing item bank. This approach was intended to (i) obtain sufficiently large response samples for stable parameter estimation, (ii) limit student testing time, and (iii) cover a broad range of CSE levels. The junior-high form contains 39 multiple-choice items and the college form contains 35 items, with 13 anchor items shared across forms to support linking. Both forms target four attributes: phonology, cohesion, vocabulary, and syntax.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e2.3 Data samples\u003c/h2\u003e \u003cp\u003eEach item record in CSEBench includes the item text and multiple annotations/empirical statistics. \u003cb\u003eAnswer\u003c/b\u003e is the keyed option. \u003cb\u003eStudent level\u003c/b\u003e indicates the intended examinee group/ability band for the item (as used in the original diagnostic test design). \u003cb\u003eSubskill\u003c/b\u003e is the primary linguistic attribute targeted by the item (phonology, cohesion, vocabulary, or syntax). \u003cb\u003eSublabel\u003c/b\u003e provides a finer-grained label under the subskill (e.g., \u003cem\u003eFixed Collocations\u003c/em\u003e under vocabulary). \u003cb\u003eCSE Difficulty Level\u003c/b\u003e is the targeted/annotated CSE level for the item. \u003cb\u003eAverage Accuracy\u003c/b\u003e is the proportion of examinees who answered the item correctly in our sample. \u003cb\u003eDiscrimination\u003c/b\u003e is the item discrimination index estimated from response data (higher values indicate better separation between higher- and lower-ability test takers; negative values may indicate a potentially problematic item).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"No\" id=\"Taba\" border=\"1\"\u003e \u003ccolgroup cols=\"7\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colspan=\"7\" nameend=\"c7\" namest=\"c1\"\u003e \u003cp\u003eI\u0026rsquo;ve got an \u0026ldquo;A\u0026rdquo; in the examination.\u003c/p\u003e \u003cp\u003eThat\u0026rsquo;s a good _____. You\u0026rsquo;ll surely win a second. \u003c/p\u003e \u003cp\u003eA. result B. start C. news D. idea\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAnswer\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eStudent level\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSubskill\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSublabel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eCSE Difficulty Level\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eAverage Accuracy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003eDiscrimination\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eB\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHigh\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003evocabulary\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eNoun\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.239\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.308\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"7\" nameend=\"c7\" namest=\"c1\"\u003e \u003cp\u003eHi, Mary, ________ beautiful day!\u003c/p\u003e \u003cp\u003eYes, it is.\u003c/p\u003e \u003cp\u003eA. How B. What a C. What D. How a\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAnswer\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eStudent level\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSubskill\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSublabel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eCSE Difficulty Level\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eAverage Accuracy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003eDiscrimination\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eB\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eLow\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003esyntax\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eExclamatory sentence\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.508\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.556\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"7\" nameend=\"c7\" namest=\"c1\"\u003e \u003cp\u003eWe can\u0026rsquo;t trust him. He always________ some excuses for doing something wrong.\u003c/p\u003e \u003cp\u003eA. makes up B. sets up\u003c/p\u003e \u003cp\u003eC. takes up D. puts up\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAnswer\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eStudent level\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSubskill\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSublabel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eCSE Difficulty Level\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eAverage Accuracy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003eDiscrimination\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eA\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eLow\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003evocabulary\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eFixed Collocations\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.467\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.517\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"7\" nameend=\"c7\" namest=\"c1\"\u003e \u003cp\u003eWhich three of the six words below have the stress on the first syllable?\u003c/p\u003e \u003cp\u003e① picture, ② maintain, ③ complete, ④ contact, ⑤ language, ⑥ defeat.\u003c/p\u003e \u003cp\u003eA. ① ④ ⑤ B. ① ② ⑥ C. ② ③ ⑤ D. ③ ④ ⑥\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAnswer\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eStudent level\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSubskill\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSublabel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eCSE Difficulty Level\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eAverage Accuracy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003eDiscrimination\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eA\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHigh\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003ephonology\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eStress\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.416\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.309\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"7\" nameend=\"c7\" namest=\"c1\"\u003e \u003cp\u003e____ disobeys the class rules will be punished.\u003c/p\u003e \u003cp\u003eA. No matter who B. Whoever C. Who D. All\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAnswer\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eStudent level\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSubskill\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSublabel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eCSE Difficulty Level\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eAverage Accuracy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003eDiscrimination\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eB\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHigh\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003esyntax\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSubject clause\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.419\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.500\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"7\" nameend=\"c7\" namest=\"c1\"\u003e \u003cp\u003eIf it _____ for his invitation the other day, I should not be here now. \u003c/p\u003e \u003cp\u003eA. had not been B. should not be C. were not to be D. should not have been\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAnswer\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eStudent level\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSubskill\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSublabel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eCSE Difficulty Level\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003eAverage Accuracy\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003eDiscrimination\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eA\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHigh\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003esyntax\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSubjunctive Mood\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e7\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.750\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.462\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e2.4 Item quality analysis\u003c/h2\u003e \u003cp\u003eTo evaluate the psychometric quality of CSEBench, we calculated item-level indices (item difficulty and discrimination index) as well as test-level reliability. For the whole item bank (CSEBench), the mean item difficulty was 0.486 and the mean discrimination index was 0.324. According to Ebel and Frisbie (\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e1991\u003c/span\u003e), item difficulty values around 0.50 are generally considered desirable, and discrimination indices above 0.30 are typically interpreted as indicating adequate item functioning. These results indicate that the item bank demonstrates appropriate difficulty level and satisfactory discriminative power.\u003c/p\u003e \u003cp\u003eFor the two large-scale diagnostic tests, the college-level form showed a mean difficulty of 0.480 and a mean discrimination index of 0.309, while the junior high form showed a mean difficulty of 0.394 and a mean discrimination index of 0.377. Test reliability, as measured by Cronbach\u0026rsquo;s \u003cem\u003eα\u003c/em\u003e, was 0.74 for the college section and 0.82 for the junior high section. Both values exceed the commonly accepted threshold of 0.70 for educational assessments (Nunnally \u0026amp; Bernstein, \u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e1994\u003c/span\u003e), indicating satisfactory reliability for both forms.\u003c/p\u003e \u003c/div\u003e"},{"header":"3. Models","content":"\u003cp\u003e\u003cstrong\u003e3.1 Model suite\u003c/strong\u003e\u003c/p\u003e\n\u003col class=\"decimal_type\"\u003e\n \u003cli\u003e\u003cstrong\u003eClosed-source models.\u003c/strong\u003e We evaluated commercially hosted models accessed via API, including GPT-4o, Claude 4, DeepSeek-R1, and Gemini 2.5 Pro, which are commonly reported to deliver strong general performance across a wide range of tasks.\u003c/li\u003e\n \u003cli\u003e\u003cstrong\u003eOpen-source models.\u003c/strong\u003e We evaluated publicly available instruction-tuned models, including Mistral-7B-Instruct, LLaMA-3-8B-Instruct, Qwen2.5-7B-Instruct, and RWKV-v6-7B. These models differ in parameterization, training data, and instruction-tuning procedures, providing a representative set of open models for comparison.\u003c/li\u003e\n \u003cli\u003e\u003cstrong\u003eEnhanced Qwen2.5 variants.\u003c/strong\u003e In addition to the base Qwen2.5 instruction model, we evaluated several enhanced variants:\u003c/li\u003e\n\u003c/ol\u003e\n\u003cul style=\"list-style-type: circle;\"\u003e\n \u003cli\u003e\u003cstrong\u003eQwen2.5-Instruct CrossAttn\u003c/strong\u003e: Qwen2.5-Instruct augmented with a cross-attention mechanism using recursive structure\u003cem\u003e\u0026nbsp;\u003c/em\u003e(Gao et al., 2026).\u003c/li\u003e\n \u003cli\u003e\u003cstrong\u003eQwen2.5-Instruct CrossAttn (Syntax)\u003c/strong\u003e: continued fine-tuning of the CrossAttn variant on CoNLL-2000 to strengthen syntactic knowledge.\u003c/li\u003e\n \u003cli\u003e\u003cstrong\u003eQwen2.5-Instruct CrossAttn (RAG)\u003c/strong\u003e: CrossAttn variant augmented with retrieval over an external knowledge graph.\u003c/li\u003e\n \u003cli\u003e\u003cstrong\u003eQwen2.5-VL CrossAttn\u003c/strong\u003e: a CrossAttn variant augmented with a visual-language (VL) component to incorporate visual knowledge.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003e3.2 Implementation details\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEnhanced variant construction.\u003c/strong\u003e Our CrossAttn variants follow the checkpoint-compatible gated tree cross-attention (GTCA) design (Gao et al., 2026), which attaches a gated cross-attention side branch to the decoder and allows token representations to attend to a cached constituency \u0026ldquo;chunk memory\u0026rdquo; while leaving the backbone architecture unchanged. Constituency parse trees are computed offline and aligned to token spans (Berkeley Neural Parser; Kitaev \u0026amp; Klein, 2018) and then cached to ensure deterministic structural inputs and to avoid parsing overhead at training and inference time.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eFor MCQA-style prompts, constituency parsing is applied within each field rather than over the entire prompt: the question stem and each candidate option are parsed independently, after which the resulting span-aligned trees are concatenated back in the original prompt order. Concretely, consider a prompt formatted as \u0026ldquo;Question: \u0026hellip; Options: A. coat B. road C. broad D. goal\u0026rdquo;. Parsing this whole string as a single sentence can cause the parser to treat the option marker (e.g., \u0026ldquo;A.\u0026rdquo;) and the option text as part of one global syntactic structure, yielding cross-field constituents that connect the stem with option fragments. Field-wise parsing avoids such artificial attachments and produces cleaner trees whose constituents respect the intended boundaries between the stem and options.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eQwen2.5-Instruct CrossAttn.\u003c/strong\u003e Starting from Qwen2.5-7B-Instruct, a checkpoint-compatible GTCA branch is attached to the language decoder so that token representations can attend to hierarchical chunk memory derived from constituency parses. The constituency parses are computed offline and cached, which yields deterministic structure signals and avoids parsing overhead during training. Continued training is then conducted on the MMLU auxiliary split (99,842 examples) using a unified multiple-choice QA (MCQA) format (Hendrycks et al., 2021), and the optimization follows the three-stage schedule in Gao et al. (2026): LoRA adapters are first trained to adapt the model to the MCQA interface with the structural pathway disabled; next, the backbone and LoRA adapters are frozen while only the GTCA branch are trained; finally, LoRA adapters and GTCA branch are jointly refined with the backbone kept frozen. Throughout this process, the learning signal is defined by the MCQA objective, i.e., maximizing the likelihood of the correct answer under the unified multiple-choice format.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eQwen2.5-Instruct CrossAttn (Syntax).\u003c/strong\u003e Building on \u003cstrong\u003eQwen2.5-Instruct CrossAttn\u003c/strong\u003e, the Syntax variant introduces an additional three-class token-level head on top of the final hidden states to provide explicit supervision over shallow syntactic structure. Training proceeds via continued finetuning on the CoNLL-2000 chunking dataset (Tjong Kim Sang \u0026amp; Buchholz, 2000), the training split contains 8,937 sentences and the test split contains 2,013 sentences. The original chunk tags are mapped to BIO labels {B, I, O}, after which the token-level cross-entropy loss is optimized to predict chunk boundaries, while the head itself remains negligible in size relative to the backbone and the GTCA branch. As a result, the training target is explicitly defined at the token level (BIO classification), while the underlying MCQA interface of the base CrossAttn model remains unchanged.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eQwen2.5-Instruct CrossAttn (RAG).\u003c/strong\u003e The RAG setting reuses the trained weights of \u003cstrong\u003eQwen2.5-Instruct CrossAttn\u003c/strong\u003e and introduces no additional training, instead, only the inference pipeline is modified. Given an item question\u0026nbsp;\u003cimg width=\"11\" height=\"19\" src=\"https://myfiles.space/user_files/58895_8739fc6c57c1c19a/58895_custom_files/img177149253195.png\" alt=\"image\"\u003e, DeepSeek-R1 generates a knowledge snippet\u0026nbsp;\u003cimg width=\"12\" height=\"19\" src=\"https://myfiles.space/user_files/58895_8739fc6c57c1c19a/58895_custom_files/img177149253193.png\" alt=\"image\"\u003e\u0026nbsp;intended to summarize rules or background facts that can support solving the item, and\u0026nbsp;\u003cimg width=\"12\" height=\"19\" src=\"https://myfiles.space/user_files/58895_8739fc6c57c1c19a/58895_custom_files/img1771492531.png\" alt=\"image\"\u003e\u0026nbsp;is concatenated with\u0026nbsp;\u003cimg width=\"11\" height=\"19\" src=\"https://myfiles.space/user_files/58895_8739fc6c57c1c19a/58895_custom_files/img177149253115.png\" alt=\"image\"\u003e\u0026nbsp;to form a knowledge-augmented input. In practice, the generated snippets are particularly effective for knowledge intensive item types (e.g., phonology, vocabulary, and syntax), and typically provide compact, option grounded cues. See Appendix for the knowledge-generation prompt and representative cases of the knowledge generated by DeepSeek-R1.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThis design is best viewed as a proxy for evidence-augmented inference: it bypasses retrieval and indexing rather than implementing a full retriever, since no external knowledge graph is constructed and no retrieval corpus is searched in this variant. The prediction target remains identical to the base setting, namely producing the correct option under the MCQA interface, but with the additional generated context available at inference time.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eQwen2.5-VL CrossAttn.\u003c/strong\u003e The multimodal variant is derived from Qwen2.5-VL-7B-Instruct by attaching the same GTCA branch to the language decoder, thereby extending the CrossAttn mechanism to a visual-language backbone. Continued training uses UAIT, a synthetic dataset of 400 images depicting uncommon action scenes (Ling \u0026amp; Ding, 2026), where each image is paired with a two-option VQA-style multiple-choice item: one option corresponds to the \u003cem\u003ecommon-text\u003c/em\u003e generated before role replacement, and the other corresponds to the \u003cem\u003euncommon-text\u003c/em\u003e obtained after swapping agent and patient roles. The model is finetuned as a visual multiple-choice task by maximizing the likelihood of the correct option letter, which encourages visual grounding for semantic role assignments in counter-common-sense scenes. For evaluation on CSEBench, the resulting model is subsequently queried using the standard text-only inputs to keep the benchmark interface consistent across model families.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEvaluation settings.\u003c/strong\u003e Open-source models are evaluated with their official chat templates and default tokenizers, while closed-source models are queried through public APIs. Decoding is deterministic (temperature = 0, top-p = 1) with a maximum of 8 generated tokens. We run each item three times under each prompting condition and take majority vote when the outputs differ (ties broken by the first run). Options are not shuffled. The predicted option is extracted as the first valid letter in {A, B, C, D}, if no valid letter is found, the item is marked incorrect.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e3.3 Prompting conditions\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe evaluate all models on \u003cstrong\u003eCSEBench\u003c/strong\u003e under two prompting conditions:\u003c/p\u003e\n\u003cul type=\"disc\"\u003e\n \u003cli\u003e\u003cstrong\u003eZero-shot\u003c/strong\u003e: the model selects the correct option and outputs \u003cstrong\u003eonly\u003c/strong\u003e the option letter.\u003c/li\u003e\n \u003cli\u003e\u003cstrong\u003eFew-shot\u003c/strong\u003e: the model is given \u003cstrong\u003etwo demonstration items\u003c/strong\u003e followed by the target item, and outputs \u003cstrong\u003eonly\u003c/strong\u003e the option letter.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" class=\"fr-table-selection-hover\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 624px;\"\u003e\n \u003ch4\u003e\u003cstrong\u003eZero-shot prompt\u003c/strong\u003e\u003c/h4\u003e\n \u003cp\u003eInstruction: Choose the correct option based on the question. Output only the corresponding letter of the option without providing a reason.\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003eInput: {input}\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003eResponse:\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 624px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u003cem\u003eFew-shot prompt\u003c/em\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003cp\u003eInstruction: Choose the correct option based on the question. Output only the corresponding letter of the option without providing a reason.\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003eInput: They said the eighteenth and last lesson ___ quite easy.\u003c/p\u003e\n \u003cp\u003eA. is B. was C. are D. were\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003eResponse: B\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003eInput: Marx left his homeland for some ___ reasons.\u003c/p\u003e\n \u003cp\u003eA. politically B. politics C. political D. politician\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003eResponse: C\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003eInput: {input}\u003c/p\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003eResponse:\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e"},{"header":"4. Methods: Mapping LLMs to Proficiency Levels","content":"\u003cp\u003e\u003cstrong\u003e4.1 Procedure\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eRQ1: What CSE proficiency levels do different models achieve?\u003c/em\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe addressed RQ1 via a standard-setting study to derive cut scores corresponding to CSE Levels 3\u0026ndash;6 on the two diagnostic test forms. We adopted the bookmark method, where panelists place a bookmark in an ordered item booklet to indicate the point at which a just\u0026nbsp;qualified candidate (JQC) at a given CSE level is expected to answer items below the bookmark correctly (i.e., items the JQC should master), but not items above it.\u003c/p\u003e\n\u003cp\u003eTo support consistent judgments, items were ordered from easiest to most difficult using IRT-estimated difficulty parameters. This ordering allowed panelists to interpret each item relative to the overall difficulty continuum before placing level-specific bookmarks.\u003c/p\u003e\n\u003cp\u003eThe standard-setting session consisted of (i) panelist training on the bookmark procedure and calibration on CSE performance-level descriptors, followed by (ii) two rounds of judgments. In Round 1, panelists placed bookmarks independently without discussion. In Round 2, they revisited their placements after discussing Round 1 rationales and reviewing summary classification information (e.g., the projected percentage of test takers classified into each level under Round 1 cuts). We use the Round 2 placements as the final cut scores for CSE Levels 3\u0026ndash;6.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eRQ2: How do LLMs perform across subskills (phonology, cohesion, vocabulary, and syntax)?\u003c/em\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe addressed RQ2 using cognitive diagnostic assessment (CDA) to estimate LLM mastery of fine-grained language attributes from item-level responses and the Q-matrix (item\u0026ndash;attribute mapping). The diagnostic framework included four attributes defined during test development: A1 phonology, A2 cohesion, A3 vocabulary, and A4 syntax. Content experts assigned each item to one or more relevant attributes: they first coded independently and then resolved disagreements through discussion to produce the final Q-matrix used in CDA.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e4.2 Statistical analysis\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eIRT for item calibration and standard setting (RQ1)\u003c/em\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTo obtain item parameters for the ordered item booklet, we fit a unidimensional two-parameter logistic (2PL) IRT model using the mirt package in R (Chalmers, 2012). The resulting difficulty estimates were used to rank items for the bookmark procedure and to ensure that cut-score decisions were made with respect to a calibrated difficulty continuum.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e\u003cem\u003eCDA for subskill mastery estimation (RQ2)\u003c/em\u003e\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTo estimate attribute mastery patterns, we conducted CDA using the GDINA package in R (v2.7.8; Ma et al., 2020). We fit five cognitive diagnostic models: the saturated G-DINA model; two compensatory models (DINO, ACDM); and two non-compensatory models (DINA, RRUM). Each model was specified with a higher-order attribute structure parameterized as 2PL (i.e., a higher-order CDM). Model selection was performed using relative fit indices. After selecting the best-fitting model, we computed attribute mastery probabilities for each LLM and compared performance across the four language attributes to identify subskill-specific strengths and weaknesses.\u003c/p\u003e"},{"header":"5. Results","content":"\u003cp\u003e\u003cstrong\u003e5.1 CSE level alignment\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe standard-setting results (see Table 1) indicate that \u003cstrong\u003eall closed-source models\u003c/strong\u003e in our study reach at least \u003cstrong\u003eCSE-6\u003c/strong\u003e under both zero-shot and few-shot prompting. Because the underlying item bank was designed for \u003cstrong\u003eGrade 8\u003c/strong\u003e and \u003cstrong\u003esecond-year university\u003c/strong\u003e EFL learners, it contains \u003cstrong\u003efew CSE-7 items\u003c/strong\u003e\u003cstrong\u003e,\u003c/strong\u003e which limits our ability to make reliable claims about proficiency above CSE-6.\u003c/p\u003e\n\u003cp\u003eIn contrast, most \u003cstrong\u003eopen-source baselines\u003c/strong\u003e cluster at \u003cstrong\u003eCSE-3 to CSE-4\u003c/strong\u003e\u003cstrong\u003e,\u003c/strong\u003e with \u003cstrong\u003eQwen2.5-7B-Instruct\u003c/strong\u003e as a notable exception, reaching \u003cstrong\u003eCSE-5\u003c/strong\u003e in both prompting conditions. Among the \u003cstrong\u003eenhanced Qwen2.5 variants\u003c/strong\u003e, performance is heterogeneous: several variants remain at CSE-5, while \u003cstrong\u003etwo models\u003c/strong\u003e show clear gains and achieve \u003cstrong\u003eCSE-6\u003c/strong\u003e, approaching the closed-source systems.\u003c/p\u003e\n\u003cp\u003eTable 1. Alignment of LLMs to CSE Levels.\u003c/p\u003e\n\u003cdiv align=\"\"\u003e\n \u003ctable border=\"0\" cellspacing=\"0\" cellpadding=\"0\" width=\"86%\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 22px;\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 46px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eModels\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eZero-shot\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eFew-shot\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"4\" valign=\"top\" style=\"width: 22px;\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003eOpen-source model\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 46px;\"\u003e\n \u003cp\u003eRWKV-v6-7b\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003eCSE-3\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003eCSE-3\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 46px;\"\u003e\n \u003cp\u003eMeta-Llama-3-8B-Instruct\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003eCSE-3\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003eCSE-4\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 46px;\"\u003e\n \u003cp\u003eQwen2.5-7B-Instruct\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003eCSE-5\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003eCSE-5\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 46px;\"\u003e\n \u003cp\u003eMistral-7B-Instruct-v0.3\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003eCSE-3\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003eCSE-3\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"4\" valign=\"top\" style=\"width: 22px;\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003eClosed-source\u003c/p\u003e\n \u003cp\u003ebaseline\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 46px;\"\u003e\n \u003cp\u003eGPT-4o\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003eCSE-6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003eCSE-6\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 46px;\"\u003e\n \u003cp\u003eClaude4\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003eCSE-6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003eCSE-6\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 46px;\"\u003e\n \u003cp\u003eDeepseek-R1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003eCSE-6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003eCSE-6\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 46px;\"\u003e\n \u003cp\u003eGemini2.5-Pro\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003eCSE-6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003eCSE-6\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd rowspan=\"4\" valign=\"top\" style=\"width: 22px;\"\u003e\n \u003cp\u003e\u0026nbsp;\u003c/p\u003e\n \u003cp\u003eEnhanced\u003c/p\u003e\n \u003cp\u003emodel\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 46px;\"\u003e\n \u003cp\u003eQwen2.5-Instruct-Cross-Attn\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003eCSE-5\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003eCSE-5\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 46px;\"\u003e\n \u003cp\u003eQwen2.5-Instruct-Cross-Attn-syntax\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003eCSE-5\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003eCSE-5\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 46px;\"\u003e\n \u003cp\u003eQwen2.5-Instruct-Cross-Attn-rag\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003eCSE-6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003eCSE-6\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 46px;\"\u003e\n \u003cp\u003eQwen2.5-vl-Cross-Attn\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003eCSE-6\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 15px;\"\u003e\n \u003cp\u003eCSE-6\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003e\u003cstrong\u003e5.2 Model selection and subskill mastery\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eTable 2 reports the relative fit statistics for five higher-order cognitive diagnostic models (HO-CDMs). For \u003cstrong\u003e\u0026minus;2LL\u003c/strong\u003e,\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003cstrong\u003eAIC\u003c/strong\u003e, and \u003cstrong\u003eBIC\u003c/strong\u003e, smaller values indicate better relative fit. The \u003cstrong\u003eHO-GDINA\u003c/strong\u003e model achieved the best overall fit (lowest AIC and \u0026minus;2LL) and was selected for subsequent mastery analysis.\u003c/p\u003e\n\u003cp\u003eTable 2. Relative model fit of CDMs.\u003c/p\u003e\n\u003cdiv align=\"\"\u003e\n \u003ctable border=\"0\" cellspacing=\"0\" cellpadding=\"0\" width=\"378\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 104px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eModel\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 90px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eAIC\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 90px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eBIC\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 94px;\"\u003e\n \u003cp\u003e\u003cstrong\u003e-2LL\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 104px;\"\u003e\n \u003cp\u003eHO-DINA\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 90px;\"\u003e\n \u003cp\u003e94568.34\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 90px;\"\u003e\n \u003cp\u003e95301.18\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 94px;\"\u003e\n \u003cp\u003e94308.34\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 104px;\"\u003e\n \u003cp\u003eHO-DINO\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 90px;\"\u003e\n \u003cp\u003e94428.75\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 90px;\"\u003e\n \u003cp\u003e95161.59\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 94px;\"\u003e\n \u003cp\u003e94168.75\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 104px;\"\u003e\n \u003cp\u003eHO-ACDM\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 90px;\"\u003e\n \u003cp\u003e94207.82\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 90px;\"\u003e\n \u003cp\u003e95064.68\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 94px;\"\u003e\n \u003cp\u003e93903.82\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 104px;\"\u003e\n \u003cp\u003eHO-RRUM\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 90px;\"\u003e\n \u003cp\u003e94205.19\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 90px;\"\u003e\n \u003cp\u003e95062.05\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 94px;\"\u003e\n \u003cp\u003e93901.19\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 104px;\"\u003e\n \u003cp\u003eHO-GDINA\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 90px;\"\u003e\n \u003cp\u003e94187.38\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 90px;\"\u003e\n \u003cp\u003e95269.73\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 94px;\"\u003e\n \u003cp\u003e93803.38\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003eUsing HO-GDINA, we estimated each model\u0026rsquo;s general ability and attribute mastery probabilities for the four subskills. As shown in Table 3, closed-source models consistently exhibit near-ceiling mastery across cohesion, vocabulary, and syntax, and also perform strongly on phonology, with only minor variation. Gemini 2.5 Pro yields the highest (near-saturated) mastery estimates among the closed-source systems.\u003c/p\u003e\n\u003cp\u003eAmong open-source baselines, Qwen2.5-7B-Instruct is the strongest overall: it exceeds the mastery threshold (probability \u0026gt; 0.5) for cohesion, vocabulary, and syntax, but remains markedly weak on phonology (0.0352 in both prompting settings). The remaining open-source baselines do not reach mastery on any attribute.\u003c/p\u003e\n\u003cp\u003eThe enhanced models show that targeted augmentation can substantially improve subskill performance. In particular, Qwen2.5-Instruct-CrossAttn (RAG) achieves near-ceiling mastery across all attributes, including phonology (0.9936), and is competitive with the closed-source models. Qwen2.5-VL CrossAttn also performs strongly, but its phonology mastery drops from 0.9679 (zero-shot) to 0.9132 (few-shot), suggesting some sensitivity to the few-shot prompt format. One plausible reason for the improvement observed with the RAG model is that this setting adds short evidence that states key facts directly. This reduces the need for the model to recall phonological and lexical facts from memory. The CrossAttn model also uses cached constituency chunks, which can help it link the stem to each option. The Qwen2.5-VL CrossAttn model improve because its training encourages careful option comparison. This objective can strengthen fine grained discrimination and role reasoning. These gains can transfer to better text only performance on CSEBench. The lower phonology mastery in few-shot prompts may come from interference from the demonstrations. Longer contexts can pull attention toward the exemplar patterns, which may weaken attention to short spelling to sound cues.\u003c/p\u003e\n\u003cp\u003eTable 3. General ability and mastery probabilities of LLMs across four language subskills.\u003c/p\u003e\n\u003cdiv align=\"center\"\u003e\n \u003ctable border=\"0\" cellspacing=\"0\" cellpadding=\"0\" width=\"102%\" class=\"fr-table-selection-hover\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 16px;\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 18px;\"\u003e\n \u003cp\u003eGeneral ability\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 16px;\"\u003e\n \u003cp\u003eA1_Cohesion\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 16px;\"\u003e\n \u003cp\u003eA2_Phonology\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 15px;\"\u003e\n \u003cp\u003eA3_Vocabulary\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd colspan=\"2\" style=\"width: 14px;\"\u003e\n \u003cp\u003eA4_Syntax\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 16px;\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003ezero\u003c/p\u003e\n \u003cp\u003e-shot\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003efew\u003c/p\u003e\n \u003cp\u003e-shot\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003ezero\u003c/p\u003e\n \u003cp\u003e-shot\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003efew\u003c/p\u003e\n \u003cp\u003e-shot\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003ezero-shot\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003efew-shot\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 7px;\"\u003e\n \u003cp\u003ezero-shot\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003efew-shot\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003ezero-shot\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 6px;\"\u003e\n \u003cp\u003efew-shot\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 16px;\"\u003e\n \u003cp\u003eRWKV-v6-7b\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003e-0.6271\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003e-0.7494\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.1689\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.0131\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.0232\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 7px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 6px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 16px;\"\u003e\n \u003cp\u003eMeta-Llama-3-8B-Instruct\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003e-0.7647\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003e0.1074\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.6875\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.0053\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.4523\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 7px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.1616\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 6px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 16px;\"\u003e\n \u003cp\u003eQwen2.5-7B-Instruct\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003e1.1456\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003e1.1456\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.0352\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.0352\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 7px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 6px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 16px;\"\u003e\n \u003cp\u003eMistral-7B-Instruct-v0.3\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003e-0.7362\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003e-0.6840\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.0387\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.0997\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 7px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 6px;\"\u003e\n \u003cp\u003e0\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 16px;\"\u003e\n \u003cp\u003eGPT-4o\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003e1.6630\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003e1.6642\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.9966\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.9987\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 7px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 6px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 16px;\"\u003e\n \u003cp\u003eClaude4\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003e1.6642\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003e1.6644\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.9987\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.9993\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 7px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 6px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 16px;\"\u003e\n \u003cp\u003eDeepseek-R1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003e1.6642\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003e1.6644\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.9987\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.9993\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 7px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 6px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 16px;\"\u003e\n \u003cp\u003eGemini2.5-Pro\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003e1.6644\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003e1.6644\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.9993\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.9993\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 7px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 6px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 16px;\"\u003e\n \u003cp\u003eQwen2.5-Instruct-Cross-Attn\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003e1.2119\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003e1.3639\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.1583\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.4409\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 7px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 6px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 16px;\"\u003e\n \u003cp\u003eQwen2.5-Instruct-Cross-Attn-syntax\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003e1.2119\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003e1.3639\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.1583\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.4409\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 7px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 6px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 16px;\"\u003e\n \u003cp\u003eQwen2.5-Instruct-Cross-Attn-rag\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003e1.6614\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003e1.6614\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.9936\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.9936\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 7px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 6px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 16px;\"\u003e\n \u003cp\u003eQwen2.5-vl-Cross-Attn\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003e1.6476\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 9px;\"\u003e\n \u003cp\u003e1.6181\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.9679\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e0.9132\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 7px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 8px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 6px;\"\u003e\n \u003cp\u003e1\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003e\u003cstrong\u003e\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e"},{"header":"6. Discussion","content":"\u003cp\u003eAcross widely used LLM evaluations, a consistent pattern has emerged: models often appear highly capable in aggregate yet exhibit substantial unevenness across task types, difficulty levels, and evaluation protocols. On broad benchmark suites such as MMLU and BIG‑bench, strong overall performance can mask sharp drops on harder reasoning subsets or compositional tasks (Hendrycks et al., \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2021\u003c/span\u003e; Srivastava et al., \u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Suzgun et al., 2023). Exam‑based evaluations similarly show that frontier models can perform competitively on portions of standardized tests while remaining brittle on multi‑step reasoning or specialized knowledge (Zhong et al., 2024). Reliability‑focused benchmarks further reveal systematic weaknesses that are obscured by surface‑level fluency: TruthfulQA demonstrates that models may produce confident but consistently false responses aligned with common misconceptions (Lin et al., 2022), while HumanEval shows that gains from scale and instruction tuning coexist with persistent functional failures on non‑trivial programming tasks (Chen et al., \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). Preference‑based and dialogue evaluations likewise indicate that perceived conversational quality can diverge substantially from task accuracy and that rankings are sensitive to judging protocols (Zheng et al., \u003cspan citationid=\"CR45\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Chiang et al., \u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). Together with broader critiques of contamination, dataset artifacts, and narrow metrics (Liang et al., \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2022\u003c/span\u003e), these findings underscore the limitations of accuracy‑centric benchmarking.\u003c/p\u003e \u003cp\u003eThese limitations become particularly consequential in educational applications, where LLM performance must be interpreted in terms of language ability rather than aggregate task success. However, most existing NLP benchmarks are not psychometrically anchored to an external proficiency scale. As a result, LLM \u0026ldquo;language ability\u0026rdquo; is typically inferred from overall accuracy on heterogeneous benchmarks or from broad comparative claims, which are poorly aligned with the proficiency-oriented judgments required in educational contexts\u0026mdash;namely, what level of English a model can reliably handle and which linguistic subskills remain fragile.\u003c/p\u003e \u003cp\u003eRather than treating uneven performance as a collection of task‑specific anomalies, we mapped LLM responses onto a calibrated CSE scale reveals systematic differences in underlying language proficiency. Under this lens, closed‑source models consistently operate above the proficiency range targeted by the learner populations used to calibrate CSEBench, whereas most open‑source baselines cluster in substantially lower CSE bands. This distinction is not easily recoverable from raw accuracy alone, but becomes salient once item difficulty, linking, and standard setting are taken into account. In this sense, CSE‑aligned reporting reframes benchmark results from how many items the model answered correctly to what level of English ability this performance credibly represents.\u003c/p\u003e \u003cp\u003eThis proficiency‑oriented perspective connects our work to a growing line of research that evaluates LLMs using human language proficiency frameworks, most prominently the Common European Framework of Reference for Languages (CEFR). Recent studies examine whether models can apply CEFR descriptors, generate text at targeted CEFR levels, or control linguistic difficulty for language learners (Benedetto et al., 2024; Benedetto et al., \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Malik et al., 2024). Notably, Benedetto et al. conceptualize CEFR not merely as a post‑hoc labeling scheme but as a behavioral target for evaluation, while Malik et al. show that combining fine‑tuning and reinforcement learning can substantially improve CEFR‑aligned generation, in some cases allowing smaller models to outperform GPT‑4 at lower cost. Related systems also assess learner writing against CEFR variants using hybrid feature‑based and model‑based approaches (Uchida \u0026amp; Negishi, \u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e2025\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eWhile complementary, most CEFR‑oriented LLM studies rely on relatively coarse validation signals, such as global level judgments, readability metrics, or indirect complexity proxies. They typically do not provide item‑level linking arguments or psychometric calibration comparable to those used in educational measurement. As a result, proficiency claims are often difficult to compare across test forms, datasets, or populations. By contrast, CSEBench integrates calibrated item difficulties, anchor‑based linking, and expert standard setting, enabling defensible and comparable proficiency‑level interpretations. Our findings therefore suggest that proficiency frameworks such as CEFR or CSE can be substantially strengthened when paired with established measurement models, rather than applied solely as descriptive labels or generation targets.\u003c/p\u003e \u003cp\u003eBeyond overall proficiency levels, our diagnostic analyses reveal systematic subskill‑level differences across models. Phonology emerges as the most discriminative and challenging subskill: even the strongest open‑source baseline shows near‑zero mastery on phonology‑related items despite high mastery in vocabulary, syntax, and cohesion/discourse. This pattern suggests that standard instruction tuning and large‑scale text‑only corpora are insufficient for developing sensitivity to stress patterns and phonological contrasts, which are only weakly encoded in orthographic text. Models augmented with external knowledge or multimodal signals substantially reduce this gap, indicating a promising direction for improvement.\u003c/p\u003e \u003cp\u003eThe robustness of proficiency estimates also warrants attention. Few‑shot prompting does not uniformly improve performance and, in some cases, slightly degrades subskill mastery, most notably for phonology. This effect is particularly visible in enhanced models such as Qwen2.5‑VL CrossAttn, suggesting sensitivity to exemplar choice, ordering, and prompt structure.\u003c/p\u003e \u003cp\u003eDespite these robustness concerns, mapping model performance to CSE levels yields interpretations that are directly actionable for educational contexts. In our setting, closed‑source systems consistently reach CSE Level 6, exceeding the proficiency range of the student populations used for calibration, whereas most open‑source baselines fall within Levels 3\u0026ndash;4, with Qwen2.5‑7B‑Instruct reaching Level 5. This stratification suggests differentiated roles for LLMs in educational applications: higher‑proficiency systems may function as effective scaffolding agents or adaptive interlocutors for upper‑intermediate learners, while lower‑proficiency models may be more appropriate for beginner‑level practice or constrained instructional settings. Importantly, these conclusions rely on proficiency‑level interpretation rather than absolute accuracy, highlighting the added value of scale‑referenced evaluation.\u003c/p\u003e \u003cp\u003eTaken together, these findings support the central claim of this work: evaluating LLMs as language users benefits from the same psychometric rigor applied to human language assessment. Proficiency‑grounded benchmarks such as CSEBench make it possible to express model capabilities in terms that are interpretable, comparable, and educationally meaningful. Beyond evaluation, such mappings open the door to new applications, including the principled use of LLMs as simulated test takers, diagnostic tools, or adaptive instructional agents\u0026mdash;provided their proficiency limits are clearly understood. More broadly, our results suggest that future LLM evaluation efforts may benefit from closer integration with educational measurement theory, especially when models are deployed in learning‑ and assessment‑critical contexts.\u003c/p\u003e"},{"header":"7. Limitation and Future Work","content":"\u003cp\u003eDespite the strengths of CSEBench and the psychometrically grounded evaluation pipeline, several limitations should be acknowledged, which also point to important directions for future work.\u003c/p\u003e \u003cp\u003e \u003cb\u003eCoverage of proficiency range and ceiling effects.\u003c/b\u003e \u003c/p\u003e \u003cp\u003eThe current version of CSEBench spans CSE Levels 2\u0026ndash;7 and is calibrated using junior-high and college EFL learners. While this range captures a broad spectrum of practical proficiency, it introduces ceiling effects for frontier-scale closed-source models, which consistently exceed the upper end of the calibrated ability range. As a result, distinctions among high-performing models at advanced levels cannot be fully resolved. Future work will expand item coverage to CSE Levels 8\u0026ndash;9, incorporating more demanding academic and professional language tasks. Extending the scale upward will enable finer-grained differentiation among advanced models and support more precise claims about near-expert or expert-level proficiency.\u003c/p\u003e \u003cp\u003e \u003cb\u003ePrompting effects and robustness of proficiency estimates.\u003c/b\u003e \u003c/p\u003e \u003cp\u003eAs observed in our results, few-shot prompting can introduce instability in subskill mastery estimates, particularly for phonology. This sensitivity raises an important limitation: proficiency estimates may vary with prompt design, exemplar selection, or formatting, even when the underlying model remains unchanged. Future work should conduct controlled robustness studies that vary exemplar number, type, and order, as well as randomize prompts, to quantify variance attributable to prompting. Developing standardized prompting protocols\u0026mdash;or prompt-invariant evaluation methods\u0026mdash;will be crucial for reliable proficiency reporting.\u003c/p\u003e \u003cp\u003e \u003cb\u003eScope of linguistic skills assessed.\u003c/b\u003e \u003c/p\u003e \u003cp\u003eCSEBench currently targets four subskills: vocabulary, syntax, phonology, and cohesion/discourse, operationalized through multiple-choice items. While this design supports psychometric calibration and diagnostic modeling, it does not capture productive skills such as speaking and writing, nor does it assess interactional competence or pragmatic appropriateness in open-ended settings. Future extensions could incorporate constructed-response items, spoken or multimodal tasks, and automated scoring models that are themselves calibrated within the same proficiency framework. This would move evaluation closer to the full construct coverage envisioned in the CSE.\u003c/p\u003e \u003cp\u003e \u003cb\u003eValidation in operational educational settings.\u003c/b\u003e \u003c/p\u003e \u003cp\u003eFinally, although CSEBench is grounded in established testing practice, it has not yet been validated in real-world educational deployments. Future work will evaluate the benchmark\u0026rsquo;s utility in operational contexts, including adaptive tutoring systems, placement decisions (e.g., assigning learners to appropriate courses or proficiency bands), and the generation of diagnostic feedback. Such studies are essential for establishing consequential validity\u0026mdash;demonstrating not only that proficiency estimates are interpretable, but also that their use leads to improved educational outcomes.\u003c/p\u003e \u003cp\u003eTaken together, these limitations highlight both the current boundaries of CSEBench and the broader research agenda it enables. Addressing them will further strengthen the case for proficiency-grounded, psychometrically principled evaluation of LLMs in language learning and assessment contexts.\u003c/p\u003e"},{"header":"8. Conclusion","content":"\u003cp\u003eWe presented \u003cb\u003eCSEBench\u003c/b\u003e, a CSE-aligned benchmark for standardized assessment of LLM English proficiency across levels and subskills. Using IRT and expert standard setting, we mapped LLMs to CSE levels and showed that closed-source models reach CSE-6, while most open-source baselines cluster around CSE-3\u0026ndash;4; Qwen2.5-7B-Instruct reaches CSE-5, and enhanced Qwen variants approach closed-source performance. A CDM analysis highlights subskill gaps\u0026mdash;especially phonology\u0026mdash;and demonstrates that targeted enhancements can substantially improve mastery.\u003c/p\u003e "},{"header":"Declarations","content":"\u003cp\u003eEthical approval: The experimental procedures were approved by the Ethics Committee of the College of Biomedical Engineering and Instrument Science, Zhejiang University (Approval No. 2024\u0026ndash;12).\u003c/p\u003e\n\u003cp\u003eConsent to participate: Due to the minimal-risk nature of this behavioral study, a waiver of documented consent (opt-out procedure) was granted by the Ethics Committee. Prior to data collection, an information letter describing the study\u0026apos;s purpose, procedures, and participants\u0026apos; rights was distributed to the parents or legal guardians of all potential participants. The letter explained that participation was entirely voluntary, that all data would be kept strictly confidential, and that no identifying information would appear in any reports or publications arising from the research. Parents or guardians who did not wish their child to participate were asked to complete and return an opt-out form. In addition, verbal assent was obtained from each child participant before the study commenced, and children were informed that they could withdraw from the study at any time without consequence. Failure to return the opt-out form, combined with the child\u0026apos;s verbal assent, was taken as consent to participate.\u003c/p\u003e\u003cp\u003e\u003cstrong\u003eEthics statement\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests:\u003c/strong\u003e The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData availability:\u003c/strong\u003e The dataset is publicly available at\u003c/p\u003e\n\u003cp\u003ehttps://osf.io/6x9sv/overview?view_only=58ace849d6e1402fbc4fd5188747d17c\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eEthical approval:\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe experimental procedures were approved by the Ethics Committee of the College of Biomedical Engineering and Instrument Science, Zhejiang University (Approval No. 2024\u0026ndash;12). \u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsent to participate:\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eDue to the minimal-risk nature of this behavioral study, a waiver of documented consent (opt-out procedure) was granted by the Ethics Committee. Prior to data collection, an information letter describing the study\u0026apos;s purpose, procedures, and participants\u0026apos; rights was distributed to the parents or legal guardians of all potential participants. The letter explained that participation was entirely voluntary, that all data would be kept strictly confidential, and that no identifying information would appear in any reports or publications arising from the research. Parents or guardians who did not wish their child to participate were asked to complete and return an opt-out form. In addition, verbal assent was obtained from each child participant before the study commenced, and children were informed that they could withdraw from the study at any time without consequence. Failure to return the opt-out form, combined with the child\u0026apos;s verbal assent, was taken as consent to participate.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthor contributions:\u003c/strong\u003e First Author: Conceptualization, Resources, Methodology, Software, Writing \u0026ndash; original draft; Second Author: Conceptualization, Resources, Investigation, Writing \u0026ndash; original draft, Review \u0026amp; Editing, Supervision, Project administration; Third Author: Modeling, Writing \u0026ndash; review \u0026amp; editing, Visualization; Fourth Author: Formal analysis, Writing \u0026ndash; review \u0026amp; editing; Fifth Author: Modeling; Sixth Author: Modeling; Seventh Author: Conceptualization, Resources, Writing \u0026ndash; review \u0026amp; editing, Supervision, Project administration.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eAnthropic. (2025). \u003cem\u003eSystem Card: Claude Opus 4 \u0026amp; Claude Sonnet 4\u003c/em\u003e. Anthropic. https://www-cdn.anthropic.com/6be99a52cb68eb70eb9572b4cafad13df32ed995.pdf\u003c/li\u003e\n\u003cli\u003eBenedetto, L., Gaudeau, G., Caines, A., \u0026amp; Buttery, P. (2025). Assessing how accurately large language models encode and apply the common European framework of reference for languages. Computers and Education: Artificial Intelligence, 8, 100353. https://doi.org/10.1016/j.caeai.2024.100353\u003c/li\u003e\n\u003cli\u003eBowman, G. and P., Samuel R. and Angeli. (2015). A large annotated corpus for learning natural language inference. In C. and S. M\u0026agrave;rquez Llu\u0026iacute;s and Callison-Burch (Ed.), Proceedings of the 2015 conference on empirical methods in natural language processing (pp. 632\u0026ndash;642). Association for Computational Linguistics. https://doi.org/10.18653/v1/D15-1075\u003c/li\u003e\n\u003cli\u003eChalmers, R. P. (2012). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1-29. https://doi.org/10.18637/jss.v048.i06\u003c/li\u003e\n\u003cli\u003eChen, M., Tworek, J., Jun, H., Yuan, Q., Ponde de Oliveira Pinto, H., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., \u0026amp; others. (2021). Evaluating large language models trained on code. arXiv E-Prints, arXiv-2107. https://arxiv.org/abs/2107.03374\u003c/li\u003e\n\u003cli\u003eChiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhu, B., Zhang, H., Jordan, M. I., Gonzalez, J. E., \u0026amp; Stoica, I. (2024). Chatbot arena: An open platform for evaluating LLMs by human preference. Proceedings of the 41st International Conference on Machine Learning, ICML\u0026rsquo; 24. https://proceedings.mlr.press/v235/chiang24b.html\u003c/li\u003e\n\u003cli\u003eCizek, G. J., \u0026amp; Bunch, M. B. (2007). Standard Setting: A Guide to Establishing and Evaluating Performance Standards on Tests. SAGE Publications. https://doi.org/10.4135/9781412985918\u003c/li\u003e\n\u003cli\u003eComanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., ... \u0026amp; Velic, M. (2025). Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. https://arxiv.org/abs/2507.06261\u003c/li\u003e\n\u003cli\u003eConneau, A., \u0026amp; Kiela, D. (2018). SentEval: An Evaluation Toolkit for Universal Sentence Representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). https://aclanthology.org/L18-1269/\u003c/li\u003e\n\u003cli\u003eDevlin, J., Chang, M. W., Lee, K., \u0026amp; Toutanova, K. (2019). Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (pp. 4171-4186). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423\u003c/li\u003e\n\u003cli\u003eDhamala, J., Sun, T., Kumar, V., Krishna, S., Pruksachatkun, Y., Chang, K.-W., \u0026amp; Gupta, R. (2021). BOLD: Dataset and metrics for measuring biases in open-ended language generation. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT\u0026rsquo; 21, 862-872. https://doi.org/10.1145/3442188.3445924\u003c/li\u003e\n\u003cli\u003eEbel, R. L., \u0026amp; Frisbie, D. A. (1991). Essentials of educational measurement (5 th ed.) Pearson College Div. ISBN 978-0132846134.\u003c/li\u003e\n\u003cli\u003eGrattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., ... \u0026amp; Vasic, P. (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783. https://arxiv.org/abs/2407.21783\u003c/li\u003e\n\u003cli\u003eGuo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., ... \u0026amp; He, Y. (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. https://arxiv.org/abs/2501.12948\u003c/li\u003e\n\u003cli\u003eHambleton, R. K., Swaminathan, H., \u0026amp; Rogers, H. J. (1991). Fundamentals of Item Response Theory. SAGE Publications. ISBN 9780803936478.\u003c/li\u003e\n\u003cli\u003eHartvigsen, S. and P., Thomas and Gabriel. (2022). ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. In P. and V. Muresan Smaranda and Nakov (Ed.), Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: Long papers) (pp. 3309\u0026ndash;3326). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.acl-long.234\u003c/li\u003e\n\u003cli\u003eHendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., \u0026amp; Steinhardt, J. (2021). Measuring massive multitask language understanding. https://arxiv.org/abs/2009.03300\u003c/li\u003e\n\u003cli\u003eHurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., ... \u0026amp; Kivlichan, I. (2024). Gpt-4o system card. arXiv preprint arXiv:2410.21276. https://arxiv.org/abs/2410.21276\u003c/li\u003e\n\u003cli\u003eJiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., ... \u0026amp; El Sayed, W. (2023). Mistral 7B. arXiv preprint arXiv:2310.06825. https://arxiv.org/abs/2310.06825\u003c/li\u003e\n\u003cli\u003eKitaev, D., Nikita and Klein. (2018). Constituency parsing with a self-attentive encoder. In Y. Gurevych Iryna and Miyao (Ed.), Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: Long papers) (pp. 2676\u0026ndash;2686). Association for Computational Linguistics. https://doi.org/10.18653/v1/P18-1249\u003c/li\u003e\n\u003cli\u003eKolen, M. J., \u0026amp; Brennan, R. L. (2014). Test Equating, Scaling, and Linking: Methods and Practices (3rd ed.). Springer. https://doi.org/10.1007/978-1-4939-0317-7\u003c/li\u003e\n\u003cli\u003eLiang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., \u0026amp; others. (2022). Holistic evaluation of language models. arXiv Preprint arXiv:2211.09110. https://arxiv.org/abs/2211.09110\u003c/li\u003e\n\u003cli\u003eLin, J. and E., Stephanie and Hilton. (2022). TruthfulQA: Measuring how models mimic human falsehoods. In P. and V. Muresan Smaranda and Nakov (Ed.), Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: Long papers) (pp. 3214\u0026ndash;3252). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.acl-long.229\u003c/li\u003e\n\u003cli\u003eLing, C., \u0026amp; Ding, N. (2026). Evaluating the encoding competence of visual language models using uncommon actions. https://arxiv.org/abs/2601.07737\u003c/li\u003e\n\u003cli\u003eLin, S., \u0026amp; Crosthwaite, P. (2024). The grass is not always greener: Teacher vs. GPT-assisted written corrective feedback. \u003cem\u003eSystem\u003c/em\u003e, \u003cem\u003e127\u003c/em\u003e, 1\u0026ndash;19. https://doi.org/10.1016/j.system.2024.103529 \u003c/li\u003e\n\u003cli\u003eMalik, S. and P., Ali and Mayhew. (2024). From tarzan to Tolkien: Controlling the language proficiency level of LLMs for content generation. In A. and S. Ku Lun-Wei and Martins (Ed.), Findings of the association for computational linguistics: ACL 2024 (pp. 15670\u0026ndash;15693). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.findings-acl.926\u003c/li\u003e\n\u003cli\u003eMcCann, B., Keskar, N. S., Xiong, C., \u0026amp; Socher, R. (2018). The natural language decathlon: Multitask learning as question answering. https://arxiv.org/abs/1806.08730\u003c/li\u003e\n\u003cli\u003eNational Education Examinations Authority (NEEA). (2018). \u003cem\u003eChina\u0026rsquo;s standards of English language ability\u003c/em\u003e. Beijing, China: Higher Education Press \u0026amp; Shanghai Foreign Language Education Press. https://cse.neea.edu.cn/html1/report/18112/9627-1.htm\u003c/li\u003e\n\u003cli\u003eNunnally, J.C. \u0026amp; Bernstein, I. H. (1994). Psychometric theory. 3rd ed. McGraw-Hill Companies. ISBN 9780070478497.\u003c/li\u003e\n\u003cli\u003ePaperno, G. and L., Denis and Kruszewski. (2016). The LAMBADA dataset: Word prediction requiring a broad discourse context. In N. A. Erk Katrin and Smith (Ed.), Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: Long papers) (pp. 1525\u0026ndash;1534). Association for Computational Linguistics. https://doi.org/10.18653/v1/P16-1144\u003c/li\u003e\n\u003cli\u003ePeng, B., Goldstein, D., Anthony, Q., Albalak, A., Alcaide, E., Biderman, S., ... \u0026amp; Zhu, R. J. (2024). Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence. arXiv preprint arXiv:2404.05892. https://arxiv.org/abs/2404.05892\u003c/li\u003e\n\u003cli\u003eQwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, Jian, Tu, J., Zhang, J., Yang, Jianxin, Yang, Jiaxi, Zhou, J., ... Qiu, Z. (2025). Qwen2.5 technical report. https://arxiv.org/abs/2412.15115\u003c/li\u003e\n\u003cli\u003eRajpurkar, J. and L., Pranav and Zhang. (2016). SQuAD: 100,000+ questions for machine comprehension of text. In K. and C. Su Jian and Duh (Ed.), Proceedings of the 2016 conference on empirical methods in natural language processing (pp. 2383\u0026ndash;2392). Association for Computational Linguistics. https://doi.org/10.18653/v1/D16-1264\u003c/li\u003e\n\u003cli\u003eRupp, A. A., Templin, J. L., \u0026amp; Henson, R. A. (2010). Diagnostic Measurement: Theory, Methods, and Applications. Guilford Press. ISBN 9781606235287.\u003c/li\u003e\n\u003cli\u003eSang, E. F. T. K., \u0026amp; Buchholz, S. (2000). Introduction to the CoNLL-2000 shared task: Chunking. https://arxiv.org/abs/cs/0009008\u003c/li\u003e\n\u003cli\u003eSrivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, Aditya, Garriga-Alonso, A., Kluska, A., Lewkowycz, A., Agarwal, A., Power, A., Ray, A., Warstadt, A., Kocurek, A. W., Safaya, A., Tazarv, A., ... Wu, Z. (2023). Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research. https://openreview.net/forum?id=uyTL5Bvosj\u003c/li\u003e\n\u003cli\u003eSuzgun, N. and S., Mirac and Scales. (2023). Challenging BIG-bench tasks and whether chain-of-thought can solve them. In J. and O. Rogers Anna and Boyd-Graber (Ed.), Findings of the association for computational linguistics: ACL 2023 (pp. 13003\u0026ndash;13051). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.findings-acl.824\u003c/li\u003e\n\u003cli\u003eThorne, A. and C., James and Vlachos. (2018). FEVER: a large-scale dataset for fact extraction and VERification. In H. and S. Walker Marilyn and Ji (Ed.), Proceedings of the 2018 conference of the north American chapter of the association for computational linguistics: Human language technologies, volume 1 (long papers) (pp. 809\u0026ndash;819). Association for Computational Linguistics. https://doi.org/10.18653/v1/N18-1074\u003c/li\u003e\n\u003cli\u003eUchida, S., \u0026amp; Negishi, M. (2025). Assigning CEFR-J levels to English learners\u0026rsquo;s writing: An approach using lexical metrics and generative AI. Research Methods in Applied Linguistics, 4(2), 100199. https://doi.org/10.1016/j.rmal.2025.100199\u003c/li\u003e\n\u003cli\u003eWan, Y., \u0026amp; Moorhouse, B. L. (2025). Using call annie as a generative artificial intelligence speaking partner for language learners. RELC Journal, 56(2), 489\u0026ndash;498. https://doi.org/10.1177/00336882231224813\u003c/li\u003e\n\u003cli\u003eWang, A. and M., Alex and Singh. (2018). GLUE: a multi-task benchmark and analysis platform for natural language understanding. In G. and A. Linzen Tal and Chrupała (Ed.), Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP (pp. 353\u0026ndash;355). Association for Computational Linguistics. https://doi.org/10.18653/v1/W18-5446\u003c/li\u003e\n\u003cli\u003eWang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., \u0026amp; Bowman, S. R. (2019). SuperGLUE: a stickier benchmark for general-purpose language understanding systems. In Proceedings of the 33rd international conference on neural information processing systems. Curran Associates Inc. https://dl.acm.org/doi/10.5555/3454287.3454581\u003c/li\u003e\n\u003cli\u003eYue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang, W., ... Chen, W. (2024). MMMU: a massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 9556\u0026ndash;9567. https://doi.org/10.1109/CVPR52733.2024.00913\u003c/li\u003e\n\u003cli\u003eZhang, W., Aljunied, S. M., Gao, C., Chia, Y. K., \u0026amp; Bing, L. (2023). M3Exam: A multilingual, multimodal, multilevel benchmark for examining large language models. Proceedings of the 37th International Conference on Neural Information Processing Systems, Nips\u0026rsquo; 23. https://arxiv.org/abs/2306.05179\u003c/li\u003e\n\u003cli\u003eZheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., \u0026amp; Stoica, I. (2023). Judging LLM-as-a-judge with MT-bench and chatbot arena. Proceedings of the 37th International Conference on Neural Information Processing Systems, Nips\u0026rsquo; 23. https://arxiv.org/abs/2306.05685\u003c/li\u003e\n\u003cli\u003eZhong, R. and G., Wanjun and Cui. (2024). AGIEval: A human-centric benchmark for evaluating foundation models. In H. and B. Duh Kevin and Gomez (Ed.), Findings of the association for computational linguistics: NAACL 2024 (pp. 2299\u0026ndash;2314). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.findings-naacl.149 \u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"proficiency assessment, CSE, item response theory, cognitive diagnosis, benchmark evaluation, large language models","lastPublishedDoi":"10.21203/rs.3.rs-8820245/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-8820245/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eLarge language models (LLMs) are increasingly used in language learning and assessment, yet their English proficiency is seldom reported against interpretable proficiency standards. We introduce the China\u0026rsquo;s Standards of English Language Ability (CSE) framework to assess the proficiency levels and subskills of LLMs. The test is referred to as the CSEBench and comprises 624 expert-annotated multiple-choice items across CSE Levels 2\u0026ndash;7. Each item is accompanied by metadata, including difficulty level and subskill labels covering vocabulary, syntax, phonology, and cohesion/discourse. Critically, the dataset includes test responses from 2,050 middle school and and sophomore college students who are learning English as a second language. We evaluate closed-source models, open-source baselines, and enhanced open-source variants incorporating additional supervision and external knowledge. Results show a clear proficiency divide: after mapping model scores to CSE levels, closed‑source models consistently reach CSE Level 6, whereas most open‑source baselines cluster around CSE Levels 3\u0026ndash;4. A follow‑up cognitive diagnostic analysis reveals that while closed‑source LLMs exhibit broad competence across subskills, open‑source models display persistent deficits\u0026mdash;most pronounced in phonology. Crucially, these weaknesses are shown to be substantially reducible through targeted enhancements. CSEBench thus offers a proficiency-interpretable testbed for reporting LLM English ability and diagnosing subskill gaps.\u003c/p\u003e","manuscriptTitle":"Standardized Assessment of LLM English Proficiency","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-02-19 09:21:56","doi":"10.21203/rs.3.rs-8820245/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"af6ba368-495c-4054-8286-8d8c8a5cc5d5","owner":[],"postedDate":"February 19th, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":63093373,"name":"Physical sciences/Mathematics and computing"},{"id":63093374,"name":"Biological sciences/Psychology"},{"id":63093375,"name":"Social science/Psychology"}],"tags":[],"updatedAt":"2026-04-30T17:54:29+00:00","versionOfRecord":[],"versionCreatedAt":"2026-02-19 09:21:56","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-8820245","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-8820245","identity":"rs-8820245","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}