Transforming science marking: A scoping review of auto-markers

Transforming science marking: A scoping review of auto-markers | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Systematic Review Transforming science marking: A scoping review of auto-markers Frank Morley, Emma Walland, Carmen Vidal Rodeiro This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5572868/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 9 You are reading this latest preprint version Abstract This scoping review explored the performance of recent transformer-based auto-markers. We followed a systematic process, adhering to relevant PRISMA guidelines. Our review included recent literature (from 2017 onwards), focusing on English natural language responses on science content in an educational assessment context. A final set of 21 articles was reviewed and coded in depth to answer our research questions which explored the types of auto-marking models being used, the datasets used to fine-tune and test them, and their performance. The most commonly used models in this context were BERT models and BERT variants, which increased in frequency in recent years reaching a peak in 2021. After 2021, papers using GPT models started to appear. The SciEntsBank dataset was the most commonly used to test auto-markers but several other datasets (e.g., ASAP SAS, Beetle) also featured in our review. BERT models generally performed better than previous models on the SciEntsBank dataset. As of yet, GPT models have not been evaluated on SciEntsBank but there was one study in the review that directly compared GPT-3.5 and BERT base and found that GPT-3.5 outperformed BERT base across different items and item types. The review also shows that models that utilise additional forms of data like textbooks and marking rubrics seem to consistently outperform models without these and that recent auto-markers may still present issues in terms of low reliability, lack of explainability and bias. Auto-marking Automatic Short Answer Grading (ASAG) science transformers large language models BERT Figures Figure 1 Figure 2 Figure 3 Introduction Auto-marking is the process of automating the marking of student responses using computational methods. Auto-marking has the potential to ease the time and resource pressures associated with human marking. In high-stakes domains where students’ results have important uses, such as determining entry to university, auto-markers could act as another check on the marks given by examiners. In low-stakes domains, such as formative assessments, auto-markers might relieve workload pressures on teachers, while also providing instant and personalised marking and feedback to students. Short answer questions in subjects such as science have traditionally been difficult to mark by computers due to the “myriad and sometimes unconventional ways in which credit-worthy answers [are] expressed” (Sukkarieh et al., 2005 , p.19). Such questions require students to produce phrases, sentences or short paragraphs of natural language in response to a question or prompt. They are not fully-objective, in the sense that there is more than one way that students could gain credit, in contrast to multiple choice questions where there is a predetermined correct answer. Recent interest and development in the field of artificial intelligence has raised the prospect that computer algorithms are now becoming sophisticated enough to mark short answer questions (Turner, 2024 ). In 2017, a seminal paper on artificial intelligence outlined a new method called ‘transformers’ (Vaswani et al., 2017 ). Transformers use complex computational and statistical methods to represent the semantic meaning of natural language in a numerical format by measuring the relevance of words in a given context. One of these transformer-based model platforms, which became available in 2022, is ChatGPT (GPT stands for Generative Pre-trained Transformer). ChatGPT is a chatbot that uses the models GPT-3.5 and GPT-4. Auto-markers which utilise transformer-based models could be both an effective and an efficient way to mark short answer responses, provided that they can be proven to perform well enough when evaluated against established performance metrics. For example, a review in 2022 found that BERT (Bidirectional Encoder Representations from Transformers), another transformer-based model, had the highest performance of any auto-marker on a widely used dataset (Kusuma et al., 2022 ). However, criticisms have been made of modern auto-markers: most do not assess domain knowledge in student responses vital for subjects such as science (Ramesh & Sanampudi, 2022 ), and neither do many prioritise explainability (i.e., explaining in human terms how the model came to the outputs it did) which is crucial in an assessment context. This article aimed to provide fresh insights into these new transformer-based auto-markers in the domain-specific marking of science responses, while also exploring metrics related to accuracy and other performance criteria such as explainability. In particular, we carried out a high-level overview and summary of recent research on the performance of transformer-based auto-markers in science. This article is primarily a technical overview to inform subsequent research and development in auto-marking through identifying challenges and opportunities with the latest technologies. It also serves as an introduction to auto-marking for interested parties without previous technical expertise in this field. The review addressed four research questions, in the context of recent transformer-based auto-markers (from 2017 onwards) for science content: Which auto-marking models were used and how do they work? Which datasets were used to fine-tune and test the auto-marking models? How well did auto-markers perform on metrics related to accuracy? What other criteria are relevant for evaluating auto-markers, and how did recent auto-markers perform on these? Method We adopted a scoping review methodology for this research. Scoping reviews follow a systematic procedure to identify and chart literature. They serve various aims, for example, to summarise a complex and heterogenous body of research evidence, or to examine the extent and nature of existing research on a specific topic (Tricco, 2016 ). Scoping reviews are a useful tool to generate an overview of existing evidence (both academic and grey literature), which can be valuable to researchers and practitioners. We followed the PRISMA-ScR guidelines for scoping reviews (Tricco et al., 2018 ) and, in particular, we conducted these main steps (Tricco, 2016 ): Formulate the research plan and review objectives (note that there is no research protocol or plan in this review, but details of our search plan and strategy are presented below). Devise eligibility criteria and search for papers (including reference list scanning and grey literature if applicable). Screen papers. Chart and present the results (extract/collect and record the relevant information; present it in tables, diagrams or descriptively). Draw conclusions and implications. Search strategy Our search for peer-reviewed articles and conference papers was conducted using Scopus ( https://www.scopus.com/ ), a comprehensive and multidisciplinary source of literature. We also carried out a non-systematic internet search using Google and Google Scholar to include any relevant grey literature. In accordance with our scope and aims, our Scopus search parameters were that the articles should be published in 2017 or later and be written in English. Table 1 shows the search terms we used. The initial search, carried out on the 18th of March 2024, generated 252 articles. Table 1 Scopus search terms Category Terms Words such as auto, automatic, automated (auto*) AND What is being marked (essay OR writing OR “short answer” OR item) AND Synonyms of marking used in the literature (grading OR scoring OR evaluation OR marking) AND Types of transformer-based models (GPT* OR BERT* OR transformer* OR LLM* OR “large language model”) Note. The asterisk (*) is a symbol that is used in search terms to find words that have the same letters Screening After our initial Scopus search, we used the following eligibility criteria to screen the 252 articles, in accordance with our research aim and scope: Auto-marking in an educational assessment context Natural language responses (i.e., not fully objective items) English responses (i.e., not other languages) Non-language domain knowledge (i.e., subject-related knowledge and understanding rather than the quality of writing) Primary study (i.e., not reviews) Science subjects (i.e., biology, chemistry and physics). Two researchers screened the articles. Abstracts were used for screening Criterion 1, 2 and 3 (and in cases where we were not sure, full papers were consulted). For Criterion 4 through 6, the full papers were used for screening. For Criterion 6, the papers were categorised by subject and those which were not in science subjects removed. MAXQDA 2024 (VERBI Software, 2024 ) was used as a tool to support the organisation, screening and charting of the literature. A summary of the search and screening processes is given in Fig. 1 , in the form of a PRISMA diagram. The search and screening process resulted in 21 articles being included in the review. Charting One researcher with expertise in data science coded the set of 21 articles in depth (see Appendix A for the full list of articles). Codes were developed through the coding process, which was done on MAXQDA 2024. The coding process began with a broad set of pre-existing ideas which were recorded in a code diary. As the coding continued, more detailed codes were added, and these changes were noted down in memos. Alongside this, the quality of the articles was interpreted. No articles were removed due to quality issues from the final pool of 21. Table 2 shows examples from the first three levels in the code hierarchy, although there were up to six levels in some cases. Table 2 Examples of the codes used in the review. Level 1 codes Level 2 codes Level 3 codes Context Item size, education level, country of primary author, publication year, subject For item size: short answer, essay Model name Non-machine learning, machine learning, deep learning, transformer-based For transformer-based: BERT, GPT, other transformer-based Model information Model purpose, type of computer used, reference answer used For model purpose: Main mark prediction, baseline, data augmentation, explainability Evaluation metrics Mark performance in datasets, explainability, bias, ease of deployment, ablation used For mark performance in datasets: SciEntsBank, ASAP SAS, Beetle Results Which auto-marking models were used and how do they work? Figure 2 : Diagram of auto-marking models in this review categorised into model groups. Transformer-based models are highlighted in grey. See Appendix B for the full list of models and studies that used them. This figure was created using SmartArt in Microsoft Word. In the following, we provide technical details of the different auto-marking models in our review, including the number of papers that present the different models or model groups (Table 4 ). Auto-markers operate through taking features from a student’s response and using these to predict the mark. This prediction is usually compared to the ‘true’ mark 1 given by a human examiner, and the difference or similarity between the true and predicted mark gives an indication of how effective the auto-marker is. Auto-markers utilise a variety of different features which can be engineered or embedded. Feature engineering relates to features that can be specified using domain knowledge and extracted by human programmers. Simple examples include word count, frequency of keywords, and number of grammatical errors (Kusuma et al., 2022 ). Responses can also be labelled with relevant aspects of the marking criteria and these can be used as features. An example of feature engineering in our review was found in Wang et al. ( 2019 ), who used a ‘Bag of Words’ approach that counted the frequencies of different words in a student response and used these as features. Feature embedding takes a different approach where features are instead automatically extracted through an algorithm (Ramesh & Sanampudi, 2022 ). The purpose of the embedding algorithm is to represent each word in a student’s response as a unique numerical vector. Each word has a number of dimensions of semantic meaning and a vector represents the location of a word in this semantic space. For example, in the word2vec algorithm, a calculation of the numerical vectors ‘King – Man + Woman’ gives a similar result to the numerical vector for the word ‘Queen’ (Christian, 2021 ). Embedding allows for the meaning of words to be encoded in a numerical representation that can be then input into a statistical model for auto-marking (see machine learning approaches below). As feature embedding is built into transformer-based models, all the studies we reviewed used this approach. Traditional approaches to auto-marking, shown in Table 4 as ‘other auto-markers’, were used prior to machine learning approaches becoming widely adopted. These cover a broad spectrum of methods and include using keywords or keyword synonyms to match to the mark scheme (Burrows et al., 2015 ). These approaches usually entailed a manually programmed, rules-based approach. Table 4 shows that eight studies in our review used these approaches (or they analysed performance metrics from previous studies) alongside transformer-based models (more details can be found in the Appendix B ). Seven studies, as seen in Table 4 and in Appendix B , used machine learning approaches, shown as ‘other machine learning’. Machine learning approaches to auto-marking replace a rules-based approach with the use of statistical models to predict a score based on the responses’ features (either engineered or embedded). An example in our review was the use of a random forest model to predict a response’s mark based on its features (Wang et al., 2019 ). Random forest models arrived at a mark prediction through aggregating the predictions of multiple decision trees. Six studies in our review used deep learning approaches (shown as ‘other deep learning’ in Table 4 and in Appendix B ). Deep learning is a type of machine learning, where the statistical approach used involves a neural network with multiple layers that updates the strength of the connections (the weights) between artificial neurons in the network based on learning from previous predictions (Chollet, 2021 ). In auto-marking, the neural network outputs a prediction of a response based on the features (either engineered or embedded) and compares this prediction to the true mark. The connections between neurons associated with a better prediction are strengthened. Through a successive, iterative process called optimisation, the neural network ‘learns’ to make better predictions. An example in our review of a deep learning model was a type of neural network called Long Short-Term Memory (LSTM) used in Chen & Li ( 2021 ) and Zhu et al. ( 2022 ). Transformers , which are the focus of our review and therefore are utilised in each paper, are a type of deep learning method that specialises in improving the meaning representation of embeddings. This is done through an attention mechanism that refines the numerical vectors in embeddings to take into account how the words, for example in a sentence, relate to each other (Vaswani et al., 2017 ). For the transformer-based model GPT-3 2 the relations between tokens (partial chunks of words) are calculated as attention scores, and these are optimised through deep learning by predicting the next token in a sentence and minimising over time the difference between the predicted and actual token. GPT-3 is trained on 499 billion tokens (Brown et al., 2020 ), roughly equivalent to 374 billion words (OpenAI, 2024 ). Going back to the previous example, while an embedding algorithm like word2vec will represent the word ‘queen’ as a single numerical vector, a transformer-based model might encode the word ‘queen’ differently when it follows the word ‘Elizabeth’ compared to when it follows the words ‘Freddie’ and ‘Mercury’. This process, known as pre-training , gives models like GPT and BERT a contextually rich representation of natural language. Fine-tuning involves training a pre-trained model for a specific task. Fine-tuning is akin to on-the-job-training for the model, allowing it to improve at a specific task like auto-marking. For BERT, an additional neural network component called a classification layer is added (Liu et al., 2023 ). The classification layer allocates the response into one of a set of mutually exclusive classes, which in this case are predicted marks. The student's response is fed into the model, the features are embedded, then the embeddings are processed through the transformers and then passed to the classification layer. The classification layer uses deep learning to improve its mark prediction in relation to the true mark. OpenAI also allows users to fine-tune GPT-3.5 and 4 through the OpenAI API 3 (OpenAI, 2024 ). Prompt engineering is an alternative to fine-tuning where instead of fine-tuning a pre-trained model on task-specific dataset, a set of well-crafted prompts allow the pre-trained model to improve (Brown et al., 2020 ). Prompt engineering lessens the need for large task specific datasets of thousands of examples that are usually needed for fine-tuning and requires less technical expertise to implement. Examples from the review of fine-tuned and prompt engineered models are presented in the next few paragraphs. Table 4 Number of papers utilising models from each model group Model group † Number of papers Other auto-markers 8 Other machine learning 7 Other deep learning 6 Other transformer-based 7 BERT 20 GPT 5 Total number of distinct papers 21 † The groups are not mutually exclusive and papers in the review could have used multiple models. Table 4 shows that 20 out of 21 studies in our review used BERT models (see Appendix B for more details), which suggests that it is a frequently used transformer-based approach for science content (although it should be noted that there is usually a lag between research and publication). Nineteen studies used the standard BERT base model created by Google, and two used BERT large, a bigger variant with 24 instead of 12 transformer layers alongside other modifications. Sixteen studies used a fine-tuned BERT base model, while four did not tune BERT base. Four studies augmented BERT base with other forms of data, including the marking rubric (Condor et al., 2022 ), an academic journal in science education (Liu et al., 2023 ), questions and/or responses (Liu et al., 2023 ; Sung et al., 2019a ) and textbooks (Sung et al., 2019a ; Wang et al., 2019 ). One study augmented BERT base by combining it with another deep learning model (Zhu et al., 2022 ). Six studies used variants of BERT, which included RoBERTa (Robustly Optimised BERT), DistilBERT (Distilled BERT), AlBERT (A Lite BERT for Self-supervised Learning of Language Representations), SciBERT (Science BERT), and Longformer base. SciBERT takes BERT base and further pre-trains it on science-related content (Liu et al., 2023 ), while the other variants use different model architectures to BERT base. Table 4 shows that five studies used GPT models (see Appendix B for details). One study used GPT-1 and GPT-2 (Zhu et al., 2022 ) and four studies used GPT-3.5. Of these, one used a fine-tuned GPT-3.5 (Li et al., 2023 ), while three used a prompt engineering approach for GPT-3.5 (Cochran et al., 2023 ; Lee et al., 2024 ; Li et al., 2023 ). Only one study used GPT-4 featuring a prompt engineering approach (Lee et al., 2024 ). There were no studies that fine-tuned GPT-4. Lee et al. ( 2024 ) also utilised the problem context and marking rubric in their testing of different prompt engineering approaches. According to Table 4 , seven studies utilised transformer-based models excluding BERT and GPT (shown as ‘other transformer-based’). More details about these models are provided in Appendix B . Our systematic search did not find any studies which utilised other new state-of-the-art transformer-based models developed by technology companies, such as Google Gemini, Anthropic’s Claude or Meta Llama. Latif and Zhai ( 2024 ) stated that they had considered using Gemini but, at the time, Gemini did not provide a public API to fine-tune the model. Over the next year we would expect to see use of these models in published studies. Frequency of models over time Figure 3 shows the frequency of each model group over time in our reviewed sample. BERT models increased in frequency, reaching a peak in 2021. GPT models have been adopted more recently. Other auto-markers, other deep learning models, and other machine learning models, were steadily used from 2020 to 2022. However, in 2023, other deep learning and machine learning models were not used, which might indicate a decrease in their popularity in relation to transformer-based methods. Which datasets were used to fine-tune and test the auto-marking models? Table 5 shows the science datasets used in the articles included in our review. Many of these articles contained datasets from other disciplines such as computer science alongside science datasets. For the purposes of our review, as the focus was on science questions, we excluded datasets from other domains. For example, we excluded the widely used Mohler / University of North Texas dataset which focuses on computer science and data science items, and which was used alongside science datasets in Chen and Li ( 2021 ) and Zhu et al. ( 2022 ). The SciEntsBank and Beetle datasets included only categorical questions (or items), taking the form of 2-way items (‘correct’ or ‘incorrect’), 3-way items (‘correct’, ‘contradictory’ or ‘incorrect’), or 5-way items (‘correct’, ‘partially correct’, ‘contradictory’, ‘irrelevant’ or ‘not in the domain’). Three test sets were available: tests of unseen-answers (UA), unseen-questions (UQ), and unseen-domains (UD). Other datasets, BEAR (Berkeley Evaluation and Assessment Research), ASAP SAS (Automated Student Assessment Prize Short Answer Scoring), and TIMMS (Trends in International Mathematics and Science Study), had either 3 or 4 mark items. The science phenomena and science argumentation datasets used levels-based marking of proficiency. The PASTA (Potential of An Automatically Scored Three-dimensional Assessment System) project and Mathematical Thinking in Science project datasets either contained questions with 3–4 marking levels or 5–10 binary marking criteria. Table 5 Science datasets selected for analysis and their characteristics Dataset Number of papers Educational level (US) Item types SciEntsBank 10 3rd to 6th grades 2-way, 3-way, 5-way Unseen-answers (UA) Unseen-questions (UQ) Unseen-domains (UD) BEAR 1 Pre-college test 4 marks ASAP SAS 2 7th to 10th grades 3 marks Beetle 2 Unclear 2-way, 5-way UA, UQ Science phenomena dataset 2 Middle school 2 or 3 levels Science argumentation tasks 1 5th to 8th grades 3 or 4 levels TIMMS items 1 4th and 8th grades 3 marks Rainwater Runoff dataset 1 6th grade 1 mark PASTA project and Mathematical Thinking in Science project datasets 1 High school 5–10 binary criteria 3 or 4 levels OpenStax Biology textbook questions 1 High school Unclear How well did auto-markers perform on metrics related to accuracy? Among the studies and datasets included in the review, the metrics described below were used. Accuracy Accuracy can be defined as the ratio of correct predictions or proportion of correctly scored answers (see, for example, Shalev-Shwartz and Ben-David ( 2014 ) for more details). M-F1 (Macro-averaged-F1) and W-F1 (Weighted-averaged-F1) These are based off the F1 score (see, for example, Sokolova et al. ( 2006 )), which is a fundamental metric for evaluating classification models, and is calculated as the harmonic mean of the precision and recall of a classification model. It is used when there is a binary classification task (e.g., present / absent; correct / incorrect). The F1 score is more robust to class imbalance (that is, when the number of instances in the different groups (or classes) is very different) than accuracy. The M-F1 and W-F1 are used in the case of multi-class classifications (e.g., correct / partially correct / incorrect). The M-F1 calculates the F1 score for each class, and then calculates the average without considering the proportion for each class in the dataset. The W-F1 calculates the F1 score for each class, and then calculates the average considering the proportion for each class in the dataset. CK (Cohen's Kappa) and QWK (Quadratic Weighted Kappa) Cohen’s Kappa (Cohen, 1960 ) is defined as the level of agreement between two ratings (e.g., between a human marker and the auto-marker) beyond random chance. This is usually a more robust measure than accuracy as it accounts for random chance. QWK is an extension of Cohen’s Kappa that is designed to handle ordinal data, where the categories have a natural order (Cohen, 1968 ). Both CK and QWK range from − 1 to 1, where 1 indicates perfect agreement, 0 indicates no agreement beyond chance, and negative values indicate less agreement than expected by chance. EAR (Exact Agreement Rate) and SER (Serious Error Rate) The EAR is the level of exact agreement between the predicted and true mark. The SER is the level of disagreement between the predicted and true mark, where the absolute difference is greater than 1 (Chang et al., 2021 ). In the following, the performance of the auto-markers used to score/classify the items in the datasets listed in Table 5 is discussed. SciEntsBank dataset Table 6 shows the performance of different models in the SciEntsBank dataset. The models shown provide “state of the art” (SOTA) performance, meaning they are the best performing in comparison to other models on the same items. Models are also compared to “baselines”, which are models that offer past SOTA performance which current studies seek to surpass. TS + SF (Token level Features + Sentence level Features) is a model proposed by Saha et al. ( 2018 ) which utilises feature engineering and deep learning. In 2-way UQ and UD tasks from the SciEntsBank dataset, TS + SF outperforms the newer BERT models, which are based on feature embedding and transformers. Their study stated that, in contrast to embeddings, “[our approach] token level hand-crafted features can be fairly domain independent and are less affected by non-sentential 4 forms” (Saha et al., 2018 , p. 2). However, TS + SF does not perform as well as the BERT models on UQ and UD on 3-way and 5-way items, suggesting that TS + SF might only be a preferable model for out-of-domain data in one mark items. MDA-ASAS (Multiple Data Augmentation Strategies for Automatic Short Answer Scoring), a BERT model enhanced with MDA, proposed by Lun et al. ( 2020 ), had SOTA performance for 2-way and 5-way items on the UA domain. This suggests that their approach of data augmentation (artificially boosting the size of the training dataset) might be a promising avenue for research. Table 6 State of the art (SOTA) models on SciEntsBank dataset Items Metrics SOTA models † Article 2-way, UA Accuracy MDA-ASAS (BERT base) Lun et al. ( 2020 ) M-F1 W-F1 AlBERT Large Poulton and Eliens ( 2022 ) 2-way, UQ Accuracy TF + SF [-question] Saha et al. ( 2018 ) M-F1 W-F1 2-way, UD Accuracy TF + SF [+ question] Saha et al. ( 2018 ) M-F1 W-F1 3-way, UA Accuracy RoBERTa Large MNLI Camus and Filighera ( 2020 ) M-F1 W-F1 3-way, UQ Accuracy BERT base uncased Ghavidel et al. ( 2020 ) M-F1 RoBERTa Large MNLI Camus and Filighera ( 2020 ) W-F1 BERT base uncased Ghavidel et al. ( 2020 ) 3-way, UD Accuracy RoBERTa Large MNLI Camus and Filighera ( 2020 ) M-F1 W-F1 5-way, UA Accuracy MDA-ASAS (BERT base) Lun et al. ( 2020 ) M-F1 BERT base hybrid Zhu et al. ( 2022 ) W-F1 MDA-ASAS (BERT base) Lun et al. ( 2020 ) 5-way, UQ Accuracy BERT base hybrid Zhu et al. ( 2022 ) M-F1 W-F1 5-way, UD Accuracy BERT base cased Ghavidel et al. ( 2020 ) M-F1 Variant of Funnel Transformer Chen and Li ( 2021 ) W-F1 BERT base uncased Ghavidel et al. ( 2020 ) UA: Unseen answers, UQ: Unseen questions, UD: Unseen domains. † SOTA is defined by being the highest performer in any one of the metrics used. The RoBERTa Large MNLI (Multi-Genre Natural Language Inference) model proposed by Camus and Filighera ( 2020 ) had SOTA performance on the 3-way dataset across UA, UQ, and UD items. This model utilises a method called transfer learning, where the pre-trained RoBERTa Large model was fine-tuned on another dataset called MNLI and then fine-tuned again on the 3-way auto-marking dataset. This model was not tested on 2-way and 5-way responses, but this could be an area for further research. Overall, this suggests that transfer learning could be a promising approach in developing auto-markers. The BERT base hybrid model which utilised BERT base alongside additional deep learning neural networks proposed by Zhu et al. ( 2022 ) had SOTA performance in 5-way UA and UQ items. A BERT base model without any additional changes was utilised in Ghavidel et al. ( 2020 ) and achieved SOTA performance in 5-way UD items (both the cased and uncased versions of the model). Finally, a (non-BERT and non-GPT) transformer-based model, Variant of Funnel Transformer, achieved SOTA in 5-way UD items. In conclusion, the BERT models (including variants such as RoBERTa) were nearly always the best performing models on tasks from the SciEntsBank dataset. Only two other models, TF + SF and Variant of Funnel Transformer, outperformed BERT models. Unfortunately, there were no studies that compared the performance of BERT to GPT-3.5 or GPT-4 on such tasks. This would be a welcome addition to the literature as the SciEntsBank dataset is commonly used to benchmark models. In the following sections, tests are of unseen answers (UA) unless specified otherwise. Table 7 shows the performance of the auto-markers in all other datasets described in Table 5 . BEAR dataset Table 7 shows that on the BEAR dataset, Modified BERT (Graph Rubric) performed best (Condor et al., 2022 ). Graph rubric was able to capture the structure of the responses, i.e., a level 2 answer being ‘higher’ than a level 1 answer. Capturing this structure was made possible through the Node2Vec algorithm which is able to convert the levels in the mark scheme, a node/graph structure, into a format readable by BERT. Node2Vec outputs were used to further pre-train BERT alongside the marking rubric text for the Graph Rubric model. ASAP SAS dataset For ASAP SAS, there were two sets of items which were used in separate articles to test different auto-markers. For the first set of items (Set A), Condor et al. ( 2022 ) modified BERT with graph rubric performed best on the W-F1 score. When using Cohen’s Kappa as a measure of agreement between the auto-marking model and the subject matter experts, modified BERT with random rubric had the highest performance. Random rubric, unlike graph rubric, did not utilise Node2Vec, but instead used random sampling during the further pre-training. The performance of a different auto-marker was also evaluated using the ASAP-SAS dataset in the article by Li et al. ( 2023 ) on another set of items (Set B). A model called T5 was fine-tuned to produce marks and rationales for unseen answers. The rationales used in training T5 were generated using GPT-3.5. The auto-marker proposed by Li et al. ( 2023 ) performed worse in all metrics than the baseline models. The Longformer baseline model performed best on the M-F1 and QWK metrics, and the BERT base model had the highest performance in terms of accuracy. Table 7 SOTA models on other datasets Dataset Items Metrics SOTA models Article BEAR 4 marks W-F1 BERT (Graph Rubric) Condor et al. ( 2022 ) CK ASAP SAS Set A (3 marks) W-F1 BERT (Graph Rubric) Condor et al. ( 2022 ) CK BERT (Random Rubric) Set B (3 marks) ‡ Accuracy BERT base uncased Li et al. ( 2023 ) M-F1 Longformer QWK Beetle 2-way, UA W-F1 AlBERT Large Poulton and Eliens ( 2022 ) 5-way, UA M-F1 Custom-made transformer Chen and Li ( 2021 ) W-F1 5-way, UQ M-F1 W-F1 BERT base Science phenomena Set A (2 levels) Accuracy GPT-4 (FS_CoT_CR) Lee et al. ( 2024 ) † F1 Set A (3 levels) Accuracy GPT-4 (ZS_CoT_CR) F1 Set B (2 levels) ‡ Accuracy SR1-BERT Liu et al. ( 2023 ) Set B (3 levels) Science argumentation 3 or 4 levels Accuracy SR2-SciBERT Liu et al. ( 2023 ) TIMMS 3 marks SER Machine Concept Map Chang et al. ( 2021 ) † EAR BERT base Rainwater Runoff 1 mark M-F1 Multilingual L12 (data generated by GPT-3.5) Cochran et al. ( 2023 ) † PASTA project & Mathematical Thinking in Science 5–10 binary criteria Accuracy GPT-3.5-Turbo Latif and Zhai ( 2024 ) 3 or 4 levels Accuracy GPT-3.5-Turbo OpenStax Biology textbook questions Unclear Accuracy ml-BERT Wang et al. ( 2019 ) F1 † For Lee et al. ( 2024 ), Chang et al. ( 2021 ) and Cochran et al. ( 2023 ), metrics were not averaged across items, meaning that the SOTA represents the highest performance on the most number of items. ‡ Different student responses were used in Lee et al. ( 2024 ) and Liu et al ( 2023 ), and in Condor et al. ( 2022 ) and Li et al. ( 2023 ), preventing direct comparison between the studies. Beetle dataset The performance of the auto-markers evaluated in the 2-way items was similar across the models, with AlBERT Large showing slightly better performance on the W-F1 score. The custom-made transformer model described in Chen and Li ( 2021 ) outperformed BERT base for 5-way UA items on the M-F1 and W-F1 metrics. This was also the case for the M-F1 score for UQ items, but for the W-F1 score for UQ items BERT base outperformed the custom-made transformer. Science phenomena dataset Lee et al. ( 2024 ) used this dataset to compare the performance of various prompt engineering approaches for GPT-4 on a set of items (Set A), and compared GPT-4 to GPT-3.5. GPT-4 yielded higher accuracy than using GPT-3.5 in all items but one, and independently of the different prompt engineering strategies used. For example, ‘Few-Shot’ learning aims for models like GPT-4 to make predictions with only a few examples, while ‘Zero-Shot’ learning aims of predictions to be made with no examples. Chain-of-Thought prompting sets out a ‘reasoning path’ for models like GPT-4 to tackle complex reasoning tasks such as marking, using prompts like “let’s think step by step” (Lee et al., 2024 ). The two most complex strategies, Zero-Shot Chain-of-Thought problem context and rubric (ZS_CoT_CR) and Few-Shot Chain-of-Thought problem context and rubric (FS_CoT_CR), had, generally, the highest performance. ZS_CoT_CR performed best on items with two levels of proficiency, while FS_CoT_CR performed best on items with three levels. Overall, these results suggest including the problem context and rubric alongside a Chain-of-Thought approach improves the performance of prompt engineering strategies. This links to other findings regarding the benefits of utilising a marking rubric for auto-marking (Condor et al., 2022 ). Liu et al. ( 2023 ) investigated, using the same dataset but on another set of items (Set B), how training language models with different contextual data can impact performance. The SR1-BERT model, trained with in-domain data, had an accuracy slightly higher than the accuracy of a base BERT model. Science argumentation tasks In Liu et al. ( 2023 ), SR2-SciBERT was the SOTA model on a dataset with science augmentation tasks. SciBERT is a BERT variant which has been further pre-trained on a scientific corpus of text, and this model was fine-tuned on SR2 (the dataset of science argumentation tasks). This study confirmed the effectiveness of using domain-specific data to train auto-marker models to improve their performance and suggests that adapting language models to science education could be worthwhile. TIMMS items Table 7 shows the performance of the auto-marker proposed by Chang et al. ( 2021 ) on just three items from TIMMS. The BERT base model outperformed the proposed method (the machine generated concept map) on the EAR metric on all items. The proposed method outperformed BERT base on the SER metric on two out of three items. The Chang et al. ( 2021 ) model does not require large quantities of training data so it might be suitable when there is not a lot of resources to train the auto-marker. Another advantage is that it utilises a non-machine learning based approach where a concept map is generated using a statistical formula, and this is then used to mark the student responses. In contrast to BERT base, the concept map and the features used to assign the mark are interpretable. Rainwater Runoff dataset In Cochran et al. ( 2023 ), two methods were used to generate synthetic data (GPT-3.5 and self-augmentation) and this data was used to fine-tune a BERT-based language model.. Performance from both models was better than performance from the BERT base model. The use of GPT-3.5 to generate data seemed to provide some advantages versus the self-augmented method. This highlights that GPT-3.5 is more versatile than BERT in being able to do a broader range of tasks such as data augmentation. PASTA project and Mathematical Thinking in Science project datasets The results from the experiments using these datasets show that the accuracy of the GPT-3.5-turbo model was higher than that of the BERT (baseline) model. In particular, GPT-3.5-turbo consistently outperformed BERT in the multi-class tasks (the 5–10 binary criteria items). Latif and Zhai ( 2024 , p. 8) suggested that “the architecture of GPT-3.5-turbo, the large amount of training data, or its innate skills to comprehend context more effectively may be responsible for its continual performance improvement”. OpenStax Biology textbook questions Wang et al. ( 2019 ) used a selection of additional data (e.g., biology textbooks) to train the BERT model through meta-learning (a step added after pre-training but before fine-tuning). This allowed their model to gain more in-domain science representation. Results indicate that the proposed ml-BERT model had better accuracy and F1 scores when compared to all baseline models. In particular, a BERT base model without meta-learning only achieved comparable performance with a random forest model, but with meta-learning it was able to outperform all baseline models. The findings from Wang et al. ( 2019 ) show the potential benefit to marking accuracy of using techniques like meta-learning to boost the model’s specific domain ‘knowledge’. What other criteria are relevant for evaluating auto-markers, and how did recent auto-markers perform on these? We found that the main focus of auto-marking research has been on classification, however, there are other evaluation criteria that should be considered when applying such technology in educational assessment settings. Aloisi ( 2023b ) discussed in detail three main threats to auto-marking in assessment (particularly in high-stakes settings) namely reliability, explainability and bias, which together address the overarching question of ethics. This serves as a useful organising framework for our results in this section. We added validity alongside reliability, as the two are closely linked. We also consider the ease of deployment of auto-markers. Reliability and validity Low reliability and validity presents an issue for recent transformer-based auto-markers. Reliability in this context refers to the consistency of results, for example, the extent to which the marks produced would be the same under similar conditions. Validity refers to the process of ensuring that the marks produced are suitable for their intended uses and interpretations. Aloisi ( 2023a , p. 1) defines low reliability in the context of AI systems as meaning that “small variations in the input may result in large differences in the output ” (italics in original), the input being the student responses and the output being the predicted mark. For example, Filighera et al. ( 2023 ) found that adding adverbs or adjectives to student answers decreased BERT’s accuracy. For human markers, reliability is ensured through training, experience, and working within communities of practice (Aloisi, 2023b ; Johnson, 2014 ). Whilst auto-markers are trained, instead of being taught how to mark , they are set the goal of improving their prediction accuracy by finding out which patterns in language are associated with high or low mark answers. This results in ‘spurious correlations’, where features which would not be relevant for the mark scheme are used by auto-markers to decide a mark (Filighera et al., 2023 ). Auto-markers may focus on superficial aspects of the text (e.g., punctuation, grammar and spelling) rather than higher-order constructs such as content, logic and coherence (Aloisi, 2023b ). This threatens construct validity as well as reliability, as auto-markers might not be rewarding the intended constructs consistently One method which may somewhat lessen the impact of construct-irrelevant features is increasing the domain-specific ‘knowledge’ of models. Four studies in our review used various techniques such as further pre-training and meta-learning to incorporate domain-specific data such as the marking rubric, science journals, and textbooks into the model (Condor et al., 2022 ; Liu et al., 2023 ; Sung et al., 2019a ; Wang et al., 2019 ). However, although adding domain-relevant information resulted in small improvements in performance, if similar levels of performance can be reached by models without this information, then this might call into question the construct-relevance of the features used in these models. New developments such as Retrieval Augmented Generation (RAG), where models such as GPT-4 are able to access a database of factual content (Lewis et al., 2020 ) could be an area of investigation. Explainability A pressing issue with many auto-markers is that they may not be able to explain how they arrived at a certain mark or provide quality feedback. Allowing students to question the marks they have received and to receive feedback is an important part of assessment ethics (Poulton & Eliens, 2022 ). Furthermore, to build trust it seems necessary that there is some transparency and that stakeholders are able to understand how auto-markers work and how marks were determined (Ghavidel et al., 2020 ; Gulson et al., 2022 ; Latif & Zhai, 2024 ; Lee et al., 2024 ). This is especially important in formative settings, where teachers need to understand the rationale behind the marks given in order to provide the right support (Lee et al., 2024 ). As mentioned, assessing the construct validity of features also depends on knowing which features the model is using. The auto-marking models in this review had various degrees of explainability. The most explainable models used simple rule-based approaches as found in Becker et al. ( 2021 ) and Chang et al. ( 2021 ). In these examples, the inner workings of the model can be directly interpreted, and the decision-making process a model undertakes can be directly understood and predicted. As the complexity of the model increases, additional methods need to be implemented to ensure explainability. For example, Poulton and Eliens ( 2022 ) use statistical methods such as SHAP 5 (see Lundberg & Lee, 2017 for more information) to try and interpret the most important features in different BERT models. However, these models do not show in detail how the different features were assessed and weighted in the same way a human marker might be able to explain. More recently, the generative aspect of GPT has led to research claiming that GPT-3.5 and GPT-4 can create explainable marks through for example ‘rationale’ generation (Lee et al., 2024 ; Li et al., 2023 ). However, it is not necessarily clear that the ‘rationale’ precisely reflects what the mark was based on. These models are also known to produce inaccurate outputs known as ‘hallucinations’, limiting the extent to which models can be trusted. More complex prompt engineering such as Chain-of-Thought reasoning lessens, but does not solve, this issue (Lee et al., 2024 ). As the model works out a problem, it is prompted to go through a set of intermediate steps that facilitate reasoning, and by doing so it notes down its own reasoning in a ‘chain of thought’ process. However, the model’s reasoning could still be wrong and this would need to be checked. Ultimately, in order for model explanations to be useful in for example formative assessment, the models will have to develop so that users can be confident that their explanations are accurate and meaningful. Bias In the context of AI systems, bias refers to the systems treating certain groups of people more favourably or discriminating against them based on their characteristics, for example, sex or ethnicity (Aloisi, 2023a ). Transformer-based auto-markers use deep learning to find language which is associated with high and low mark answers and base their mark on this. This can lead to bias because if certain demographic groups are more likely to use some words over others, for example more masculine or feminine examples, these patterns could lead to bias against certain groups (Christian, 2021 ). Another example is where students sharing a particular linguistic background will use vocabulary or make errors in English which were not well represented in the training data, and so their correct answers are not recognised as correct. Filighera et al. ( 2023 ) notes that students with developmental language disorder might express themselves differently and be systematically disadvantaged. There are other documented cases where artificial intelligence and machine learning systems are known to be biased towards or against certain groups (Aloisi, 2023b ; Latif & Zhai, 2024 ). Understanding how models are trained and how decisions are arrived at is likely a step towards evaluating how bias is introduced and replicated. Rather than treat the models as black boxes, we need to consider the consequences of the training procedures (Filighera et al., 2023 ). Only one study in our review, the Latif and Zhai ( 2024 ) study, mentioned that they had taken into account bias against demographic groups within their model training. However, the study did not give much detail as to how they did this. This relationship between construct relevance, explainability, and bias can be formulated in five questions to think about in relation to auto-marking: Can the mark be explained? How useful and accurate is this explanation? Was the mark based on construct-relevant features? How susceptible are the features to adversarial input / gaming the system? Do the features lead to bias against certain demographic groups? Other ethical issues Although technology is advancing, a question remains as to whether auto-markers will ever be able to replicate human intelligence accurately, and whether indeed they should (Aloisi, 2023b ). In shifting to auto-marking, we risk losing the positive human elements of marking. In many contexts, such as marking high stakes exams, when humans mark student work, they may feel morally responsible for the marks they award. There could be negative consequences for them and the students if they make poor decisions. A shift to auto-markers might lead to a loss of valuable human elements of marking, such as morality, responsibility, ethics and conscience. Previous research on examiner cognition in the context of high stakes exam marking in England has shown that good examiners are likely to take a lot of care and responsibility when marking student work, as they are aware of the implications of the results for students’ life chances (Lockyer, 2019 ). There are various other ethical issues to consider in relation to auto-markers attempting to mimic human marking. For example, the role that marking plays for human teachers. It is possible that the process of teachers marking student work is beneficial to their practice and that using auto-markers may deskill teachers (Gulson et al., 2022 ). Some authors, as cited in Latif and Zhai ( 2024 ), are concerned about auto-markers leading to reduced human critical engagement in learning, causing a loss of critical thinking skills and personal interactions in education. On the other hand, there could be benefits of teachers’ changing roles due to auto-markers, for example, they could shift their attention to offering more personalised support. Further research exploring the interaction between humans and auto-markers and the relationship with learning is needed. The issues discussed above highlight some limitations of auto-markers. The evidence currently suggests that humans need to remain involved, and that auto-markers could support rather than replace humans. This view was indeed expressed by many authors in our review (see, for example, Condor, 2020 ; Filighera et al., 2023 ). Ease of deployment The ease in which auto-markers can be deployed is another aspect of performance that should be considered. There are three aspects to this: availability of data, computation, and expertise. Transformer-based models such as BERT can take hours, days, or more to fine-tune depending on the size of the dataset, the computational power available, and the expertise of the human programmer (Brown et al., 2020 ). Fine-tuning often involves a lot of trial-by-error and experimentation before satisfactory performance can be gained (Chollet, 2021 ). As more aspects are added to the model, such as further pre-training (Liu et al., 2023 ), or a mark scheme rubric (Condor et al., 2022 ), these extra levels of complexity make the model more cumbersome to train and deploy. These are not necessarily difficult problems to overcome but are issues to think about when, for example, designing pilot projects. Availability of publicly accessible data is also an issue mentioned in studies we reviewed (e.g., in Sung et al. ( 2019a )). GPT-3.5 and GPT-4 represent a general advance over BERT in ease of deployment. Prompt engineering allows these models to be operationalised with comparatively little data, less expertise, and fewer computational resources (Brown et al., 2020 ). Conclusion and discussion This review set out to explore the performance of recent transformer-based auto-markers on science content. Overall, we found that the most commonly used set of models in this context were BERT, which increased in frequency in recent years reaching a peak in 2021. After 2021, papers using GPT models started to appear and we anticipate more papers will be published in this area in the near future. GPT-3.5 and GPT-4 potentially offer benefits over BERT in terms of marking performance, ease of deployment with prompt engineering, and other capabilities such as generative explainability and data augmentation. Generally, larger models such as GPT-4 tend to have increased computational time. Larger models have increased financial costs over smaller models like BERT, for both model training and implementation in marking. However, research is only just beginning to be published using GPT-3.5 and GPT-4, making conclusions on marking performance tentative, but also meaning that the full benefits of these GPT models have not yet been realised. The results show that all of the datasets used in the articles included in the review were collected in the United States, with the most frequent being the SciEntsBank dataset. The SciEntsBank dataset covers science topics, and includes 2-way, 3-way and 5-way items. There are tests of unseen answers, domains and questions and covering US grades 3 to 6. Our review highlighted the various metrics that can be used to evaluate auto-marker performance, depending on the type of model, with the most commonly used in our review being accuracy, M-F1 and W-F1. This indicates that most of the models were evaluated in terms of classification tasks (e.g., whether an outcome is a categorical value such as correct, incorrect or another category such as partially correct), rather than their ability to predict multiple numerical marks in a regression task. Evaluating a response as correct or incorrect might have limited applications in terms of assessment, and these models might only be appropriate for quite narrow science content questions with a low tariff. There are also some methodological constraints in the accuracy metric due to imbalanced classes. In addition, while auto-markers were compared to the ‘true mark’, how accurate a human marking the same content would be by comparison was not measured in the studies we reviewed. BERT models performed better than previous models on the SciEntsBank dataset. Unfortunately, there were no studies that compared the performance of BERT to GPT-3.5 or GPT-4 on the SciEntsBank dataset, and we suggest this would be a welcome addition to the literature as this dataset is commonly used to benchmark models. In a case where BERT and GPT-3.5 were compared on the same dataset, GPT-3.5 performed better. On another dataset, GPT-4 generally outperformed GPT-3.5, while GPT-4 models which utilised the problem context and marking rubric along with a Chain-of-Thought prompt engineering approach outperformed other prompting strategies. In addition, Few-Shot learning provided gains over Zero-Shot learning in some but not all cases. In both cases (BERT and GPT), the performance was higher when there were fewer classifications, which is to be expected as classifying answers into fewer categories is an easier task and there is less scope for error. For BERT models, the results indicate that using additional training data like textbooks and marking rubrics could lead to improved performance. Overall, models that utilise other forms of data like textbooks and marking rubrics seem to consistently outperform models without these. However, using textbooks introduces a risk of bias, as models could benefit students using those textbooks rather than others. Other than using metrics such as accuracy or F1 scores to evaluate auto-markers, these should also be evaluated in other ways, for example, in terms of ethics, explainability and accountability. The review has shown that transformer-based auto-markers may have issues in terms of low reliability, lack of explainability and bias. For example, low reliability can be caused by auto-markers assigning different marks based on slightly different inputs. This also presents a threat to validity, for example where decisions are made based on ‘spurious correlations’. Methods that can lessen (although as of yet not completely eliminate) this threat include increasing the domain-specific ‘knowledge’ of the models, e.g., through pre-training and meta-learning. Four studies that were reviewed tested methods of improving the explainability of auto-markers, but they still did not reach the level of being able to explain their decisions as well as human markers could. We found that GPT models can generate ‘explanations’ but these need to be checked and verified, thus weakening their utility in practice. Finally, bias in auto-markers remains a concern. These considerations have so far been secondary to increasing the marking accuracy of auto-markers. As auto-markers are increasingly used outside of research settings and in real-world situations, issues relating to ethics, explainability and accountability should become more central to how auto-markers are developed. In our closing paragraph, we note the limitations of this review. Firstly, our search was limited to one database (Scopus), and we may have missed some articles that were not indexed in it. As a scoping review, rather than a full systematic review, we have not comprehensively covered all available literature in this area. Secondly, our scope was limited to science content; there may be developments in auto-markers in other subject domains that were not reflected in this review. Finally, we focused our search on transformer-based technology. There might be other technologies that do not use transformers that could offer something of value for auto-marking. Declarations Competing interests The authors have no relevant financial or non-financial interests to disclose. Funding The authors declare that no funds, grants, or other support were received during the preparation of this manuscript. Author Contribution F.M. and E.W. performed study design, search, and screening. F.M. coded the studies. F.M., E.W., and C.V.R. analysed and wrote the results. All authors read, reviewed, and approved of the final manuscript. Data Availability The findings used for this scoping review are available in Appendix A, which shows the full list of reviewed papers. These can be accessed through Scopus. References Aloisi, C. (2023a). AI and exam marking: Exploring the difficult questions of trust and accountability. AQA . https://www.aqa.org.uk/about-us/our-research/blog/ai-and-exam-marking-exploring-the-difficult-questions-of-trust-and-accountability. Aloisi, C. (2023b). The future of standardised assessment: Validity and trust in algorithms for assessment and scoring. European Journal of Education, 58 (1), 98-110. https://doi.org/10.1111/ejed.12542. Becker, J. P., Kahanda, I., Kazi, N. H. (2021). WIP: Detection of Student Misconceptions of Electrical Circuit Concepts in a Short Answer Question Using NLP . 2021 American Society for Engineering Education (ASEE) Annual Conference, Virtual Meeting. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D. (2020). Language models are few-shot learners . https://arxiv.org/abs/2005.14165. Burrows, S., Gurevych, I., & Stein, B. (2015). The eras and trends of automatic short answer grading. International Journal of Artificial Intelligence in Education, 25 , 60-117. Camus, L., & Filighera, A. (2020). Investigating transformers for automatic short answer grading. Artificial Intelligence in Education: 21st International Conference, Ifrane, Morocco. Chang, T. H., Chen, J. L., Chou, H. M., Bai, M. H., Hsu, F. Y., & Chen, Y. C. (2021). Automatic Scoring Method of Short-Answer Questions in the Context of Low-Resource Corpora. 2021 International Conference on Asian Language Processing (IALP), Singapore. Chen, S., & Li, L. (2021). Incorporating question information to enhance the performance of automatic short answer grading. Knowledge Science, Engineering and Management: 14th International Conference, KSEM 2021, Tokyo, Japan. Chollet, F. (2021). Deep learning with Python . Simon and Schuster. Christian, B. (2021). The alignment problem: How can machines learn human values? Atlantic Books. Cochran, K., Cohn, C., Rouet, J. F., & Hastings, P. (2023). Improving automated evaluation of student text responses using gpt-3.5 for text data augmentation. International Conference on Artificial Intelligence in Education, Tokyo, Japan. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20 (1), 37-46. Cohen, J. (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70 (4), 213. Condor, A. (2020). Exploring Automatic Short Answer Grading as a tool to assist in human rating. In I. I. Bittencourt, M. Cukurova, K. Muldner, R. Luckin, & E. Millán, Artificial Intelligence in Education AIED 2020, Cham. Condor, A., Pardos, Z., & Linn, M. (2022). Representing scoring rubrics as graphs for automatic short answer grading. International Conference on Artificial Intelligence in Education, Durham, UK. Filighera, A., Ochs, S., Steuer, T., & Tregel, T. (2023). Cheating Automatic Short Answer Grading with the adversarial usage of adjectives and adverbs. International Journal of Artificial Intelligence in Education . https://doi.org/10.1007/s40593-023-00361-2. Ghavidel, H. A., Zouaq, A., & Desmarais, M. C. (2020). Using BERT and XLNET for the Automatic Short Answer Grading task. International Conference on Computer Supported Education, https://www.semanticscholar.org/paper/Using-BERT-and-XLNET-for-the-Automatic-Short-Answer-Ghavidel-Zouaq/96a3461e5f05973db7aec78afdb763304bd2e195. Gulson, K., Thompson, G., Swist, T., Kitto, K., Rutkowski, L., Rutkowski, D., Hogan, A., Zhang, V., & Knight, S. (2022). Automated Essay Scoring in Australian schools: Key issues and recommendations . University of Sydney Social Sciences and Humanities Advanced Research Centre. https://cic.uts.edu.au/wp-content/uploads/2022/11/AutoEssayScoring_WhitePaper.pdf. Johnson, M. (2014). Insights into contextualised learning: How do professional examiners construct shared understanding through feedback? E-Learning and Digital Media, 11 (4), 363-378. https://doi.org/10.2304/elea.2014.11.4.363. Kusuma, J. S., Halim, K., Pranoto, E. J. P., Kanigoro, B., & Irwansyah, E. (2022). Automated essay scoring using machine learning. 2022 4th International Conference on Cybernetics and Intelligent System (ICORIS) , 1-5. Latif, E., & Zhai, X. (2024). Fine-tuning ChatGPT for automatic scoring. Computers and Education: Artificial Intelligence, 6 , 100210. https://doi.org/10.1016/j.caeai.2024.100210. Lee, G. G., Latif, E., Wu, X., Liu, N., & Zhai, X. (2024). Applying large language models and chain-of-thought for automatic scoring. Computers and Education: Artificial Intelligence, 6 , 100213. https://doi.org/10.1016/j.caeai.2024.100213. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W. T., & Rocktäschel, T. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33 , 9459-9474. Li, J., Gui, L., Zhou, Y., West, D., & Aloisi, C. (2023). Distilling ChatGPT for explainable Automated Student Answer Assessment. Conference on Empirical Methods in Natural Language Processing, Singapore. https://www.semanticscholar.org/paper/Distilling-ChatGPT-for-Explainable-Automated-Answer-Li-Gui/c329ddddc53b9df84cf174d49a37c0bae585ea7a. Liu, Z., He, X., Liu, L., Liu, T., & Zhai, X. (2023). Context matters: A strategy to pre-train language model for science education. International Conference on Artificial Intelligence in Education, Toyko, Japan. Lockyer, C. (2019). Examining the examiners. The Ofqual blog . https://ofqual.blog.gov.uk/2019/03/13/examining-the-examiners/. Lun, J., Zhu, J., Tang, Y., & Yang, M. (2020). Multiple data augmentation strategies for improving performance on automatic short answer scoring. Proceedings of the AAAI Conference on Artificial Intelligence, New York, USA. Lundberg, S. M., & Lee, S. I. (2017). A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems, 30 . Lv, G., Song, W., Cheng, M., & Liu, L. (2021). Exploring the effectiveness of question for neural short answer scoring system. 2021 IEEE 11th International Conference on Electronics Information and Emergency Communication (ICEIEC), Beijing, China. OpenAI. (2023). GPT-4 Technical Report. https://arxiv.org/abs/2303.08774. OpenAI. (2024). Fine-tuning . https://platform.openai.com/docs/guides/fine-tuning. Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., McGuinness, L. A., Stewart, L. A., Thomas, J., Tricco, A. C., Welch, V. A., Whiting, P., & Moher, D. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. Bmj, 372 , 71. https://doi.org/10.1136/bmj.n71. Poulton, A., & Eliens, S. (2022). Explaining transformer-based models for automatic short answer grading. Proceedings of the 5th International Conference on Digital Technology in Education, Busan, Republic of Korea. https://doi.org/10.1145/3488466.3488479. Ramesh, D., & Sanampudi, S. K. (2022). An automated essay scoring systems: a systematic literature review. Artificial Intelligence Review , 2495–2527. Saha, S., Dhamecha, T. I., Marvaniya, S., Sindhgatta, R., & Sengupta, B. (2018). Sentence level or token level features for automatic short answer grading? Use both. Artificial Intelligence in Education: 19th International Conference, AIED 2018, London, UK. Sayeed, M. A., & Gupta, D. (2022). Automate Descriptive Answer Grading using Reference based Models. 2022 OITS International Conference on Information Technology (OCIT), Bhubaneswar, India. Shalev-Shwartz, S., & Ben-David, S. (2014). Understanding machine learning: From theory to algorithms . Cambridge University Press. Sokolova, M., Japkowicz, N., & Szpakowicz, S. (2006). Beyond accuracy, F-score and ROC: a family of discriminant measures for performance evaluation. Australasian Joint Conference on Artificial Intelligence, Hobart, Australia. Sukkarieh, J. Z., Pulman, S. G., & Raikes, N. (2005). Automatic marking of short, free text responses. Research Matters: A Cambridge Assessment Publication (1), 19-22. Sung, C., Dhamecha, T. I., Saha, S., Ma, T., Reddy, V., & Arora, R. (2019a). Pre-training BERT on domain resources for short answer grading. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China. Sung, C., Dhamecha, T. I., Mukhi, N. (2019b). Improving Short Answer Grading Using Transformer-Based Pre-training. Artificial Intelligence in Education 20 th International Conference, Proceedings Part 1, Chicago, IL, USA. https://doi.org/10.1007/978-3-030-23204-7_39. Tricco, A. C. (2016). Scoping reviews: What are they and how you can do them . https://training.cochrane.org/sites/training.cochrane.org/files/public/uploads/resources/downloadable_resources /English/Scoping%20reviews%20webinar%20Andrea%20Tricco%20PDF.pdf. Tricco, A. C., Lillie, E., Zarin, W., & et al. (2018). PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Annals of Internal Medicine, 169 (7), 467-473. https://doi.org/10.7326/m18-0850. Turner, C. (2024). GCSEs 2024: Exam board to trial AI in summer exams. Tes Magazine . https://www.tes.com/magazine/news/general/gcses-2024-exam-board-trial-ai-summer-exams. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems , 5998-6008. VERBI Software. (2024). MAXQDA 2024. In Berlin, Germany. www.maxqda.com. Wang, Z., Lan, A. S., Waters, A. E., Grimaldi, P., & Baraniuk, R. G. (2019). A Meta-Learning Augmented Bidirectional Transformer Model for Automatic Short Answer Grading. EDM, Montreal, Canada. Zhu, X., Wu, H., & Zhang, L. (2022). Automatic short-answer grading via BERT-based deep neural networks. IEEE Transactions on Learning Technologies, 15 (3), 364-375. Footnotes The ‘true’ mark was defined in various ways across the studies included in our review. For example, as a consensus mark across a group of markers, or the mark from one expert marker. GPT-3.5 and 4 have less information published by OpenAI on their training than GPT-3, although next token prediction is used to train GPT-4 (OpenAI, 2023 ). API stands for Application Programme Interface, and allows for two or more software components to communicate with each other. An OpenAI API allows developers to interact with OpenAI’s language models through programming languages such as Python. Non-sentential refers to responses which are not formed as part of a whole sentence but where the meaning can be inferred from the question. SHAP stands for SHapley Additive exPlanations. Additional Declarations No competing interests reported. Supplementary Files Appendices.docx Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 16 Oct, 2025 Reviews received at journal 21 May, 2025 Reviewers agreed at journal 06 May, 2025 Reviews received at journal 14 Apr, 2025 Reviewers agreed at journal 09 Apr, 2025 Reviewers invited by journal 06 Apr, 2025 Editor assigned by journal 09 Dec, 2024 Submission checks completed at journal 09 Dec, 2024 First submitted to journal 03 Dec, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5572868","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Systematic Review","associatedPublications":[],"authors":[{"id":440877865,"identity":"2bd1da6f-794d-4aa1-8736-e72e737c058f","order_by":0,"name":"Frank Morley","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABHUlEQVRIiWNgGAWjYBAC9gYGBsYGMAuIPwCxAZAvgU8LzwGYFmYgYwbJWph5wFoYGPBrkT787OEMhm1y5s3Mx6RtauyizaUPN978UsEgZ96/ALsWvjRzww0Mt41lDrOlSeccS87d2ZfYbC1zhsFY5sYDrFrseRjMJB8w3E6cwcxjJp3bwJy74Qxjm7RkG0PiDIkD2G3hYf8G1cL/TdqyoR6q5R8+LTxmkhsgtrBJMzYcBmuR/NgA1MLfgEtLmeQMg9vGEsxsxpY9x47n7uxhbLZmOCZhLIEj3IAO2ybZU3FbToK9+eGNHzXVudt52B/e/FFjIyfBj91hEGAAJlngxgIjCMiWSMCjBarwA4zF+ANE4rVlFIyCUTAKRhAAAFoZV4hVVG6WAAAAAElFTkSuQmCC","orcid":"","institution":"Cambridge University Press \u0026 Assessment","correspondingAuthor":true,"prefix":"","firstName":"Frank","middleName":"","lastName":"Morley","suffix":""},{"id":440877866,"identity":"79a8301d-0403-41fc-a317-71ff5c8f6303","order_by":1,"name":"Emma Walland","email":"","orcid":"","institution":"Cambridge University Press \u0026 Assessment","correspondingAuthor":false,"prefix":"","firstName":"Emma","middleName":"","lastName":"Walland","suffix":""},{"id":440877867,"identity":"cd145e14-1208-4281-9a18-fa8602d94444","order_by":2,"name":"Carmen Vidal Rodeiro","email":"","orcid":"","institution":"Cambridge University Press \u0026 Assessment","correspondingAuthor":false,"prefix":"","firstName":"Carmen","middleName":"Vidal","lastName":"Rodeiro","suffix":""}],"badges":[],"createdAt":"2024-12-03 13:53:34","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5572868/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5572868/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":80389888,"identity":"ffe99ba7-1366-4a16-9fd9-a22a1aaa2cb5","added_by":"auto","created_at":"2025-04-11 11:05:14","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":63472,"visible":true,"origin":"","legend":"\u003cp\u003ePRISMA diagram (Page et al., 2021) of the search process. This figure was created in Microsoft Visio.\u003c/p\u003e","description":"","filename":"1.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5572868/v1/5822a4878516d43aaa17fa82.jpg"},{"id":80389889,"identity":"72ec162b-bb2c-41bc-a1e9-da71992cbc6b","added_by":"auto","created_at":"2025-04-11 11:05:14","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":35829,"visible":true,"origin":"","legend":"\u003cp\u003eDiagram of auto-marking models in this review categorised into model groups. Transformer-based models are highlighted in grey. See Appendix B for the full list of models and studies that used them. This figure was created using SmartArt in Microsoft Word.\u003c/p\u003e","description":"","filename":"2.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5572868/v1/1645b28b64f41be3bc4e3c2d.jpg"},{"id":80389892,"identity":"bf6d7db7-ee38-400d-ab92-9451b35811a9","added_by":"auto","created_at":"2025-04-11 11:05:14","extension":"jpg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":54826,"visible":true,"origin":"","legend":"\u003cp\u003eFrequency of each model group across years. Number of papers a model group is used in each year (2024 was recorded up until the 18th of March when the literature search was undertaken). The R packages ggplot2 and ggpattern were used to create this figure.\u003c/p\u003e","description":"","filename":"3.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5572868/v1/bffbbef7019018d582ac8475.jpg"},{"id":80391311,"identity":"fec955f8-7e5a-42b5-a7ca-9374757bf018","added_by":"auto","created_at":"2025-04-11 11:21:14","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1354005,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5572868/v1/9cdf7191-f180-430a-9008-546322a5e537.pdf"},{"id":80389901,"identity":"43bac5ee-f934-4eee-a20d-c89ea90e707a","added_by":"auto","created_at":"2025-04-11 11:05:14","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":27028,"visible":true,"origin":"","legend":"","description":"","filename":"Appendices.docx","url":"https://assets-eu.researchsquare.com/files/rs-5572868/v1/97665f4c4b10eb73f73df0fd.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Transforming science marking: A scoping review of auto-markers","fulltext":[{"header":"Introduction","content":"\u003cp\u003eAuto-marking is the process of automating the marking of student responses using computational methods. Auto-marking has the potential to ease the time and resource pressures associated with human marking. In high-stakes domains where students\u0026rsquo; results have important uses, such as determining entry to university, auto-markers could act as another check on the marks given by examiners. In low-stakes domains, such as formative assessments, auto-markers might relieve workload pressures on teachers, while also providing instant and personalised marking and feedback to students.\u003c/p\u003e \u003cp\u003eShort answer questions in subjects such as science have traditionally been difficult to mark by computers due to the \u0026ldquo;myriad and sometimes unconventional ways in which credit-worthy answers [are] expressed\u0026rdquo; (Sukkarieh et al., \u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e2005\u003c/span\u003e, p.19). Such questions require students to produce phrases, sentences or short paragraphs of natural language in response to a question or prompt. They are not fully-objective, in the sense that there is more than one way that students could gain credit, in contrast to multiple choice questions where there is a predetermined correct answer.\u003c/p\u003e \u003cp\u003eRecent interest and development in the field of artificial intelligence has raised the prospect that computer algorithms are now becoming sophisticated enough to mark short answer questions (Turner, \u003cspan citationid=\"CR44\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). In 2017, a seminal paper on artificial intelligence outlined a new method called \u0026lsquo;transformers\u0026rsquo; (Vaswani et al., \u003cspan citationid=\"CR45\" class=\"CitationRef\"\u003e2017\u003c/span\u003e). Transformers use complex computational and statistical methods to represent the semantic meaning of natural language in a numerical format by measuring the relevance of words in a given context. One of these transformer-based model platforms, which became available in 2022, is ChatGPT (GPT stands for Generative Pre-trained Transformer). ChatGPT is a chatbot that uses the models GPT-3.5 and GPT-4.\u003c/p\u003e \u003cp\u003eAuto-markers which utilise transformer-based models could be both an effective and an efficient way to mark short answer responses, provided that they can be proven to perform well enough when evaluated against established performance metrics. For example, a review in 2022 found that BERT (Bidirectional Encoder Representations from Transformers), another transformer-based model, had the highest performance of any auto-marker on a widely used dataset (Kusuma et al., \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e2022\u003c/span\u003e). However, criticisms have been made of modern auto-markers: most do not assess domain knowledge in student responses vital for subjects such as science (Ramesh \u0026amp; Sanampudi, \u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e2022\u003c/span\u003e), and neither do many prioritise explainability (i.e., explaining in human terms how the model came to the outputs it did) which is crucial in an assessment context.\u003c/p\u003e \u003cp\u003eThis article aimed to provide fresh insights into these new transformer-based auto-markers in the domain-specific marking of science responses, while also exploring metrics related to accuracy and other performance criteria such as explainability. In particular, we carried out a high-level overview and summary of recent research on the performance of transformer-based auto-markers in science. This article is primarily a technical overview to inform subsequent research and development in auto-marking through identifying challenges and opportunities with the latest technologies. It also serves as an introduction to auto-marking for interested parties without previous technical expertise in this field.\u003c/p\u003e \u003cp\u003eThe review addressed four research questions, in the context of recent transformer-based auto-markers (from 2017 onwards) for science content:\u003c/p\u003e \u003cp\u003e \u003col\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eWhich auto-marking models were used and how do they work?\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eWhich datasets were used to fine-tune and test the auto-marking models?\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eHow well did auto-markers perform on metrics related to accuracy?\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eWhat other criteria are relevant for evaluating auto-markers, and how did recent auto-markers perform on these?\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e \u003c/p\u003e"},{"header":"Method","content":"\u003cp\u003eWe adopted a scoping review methodology for this research. Scoping reviews follow a systematic procedure to identify and chart literature. They serve various aims, for example, to summarise a complex and heterogenous body of research evidence, or to examine the extent and nature of existing research on a specific topic (Tricco, \u003cspan citationid=\"CR42\" class=\"CitationRef\"\u003e2016\u003c/span\u003e). Scoping reviews are a useful tool to generate an overview of existing evidence (both academic and grey literature), which can be valuable to researchers and practitioners. We followed the PRISMA-ScR guidelines for scoping reviews (Tricco et al., \u003cspan citationid=\"CR43\" class=\"CitationRef\"\u003e2018\u003c/span\u003e) and, in particular, we conducted these main steps (Tricco, \u003cspan citationid=\"CR42\" class=\"CitationRef\"\u003e2016\u003c/span\u003e):\u003c/p\u003e \u003cp\u003e \u003cul\u003e \u003cli\u003e \u003cp\u003eFormulate the research plan and review objectives (note that there is no research protocol or plan in this review, but details of our search plan and strategy are presented below).\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eDevise eligibility criteria and search for papers (including reference list scanning and grey literature if applicable).\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eScreen papers.\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eChart and present the results (extract/collect and record the relevant information; present it in tables, diagrams or descriptively).\u003c/p\u003e \u003c/li\u003e \u003cli\u003e \u003cp\u003eDraw conclusions and implications.\u003c/p\u003e \u003c/li\u003e \u003c/ul\u003e \u003c/p\u003e \u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eSearch strategy\u003c/h2\u003e \u003cp\u003eOur search for peer-reviewed articles and conference papers was conducted using Scopus (\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.scopus.com/\u003c/span\u003e\u003cspan address=\"https://www.scopus.com/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e), a comprehensive and multidisciplinary source of literature. We also carried out a non-systematic internet search using Google and Google Scholar to include any relevant grey literature.\u003c/p\u003e \u003cp\u003eIn accordance with our scope and aims, our Scopus search parameters were that the articles should be published in 2017 or later and be written in English. Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e shows the search terms we used. The initial search, carried out on the 18th of March 2024, generated 252 articles.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eScopus search terms\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCategory\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTerms\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eWords such as auto, automatic, automated\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(auto*) AND\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eWhat is being marked\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(essay OR writing OR \u0026ldquo;short answer\u0026rdquo; OR item) AND\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSynonyms of marking used in the literature\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(grading OR scoring OR evaluation OR marking) AND\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTypes of transformer-based models\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e(GPT* OR BERT* OR transformer* OR LLM* OR \u0026ldquo;large language model\u0026rdquo;)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"2\"\u003e\u003cem\u003eNote.\u003c/em\u003e The asterisk (*) is a symbol that is used in search terms to find words that have the same letters\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003eScreening\u003c/h3\u003e\n\u003cp\u003eAfter our initial Scopus search, we used the following eligibility criteria to screen the 252 articles, in accordance with our research aim and scope:\u003c/p\u003e \u003cp\u003e \u003col\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eAuto-marking in an educational assessment context\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eNatural language responses (i.e., not fully objective items)\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eEnglish responses (i.e., not other languages)\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eNon-language domain knowledge (i.e., subject-related knowledge and understanding rather than the quality of writing)\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003ePrimary study (i.e., not reviews)\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eScience subjects (i.e., biology, chemistry and physics).\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e \u003c/p\u003e \u003cp\u003eTwo researchers screened the articles. Abstracts were used for screening Criterion 1, 2 and 3 (and in cases where we were not sure, full papers were consulted). For Criterion 4 through 6, the full papers were used for screening. For Criterion 6, the papers were categorised by subject and those which were not in science subjects removed. MAXQDA 2024 (VERBI Software, \u003cspan citationid=\"CR46\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) was used as a tool to support the organisation, screening and charting of the literature.\u003c/p\u003e \u003cp\u003eA summary of the search and screening processes is given in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e, in the form of a PRISMA diagram. The search and screening process resulted in 21 articles being included in the review.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\n\u003ch3\u003eCharting\u003c/h3\u003e\n\u003cp\u003eOne researcher with expertise in data science coded the set of 21 articles in depth (see Appendix \u003cspan refid=\"Sec26\" class=\"InternalRef\"\u003eA\u003c/span\u003e for the full list of articles). Codes were developed through the coding process, which was done on MAXQDA 2024. The coding process began with a broad set of pre-existing ideas which were recorded in a code diary. As the coding continued, more detailed codes were added, and these changes were noted down in memos. Alongside this, the quality of the articles was interpreted. No articles were removed due to quality issues from the final pool of 21.\u003c/p\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e shows examples from the first three levels in the code hierarchy, although there were up to six levels in some cases.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eExamples of the codes used in the review.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eLevel 1 codes\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eLevel 2 codes\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eLevel 3 codes\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eContext\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eItem size, education level, country of primary author, publication year, subject\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eFor item size: short answer, essay\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel name\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eNon-machine learning, machine learning, deep learning, transformer-based\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eFor transformer-based: BERT, GPT, other transformer-based\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel information\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eModel purpose, type of computer used, reference answer used\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eFor model purpose: Main mark prediction, baseline, data augmentation, explainability\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eEvaluation metrics\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMark performance in datasets, explainability, bias, ease of deployment, ablation used\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eFor mark performance in datasets: SciEntsBank, ASAP SAS, Beetle\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e"},{"header":"Results","content":"\u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003eWhich auto-marking models were used and how do they work?\u003c/h2\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eFigure \u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e: Diagram of auto-marking models in this review categorised into model groups. Transformer-based models are highlighted in grey. See Appendix \u003cspan refid=\"Sec27\" class=\"InternalRef\"\u003eB\u003c/span\u003e for the full list of models and studies that used them. This figure was created using SmartArt in Microsoft Word.\u003c/p\u003e \u003cp\u003eIn the following, we provide technical details of the different auto-marking models in our review, including the number of papers that present the different models or model groups (Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e4\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cem\u003eAuto-markers\u003c/em\u003e operate through taking features from a student’s response and using these to predict the mark. This prediction is usually compared to the ‘true’ mark\u003csup\u003e1\u003c/sup\u003e given by a human examiner, and the difference or similarity between the true and predicted mark gives an indication of how effective the auto-marker is.\u003c/p\u003e \u003cp\u003eAuto-markers utilise a variety of different features which can be engineered or embedded.\u003c/p\u003e \u003cp\u003e \u003cem\u003eFeature engineering\u003c/em\u003e relates to features that can be specified using domain knowledge and extracted by human programmers. Simple examples include word count, frequency of keywords, and number of grammatical errors (Kusuma et al., \u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e2022\u003c/span\u003e). Responses can also be labelled with relevant aspects of the marking criteria and these can be used as features. An example of feature engineering in our review was found in Wang et al. (\u003cspan citationid=\"CR47\" class=\"CitationRef\"\u003e2019\u003c/span\u003e), who used a ‘Bag of Words’ approach that counted the frequencies of different words in a student response and used these as features.\u003c/p\u003e \u003cp\u003e \u003cem\u003eFeature embedding\u003c/em\u003e takes a different approach where features are instead automatically extracted through an algorithm (Ramesh \u0026amp; Sanampudi, \u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e2022\u003c/span\u003e). The purpose of the embedding algorithm is to represent each word in a student’s response as a unique numerical vector. Each word has a number of dimensions of semantic meaning and a vector represents the location of a word in this semantic space. For example, in the word2vec algorithm, a calculation of the numerical vectors ‘King – Man + Woman’ gives a similar result to the numerical vector for the word ‘Queen’ (Christian, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). Embedding allows for the meaning of words to be encoded in a numerical representation that can be then input into a statistical model for auto-marking (see machine learning approaches below). As feature embedding is built into transformer-based models, all the studies we reviewed used this approach.\u003c/p\u003e \u003cp\u003eTraditional approaches to auto-marking, shown in Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e4\u003c/span\u003e as ‘other auto-markers’, were used prior to machine learning approaches becoming widely adopted. These cover a broad spectrum of methods and include using keywords or keyword synonyms to match to the mark scheme (Burrows et al., \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2015\u003c/span\u003e). These approaches usually entailed a manually programmed, rules-based approach. Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e4\u003c/span\u003e shows that eight studies in our review used these approaches (or they analysed performance metrics from previous studies) alongside transformer-based models (more details can be found in the Appendix \u003cspan refid=\"Sec27\" class=\"InternalRef\"\u003eB\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eSeven studies, as seen in Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e4\u003c/span\u003e and in Appendix \u003cspan refid=\"Sec27\" class=\"InternalRef\"\u003eB\u003c/span\u003e, used machine learning approaches, shown as ‘other machine learning’. \u003cem\u003eMachine learning\u003c/em\u003e approaches to auto-marking replace a rules-based approach with the use of statistical models to predict a score based on the responses’ features (either engineered or embedded). An example in our review was the use of a random forest model to predict a response’s mark based on its features (Wang et al., \u003cspan citationid=\"CR47\" class=\"CitationRef\"\u003e2019\u003c/span\u003e). Random forest models arrived at a mark prediction through aggregating the predictions of multiple decision trees.\u003c/p\u003e \u003cp\u003eSix studies in our review used deep learning approaches (shown as ‘other deep learning’ in Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e4\u003c/span\u003e and in Appendix \u003cspan refid=\"Sec27\" class=\"InternalRef\"\u003eB\u003c/span\u003e). \u003cem\u003eDeep learning\u003c/em\u003e is a type of machine learning, where the statistical approach used involves a neural network with multiple layers that updates the strength of the connections (the weights) between artificial neurons in the network based on learning from previous predictions (Chollet, \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). In auto-marking, the neural network outputs a prediction of a response based on the features (either engineered or embedded) and compares this prediction to the true mark. The connections between neurons associated with a better prediction are strengthened. Through a successive, iterative process called optimisation, the neural network ‘learns’ to make better predictions. An example in our review of a deep learning model was a type of neural network called Long Short-Term Memory (LSTM) used in Chen \u0026amp; Li (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2021\u003c/span\u003e) and Zhu et al. (\u003cspan citationid=\"CR48\" class=\"CitationRef\"\u003e2022\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cem\u003eTransformers\u003c/em\u003e, which are the focus of our review and therefore are utilised in each paper, are a type of deep learning method that specialises in improving the meaning representation of embeddings. This is done through an attention mechanism that refines the numerical vectors in embeddings to take into account how the words, for example in a sentence, relate to each other (Vaswani et al., \u003cspan citationid=\"CR45\" class=\"CitationRef\"\u003e2017\u003c/span\u003e). For the transformer-based model GPT-3\u003csup\u003e2\u003c/sup\u003e the relations between tokens (partial chunks of words) are calculated as attention scores, and these are optimised through deep learning by predicting the next token in a sentence and minimising over time the difference between the predicted and actual token. GPT-3 is trained on 499\u0026nbsp;billion tokens (Brown et al., \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2020\u003c/span\u003e), roughly equivalent to 374\u0026nbsp;billion words (OpenAI, \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). Going back to the previous example, while an embedding algorithm like word2vec will represent the word ‘queen’ as a single numerical vector, a transformer-based model might encode the word ‘queen’ differently when it follows the word ‘Elizabeth’ compared to when it follows the words ‘Freddie’ and ‘Mercury’. This process, known as \u003cem\u003epre-training\u003c/em\u003e, gives models like GPT and BERT a contextually rich representation of natural language.\u003c/p\u003e \u003cp\u003e \u003cem\u003eFine-tuning\u003c/em\u003e involves training a pre-trained model for a specific task. Fine-tuning is akin to on-the-job-training for the model, allowing it to improve at a specific task like auto-marking. For BERT, an additional neural network component called a classification layer is added (Liu et al., \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). The classification layer allocates the response into one of a set of mutually exclusive classes, which in this case are predicted marks. The student's response is fed into the model, the features are embedded, then the embeddings are processed through the transformers and then passed to the classification layer. The classification layer uses deep learning to improve its mark prediction in relation to the true mark. OpenAI also allows users to fine-tune GPT-3.5 and 4 through the OpenAI API\u003csup\u003e3\u003c/sup\u003e (OpenAI, \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e2024\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cem\u003ePrompt engineering\u003c/em\u003e is an alternative to fine-tuning where instead of fine-tuning a pre-trained model on task-specific dataset, a set of well-crafted prompts allow the pre-trained model to improve (Brown et al., \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2020\u003c/span\u003e). Prompt engineering lessens the need for large task specific datasets of thousands of examples that are usually needed for fine-tuning and requires less technical expertise to implement. Examples from the review of fine-tuned and prompt engineered models are presented in the next few paragraphs.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eNumber of papers utilising models from each model group\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e\u003ccolgroup cols=\"2\"\u003e\u003c/colgroup\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eModel group \u003csup\u003e†\u003c/sup\u003e\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eNumber of papers\u003c/p\u003e \u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOther auto-markers\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e8\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOther machine learning\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e7\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOther deep learning\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOther transformer-based\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e7\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBERT\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e20\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eGPT\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eTotal number of distinct papers\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e21\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003ctfoot\u003e\u003ctr\u003e\u003ctd colspan=\"2\"\u003e\u003csup\u003e†\u003c/sup\u003e The groups are not mutually exclusive and papers in the review could have used multiple models.\u003c/td\u003e\u003c/tr\u003e\u003c/tfoot\u003e\u003c/table\u003e\u003c/div\u003e \u003cp\u003e\u003c/p\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e4\u003c/span\u003e shows that 20 out of 21 studies in our review used BERT models (see Appendix \u003cspan refid=\"Sec27\" class=\"InternalRef\"\u003eB\u003c/span\u003e for more details), which suggests that it is a frequently used transformer-based approach for science content (although it should be noted that there is usually a lag between research and publication). Nineteen studies used the standard BERT base model created by Google, and two used BERT large, a bigger variant with 24 instead of 12 transformer layers alongside other modifications. Sixteen studies used a fine-tuned BERT base model, while four did not tune BERT base. Four studies augmented BERT base with other forms of data, including the marking rubric (Condor et al., \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2022\u003c/span\u003e), an academic journal in science education (Liu et al., \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2023\u003c/span\u003e), questions and/or responses (Liu et al., \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Sung et al., \u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e2019a\u003c/span\u003e) and textbooks (Sung et al., \u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e2019a\u003c/span\u003e; Wang et al., \u003cspan citationid=\"CR47\" class=\"CitationRef\"\u003e2019\u003c/span\u003e). One study augmented BERT base by combining it with another deep learning model (Zhu et al., \u003cspan citationid=\"CR48\" class=\"CitationRef\"\u003e2022\u003c/span\u003e). Six studies used variants of BERT, which included RoBERTa (Robustly Optimised BERT), DistilBERT (Distilled BERT), AlBERT (A Lite BERT for Self-supervised Learning of Language Representations), SciBERT (Science BERT), and Longformer base. SciBERT takes BERT base and further pre-trains it on science-related content (Liu et al., \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2023\u003c/span\u003e), while the other variants use different model architectures to BERT base.\u003c/p\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e4\u003c/span\u003e shows that five studies used GPT models (see Appendix \u003cspan refid=\"Sec27\" class=\"InternalRef\"\u003eB\u003c/span\u003e for details). One study used GPT-1 and GPT-2 (Zhu et al., \u003cspan citationid=\"CR48\" class=\"CitationRef\"\u003e2022\u003c/span\u003e) and four studies used GPT-3.5. Of these, one used a fine-tuned GPT-3.5 (Li et al., \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e2023\u003c/span\u003e), while three used a prompt engineering approach for GPT-3.5 (Cochran et al., \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Lee et al., \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Li et al., \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). Only one study used GPT-4 featuring a prompt engineering approach (Lee et al., \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). There were no studies that fine-tuned GPT-4. Lee et al. (\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) also utilised the problem context and marking rubric in their testing of different prompt engineering approaches.\u003c/p\u003e \u003cp\u003eAccording to Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e4\u003c/span\u003e, seven studies utilised transformer-based models excluding BERT and GPT (shown as ‘other transformer-based’). More details about these models are provided in Appendix \u003cspan refid=\"Sec27\" class=\"InternalRef\"\u003eB\u003c/span\u003e.\u003c/p\u003e \u003cp\u003eOur systematic search did not find any studies which utilised other new state-of-the-art transformer-based models developed by technology companies, such as Google Gemini, Anthropic’s Claude or Meta Llama. Latif and Zhai (\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) stated that they had considered using Gemini but, at the time, Gemini did not provide a public API to fine-tune the model. Over the next year we would expect to see use of these models in published studies.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003eFrequency of models over time\u003c/h2\u003e \u003cp\u003eFigure \u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e shows the frequency of each model group over time in our reviewed sample. BERT models increased in frequency, reaching a peak in 2021. GPT models have been adopted more recently. Other auto-markers, other deep learning models, and other machine learning models, were steadily used from 2020 to 2022. However, in 2023, other deep learning and machine learning models were not used, which might indicate a decrease in their popularity in relation to transformer-based methods.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e\n\u003ch3\u003eWhich datasets were used to fine-tune and test the auto-marking models?\u003c/h3\u003e\n\u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e5\u003c/span\u003e shows the science datasets used in the articles included in our review. Many of these articles contained datasets from other disciplines such as computer science alongside science datasets. For the purposes of our review, as the focus was on science questions, we excluded datasets from other domains. For example, we excluded the widely used Mohler / University of North Texas dataset which focuses on computer science and data science items, and which was used alongside science datasets in Chen and Li (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2021\u003c/span\u003e) and Zhu et al. (\u003cspan citationid=\"CR48\" class=\"CitationRef\"\u003e2022\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eThe SciEntsBank and Beetle datasets included only categorical questions (or items), taking the form of 2-way items (‘correct’ or ‘incorrect’), 3-way items (‘correct’, ‘contradictory’ or ‘incorrect’), or 5-way items (‘correct’, ‘partially correct’, ‘contradictory’, ‘irrelevant’ or ‘not in the domain’). Three test sets were available: tests of unseen-answers (UA), unseen-questions (UQ), and unseen-domains (UD). Other datasets, BEAR (Berkeley Evaluation and Assessment Research), ASAP SAS (Automated Student Assessment Prize Short Answer Scoring), and TIMMS (Trends in International Mathematics and Science Study), had either 3 or 4 mark items. The science phenomena and science argumentation datasets used levels-based marking of proficiency. The PASTA (Potential of An Automatically Scored Three-dimensional Assessment System) project and Mathematical Thinking in Science project datasets either contained questions with 3–4 marking levels or 5–10 binary marking criteria.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 5\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eScience datasets selected for analysis and their characteristics\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e\u003ccolgroup cols=\"4\"\u003e\u003c/colgroup\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDataset\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eNumber of papers\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eEducational level (US)\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eItem types\u003c/p\u003e \u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSciEntsBank\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e10\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e3rd to 6th grades\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2-way, 3-way, 5-way\u003c/p\u003e \u003cp\u003eUnseen-answers (UA)\u003c/p\u003e \u003cp\u003eUnseen-questions (UQ)\u003c/p\u003e \u003cp\u003eUnseen-domains (UD)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBEAR\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003ePre-college test\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e4 marks\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eASAP SAS\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e7th to 10th grades\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e3 marks\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBeetle\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eUnclear\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2-way, 5-way\u003c/p\u003e \u003cp\u003eUA, UQ\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eScience phenomena dataset\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eMiddle school\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2 or 3 levels\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eScience argumentation tasks\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e5th to 8th grades\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e3 or 4 levels\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTIMMS items\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e4th and 8th grades\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e3 marks\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRainwater Runoff dataset\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e6th grade\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e1 mark\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePASTA project and Mathematical Thinking in Science project datasets\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHigh school\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e5–10 binary criteria\u003c/p\u003e \u003cp\u003e3 or 4 levels\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eOpenStax Biology textbook questions\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eHigh school\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eUnclear\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/table\u003e\u003c/div\u003e \u003cp\u003e\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\u003cdiv class=\"BlockQuote\"\u003e \u003cp\u003e \u003cb\u003eHow well did auto-markers perform on metrics related to accuracy?\u003c/b\u003e \u003c/p\u003e \u003c/div\u003e \u003cp\u003e\u003c/p\u003e \u003cp\u003eAmong the studies and datasets included in the review, the metrics described below were used.\u003c/p\u003e \u003cp\u003e \u003cstrong\u003eAccuracy\u003c/strong\u003e \u003c/p\u003e\u003cp\u003eAccuracy can be defined as the ratio of correct predictions or proportion of correctly scored answers (see, for example, Shalev-Shwartz and Ben-David (\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e2014\u003c/span\u003e) for more details).\u003c/p\u003e \u003cp\u003e\u003c/p\u003e \u003cp\u003e \u003cstrong\u003eM-F1 (Macro-averaged-F1) and W-F1 (Weighted-averaged-F1)\u003c/strong\u003e \u003c/p\u003e\u003cp\u003eThese are based off the F1 score (see, for example, Sokolova et al. (\u003cspan citationid=\"CR38\" class=\"CitationRef\"\u003e2006\u003c/span\u003e)), which is a fundamental metric for evaluating classification models, and is calculated as the harmonic mean of the precision and recall of a classification model. It is used when there is a binary classification task (e.g., present / absent; correct / incorrect). The F1 score is more robust to class imbalance (that is, when the number of instances in the different groups (or classes) is very different) than accuracy. The M-F1 and W-F1 are used in the case of multi-class classifications (e.g., correct / partially correct / incorrect). The M-F1 calculates the F1 score for each class, and then calculates the average without considering the proportion for each class in the dataset. The W-F1 calculates the F1 score for each class, and then calculates the average considering the proportion for each class in the dataset.\u003c/p\u003e \u003cp\u003e\u003c/p\u003e \u003cp\u003e \u003cstrong\u003eCK (Cohen's Kappa) and QWK (Quadratic Weighted Kappa)\u003c/strong\u003e \u003c/p\u003e\u003cp\u003eCohen’s Kappa (Cohen, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e1960\u003c/span\u003e) is defined as the level of agreement between two ratings (e.g., between a human marker and the auto-marker) beyond random chance. This is usually a more robust measure than accuracy as it accounts for random chance. QWK is an extension of Cohen’s Kappa that is designed to handle ordinal data, where the categories have a natural order (Cohen, \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e1968\u003c/span\u003e). Both CK and QWK range from − 1 to 1, where 1 indicates perfect agreement, 0 indicates no agreement beyond chance, and negative values indicate less agreement than expected by chance.\u003c/p\u003e \u003cp\u003e\u003c/p\u003e \u003cp\u003e \u003cstrong\u003eEAR (Exact Agreement Rate) and SER (Serious Error Rate)\u003c/strong\u003e \u003c/p\u003e\u003cp\u003eThe EAR is the level of exact agreement between the predicted and true mark. The SER is the level of disagreement between the predicted and true mark, where the absolute difference is greater than 1 (Chang et al., \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2021\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e\u003c/p\u003e \u003cp\u003eIn the following, the performance of the auto-markers used to score/classify the items in the datasets listed in Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e5\u003c/span\u003e is discussed.\u003c/p\u003e\n\u003ch3\u003eSciEntsBank dataset\u003c/h3\u003e\n\u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab5\" class=\"InternalRef\"\u003e6\u003c/span\u003e shows the performance of different models in the SciEntsBank dataset. The models shown provide “state of the art” (SOTA) performance, meaning they are the best performing in comparison to other models on the same items. Models are also compared to “baselines”, which are models that offer past SOTA performance which current studies seek to surpass.\u003c/p\u003e \u003cp\u003eTS + SF (Token level Features + Sentence level Features) is a model proposed by Saha et al. (\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e2018\u003c/span\u003e) which utilises feature engineering and deep learning. In 2-way UQ and UD tasks from the SciEntsBank dataset, TS + SF outperforms the newer BERT models, which are based on feature embedding and transformers. Their study stated that, in contrast to embeddings, “[our approach] token level hand-crafted features can be fairly domain independent and are less affected by non-sentential\u003csup\u003e4\u003c/sup\u003e forms” (Saha et al., \u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e2018\u003c/span\u003e, p. 2). However, TS + SF does not perform as well as the BERT models on UQ and UD on 3-way and 5-way items, suggesting that TS + SF might only be a preferable model for out-of-domain data in one mark items.\u003c/p\u003e \u003cp\u003eMDA-ASAS (Multiple Data Augmentation Strategies for Automatic Short Answer Scoring), a BERT model enhanced with MDA, proposed by Lun et al. (\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e2020\u003c/span\u003e), had SOTA performance for 2-way and 5-way items on the UA domain. This suggests that their approach of data augmentation (artificially boosting the size of the training dataset) might be a promising avenue for research.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e\u003ctable float=\"Yes\" id=\"Tab5\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 6\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eState of the art (SOTA) models on SciEntsBank dataset\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e\u003ccolgroup cols=\"4\"\u003e\u003c/colgroup\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eItems\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMetrics\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSOTA models \u003csup\u003e†\u003c/sup\u003e\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eArticle\u003c/p\u003e \u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003e2-way, UA\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eMDA-ASAS (BERT base)\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eLun et al. (\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e2020\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eM-F1\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eW-F1\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAlBERT Large\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003ePoulton and Eliens (\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e2022\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003e2-way, UQ\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003eTF + SF [-question]\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003eSaha et al. (\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e2018\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eM-F1\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eW-F1\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003e2-way, UD\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003eTF + SF [+ question]\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003eSaha et al. (\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e2018\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eM-F1\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eW-F1\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003e3-way, UA\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003eRoBERTa Large MNLI\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003eCamus and Filighera (\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2020\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eM-F1\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eW-F1\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003e3-way, UQ\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eBERT base uncased\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eGhavidel et al. (\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2020\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eM-F1\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eRoBERTa Large MNLI\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eCamus and Filighera (\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2020\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eW-F1\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eBERT base uncased\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eGhavidel et al. (\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2020\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003e3-way, UD\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003eRoBERTa Large MNLI\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003eCamus and Filighera (\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2020\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eM-F1\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eW-F1\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003e5-way, UA\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eMDA-ASAS (BERT base)\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eLun et al. (\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e2020\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eM-F1\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eBERT base hybrid\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eZhu et al. (\u003cspan citationid=\"CR48\" class=\"CitationRef\"\u003e2022\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eW-F1\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eMDA-ASAS (BERT base)\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eLun et al. (\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e2020\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003e5-way, UQ\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003eBERT base hybrid\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003eZhu et al. (\u003cspan citationid=\"CR48\" class=\"CitationRef\"\u003e2022\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eM-F1\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eW-F1\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003e5-way, UD\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eBERT base cased\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eGhavidel et al. (\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2020\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eM-F1\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eVariant of Funnel Transformer\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eChen and Li (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2021\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eW-F1\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eBERT base uncased\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eGhavidel et al. (\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2020\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/table\u003e\u003c/div\u003e \u003cp\u003e\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\u003cdiv class=\"BlockQuote\"\u003e \u003cp\u003eUA: Unseen answers, UQ: Unseen questions, UD: Unseen domains.\u003c/p\u003e \u003c/div\u003e \u003cp\u003e\u003c/p\u003e \u003cp\u003e \u003csup\u003e†\u003c/sup\u003e SOTA is defined by being the highest performer in any one of the metrics used.\u003c/p\u003e \u003cp\u003eThe RoBERTa Large MNLI (Multi-Genre Natural Language Inference) model proposed by Camus and Filighera (\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e2020\u003c/span\u003e) had SOTA performance on the 3-way dataset across UA, UQ, and UD items. This model utilises a method called transfer learning, where the pre-trained RoBERTa Large model was fine-tuned on another dataset called MNLI and then fine-tuned again on the 3-way auto-marking dataset. This model was not tested on 2-way and 5-way responses, but this could be an area for further research. Overall, this suggests that transfer learning could be a promising approach in developing auto-markers.\u003c/p\u003e \u003cp\u003eThe BERT base hybrid model which utilised BERT base alongside additional deep learning neural networks proposed by Zhu et al. (\u003cspan citationid=\"CR48\" class=\"CitationRef\"\u003e2022\u003c/span\u003e) had SOTA performance in 5-way UA and UQ items. A BERT base model without any additional changes was utilised in Ghavidel et al. (\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2020\u003c/span\u003e) and achieved SOTA performance in 5-way UD items (both the cased and uncased versions of the model). Finally, a (non-BERT and non-GPT) transformer-based model, Variant of Funnel Transformer, achieved SOTA in 5-way UD items.\u003c/p\u003e \u003cp\u003eIn conclusion, the BERT models (including variants such as RoBERTa) were nearly always the best performing models on tasks from the SciEntsBank dataset. Only two other models, TF + SF and Variant of Funnel Transformer, outperformed BERT models. Unfortunately, there were no studies that compared the performance of BERT to GPT-3.5 or GPT-4 on such tasks. This would be a welcome addition to the literature as the SciEntsBank dataset is commonly used to benchmark models.\u003c/p\u003e \u003cp\u003eIn the following sections, tests are of unseen answers (UA) unless specified otherwise. Table\u0026nbsp;\u003cspan refid=\"Tab6\" class=\"InternalRef\"\u003e7\u003c/span\u003e shows the performance of the auto-markers in all other datasets described in Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e5\u003c/span\u003e.\u003c/p\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003eBEAR dataset\u003c/h2\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab6\" class=\"InternalRef\"\u003e7\u003c/span\u003e shows that on the BEAR dataset, Modified BERT (Graph Rubric) performed best (Condor et al., \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2022\u003c/span\u003e). Graph rubric was able to capture the structure of the responses, i.e., a level 2 answer being ‘higher’ than a level 1 answer. Capturing this structure was made possible through the Node2Vec algorithm which is able to convert the levels in the mark scheme, a node/graph structure, into a format readable by BERT. Node2Vec outputs were used to further pre-train BERT alongside the marking rubric text for the Graph Rubric model.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003eASAP SAS dataset\u003c/h2\u003e \u003cp\u003eFor ASAP SAS, there were two sets of items which were used in separate articles to test different auto-markers. For the first set of items (Set A), Condor et al. (\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2022\u003c/span\u003e) modified BERT with graph rubric performed best on the W-F1 score. When using Cohen’s Kappa as a measure of agreement between the auto-marking model and the subject matter experts, modified BERT with random rubric had the highest performance. Random rubric, unlike graph rubric, did not utilise Node2Vec, but instead used random sampling during the further pre-training.\u003c/p\u003e \u003cp\u003eThe performance of a different auto-marker was also evaluated using the ASAP-SAS dataset in the article by Li et al. (\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e2023\u003c/span\u003e) on another set of items (Set B). A model called T5 was fine-tuned to produce marks and rationales for unseen answers. The rationales used in training T5 were generated using GPT-3.5. The auto-marker proposed by Li et al. (\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e2023\u003c/span\u003e) performed worse in all metrics than the baseline models. The Longformer baseline model performed best on the M-F1 and QWK metrics, and the BERT base model had the highest performance in terms of accuracy.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e\u003ctable float=\"Yes\" id=\"Tab6\" border=\"1\"\u003e\u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 7\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eSOTA models on other datasets\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e\u003ccolgroup cols=\"5\"\u003e\u003c/colgroup\u003e\u003cthead\u003e\u003ctr\u003e\u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDataset\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eItems\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eMetrics\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSOTA models\u003c/p\u003e \u003c/th\u003e\u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eArticle\u003c/p\u003e \u003c/th\u003e\u003c/tr\u003e\u003c/thead\u003e\u003ctbody\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eBEAR\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e4 marks\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eW-F1\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eBERT (Graph Rubric)\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eCondor et al. (\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2022\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCK\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\" morerows=\"4\" rowspan=\"5\"\u003e \u003cp\u003eASAP SAS\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eSet A\u003c/p\u003e \u003cp\u003e(3 marks)\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eW-F1\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eBERT (Graph Rubric)\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eCondor et al. (\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2022\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCK\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eBERT (Random Rubric)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003eSet B\u003c/p\u003e \u003cp\u003e(3 marks) \u003csup\u003e‡\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eBERT base uncased\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003eLi et al. (\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e2023\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eM-F1\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eLongformer\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eQWK\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\" morerows=\"4\" rowspan=\"5\"\u003e \u003cp\u003eBeetle\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2-way, UA\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eW-F1\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eAlBERT Large\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003ePoulton and Eliens (\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e2022\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e5-way, UA\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eM-F1\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\" morerows=\"2\" rowspan=\"3\"\u003e \u003cp\u003eCustom-made transformer\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\" morerows=\"3\" rowspan=\"4\"\u003e \u003cp\u003eChen and Li (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2021\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eW-F1\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e5-way, UQ\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eM-F1\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eW-F1\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eBERT base\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\" morerows=\"5\" rowspan=\"6\"\u003e \u003cp\u003eScience phenomena\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eSet A\u003c/p\u003e \u003cp\u003e(2 levels)\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eGPT-4 (FS_CoT_CR)\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\" morerows=\"3\" rowspan=\"4\"\u003e \u003cp\u003eLee et al. (\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) \u003csup\u003e†\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eF1\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eSet A\u003c/p\u003e \u003cp\u003e(3 levels)\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eGPT-4 (ZS_CoT_CR)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eF1\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSet B\u003c/p\u003e \u003cp\u003e(2 levels) \u003csup\u003e‡\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eSR1-BERT\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eLiu et al. (\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2023\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSet B\u003c/p\u003e \u003cp\u003e(3 levels)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eScience argumentation\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e3 or 4 levels\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSR2-SciBERT\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eLiu et al. (\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2023\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eTIMMS\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003e3 marks\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSER\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eMachine Concept Map\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eChang et al. (\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2021\u003c/span\u003e) \u003csup\u003e†\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eEAR\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u003cem\u003eBERT base\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRainwater Runoff\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e1 mark\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eM-F1\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eMultilingual L12 (data generated by GPT-3.5)\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eCochran et al. (\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2023\u003c/span\u003e) \u003csup\u003e†\u003c/sup\u003e\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003ePASTA project \u0026amp; Mathematical Thinking in Science\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e5–10 binary criteria\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eGPT-3.5-Turbo\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eLatif and Zhai (\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2024\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e3 or 4 levels\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eGPT-3.5-Turbo\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eOpenStax Biology textbook questions\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c2\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eUnclear\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c4\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eml-BERT\u003c/p\u003e \u003c/td\u003e\u003ctd align=\"left\" colname=\"c5\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eWang et al. (\u003cspan citationid=\"CR47\" class=\"CitationRef\"\u003e2019\u003c/span\u003e)\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003ctr\u003e\u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eF1\u003c/p\u003e \u003c/td\u003e\u003c/tr\u003e\u003c/tbody\u003e\u003c/table\u003e\u003c/div\u003e \u003cp\u003e\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\u003cdiv class=\"BlockQuote\"\u003e \u003cp\u003e \u003csup\u003e†\u003c/sup\u003e For Lee et al. (\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2024\u003c/span\u003e), Chang et al. (\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2021\u003c/span\u003e) and Cochran et al. (\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2023\u003c/span\u003e), metrics were not averaged across items, meaning that the SOTA represents the highest performance on the most number of items.\u003c/p\u003e \u003cp\u003e \u003csup\u003e‡\u003c/sup\u003e Different student responses were used in Lee et al. (\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) and Liu et al (\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2023\u003c/span\u003e), and in Condor et al. (\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2022\u003c/span\u003e) and Li et al. (\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e2023\u003c/span\u003e), preventing direct comparison between the studies.\u003c/p\u003e \u003c/div\u003e \u003cp\u003e\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003eBeetle dataset\u003c/h2\u003e \u003cp\u003eThe performance of the auto-markers evaluated in the 2-way items was similar across the models, with AlBERT Large showing slightly better performance on the W-F1 score. The custom-made transformer model described in Chen and Li (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e2021\u003c/span\u003e) outperformed BERT base for 5-way UA items on the M-F1 and W-F1 metrics. This was also the case for the M-F1 score for UQ items, but for the W-F1 score for UQ items BERT base outperformed the custom-made transformer.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003eScience phenomena dataset\u003c/h2\u003e \u003cp\u003eLee et al. (\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) used this dataset to compare the performance of various prompt engineering approaches for GPT-4 on a set of items (Set A), and compared GPT-4 to GPT-3.5. GPT-4 yielded higher accuracy than using GPT-3.5 in all items but one, and independently of the different prompt engineering strategies used. For example, ‘Few-Shot’ learning aims for models like GPT-4 to make predictions with only a few examples, while ‘Zero-Shot’ learning aims of predictions to be made with no examples. Chain-of-Thought prompting sets out a ‘reasoning path’ for models like GPT-4 to tackle complex reasoning tasks such as marking, using prompts like “let’s think step by step” (Lee et al., \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2024\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eThe two most complex strategies, Zero-Shot Chain-of-Thought problem context and rubric (ZS_CoT_CR) and Few-Shot Chain-of-Thought problem context and rubric (FS_CoT_CR), had, generally, the highest performance. ZS_CoT_CR performed best on items with two levels of proficiency, while FS_CoT_CR performed best on items with three levels. Overall, these results suggest including the problem context and rubric alongside a Chain-of-Thought approach improves the performance of prompt engineering strategies. This links to other findings regarding the benefits of utilising a marking rubric for auto-marking (Condor et al., \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2022\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eLiu et al. (\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2023\u003c/span\u003e) investigated, using the same dataset but on another set of items (Set B), how training language models with different contextual data can impact performance. The SR1-BERT model, trained with in-domain data, had an accuracy slightly higher than the accuracy of a base BERT model.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003eScience argumentation tasks\u003c/h2\u003e \u003cp\u003eIn Liu et al. (\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2023\u003c/span\u003e), SR2-SciBERT was the SOTA model on a dataset with science augmentation tasks. SciBERT is a BERT variant which has been further pre-trained on a scientific corpus of text, and this model was fine-tuned on SR2 (the dataset of science argumentation tasks). This study confirmed the effectiveness of using domain-specific data to train auto-marker models to improve their performance and suggests that adapting language models to science education could be worthwhile.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003eTIMMS items\u003c/h2\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab6\" class=\"InternalRef\"\u003e7\u003c/span\u003e shows the performance of the auto-marker proposed by Chang et al. (\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2021\u003c/span\u003e) on just three items from TIMMS. The BERT base model outperformed the proposed method (the machine generated concept map) on the EAR metric on all items. The proposed method outperformed BERT base on the SER metric on two out of three items.\u003c/p\u003e \u003cp\u003eThe Chang et al. (\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2021\u003c/span\u003e) model does not require large quantities of training data so it might be suitable when there is not a lot of resources to train the auto-marker. Another advantage is that it utilises a non-machine learning based approach where a concept map is generated using a statistical formula, and this is then used to mark the student responses. In contrast to BERT base, the concept map and the features used to assign the mark are interpretable.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec17\" class=\"Section2\"\u003e \u003ch2\u003eRainwater Runoff dataset\u003c/h2\u003e \u003cp\u003eIn Cochran et al. (\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e2023\u003c/span\u003e), two methods were used to generate synthetic data (GPT-3.5 and self-augmentation) and this data was used to fine-tune a BERT-based language model.. Performance from both models was better than performance from the BERT base model. The use of GPT-3.5 to generate data seemed to provide some advantages versus the self-augmented method. This highlights that GPT-3.5 is more versatile than BERT in being able to do a broader range of tasks such as data augmentation.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec18\" class=\"Section2\"\u003e \u003ch2\u003ePASTA project and Mathematical Thinking in Science project datasets\u003c/h2\u003e \u003cp\u003eThe results from the experiments using these datasets show that the accuracy of the GPT-3.5-turbo model was higher than that of the BERT (baseline) model. In particular, GPT-3.5-turbo consistently outperformed BERT in the multi-class tasks (the 5–10 binary criteria items). Latif and Zhai (\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2024\u003c/span\u003e, p. 8) suggested that “the architecture of GPT-3.5-turbo, the large amount of training data, or its innate skills to comprehend context more effectively may be responsible for its continual performance improvement”.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec19\" class=\"Section2\"\u003e \u003ch2\u003eOpenStax Biology textbook questions\u003c/h2\u003e \u003cp\u003eWang et al. (\u003cspan citationid=\"CR47\" class=\"CitationRef\"\u003e2019\u003c/span\u003e) used a selection of additional data (e.g., biology textbooks) to train the BERT model through meta-learning (a step added after pre-training but before fine-tuning). This allowed their model to gain more in-domain science representation. Results indicate that the proposed ml-BERT model had better accuracy and F1 scores when compared to all baseline models. In particular, a BERT base model without meta-learning only achieved comparable performance with a random forest model, but with meta-learning it was able to outperform all baseline models. The findings from Wang et al. (\u003cspan citationid=\"CR47\" class=\"CitationRef\"\u003e2019\u003c/span\u003e) show the potential benefit to marking accuracy of using techniques like meta-learning to boost the model’s specific domain ‘knowledge’.\u003c/p\u003e\u003cdiv class=\"BlockQuote\"\u003e\u003cp\u003e \u003cb\u003eWhat other criteria are relevant for evaluating auto-markers, and how did recent auto-markers perform on these?\u003c/b\u003e \u003c/p\u003e\u003c/div\u003e\u003cp\u003e\u003c/p\u003e \u003cp\u003eWe found that the main focus of auto-marking research has been on classification, however, there are other evaluation criteria that should be considered when applying such technology in educational assessment settings. Aloisi (\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2023b\u003c/span\u003e) discussed in detail three main threats to auto-marking in assessment (particularly in high-stakes settings) namely reliability, explainability and bias, which together address the overarching question of ethics. This serves as a useful organising framework for our results in this section. We added validity alongside reliability, as the two are closely linked. We also consider the ease of deployment of auto-markers.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec20\" class=\"Section2\"\u003e \u003ch2\u003eReliability and validity\u003c/h2\u003e \u003cp\u003eLow reliability and validity presents an issue for recent transformer-based auto-markers. Reliability in this context refers to the consistency of results, for example, the extent to which the marks produced would be the same under similar conditions. Validity refers to the process of ensuring that the marks produced are suitable for their intended uses and interpretations.\u003c/p\u003e \u003cp\u003eAloisi (\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2023a\u003c/span\u003e, p. 1) defines low reliability in the context of AI systems as meaning that “small variations in the \u003cem\u003einput\u003c/em\u003e may result in large differences in the \u003cem\u003eoutput\u003c/em\u003e” (italics in original), the input being the student responses and the output being the predicted mark. For example, Filighera et al. (\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e2023\u003c/span\u003e) found that adding adverbs or adjectives to student answers decreased BERT’s accuracy.\u003c/p\u003e \u003cp\u003eFor human markers, reliability is ensured through training, experience, and working within communities of practice (Aloisi, \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2023b\u003c/span\u003e; Johnson, \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e2014\u003c/span\u003e). Whilst auto-markers are trained, instead of being taught \u003cem\u003ehow to mark\u003c/em\u003e, they are set the goal of improving their prediction accuracy by finding out which patterns in language are associated with high or low mark answers. This results in ‘spurious correlations’, where features which would not be relevant for the mark scheme are used by auto-markers to decide a mark (Filighera et al., \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). Auto-markers may focus on superficial aspects of the text (e.g., punctuation, grammar and spelling) rather than higher-order constructs such as content, logic and coherence (Aloisi, \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2023b\u003c/span\u003e). This threatens construct validity as well as reliability, as auto-markers might not be rewarding the intended constructs consistently\u003c/p\u003e \u003cp\u003eOne method which may somewhat lessen the impact of construct-irrelevant features is increasing the domain-specific ‘knowledge’ of models. Four studies in our review used various techniques such as further pre-training and meta-learning to incorporate domain-specific data such as the marking rubric, science journals, and textbooks into the model (Condor et al., \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2022\u003c/span\u003e; Liu et al., \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2023\u003c/span\u003e; Sung et al., \u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e2019a\u003c/span\u003e; Wang et al., \u003cspan citationid=\"CR47\" class=\"CitationRef\"\u003e2019\u003c/span\u003e). However, although adding domain-relevant information resulted in small improvements in performance, if similar levels of performance can be reached by models without this information, then this might call into question the construct-relevance of the features used in these models. New developments such as Retrieval Augmented Generation (RAG), where models such as GPT-4 are able to access a database of factual content (Lewis et al., \u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e2020\u003c/span\u003e) could be an area of investigation.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec21\" class=\"Section2\"\u003e \u003ch2\u003eExplainability\u003c/h2\u003e \u003cp\u003eA pressing issue with many auto-markers is that they may not be able to explain how they arrived at a certain mark or provide quality feedback. Allowing students to question the marks they have received and to receive feedback is an important part of assessment ethics (Poulton \u0026amp; Eliens, \u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e2022\u003c/span\u003e). Furthermore, to build trust it seems necessary that there is some transparency and that stakeholders are able to understand how auto-markers work and how marks were determined (Ghavidel et al., \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e2020\u003c/span\u003e; Gulson et al., \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2022\u003c/span\u003e; Latif \u0026amp; Zhai, \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Lee et al., \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). This is especially important in formative settings, where teachers need to understand the rationale behind the marks given in order to provide the right support (Lee et al., \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). As mentioned, assessing the construct validity of features also depends on knowing which features the model is using.\u003c/p\u003e \u003cp\u003eThe auto-marking models in this review had various degrees of explainability. The most explainable models used simple rule-based approaches as found in Becker et al. (\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e2021\u003c/span\u003e) and Chang et al. (\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). In these examples, the inner workings of the model can be directly interpreted, and the decision-making process a model undertakes can be directly understood and predicted. As the complexity of the model increases, additional methods need to be implemented to ensure explainability. For example, Poulton and Eliens (\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e2022\u003c/span\u003e) use statistical methods such as SHAP\u003csup\u003e5\u003c/sup\u003e (see Lundberg \u0026amp; Lee, \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e2017\u003c/span\u003e for more information) to try and interpret the most important features in different BERT models. However, these models do not show in detail how the different features were assessed and weighted in the same way a human marker might be able to explain.\u003c/p\u003e \u003cp\u003eMore recently, the generative aspect of GPT has led to research claiming that GPT-3.5 and GPT-4 can create explainable marks through for example ‘rationale’ generation (Lee et al., \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2024\u003c/span\u003e; Li et al., \u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). However, it is not necessarily clear that the ‘rationale’ precisely reflects what the mark was based on. These models are also known to produce inaccurate outputs known as ‘hallucinations’, limiting the extent to which models can be trusted.\u003c/p\u003e \u003cp\u003eMore complex prompt engineering such as Chain-of-Thought reasoning lessens, but does not solve, this issue (Lee et al., \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2024\u003c/span\u003e). As the model works out a problem, it is prompted to go through a set of intermediate steps that facilitate reasoning, and by doing so it notes down its own reasoning in a ‘chain of thought’ process. However, the model’s reasoning could still be wrong and this would need to be checked. Ultimately, in order for model explanations to be useful in for example formative assessment, the models will have to develop so that users can be confident that their explanations are accurate and meaningful.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec22\" class=\"Section2\"\u003e \u003ch2\u003eBias\u003c/h2\u003e \u003cp\u003eIn the context of AI systems, bias refers to the systems treating certain groups of people more favourably or discriminating against them based on their characteristics, for example, sex or ethnicity (Aloisi, \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2023a\u003c/span\u003e). Transformer-based auto-markers use deep learning to find language which is associated with high and low mark answers and base their mark on this. This can lead to bias because if certain demographic groups are more likely to use some words over others, for example more masculine or feminine examples, these patterns could lead to bias against certain groups (Christian, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). Another example is where students sharing a particular linguistic background will use vocabulary or make errors in English which were not well represented in the training data, and so their correct answers are not recognised as correct. Filighera et al. (\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e2023\u003c/span\u003e) notes that students with developmental language disorder might express themselves differently and be systematically disadvantaged. There are other documented cases where artificial intelligence and machine learning systems are known to be biased towards or against certain groups (Aloisi, \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2023b\u003c/span\u003e; Latif \u0026amp; Zhai, \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2024\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eUnderstanding how models are trained and how decisions are arrived at is likely a step towards evaluating how bias is introduced and replicated. Rather than treat the models as black boxes, we need to consider the consequences of the training procedures (Filighera et al., \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). Only one study in our review, the Latif and Zhai (\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2024\u003c/span\u003e) study, mentioned that they had taken into account bias against demographic groups within their model training. However, the study did not give much detail as to how they did this.\u003c/p\u003e \u003cp\u003eThis relationship between construct relevance, explainability, and bias can be formulated in five questions to think about in relation to auto-marking:\u003c/p\u003e \u003cp\u003e \u003c/p\u003e\u003col\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eCan the mark be explained?\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eHow useful and accurate is this explanation?\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eWas the mark based on construct-relevant features?\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eHow susceptible are the features to adversarial input / gaming the system?\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eDo the features lead to bias against certain demographic groups?\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e \u003cp\u003e\u003c/p\u003e \u003cdiv id=\"Sec23\" class=\"Section3\"\u003e \u003ch2\u003eOther ethical issues\u003c/h2\u003e \u003cp\u003eAlthough technology is advancing, a question remains as to whether auto-markers will ever be able to replicate human intelligence accurately, and whether indeed they should (Aloisi, \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2023b\u003c/span\u003e). In shifting to auto-marking, we risk losing the positive human elements of marking. In many contexts, such as marking high stakes exams, when humans mark student work, they may feel morally responsible for the marks they award. There could be negative consequences for them and the students if they make poor decisions. A shift to auto-markers might lead to a loss of valuable human elements of marking, such as morality, responsibility, ethics and conscience. Previous research on examiner cognition in the context of high stakes exam marking in England has shown that good examiners are likely to take a lot of care and responsibility when marking student work, as they are aware of the implications of the results for students’ life chances (Lockyer, \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e2019\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eThere are various other ethical issues to consider in relation to auto-markers attempting to mimic human marking. For example, the role that marking plays for human teachers. It is possible that the process of teachers marking student work is beneficial to their practice and that using auto-markers may deskill teachers (Gulson et al., \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2022\u003c/span\u003e). Some authors, as cited in Latif and Zhai (\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e2024\u003c/span\u003e), are concerned about auto-markers leading to reduced human critical engagement in learning, causing a loss of critical thinking skills and personal interactions in education. On the other hand, there could be benefits of teachers’ changing roles due to auto-markers, for example, they could shift their attention to offering more personalised support. Further research exploring the interaction between humans and auto-markers and the relationship with learning is needed.\u003c/p\u003e \u003cp\u003eThe issues discussed above highlight some limitations of auto-markers. The evidence currently suggests that humans need to remain involved, and that auto-markers could support rather than replace humans. This view was indeed expressed by many authors in our review (see, for example, Condor, \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e2020\u003c/span\u003e; Filighera et al., \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e2023\u003c/span\u003e).\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv id=\"Sec24\" class=\"Section2\"\u003e \u003ch2\u003eEase of deployment\u003c/h2\u003e \u003cp\u003eThe ease in which auto-markers can be deployed is another aspect of performance that should be considered. There are three aspects to this: availability of data, computation, and expertise. Transformer-based models such as BERT can take hours, days, or more to fine-tune depending on the size of the dataset, the computational power available, and the expertise of the human programmer (Brown et al., \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2020\u003c/span\u003e). Fine-tuning often involves a lot of trial-by-error and experimentation before satisfactory performance can be gained (Chollet, \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e2021\u003c/span\u003e). As more aspects are added to the model, such as further pre-training (Liu et al., \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e2023\u003c/span\u003e), or a mark scheme rubric (Condor et al., \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2022\u003c/span\u003e), these extra levels of complexity make the model more cumbersome to train and deploy. These are not necessarily difficult problems to overcome but are issues to think about when, for example, designing pilot projects. Availability of publicly accessible data is also an issue mentioned in studies we reviewed (e.g., in Sung et al. (\u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e2019a\u003c/span\u003e)). GPT-3.5 and GPT-4 represent a general advance over BERT in ease of deployment. Prompt engineering allows these models to be operationalised with comparatively little data, less expertise, and fewer computational resources (Brown et al., \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2020\u003c/span\u003e).\u003c/p\u003e \u003cdiv id=\"Sec25\" class=\"Section3\"\u003e \u003c/div\u003e \u003c/div\u003e"},{"header":"Conclusion and discussion","content":"\u003cp\u003eThis review set out to explore the performance of recent transformer-based auto-markers on science content. Overall, we found that the most commonly used set of models in this context were BERT, which increased in frequency in recent years reaching a peak in 2021. After 2021, papers using GPT models started to appear and we anticipate more papers will be published in this area in the near future. GPT-3.5 and GPT-4 potentially offer benefits over BERT in terms of marking performance, ease of deployment with prompt engineering, and other capabilities such as generative explainability and data augmentation. Generally, larger models such as GPT-4 tend to have increased computational time. Larger models have increased financial costs over smaller models like BERT, for both model training and implementation in marking. However, research is only just beginning to be published using GPT-3.5 and GPT-4, making conclusions on marking performance tentative, but also meaning that the full benefits of these GPT models have not yet been realised.\u003c/p\u003e\u003cp\u003eThe results show that all of the datasets used in the articles included in the review were collected in the United States, with the most frequent being the SciEntsBank dataset. The SciEntsBank dataset covers science topics, and includes 2-way, 3-way and 5-way items. There are tests of unseen answers, domains and questions and covering US grades 3 to 6.\u003c/p\u003e\u003cp\u003eOur review highlighted the various metrics that can be used to evaluate auto-marker performance, depending on the type of model, with the most commonly used in our review being accuracy, M-F1 and W-F1. This indicates that most of the models were evaluated in terms of classification tasks (e.g., whether an outcome is a categorical value such as correct, incorrect or another category such as partially correct), rather than their ability to predict multiple numerical marks in a regression task. Evaluating a response as correct or incorrect might have limited applications in terms of assessment, and these models might only be appropriate for quite narrow science content questions with a low tariff. There are also some methodological constraints in the accuracy metric due to imbalanced classes. In addition, while auto-markers were compared to the ‘true mark’, how accurate a human marking the same content would be by comparison was not measured in the studies we reviewed.\u003c/p\u003e\u003cp\u003eBERT models performed better than previous models on the SciEntsBank dataset. Unfortunately, there were no studies that compared the performance of BERT to GPT-3.5 or GPT-4 on the SciEntsBank dataset, and we suggest this would be a welcome addition to the literature as this dataset is commonly used to benchmark models. In a case where BERT and GPT-3.5 were compared on the same dataset, GPT-3.5 performed better. On another dataset, GPT-4 generally outperformed GPT-3.5, while GPT-4 models which utilised the problem context and marking rubric along with a Chain-of-Thought prompt engineering approach outperformed other prompting strategies. In addition, Few-Shot learning provided gains over Zero-Shot learning in some but not all cases. In both cases (BERT and GPT), the performance was higher when there were fewer classifications, which is to be expected as classifying answers into fewer categories is an easier task and there is less scope for error.\u003c/p\u003e\u003cp\u003eFor BERT models, the results indicate that using additional training data like textbooks and marking rubrics could lead to improved performance. Overall, models that utilise other forms of data like textbooks and marking rubrics seem to consistently outperform models without these. However, using textbooks introduces a risk of bias, as models could benefit students using those textbooks rather than others.\u003c/p\u003e\u003cp\u003eOther than using metrics such as accuracy or F1 scores to evaluate auto-markers, these should also be evaluated in other ways, for example, in terms of ethics, explainability and accountability.\u003c/p\u003e\u003cp\u003eThe review has shown that transformer-based auto-markers may have issues in terms of low reliability, lack of explainability and bias. For example, low reliability can be caused by auto-markers assigning different marks based on slightly different inputs. This also presents a threat to validity, for example where decisions are made based on ‘spurious correlations’. Methods that can lessen (although as of yet not completely eliminate) this threat include increasing the domain-specific ‘knowledge’ of the models, e.g., through pre-training and meta-learning. Four studies that were reviewed tested methods of improving the explainability of auto-markers, but they still did not reach the level of being able to explain their decisions as well as human markers could. We found that GPT models can generate ‘explanations’ but these need to be checked and verified, thus weakening their utility in practice. Finally, bias in auto-markers remains a concern.\u003c/p\u003e\u003cp\u003eThese considerations have so far been secondary to increasing the marking accuracy of auto-markers. As auto-markers are increasingly used outside of research settings and in real-world situations, issues relating to ethics, explainability and accountability should become more central to how auto-markers are developed.\u003c/p\u003e\u003cp\u003eIn our closing paragraph, we note the limitations of this review. Firstly, our search was limited to one database (Scopus), and we may have missed some articles that were not indexed in it. As a scoping review, rather than a full systematic review, we have not comprehensively covered all available literature in this area. Secondly, our scope was limited to science content; there may be developments in auto-markers in other subject domains that were not reflected in this review. Finally, we focused our search on transformer-based technology. There might be other technologies that do not use transformers that could offer something of value for auto-marking.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e \u003ch2\u003eCompeting interests\u003c/h2\u003e \u003cp\u003eThe authors have no relevant financial or non-financial interests to disclose.\u003c/p\u003e \u003c/p\u003e\u003ch2\u003eFunding\u003c/h2\u003e \u003cp\u003eThe authors declare that no funds, grants, or other support were received during the preparation of this manuscript.\u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eF.M. and E.W. performed study design, search, and screening. F.M. coded the studies. F.M., E.W., and C.V.R. analysed and wrote the results. All authors read, reviewed, and approved of the final manuscript.\u003c/p\u003e\u003ch2\u003eData Availability\u003c/h2\u003e\u003cp\u003eThe findings used for this scoping review are available in Appendix A, which shows the full list of reviewed papers. These can be accessed through Scopus.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eAloisi, C. (2023a). AI and exam marking: Exploring the difficult questions of trust and accountability. \u003cem\u003eAQA\u003c/em\u003e. https://www.aqa.org.uk/about-us/our-research/blog/ai-and-exam-marking-exploring-the-difficult-questions-of-trust-and-accountability.\u003c/li\u003e\n\u003cli\u003eAloisi, C. (2023b). The future of standardised assessment: Validity and trust in algorithms for assessment and scoring. \u003cem\u003eEuropean Journal of Education, 58\u003c/em\u003e(1), 98-110. https://doi.org/10.1111/ejed.12542. \u003c/li\u003e\n\u003cli\u003eBecker, J. P., Kahanda, I., Kazi, N. H. (2021). \u003cem\u003eWIP: Detection of Student Misconceptions of Electrical Circuit Concepts in a Short Answer Question Using NLP\u003c/em\u003e. 2021 American Society for Engineering Education (ASEE) Annual Conference, Virtual Meeting. \u003c/li\u003e\n\u003cli\u003eBrown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D. (2020). \u003cem\u003eLanguage models are few-shot learners\u003c/em\u003e. https://arxiv.org/abs/2005.14165.\u003c/li\u003e\n\u003cli\u003eBurrows, S., Gurevych, I., \u0026amp; Stein, B. (2015). The eras and trends of automatic short answer grading. \u003cem\u003eInternational Journal of Artificial Intelligence in Education, 25\u003c/em\u003e, 60-117. \u003c/li\u003e\n\u003cli\u003eCamus, L., \u0026amp; Filighera, A. (2020). \u003cem\u003eInvestigating transformers for automatic short answer grading.\u003c/em\u003e Artificial Intelligence in Education: 21st International Conference, Ifrane, Morocco. \u003c/li\u003e\n\u003cli\u003eChang, T. H., Chen, J. L., Chou, H. M., Bai, M. H., Hsu, F. Y., \u0026amp; Chen, Y. C. (2021). \u003cem\u003eAutomatic Scoring Method of Short-Answer Questions in the Context of Low-Resource Corpora.\u003c/em\u003e 2021 International Conference on Asian Language Processing (IALP), Singapore.\u003c/li\u003e\n\u003cli\u003eChen, S., \u0026amp; Li, L. (2021). \u003cem\u003eIncorporating question information to enhance the performance of automatic short answer grading.\u003c/em\u003e Knowledge Science, Engineering and Management: 14th International Conference, KSEM 2021, Tokyo, Japan. \u003c/li\u003e\n\u003cli\u003eChollet, F. (2021). \u003cem\u003eDeep learning with Python\u003c/em\u003e. Simon and Schuster. \u003c/li\u003e\n\u003cli\u003eChristian, B. (2021). \u003cem\u003eThe alignment problem: How can machines learn human values?\u003c/em\u003e Atlantic Books. \u003c/li\u003e\n\u003cli\u003eCochran, K., Cohn, C., Rouet, J. F., \u0026amp; Hastings, P. (2023). \u003cem\u003eImproving automated evaluation of student text responses using gpt-3.5 for text data augmentation.\u003c/em\u003e International Conference on Artificial Intelligence in Education, Tokyo, Japan. \u003c/li\u003e\n\u003cli\u003eCohen, J. (1960). A coefficient of agreement for nominal scales. \u003cem\u003eEducational and Psychological Measurement, 20\u003c/em\u003e(1), 37-46. \u003c/li\u003e\n\u003cli\u003eCohen, J. (1968). Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. \u003cem\u003ePsychological Bulletin, 70\u003c/em\u003e(4), 213. \u003c/li\u003e\n\u003cli\u003eCondor, A. (2020). Exploring Automatic Short Answer Grading as a tool to assist in human rating. In I. I. Bittencourt, M. Cukurova, K. Muldner, R. Luckin, \u0026amp; E. Mill\u0026aacute;n, \u003cem\u003eArtificial Intelligence in Education\u003c/em\u003e AIED 2020, Cham.\u003c/li\u003e\n\u003cli\u003eCondor, A., Pardos, Z., \u0026amp; Linn, M. (2022). \u003cem\u003eRepresenting scoring rubrics as graphs for automatic short answer grading.\u003c/em\u003e International Conference on Artificial Intelligence in Education, Durham, UK.\u003c/li\u003e\n\u003cli\u003eFilighera, A., Ochs, S., Steuer, T., \u0026amp; Tregel, T. (2023). Cheating Automatic Short Answer Grading with the adversarial usage of adjectives and adverbs. \u003cem\u003eInternational Journal of Artificial Intelligence in Education\u003c/em\u003e. https://doi.org/10.1007/s40593-023-00361-2. \u003c/li\u003e\n\u003cli\u003eGhavidel, H. A., Zouaq, A., \u0026amp; Desmarais, M. C. (2020). \u003cem\u003eUsing BERT and XLNET for the Automatic Short Answer Grading task.\u003c/em\u003e International Conference on Computer Supported Education, https://www.semanticscholar.org/paper/Using-BERT-and-XLNET-for-the-Automatic-Short-Answer-Ghavidel-Zouaq/96a3461e5f05973db7aec78afdb763304bd2e195.\u003c/li\u003e\n\u003cli\u003eGulson, K., Thompson, G., Swist, T., Kitto, K., Rutkowski, L., Rutkowski, D., Hogan, A., Zhang, V., \u0026amp; Knight, S. (2022). \u003cem\u003eAutomated Essay Scoring in Australian schools: Key issues and recommendations\u003c/em\u003e. University of Sydney Social Sciences and Humanities Advanced Research Centre. https://cic.uts.edu.au/wp-content/uploads/2022/11/AutoEssayScoring_WhitePaper.pdf.\u003c/li\u003e\n\u003cli\u003eJohnson, M. (2014). Insights into contextualised learning: How do professional examiners construct shared understanding through feedback? \u003cem\u003eE-Learning and Digital Media, 11\u003c/em\u003e(4), 363-378. https://doi.org/10.2304/elea.2014.11.4.363. \u003c/li\u003e\n\u003cli\u003eKusuma, J. S., Halim, K., Pranoto, E. J. P., Kanigoro, B., \u0026amp; Irwansyah, E. (2022). Automated essay scoring using machine learning. \u003cem\u003e2022 4th International Conference on Cybernetics and Intelligent System (ICORIS)\u003c/em\u003e, 1-5. \u003c/li\u003e\n\u003cli\u003eLatif, E., \u0026amp; Zhai, X. (2024). Fine-tuning ChatGPT for automatic scoring. \u003cem\u003eComputers and Education: Artificial Intelligence, 6\u003c/em\u003e, 100210. https://doi.org/10.1016/j.caeai.2024.100210. \u003c/li\u003e\n\u003cli\u003eLee, G. G., Latif, E., Wu, X., Liu, N., \u0026amp; Zhai, X. (2024). Applying large language models and chain-of-thought for automatic scoring. \u003cem\u003eComputers and Education: Artificial Intelligence, 6\u003c/em\u003e, 100213. https://doi.org/10.1016/j.caeai.2024.100213. \u003c/li\u003e\n\u003cli\u003eLewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K\u0026uuml;ttler, H., Lewis, M., Yih, W. T., \u0026amp; Rockt\u0026auml;schel, T. (2020). Retrieval-augmented generation for knowledge-intensive nlp tasks. \u003cem\u003eAdvances in Neural Information Processing Systems, 33\u003c/em\u003e, 9459-9474. \u003c/li\u003e\n\u003cli\u003eLi, J., Gui, L., Zhou, Y., West, D., \u0026amp; Aloisi, C. (2023). \u003cem\u003eDistilling ChatGPT for explainable Automated Student Answer Assessment.\u003c/em\u003e Conference on Empirical Methods in Natural Language Processing, Singapore. https://www.semanticscholar.org/paper/Distilling-ChatGPT-for-Explainable-Automated-Answer-Li-Gui/c329ddddc53b9df84cf174d49a37c0bae585ea7a.\u003c/li\u003e\n\u003cli\u003eLiu, Z., He, X., Liu, L., Liu, T., \u0026amp; Zhai, X. (2023). \u003cem\u003eContext matters: A strategy to pre-train language model for science education.\u003c/em\u003e International Conference on Artificial Intelligence in Education, Toyko, Japan. \u003c/li\u003e\n\u003cli\u003eLockyer, C. (2019). Examining the examiners. \u003cem\u003eThe Ofqual blog\u003c/em\u003e. https://ofqual.blog.gov.uk/2019/03/13/examining-the-examiners/.\u003c/li\u003e\n\u003cli\u003eLun, J., Zhu, J., Tang, Y., \u0026amp; Yang, M. (2020). \u003cem\u003eMultiple data augmentation strategies for improving performance on automatic short answer scoring.\u003c/em\u003e Proceedings of the AAAI Conference on Artificial Intelligence, New York, USA. \u003c/li\u003e\n\u003cli\u003eLundberg, S. M., \u0026amp; Lee, S. I. (2017). A unified approach to interpreting model predictions. \u003cem\u003eAdvances in Neural Information Processing Systems, 30\u003c/em\u003e. \u003c/li\u003e\n\u003cli\u003eLv, G., Song, W., Cheng, M., \u0026amp; Liu, L. (2021). \u003cem\u003eExploring the effectiveness of question for neural short answer scoring system.\u003c/em\u003e 2021 IEEE 11th International Conference on Electronics Information and Emergency Communication (ICEIEC), Beijing, China. \u003c/li\u003e\n\u003cli\u003eOpenAI. (2023). \u003cem\u003eGPT-4 Technical Report. \u003c/em\u003ehttps://arxiv.org/abs/2303.08774.\u003c/li\u003e\n\u003cli\u003eOpenAI. (2024). \u003cem\u003eFine-tuning\u003c/em\u003e. https://platform.openai.com/docs/guides/fine-tuning.\u003c/li\u003e\n\u003cli\u003ePage, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hr\u0026oacute;bjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., McGuinness, L. A., Stewart, L. A., Thomas, J., Tricco, A. C., Welch, V. A., Whiting, P., \u0026amp; Moher, D. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. \u003cem\u003eBmj, 372\u003c/em\u003e, 71. https://doi.org/10.1136/bmj.n71. \u003c/li\u003e\n\u003cli\u003ePoulton, A., \u0026amp; Eliens, S. (2022). \u003cem\u003eExplaining transformer-based models for automatic short answer grading.\u003c/em\u003e Proceedings of the 5th International Conference on Digital Technology in Education, Busan, Republic of Korea. https://doi.org/10.1145/3488466.3488479.\u003c/li\u003e\n\u003cli\u003eRamesh, D., \u0026amp; Sanampudi, S. K. (2022). An automated essay scoring systems: a systematic literature review. \u003cem\u003eArtificial Intelligence Review\u003c/em\u003e, 2495\u0026ndash;2527.\u003c/li\u003e\n\u003cli\u003eSaha, S., Dhamecha, T. I., Marvaniya, S., Sindhgatta, R., \u0026amp; Sengupta, B. (2018). \u003cem\u003eSentence level or token level features for automatic short answer grading? Use both.\u003c/em\u003e Artificial Intelligence in Education: 19th International Conference, AIED 2018, London, UK.\u003c/li\u003e\n\u003cli\u003eSayeed, M. A., \u0026amp; Gupta, D. (2022). \u003cem\u003eAutomate Descriptive Answer Grading using Reference based Models.\u003c/em\u003e 2022 OITS International Conference on Information Technology (OCIT), Bhubaneswar, India. \u003c/li\u003e\n\u003cli\u003eShalev-Shwartz, S., \u0026amp; Ben-David, S. (2014). \u003cem\u003eUnderstanding machine learning: From theory to algorithms\u003c/em\u003e. Cambridge University Press. \u003c/li\u003e\n\u003cli\u003eSokolova, M., Japkowicz, N., \u0026amp; Szpakowicz, S. (2006). \u003cem\u003eBeyond accuracy, F-score and ROC: a family of discriminant measures for performance evaluation.\u003c/em\u003e Australasian Joint Conference on Artificial Intelligence, Hobart, Australia. \u003c/li\u003e\n\u003cli\u003eSukkarieh, J. Z., Pulman, S. G., \u0026amp; Raikes, N. (2005). Automatic marking of short, free text responses. \u003cem\u003eResearch Matters: A Cambridge Assessment Publication\u003c/em\u003e(1), 19-22. \u003c/li\u003e\n\u003cli\u003eSung, C., Dhamecha, T. I., Saha, S., Ma, T., Reddy, V., \u0026amp; Arora, R. (2019a). \u003cem\u003ePre-training BERT on domain resources for short answer grading.\u003c/em\u003e Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China. \u003c/li\u003e\n\u003cli\u003eSung, C., Dhamecha, T. I., Mukhi, N. (2019b). \u003cem\u003eImproving Short Answer Grading Using Transformer-Based Pre-training.\u003c/em\u003e Artificial Intelligence in Education 20\u003csup\u003eth\u003c/sup\u003e International Conference, Proceedings Part 1, Chicago, IL, USA. https://doi.org/10.1007/978-3-030-23204-7_39. \u003c/li\u003e\n\u003cli\u003eTricco, A. C. (2016). \u003cem\u003eScoping reviews: What are they and how you can do them\u003c/em\u003e. https://training.cochrane.org/sites/training.cochrane.org/files/public/uploads/resources/downloadable_resources\u003cbr\u003e/English/Scoping%20reviews%20webinar%20Andrea%20Tricco%20PDF.pdf.\u003c/li\u003e\n\u003cli\u003eTricco, A. C., Lillie, E., Zarin, W., \u0026amp; et al. (2018). PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. \u003cem\u003eAnnals of Internal Medicine, 169\u003c/em\u003e(7), 467-473. https://doi.org/10.7326/m18-0850. \u003c/li\u003e\n\u003cli\u003eTurner, C. (2024). GCSEs 2024: Exam board to trial AI in summer exams. \u003cem\u003eTes Magazine\u003c/em\u003e. https://www.tes.com/magazine/news/general/gcses-2024-exam-board-trial-ai-summer-exams. \u003c/li\u003e\n\u003cli\u003eVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., \u0026amp; Polosukhin, I. (2017). Attention is all you need. \u003cem\u003eAdvances in Neural Information Processing Systems\u003c/em\u003e, 5998-6008. \u003c/li\u003e\n\u003cli\u003eVERBI Software. (2024). \u003cem\u003eMAXQDA 2024. \u003c/em\u003eIn Berlin, Germany. www.maxqda.com.\u003c/li\u003e\n\u003cli\u003eWang, Z., Lan, A. S., Waters, A. E., Grimaldi, P., \u0026amp; Baraniuk, R. G. (2019). \u003cem\u003eA Meta-Learning Augmented Bidirectional Transformer Model for Automatic Short Answer Grading.\u003c/em\u003e EDM, Montreal, Canada. \u003c/li\u003e\n\u003cli\u003eZhu, X., Wu, H., \u0026amp; Zhang, L. (2022). Automatic short-answer grading via BERT-based deep neural networks. \u003cem\u003eIEEE Transactions on Learning Technologies, 15\u003c/em\u003e(3), 364-375. \u003c/li\u003e\n\u003c/ol\u003e"},{"header":"Footnotes","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003e The \u0026lsquo;true\u0026rsquo; mark was defined in various ways across the studies included in our review. For example, as a consensus mark across a group of markers, or the mark from one expert marker.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e GPT-3.5 and 4 have less information published by OpenAI on their training than GPT-3, although next token prediction is used to train GPT-4 (OpenAI, \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e2023\u003c/span\u003e).\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e API stands for Application Programme Interface, and allows for two or more software components to communicate with each other. An OpenAI API allows developers to interact with OpenAI\u0026rsquo;s language models through programming languages such as Python.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e Non-sentential refers to responses which are not formed as part of a whole sentence but where the meaning can be inferred from the question.\u003c/span\u003e\u003c/li\u003e\u003cli\u003e\u003cspan\u003e SHAP stands for SHapley Additive exPlanations.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"international-journal-of-artificial-intelligence-in-education","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"aied","sideBox":"Learn more about [International Journal of Artificial Intelligence in Education](http://link.springer.com/journal/40593)","snPcode":"40593","submissionUrl":"https://submission.nature.com/new-submission/40593/3","title":"International Journal of Artificial Intelligence in Education","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"Auto-marking, Automatic Short Answer Grading (ASAG), science, transformers, large language models, BERT","lastPublishedDoi":"10.21203/rs.3.rs-5572868/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5572868/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThis scoping review explored the performance of recent transformer-based auto-markers. We followed a systematic process, adhering to relevant PRISMA guidelines. Our review included recent literature (from 2017 onwards), focusing on English natural language responses on science content in an educational assessment context. A final set of 21 articles was reviewed and coded in depth to answer our research questions which explored the types of auto-marking models being used, the datasets used to fine-tune and test them, and their performance.\u003c/p\u003e \u003cp\u003eThe most commonly used models in this context were BERT models and BERT variants, which increased in frequency in recent years reaching a peak in 2021. After 2021, papers using GPT models started to appear. The SciEntsBank dataset was the most commonly used to test auto-markers but several other datasets (e.g., ASAP SAS, Beetle) also featured in our review. BERT models generally performed better than previous models on the SciEntsBank dataset. As of yet, GPT models have not been evaluated on SciEntsBank but there was one study in the review that directly compared GPT-3.5 and BERT base and found that GPT-3.5 outperformed BERT base across different items and item types. The review also shows that models that utilise additional forms of data like textbooks and marking rubrics seem to consistently outperform models without these and that recent auto-markers may still present issues in terms of low reliability, lack of explainability and bias.\u003c/p\u003e","manuscriptTitle":"Transforming science marking: A scoping review of auto-markers","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-04-11 11:05:09","doi":"10.21203/rs.3.rs-5572868/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-10-17T02:38:20+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-05-21T22:06:37+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"302460927917898907601004890642166328346","date":"2025-05-06T13:06:11+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-04-14T16:05:27+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"257427092915803647934586826851457203646","date":"2025-04-09T13:49:58+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-04-07T01:11:41+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-12-09T13:02:44+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-12-09T12:59:20+00:00","index":"","fulltext":""},{"type":"submitted","content":"International Journal of Artificial Intelligence in Education","date":"2024-12-03T13:45:19+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"international-journal-of-artificial-intelligence-in-education","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"aied","sideBox":"Learn more about [International Journal of Artificial Intelligence in Education](http://link.springer.com/journal/40593)","snPcode":"40593","submissionUrl":"https://submission.nature.com/new-submission/40593/3","title":"International Journal of Artificial Intelligence in Education","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Springer Hybrid","inReviewEnabled":true,"inReviewRevisionsEnabled":false}}],"origin":"","ownerIdentity":"3d797dcd-b864-4079-92ec-e78dbef76b80","owner":[],"postedDate":"April 11th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-02-12T19:09:40+00:00","versionOfRecord":[],"versionCreatedAt":"2025-04-11 11:05:09","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-5572868","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5572868","identity":"rs-5572868","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}