Language-specific embeddings of Old English with character-level processing

Language-specific embeddings of Old English with character-level processing | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Language-specific embeddings of Old English with character-level processing Javier Martín Arista, Darío Metola Rodríguez This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6473289/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract This article seeks to contribute to narrowing the gap between philological research and computational linguistics by providing a neural network model for Old English analysis. The article makes three key innovations: (i) the development of language-specific word embeddings derived directly from The Dictionary of Old English Corpus ; (ii) a comparative analysis of character-level versus word-level models that demonstrates the superior performance of character-level processing for morphologically rich historical languages; and (iii) a comprehensive Stanza-based pipeline that outperforms previous approaches to Old English parsing. Our model achieves 88.92% Unlabeled Attachment Score and 79.65% Labeled Attachment Score on dependency parsing tasks, which represents approximately 20 percentage point improvement over previous state-of-the-art multilingual approaches. The main conclusion of this work is that language-specific resources and character-level modeling are more effective for Old English processing than cross-linguistic transfer learning. This opens new avenues for computational research in historical linguistics and digital humanities. Linguistics Neural network models Language-specific word embeddings Character-level embeddings Old English processing Dependency parsing Figures Figure 1 Figure 2 1. Aims and scope The aim of this study is to present and evaluate a comprehensive neural network model for Old English linguistic analysis which has been designed to provide researchers with an effective computational tool adhering to principles of availability, scalability, and high performance. Building upon previous approaches to Old English language modeling, our work takes a new line: the incorporation of language-specific word embeddings derived directly from The Dictionary of Old English Corpus (DOEC). This is a significant departure from previous studies based on the idea that low-resource historical languages like Old English must rely primarily on cross-lingual transfer learning from related modern languages.[1] Our research addresses three questions: (i) Can language-specific word embeddings derived from historical corpora improve parsing accuracy compared to multilingual transfer learning? (ii) Do character-level models offer significant advantages over word-level models for processing morphologically complex historical languages? (iii) How do the different components of the neural pipeline perform on specific linguistic tasks of Old English processing? Through model comparison and assessment of performance metrics, the article demonstrates that a neural architecture combining language-specific embeddings with character-level processing outperforms previous approaches to Old English parsing. Our model achieves state-of-the-art results while remaining computationally efficient and readily adaptable for research applications in historical linguistics and digital humanities. The article is structured as follows. Section 2 reviews embedding technologies in NLP libraries. Section 3 surveys recent advances in the computational processing of historical languages, with particular focus on Old English. Section 4 details our methodology and presents comprehensive performance results for each component of our pipeline, while Section 5 shows the performance by component. Section 6 discusses the results by comparing our approach with previous work on Old English parsing. Section 7 summarises the main conclusions and makes an assessment of our contribution for future research in the computational processing of historical languages. [1] British English spelling is kept throughout the article except when referring to components and tasks of PLN, such as tokenizer or normalization , where the -z- orthography, far more frequent in NLP studies, has been preferred. 2. Embeddings in NLP libraries The NLP libraries NLTK, spaCy, and Stanza represent three different generations and designs, in such a way that each has advantages for different applications. NLTK, the oldest library, serves primarily educational purposes with extensive algorithmic implementations and corpus access but has performance limitations in production environments, as pointed out by Bird et al. (2019). spaCy, developed by Explosion AI, addresses these shortcomings through Cython-optimised code and an intuitive object-oriented design, which makes it the preferred choice for industry applications requiring high-performance English text processing (Honnibal et al., 2020). Stanza, created by Stanford NLP Group, takes a different approach by implementing state-of-the-art neural network models with an emphasis on linguistic accuracy and multilingual support (Qi et al., 2020). Stanza currently covers 66 languages across 112 datasets and provides a Python interface to Stanford CoreNLP for advanced functionalities like coreference resolution. The main difference among the three NLP libraries lies in their priorities: spaCy optimises speed and developer experience, whereas Stanza prioritises linguistic accuracy and language coverage, which leads many users to adopt hybrid approaches that leverage the strengths of each library. These libraries perform a set of text analysis tasks, typically beginning with fundamental operations like tokenization, part-of-speech tagging, and lemmatization; moving to dependency parsing and named entity recognition; and finishing up in advanced semantic analysis including sentiment analysis, coreference resolution, and relation extraction. While basic tasks like tokenization and simple rule-based tagging can function without embeddings, most modern approaches to NLP rely on embeddings for optimal performance. NLP embeddings are based on the hypothesis that words that occur in similar contexts tend to have similar meanings (Turney & Pantel, 2010). An embedding is a dense vector representation of a linguistic unit (usually a word) in a continuous, low-dimensional space where the semantic and syntactic properties of linguistic units are captured by vector proximity (Mikolov et al., 2013). While traditional vector representations like one-hot encoding treat words as isolated, discrete symbols in high-dimensional sparse vectors (Nadkarni et al., 2011), embeddings encode semantic relations in dense, lower-dimensional vectors where similar words cluster together in the vector space (Goldberg, 2016). Embeddings, therefore, enable generalisation across meanings and provide computational efficiency (Jurafsky & Martin, 2023). Word embeddings have undergone rapid evolution over the past decade, with several breakthrough papers establishing the foundations of modern approaches: Mikolov et al. (2013a) revolutionised the field with Word2Vec, which introduced efficient models for learning high-quality word vectors from large datasets. Their follow-up work (Mikolov et al., 2013b) demonstrated the ability of these vectors to capture semantic and syntactic regularities. Pennington et al. (2014) advanced the field further with GloVe (Global Vectors). GloVe combines global matrix factorisation with local context window methods to efficiently leverage statistical information from entire corpora. The paradigm shift toward contextual embeddings began with Peters et al. (2018), who introduced ELMo (Embeddings from Language Models). Peters et al. (2018) proved that deep bidirectional language models could generate context-sensitive representations that significantly improve performance across various NLP tasks. This evolution led to Devlin et al.´s (2019) introduction of BERT (Bidirectional Encoder Representations from Transformers), which established the state-of-the-art of pre-training and fine-tuning that has transformed NLP approaches and performance standards. Word embeddings fall into three main categories, namely, static word embeddings, contextual embeddings and sub-lexical embeddings. Static word embeddings include Word2Vec (Mikolov et al.,, 2013a), which employs CBOW and Skip-gram architectures to learn single-vector word representations by predicting words from context or vice versa; GloVe (Pennington et al., 2014), which combines global matrix factorisation with local context window methods to capture both statistical and contextual information; and FastText (Bojanowski et al., 2017), which extends Word2Vec by representing words as bags of character n-grams to handle morphologically rich languages and unseen words. Contextual embeddings generate dynamic representations based on surrounding text: ELMo (Peters et al., 2018) uses bidirectional LSTMs to produce context-dependent vectors; BERT (Devlin et al., 2019) employs transformer architecture with masked language modeling for deeply bidirectional representations; and GPT models (Radford et al., 2018, 2019; Brown et al., 2020) resort to unidirectional transformers trained on next-word prediction. This evolution from static to contextual representations marks a fundamental shift in NLP because it reconceptualises words from isolated units with fixed meanings to dynamic entities whose representation varies with linguistic context. Against this backdrop, specialised embedding types include cross-lingual embeddings (Conneau et al., 2018) that align vector spaces across languages; domain-specific embeddings trained on specialised corpora; and multi-modal embeddings that integrate text with other data types like images (CLIP by Radford et al., 2021) or audio Sub-lexical embedding approaches operate at multiple granularity levels, including character-level, subword level and hierarchical approaches. Character-level embeddings provide fundamental sub-word information. For example, Kim et al. (2016) demonstrate effective character-level CNNs for language modeling and text classification by applying convolutional filters to character embeddings to capture morphological patterns without explicit linguistic knowledge. In the same line, Ma & Hovy (2016) show significant improvements in sequence labeling tasks by combining character-level CNNs with word embeddings. Subword tokenization occupies the middle ground with three major approaches: Byte-Pair Encoding, which merges frequent character pairs to create subword units and is now used by many transformer models (Sennrich et al., 2016); WordPiece (Schuster & Nakajima, 2012), adopted in BERT, which uses a likelihood maximisation criterion for merging; and Unigram Language Model (Kudo, 2018), which treats segmentation as a probabilistic problem. Hierarchical approaches combine multiple representation levels and are contextualised. Ling et al. (2015), for instance, propose character-to-word models using bidirectional LSTMs (Long Short-Term Memory recurrent neural networks), and Peters et al. (2018) implement multi-level representations derived from LSTMs in ELMo (Embeddings from Language Models). The picture that emerges from a brief review of recent reseach is that sub-lexical approaches offer substantial advantages: vocabulary efficiency (Mielke et al., 2021), morphological awareness for morphologically rich languages (Çetinoğlu & Çöltekin, 2022; Shareghi et al., 2023), efficient handling of out-of-vocabulary words (Hofmann, 2022), sensitivity to spelling variations (Clark et al., 2022), and compatibility with cross-lingual transfer learning (Chung et al., 2023). As Rust et al. (2024) remark, modern transformer models typically employ subword tokenization, while specialised NLP applications with high morphological demands often benefit from dealing with character, subword, and word levels simultaneously (Shareghi et al., 2023). Major NLP libraries incorporate embeddings as core components. spaCy integrates pre-trained word embeddings directly into its processing pipeline. As described by Honnibal et al. (2020), spaCy offers multiple embedding options, including static GloVe vectors and transformers-based contextual embeddings. These embeddings serve as input features for its neural network models that handle tasks like parsing and named entity recognition. The architecture of spaCy allows computational linguists to swap embedding models based on their specific requirements and resource constraints. Stanza, according to Qi et al. (2020), takes a more comprehensive approach to embeddings. It implements a multi-level embedding strategy that combines word-level, character-level, and when available, BERT-based contextual embeddings. This approach guarantees the handling of morphologically rich languages and out-of-vocabulary words. The neural models of Stanza rely on these embeddings for all analysis tasks. Unlike spaCy’s more modular approach, the embedding strategy of Stanza is more deeply integrated within its neural architecture, thus prioritising linguistic accuracy over customisability (Qi et al., 2020). 3. The computational parsing of Old English Recent advances in the computational processing of historical languages have addressed the questions raised by ancient texts through different techniques and methodologies. For diachronic semantics, Hamilton et al. (2018) apply diachronic word embeddings to detect linguistic shifts, while Dubossarsky et al. (2019) develop a temporal referencing framework that distinguishes genuine semantic change from random sampling fluctuations. In text normalization, Bollmann (2019) demonstrates that character-based neural machine translation approaches outperform rule-based systems for handling non-standardised orthography. Manjavacas et al. (2019) develop a joint-learning approach for lemmatization that incorporates character-level embeddings and contextual information. In the field of Old English studies, a relatively wide variety of digital resources can be used for computational analysis. The Dictionary of Old English Corpus (DOEC) contains approximately 3.1 million words with minimal textual annotation. The York-Toronto-Helsinki Parsed Corpus of Old English Prose (YCOE) offers approximately 1.5 million words with POS tagging and constituent-oriented syntactic annotation, while its poetic counterpart comprises 71,000 words of verse annotated with the same tagset. The Parallel Corpus of Old English Prose provides sentence-level and word-level alignment between the Old and Present-Day English versions of selected texts comprising 300,000 words in the source language. These corpora are complemented by lexicographical sources that include The Dictionary of Old English (DOE), covering the headwords A-I with exhaustive morphological, grammatical and semantic information, and the digital version of Bosworth-Toller Anglo-Saxon Dictionary , with approximately 35,000 machine-readable headwords. Although these resources constitute a comparatively large dataset if compared with other old Germanic languages, they do not suffice to meet the requirements of up-to-date computational resources like transformer-based models (Martín Arista et al., 2025). Beyond resource scarcity, the computational parsing of Old English presents other difficulties like morphological complexity, non-standardised spelling, decaying inflections and flexible word order. Recent work has made some progress in applying modern NLP techniques to this diachronic stage of the English language. Faulkner (2023), for example, introduces a corpus philology approach that extracts large datasets for analysing spelling variations, effectively challenging traditional philological claims about Old English spelling through the analysis of over 216,000 spellings for words beginning with followed by a vowel. Brigada Villa & Giarda (2023a) test multilingual parsing performance on Old English using different language sets for training. Their findings show that models incorporating Old English data significantly outperform those without it and that Icelandic and German combinations perform best due to their retention of morphosyntactic features or, in the case of Icelandic, certain graphemes. In a follow-up study, Brigada Villa & Giarda (2023b) develop a rule-based approach for root identification in Old English which achieves 89.49% accuracy. This significantly outperforms their previous multilingual parser approach, although it is limited to the adaptation of the YCOE constituent parsing to dependency parsing. Martín Arista (2022) establishes the foundations for applying Universal Dependencies to Old English. The thrust of the argument is that standard UD annotation lacks sufficient representation of the associative lexicon of Old English. This author adds gloss and morphological relatedness fields to the annotation scheme. Building on this framework, Martín Arista (2024) demonstrates that adding a MORPHREL field in the CoNLL-U Plus format can capture both short-distance and long-distance morphological relations among un -derivatives in Old English. Martín Arista et al. (2025) evaluate different training procedures for automatic UD annotation of Old English using spaCy and MobileBERT. By testing various corpus sizes, these authors find that larger training corpora improve performance across all pipeline stages and that pre-training the tok2vec stage yields better results than the default pipeline. Martín Arista et al.’s (2025) model achieves precision rates of approximately 95% for lexical categories and 80% for dependency relations, thus outperforming previous multilingual approaches. These advances in the computational processing of Old English are compared with the proposal made in this article in Section 6. 4. Methodology The main innovation of our approach is the development of language-specific word embeddings for Old English. Previous approaches to Old English language model building either omitted this aspect entirely or attempted to leverage word vectors from typologically related languages (Brigada Villa & Giarda, 2023). Our model, by contrast, incorporates word embeddings constructed directly from the DOEC, comprising approximately 57,000 tokens distributed across 100 dimensions. Word embeddings represent a dense, low-dimensional projection of the sparse high-dimensional word vector space. While traditional word vectors are typically sparse with most entries being zero counts, embeddings compress this information into dense vectors with continuous real values (both positive and negative), ranging in dimensionality from 50 to 1000 (Jurafsky et al., 2023). The dimensionality selection represents a critical balance between computational efficiency and semantic expressiveness. Lower-dimensional embeddings (50-100 dimensions) consume less memory and process more quickly but may lose subtle semantic distinctions. Higher-dimensional embeddings (300-1000 dimensions) capture fine-grain relations but require more computational resources. For Old English, we empirically determined that 100-dimensional embeddings provide an optimal balance, as they capture sufficient semantic information while being compatible with the relatively small corpus size. This dimensionality allows the model to encode various linguistic aspects such as grammatical gender, verbal tense, and number; and maintains the distributional semantics necessary for contextual prediction. Figure 1 illustrates these embeddings with randomly selected words positioned according to their contextual similarity: words appearing at the center of the scatter plot exhibit stronger contextual relations than those at the periphery. The embeddings were created using word2vec, which used the continuous bag-of-words (CBOW) architecture with a context window of 5 words. This approach allows the model to learn word representations based on their distributional properties in the corpus, in such a way that words appearing in similar contexts tend to have similar meanings (Chaubard et al., 2019). The resulting embeddings serve as the point of departure for the POS tagger and dependency parser processors. This substantially improves their performance compared to models without language-specific embeddings. Our Old English language model was implemented using the Stanza Python library (Qi et al., 2020), which provides a neural network pipeline built on PyTorch. The complete model comprises four core processors: tokenizer, lemmatizer, POS tagger, and dependency parser. A distinctive feature of our approach is the comparison between traditional word-level tokenization and character-level models, which operate on individual characters rather than complete words. Character-level models have gained prominence for their effectiveness in handling languages with complex morphology (Adel et al., 2018). Unlike word-level models that treat each word as an atomic unit, character-level models process text character by character, thus allowing them to capture subword information and morphological patterns. This approach is particularly valuable for historical languages like Old English that exhibit rich inflectional morphology and orthographic variation. The Stanza library supports both character-level and non-character-level training options. To evaluate their relative effectiveness for Old English, we trained each processor (except the tokenizer, which cannot be implemented as a character-level model in Stanza) in both configurations. This comparative approach allowed us to quantify the performance benefits of character-level processing specifically for Old English. For training and evaluation, we used a dataset of 25,000 annotated Old English words (Martín Arista et al., 2023), divided into training (80%), development (10%), and test (10%) sets. This three-way split helps prevent overfitting by providing separate evaluation criteria during and after training. Each processor was trained sequentially on the training set. The fine-tuning of parameters was based on development set performance before final evaluation on the test set. 5. Results The tokenizer demonstrated excellent performance in word segmentation but faced difficulties with sentence boundaries. Table 1 tabulates the metrics of the performance of the tokenizer. The relatively poor performance in sentence segmentation (64.06% on the test set) can be attributed to the inconsistent punctuation in Old English texts. Unlike contemporary English, which follows standardised punctuation conventions, Old English manuscripts exhibit considerable variation in punctuation usage (see, for instance, Scragg 1974; Gneuss & Lapidge, 2014; Parkes, 2016). The 14-percentage-point drop between development and test set sentence segmentation (78.08% vs. 64.06%) indicates that the test set contains more complex or ambiguous sentence structures that pose additional difficulties. The lemmatizer was the first processor for which we could implement and compare character-level and non-character-level approaches. Table 2 provides a performance comparison between these implementations. The character-level model demonstrated greater training efficiency because it achieved higher accuracy (72.49% vs. 71.61% on the development set) and, moreover, converged twice as quickly (5 epochs vs. 10 epochs). The superior results of the character-level model is related to its ability to grasp morphological features at the sub-word level, which are widespread through the inflectional morphology of Old English. The efficiency advantage is particularly significant given the computational constraints often faced when processing historical languages like Old English. In this respect, the modest 1.08 percentage point improvement in accuracy represents hundreds of additional correctly lemmatized words in a typical corpus section, which improves downstream task performance. The POS tagger was the first processor to incorporate both character-level processing and the word embeddings that we developed. Table 3 presents a detailed performance comparison across different tagging categories. As shown in Table, 3, the character-level model consistently outperformed the non-character-level approach across all tagging categories. The high UPOS accuracy (94.21%) shows that basic part-of-speech categorisation is reliable and can provide a solid foundation for downstream syntactic analysis. The 1.95 percentage point advantage in UFeats accuracy (90.12% vs. 88.17%) highlights the ability of the character-level model to capture inflectional morphology, which is essential for identifying agreement patterns at the noun phrase and sentence level. The dependency parser represents the most complex component of our pipeline and the one that most directly benefits from both character-level processing and word embeddings. Table 4 presents the performance across various parsing metrics. As can be seen in Table 4, the character-level model achieved superior scores across all metrics (UAS: 88.92% vs. 85.01%; LAS: 79.65% vs. 74.78% on the test set) and converged in half the time (18 batches vs. 36 batches). This efficiency advantage results in significant time and resource savings when working with large corpora or when retraining the model with additional data. The consistent gap between UAS and LAS scores (approximately 9-10 percentage points) evidences that the model is more effective at identifying syntactic relationships than correctly labeling them. The lower MLAS (68.23%) and BLEX (66.72%) scores further demonstrate that performance decreases when morphological features and lemmatization are incorporated into evaluation metrics, which, in turn, highlights the difficulties of the combined analysis of Old English morphology and syntax. Figure 2 presents an overall representation of the performance of each model, evaluated agains the dev and test sets after being trained. To gain a deeper understanding of the performance of the dependency parser, we analysed precision, recall, and F1-scores for specific dependency relations. Table 5 highlights performance for selected relations using the character-level model on the test set. Three levels of performance emerge in Table 5. High-performing relations (F1 > 0.80) typically involve limited structural variation and occur frequently in the training data. They include coordinating conjunctions (cc, 0.9060), numeral modifiers (nummod, 0.9442), and adverbial modifiers (advmod, 0.8221). Average-performing relations (F1 0.60-0.80) comprise core arguments like nominal subjects (nsubj, 0.7716) and root relations (root, 0.8357). They show reasonably good performance but present shortcomings in complex sentences. The relative clause relation (acl:relcl, 0.6667) performs adequately despite involving subordination. Low-performing relations (F1 < 0.40) include complex clausal relations (ccomp, 0.2609; xcomp, 0.0800), auxiliaries (aux, 0.2222), and orphaned elements (orphan, 0.0889). These configurations often involve long-distance dependencies, discontinuous constituents, or ambiguous syntactic relations. The particularly poor performance on auxiliary relations (aux, F1=0.2222) may reflect the difficulty in distinguishing auxiliary verbs from main verbs in Old English, where the grammaticalisation of auxiliaries was still in progress (Martín Arista & Ojanguren López, 2018; Martín Arista, 2020). Similarly, the issues with oblique nominals (obl, F1=0.4000) may result from the case-marking system, where the function of noun phrases is indicated morphologically rather than positionally, which creates ambiguity for the parser. The extremely low performance on open clausal complements (xcomp, F1=0.0800) and orphaned constituents (orphan, F1=0.0889) points to specific syntactic constructions that require special attention in the future development of the model. These constructions often involve ellipsis or discontinuous dependencies that defy the sequential processing approach of neural models. 6. Discussion: innovations and performance This section compares the approaches to computational parsing and language modeling of Old English taken by Brigada Villa & Giarda (2023) and Martín Arista et al. (2025) with this study. The focus in on methodological innovations and performance metrics. The three papers represent distinct but complementary approaches to the computational analysis of Old English. Brigada Villa & Giarda (2023) rely on a multilingual parser to evaluate cross-lingual transfer learning approaches for parsing Old English. These authors train UUParser v2.4 by using various combinations of modern Germanic languages (Swedish, German, and Icelandic) both with and without Old English data. Then, they examine how related languages might enhance parsing performance for this low-resource historical language. Martín Arista et al. (2025) train a model from scratch on Old English data exclusively, testing different configurations for a spaCy pipeline to automatically annotate Old English with Universal Dependencies. They compare three approaches (default pipeline, pre-trained tok2vec, and transformer-based) with training corpora of varying sizes (1,000-20,000 words). The present study implements a comprehensive Old English language model built using the Stanza Python library. It incorporates word embeddings derived from the DOEC. This study contrasts character-level models with traditional word tokenization approaches across a 25,000-word dataset, with detailed performance metrics for each processor in the pipeline. A crucial difference among these studies is their approach to training data and model architecture. Brigada Villa & Giarda (2023) worked with a small dataset (292 sentences, 5,315 tokens) from religious prose texts, employing cross-lingual transfer learning with modern Germanic languages (Icelandic, German, and Swedish). These languages were chosen for their typological similarities to Old English, such as the case inflection of Icelandic and the flexible word order of German. The authors intentionally limited the support language treebanks to 60,000 tokens each to prevent size-related bias. Their model training involved a 30-epoch process, selecting the best performing epoch based on LAS scores on the development set. Martín Arista et al. (2025) used a larger dataset (25,000 words) from various Old English texts, including homilies, chronicles, and history, as well as biblical and legal documents. They carried out monolingual training using spaCy pipelines, testing three approaches: a baseline tok2vec model, a pretrained model initialised on unannotated Old English text, and a MobileBERT transformer. For the pre-trained model, they used the full Old English corpus (3 million words) to initialise the tok2vec stage, and for the transformer model, they trained a custom tokenizer to handle Old English-specific graphemes (æ, Æ, ð, Ð, þ, Þ). This study utilises the Stanza neural network pipeline with both character-level and non-character-level models. The 25,000-word dataset is divided into training (80%), development (10%), and test (10%) sets. A distinctive feature of this approach is the creation of language-specific word embeddings for Old English using word2vec on the DOEC, representing approximately 57,000 tokens across 100 dimensions. Focusing to the accuracy metrics, the monolingual approach adopted in Martín Arista et al. (2025) and in this study turns out more accurate results, in line with Meechan-Maddon & Nivre (2019), who demonstrate that in the parsing of low-resource languages, annotation of the target language yields more accurate results than cross-lingual transfer models. Brigada Villa & Giarda (2023) report the following key metrics for their best models: Old English monolingual model: 60.79% UAS, 64.39% LA, 47.23% LAS; OE + Icelandic: 68.44% UAS, 73.76% LA, 58.70% LAS (best overall UAS and LAS); OE + German + Icelandic: 66.34% UAS, 74.29% LA, 57.42% LAS (best LA). Their best model, combining Old English with Icelandic, achieved 68.44% UAS and 58.70% LAS. Their approach demonstrated that related languages could improve parsing accuracy, though adding German or Swedish showed diminishing returns. Martín Arista et al. (2025) show performance increasing with larger training datasets across all three of their model configurations. The pretrained tok2vec model achieves superior results (83.24% UAS, 74.23% LAS) with 20k words of training data. The findings emphasise the importance of dataset size and domain-specific pretraining over cross-linguistic transfer. These authors achieve the following results: TAG_ACC (XPOS): 75-95% (improving with larger datasets); POS_ACC (UPOS): 75-95% (improving with larger datasets); MORPH_ACC: 75-88% (improving with larger datasets); LEMMA_ACC: 75-87% (improving with larger datasets); DEP_UAS: 70-82% (improving with larger datasets); DEP_LAS: 65-73% (improving with larger datasets); SENTS_F: 75-78% (relatively stable across all dataset sizes). The present study shows metrics for specific processors. It also provides a detailed description of precision, recall, and F1 scores for every dependency relation type: Tokenizer: 99.63% (dev)/99.84% (test) accuracy for word tokenization; 78.08% (dev)/64.06% (test) for sentence segmentation; Lemmatizer: 72.49% with character model (best score at epoch 5) vs. 71.61% with no-character model (best score at epoch 10); POS Tagger: Character models show consistent advantages across all tag categories; Dependency Parser: UAS: 84-89% (character model), 81-85% (no-character model); LAS: 75-80% (character model), 71-75% (no-character model); Root: 83.57% accuracy (test charlm). When directly comparing performance metrics, it turns out that with respect to overall dependency parsing accuracy Brigada Villa & Giarda´s (2023) best model achieved 68.44% UAS and 58.70% LAS. The best model by Martín Arista et al. (2025) achieved 83.24% UAS and 74.23% LAS, while the best model presented in this study achieves ~84-89% UAS and ~75-80% LAS. The persistent gap between UAS and LAS scores evidenced by all studies indicates the difficulty of correctly labeling dependency relations compared to identifying head-dependent attachments. Nevertheless, the clear progression in performance demonstrates that both the specific Old English word embeddings and the character-level modeling in the later studies improve parsing accuracy. Table 6 compares the performance metrics across the three Old English parsing studies. All three studies identify problematic structures and functions in parsing Old English. Brigada Villa & Giarda (2023) highlight several key error sources: words with multiple grammatical functions (e.g., ne ‘not’ as both adverb and conjunction), postpositions, which none of their models could correctly handle, relative clauses with varying pronouns (only correctly identifying those with þe ‘that’), and discontinuous relative clauses, where the models incorrectly attached clauses to the nearest noun rather than their true antecedent. Martín Arista et al. (2025) find more structural challenges: negative contractions of verbs, pronouns, and adverbs, noun and prepositional phrases, especially flat multiword expressions, roots in copulative, existential, and conditional sentences, oblique nominals without prepositional marking, foreign words, complex sentences with orphans, clausal complements, and adverbial modifiers. This study provides the most detailed error analysis, with specific dependency relations showing particularly low performance, like auxiliary relations (aux/aux:pass, 18-33% precision), clausal relations (ccomp/xcomp, very low precision) and oblique relations (obl, 34-40% precision in some contexts). Each of the three studies in computational approaches to Old English contributes some methodological innovations that collectively advance historical language processing. Brigada Villa & Giarda (2023) demonstrate that cross-lingual transfer learning can effectively leverage knowledge from modern Germanic languages for Old English parsing. Martín Arista et al. (2025) contribute through monolingual training approaches optimised for historical language processing. The present study introduces two basic innovations: word embeddings derived from the entire DOEC and systematic comparison between character-level and token-level models. The accuracy metrics underscore the importance of the specific embedding and the superiority of the character-level model. The present study also demonstrates that the Stanza-based pipeline can provide an integrated solution and identifies additional challenging dependency relations for future research. 7. Conclusion Our study makes several contributions to computational linguistics for historical languages. We addressed three research questions, with results that demonstrate some advantages of our approach. First, we show that language-specific word embeddings substantially improve parsing accuracy for Old English, which parts company with the multilingual metod based on the idea that transfer from related languages is the only viable approach for low-resource historical languages. Our 100-dimensional embeddings derived from the DOEC capture semantic relations specific to Old English and enables more accurate contextual prediction than would be possible with embeddings from typologically or areally related languages. Second, our comparative analysis provides empirical evidence for the superiority of character-level approaches in languages with complex morphology and orthographic variation. For Old English, character-level models consistently outperformed their word-level counterparts across all processors, with accuracy improvements of 1.08% for lemmatization, 1.97% for POS tagging, and 4.87% for dependency parsing (LAS). Moreover, character-level models exhibited greater training efficiency, converging in half the time and requiring fewer computational resources. The performance of our model represents a substantial improvement over previous approaches. While Brigada Villa & Giarda (2023) reported a maximum Labeled Attachment Score of 58.70% using a multilingual parser combining Old English with Icelandic, our character-level model achieves 79.65% LAS on the test set. This 20.95 percentage point improvement underscores the value of our combined approach compared to previous methods. Third, our analysis of component-specific performance reveals that while common syntactic relations are modeled with high accuracy (>80% F1 for determiners, subjects, and case marking), complex structures remain challenging (<30% F1 for clausal complements and orphans). This understanding of where the model does not completely succeed can guide future research. The scalability of the model constitutes another advantage. It can be continuously improved through additional training data, integrated with annotation interfaces to facilitate human workflows, and combined with other NLP tools through the spaCy-stanza interface. This interoperability enhances its practical utility for researchers. Despite these advances, our research has several limitations that should be acknowledged. Our training dataset, while larger than those used in previous studies, still represents a small fraction of the extant Old English corpus and is biased toward prose texts, potentially limiting performance on poetic texts with their distinctive syntax and vocabulary. The model also struggles with sentence boundary detection (64.06% accuracy), which affects downstream processing, particularly for complex multi-clause sentences. Additionally, our approach does not address dialectal variation within Old English, treating it as a homogeneous language despite documented regional and temporal differences that impact morphology and syntax. Finally, our evaluation metrics, while standard for computational linguistics, may not fully capture the nuanced linguistic phenomena of interest to historical linguists. Future work should focus on expanding the Old English treebank to enhance performance for sentence segmentation and less common dependency relations, on normalising punctuation to improve sentence boundary detection, and on developing named entity recognition capabilities. Experimenting with different embedding dimensionalities and architectures -particularly larger dimensions (300-500) for capturing subtle semantic distinctions- could improve performance on complex relations like clausal complements. Declarations Acknowledgement: We gratefully acknowledge the grant PID2023-149762NB-100, funded by MCIN / AEI / 10.13039/501100011033 References Adel, H., Asgari, E., & Schütze, H. (2018). Overview of character-based models for natural language processing. In A. Gelbukh (Ed.), Computational Linguistics and Intelligent Text Processing. CICLing 2017 . Lecture Notes in Computer Science (Vol. 10761). Springer. https://doi.org/10.1007/978-3-319-77113-7_1 Bird, S., Klein, E., & Loper, E. (2019). Natural language processing with Python: Analyzing text with the Natural Language Toolkit (2nd ed.). O'Reilly Media. Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017). Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics , 5, 135-146. https://doi.org/10.1162/tacl_a_00051 Bollmann, M. (2019). A large-scale comparison of historical text normalization systems. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 3885-3898). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1389 Bosworth, J., & Toller, T. N. (2018). An Anglo-Saxon Dictionary Online (O. Tichý, & A. Boer, Eds.). Faculty of Arts, Charles University. http://bosworth.V.cuni.cz Brigada Villa, L., & Giarda, M. (2023a). Using modern languages to parse ancient ones: a test on Old English. In Proceedings of the 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP (pp. 30-41). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.sigtyp-1.4 Brigada Villa, L., & Giarda, M. (2023b). From YCOE to UD: rule-based root identification in Old English. In Proceedings of the Third Workshop on Language Technology for Digital Historical Archives (LT4HALA 2024) (pp. 22-29). European Language Resources Association. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., & Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems , 33, 1877-1901. https://doi.org/10.48550/arXiv.2005.14165 Cameron, A., Amos, A. C., Healey, A. D., Holland, J., McDougall, D., McDougall, I., Speirs, N., & Thompson, P. (Eds.). (2018). Dictionary of Old English: A to I online . Dictionary of Old English Project, University of Toronto. https://www.doe.utoronto.ca Çetinoğlu, Ö., & Çöltekin, Ç. (2022). A thorough evaluation of character-level models for Turkish natural language processing. Natural Language Engineering , 28(3), 275-301. https://doi.org/10.1017/S1351324921000218 Chaubard, F., Fang, M., Genthial, G., Mundra, R., & Socher, R. (2019). CS224n: Natural language processing with deep learning, lecture notes: Part I, Winter 2019 . Retrieved January 26, 2024, from https://web.stanford.edu/class/cs224n/ Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X. V., Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., Rose, T., Krakover, I., Braz, P. E., Roberts, A., Dyer, J., Ser, D., Fernando, J., Zoph, B., Welinder, P., Baljekar, P., Firat, O., Zettlemoyer, L., Hughes, M., Austin, J., Norouzi, M., Tan, M., Sohl-Dickstein, J., Dean, J., & Wei, J. (2023). Scaling instruction-finetuned language models. Computational Linguistics, 49(4), 965-985. https://doi.org/10.1162/coli_a_00474 Clark, J. H., Garrette, D., Turc, I., & Wieting, J. (2022). Canine: Pre-training an efficient tokenization-free encoder for language representation. Transactions of the Association for Computational Linguistics , 10, 73-91. https://doi.org/10.1162/tacl_a_00448 Conneau, A., Lample, G., Ranzato, M., Denoyer, L., & Jégou, H. (2018). Word translation without parallel data. In International Conference on Learning Representations . https://doi.org/10.48550/arXiv.1710.04087 de Marneffe, M. C., Manning, C. D., Nivre, J., & Zeman, D. (2021). Universal Dependencies. Computational Linguistics , 47(2), 255-308. https://doi.org/10.1162/coli_a_00402 Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (pp. 4171-4186). https://doi.org/10.18653/v1/N19-1423 Dubossarsky, H., Hengchen, S., Tahmasebi, N., & Schlechtweg, D. (2019). Time-out: Temporal referencing for robust modeling of lexical semantic change. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 457-470). Association for Computational Linguistics. https://doi.org/10.18653/v1/P19-1044 Faulkner, M. (2023). Corpus philology: Using the Dictionary of Old English to get bigger data for Old English spelling variation. Digital Scholarship in the Humanities , 38(4), 1508-1521. https://doi.org/10.1093/llc/fqad023 Gneuss, H., & Lapidge, M. (2014). Anglo-Saxon Manuscripts: A Bibliographical Handbook . University of Toronto Press. Goldberg, Y. (2016). A primer on neural network models for natural language processing. Journal of Artificial Intelligence Research , 57, 345-420. https://doi.org/10.1613/jair.4992 Hamilton, W. L., Leskovec, J., & Jurafsky, D. (2018). Diachronic word embeddings reveal statistical laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (pp. 1489-1501). Association for Computational Linguistics. https://doi.org/10.18653/v1/P16-1141 Haug, D. T. T., & Jøhndal, M. (2021). Creating a parallel corpus of Old and Modern French. Journal of Data Mining and Digital Humanities , 2021. https://doi.org/10.46298/jdmdh.6566 Healey, A. diPaolo. (Ed.). (2018 ). The Dictionary of Old English Corpus in Electronic Form . Dictionary of Old English Project, University of Toronto. https://www.doe.utoronto.ca/pages/pub/web-corpus.html Hofmann, V., Pierrehumbert, J. B., & Schütze, H. (2022). Modeling morphological processing in human language understanding. Computational Linguistics , 48(2), 219-271. https://doi.org/10.1162/coli_a_00437 Honnibal, M., Montani, I., Van Landeghem, S., & Boyd, A. (2020). spaCy: Industrial-strength Natural Language Processing in Python . https://doi.org/10.5281/zenodo.1212303 Jurafsky, D., & Martin, J. H. (2023). Speech and language processing (3rd ed.). Pearson. Kim, Y., Jernite, Y., Sontag, D., & Rush, A. M. (2016). Character-aware neural language models. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (pp. 2741-2749). https://doi.org/10.5555/3016100.3016285 Kudo, T. (2018). Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (pp. 66-75). https://doi.org/10.18653/v1/P18-1007 Ling, W., Luís, T., Marujo, L., Astudillo, R. F., Amir, S., Dyer, C., Black, A. W., & Trancoso, I. (2015). Finding function in form: Compositional character models for open vocabulary word representation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (pp. 1520-1530). https://doi.org/10.18653/v1/D15-1176 Ma, X., & Hovy, E. (2016). End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (pp. 1064-1074). https://doi.org/10.18653/v1/P16-1101 Manjavacas, E., Kádár, Á., & Kestemont, M. (2019). Improving lemmatization of non-standard languages with joint learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 1493-1503). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1153 Martín Arista, J. (2020). Further remarks on the deflexion and grammaticalization of the Old English past participle with habban. International Journal of English Studies , 20(1): 51-71. https://doi.org/10.6018/ijes.404881 Martín Arista, J. (2022). Old English Universal Dependencies: Categories, Functions and Specific Fields. In Proceedings of the 14th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART (pp. 945-951). SCITEPRESS. https://doi.org/10.5220/0010896700003116 Martín Arista, J. (2024). Toward a Universal Dependencies Treebank of Old English: Representing the Morphological Relatedness of Un-Derivatives. Languages , 9(3), 76. https://doi.org/10.3390/languages9030076 Martín Arista, J., & Ojanguren López, A. E. (2018). Grammaticalization and deflexion in progress. The past participle in the Old English passive. Studia Neophilologica , 90(2), 155-175. https://doi.org/10.1080/00393274.2018.1444421 Martín Arista, J., Domínguez Barragán, S., Fidalgo Allo, L., García Fernández, L., Hamdoun Bghiyel, Y., Lacalle Palacios, M., Mateo Mendaza, R., Novo Urraca, C., Ojanguren López, A. E., Ruíz Narbona, E., Torre Alonso, R., & Vea Escarza, R. (2023). ParCorOEv3. An open access annotated parallel corpus Old English-English . Nerthus Project, Universidad de La Rioja. www.nerthusproject.com Martín Arista, J., Ojanguren López, A. E., & Domínguez Barragán, S. (2025). Universal Dependencies annotation of Old English with spaCy and MobileBERT. Evaluation and perspectives. Procesamiento del Lenguaje Natural (forthcoming). Meechan-Maddon, A., & Nivre, J. (2019). How to parse low-resource languages: Cross-lingual parsing, target language annotation, or both? In Proceedings of the Fifth International Conference on Dependency Linguistics (DepLing 2019) (pp. 112-120). Association for Computational Linguistics. https://doi.org/10.48550/arXiv.1908.10626 Mielke, S. J., Barrault, L., Liu, F., & Bapna, A. (2021). Between words and characters: A Brief history of open-vocabulary modeling and tokenization in NLP. Transactions of the Association for Computational Linguistics , 9, 1153-1175. https://doi.org/10.1162/tacl_a_00416 Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013a). Efficient estimation of word representations in vector space. In International Conference on Learning Representations . https://doi.org/10.48550/arXiv.1301.3781 Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., & Dean, J. (2013b). Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems (pp. 3111-3119). https://doi.org/10.48550/arXiv.1310.4546 Nadkarni, P. M., Ohno-Machado, L., & Chapman, W. W. (2011). Natural language processing: An introduction. Journal of the American Medical Informatics Association , 18(5), 544-551. https://doi.org/10.1136/amiajnl-2011-000464 Parkes, M. B. (2016). Pause and Effect: An Introduction to the History of Punctuation in the West . Routledge. Pennington, J., Socher, R., & Manning, C. D. (2014). GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (pp. 1532-1543). https://doi.org/10.3115/v1/D14-1162 Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018). Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics (pp. 2227-2237). https://doi.org/10.18653/v1/N18-1202 Pintzuk, S., & Plug, L. (Eds.). (2001). The York-Helsinki Parsed Corpus of Old English Poetry . Department of Language and Linguistic Science, University of York. http://www-users.york.ac.uk/~lang18/pcorpus.html Qi, P., Zhang, Y., Zhang, Y., Bolton, J., & Manning, C. D. (2020). Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations (pp. 101-108). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-demos.14 Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training . OpenAI. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog , 1(8). Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., & Sutskever, I. (2021). Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (pp. 8748-8763). https://doi.org/10.48550/arXiv.2103.00020 Rust, P., Pfeifer, J., Vulić, I., Ruder, S., & Gurevych, I. (2024). How does vocabulary design impact the utility of large language models? Transactions of the Association for Computational Linguistics , 12, 167-188. https://doi.org/10.1162/tacl_a_00609 Schuster, M., & Nakajima, K. (2012). Japanese and Korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 5149-5152). https://doi.org/10.1109/ICASSP.2012.6289079 Scragg, D. G. (1974). A History of English Spelling . Manchester University Press. Sennrich, R., Haddow, B., & Birch, A. (2016). Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (pp. 1715-1725). https://doi.org/10.18653/v1/P16-1162 Shareghi, E., Bollegala, D., Kann, K., & Bisk, Y. (2023). Advancements and challenges in morphologically rich languages. Annual Review of Linguistics , 9, 251-274. https://doi.org/10.1146/annurev-linguistics-031720-123720 Taylor, A., Warner, A., Pintzuk, S., & Beths, F. (2003). The York-Toronto-Helsinki Parsed Corpus of Old English Prose . Department of Language and Linguistic Science, University of York. http://www-users.york.ac.uk/~lang22/YCOE/YcoeHome.htm Turney, P. D., & Pantel, P. (2010). From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research , 37, 141-188. https://doi.org/10.1613/jair.2934 Tables Tables 1 to 6 are available in the Supplementary Files section Additional Declarations The authors declare no competing interests. Supplementary Files Tables.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6473289","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":444499757,"identity":"02b859c3-d623-4731-8b18-92995197cdc5","order_by":0,"name":"Javier Martín Arista","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABDklEQVRIiWNgGAWjYBACxgYGhgMwzgGGCgYGNpCYBHFamIGMM0RoQQLMQAPaiFHXfsbwwMc9DNH8s88fPFw473Aen/ThxgcWNQxy8g04HNaTY3BwxjOG3BnnkhkOz9x2uJiNL7HZQOIYg7HBARxaGtISDvMcYMhtOMPMcJh32+HENh7GNgkJNobEDTgcxtj/LOHwH6CW+WAtc2Ba/jHUz8flsBnJBw4Dgyp3A1hLA1SLZBtDAiLw0bU8PnCw54BE7sYzzAaHeY6lg7Q0G0j2SRhuwKHFsD+x+cOPAza5884wPv7MU2OdOL+H/eFjiW828rhCzBAijhZ1zBJ4IlMeu3M/4NQwCkbBKBgFIxAAALamXiB/j6iVAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0001-9900-0104","institution":"University of La Rioja","correspondingAuthor":true,"prefix":"","firstName":"Javier","middleName":"Martín","lastName":"Arista","suffix":""},{"id":444501289,"identity":"7b3b1dbe-84f8-4895-96a0-28e756defe15","order_by":1,"name":"Darío Metola Rodríguez","email":"","orcid":"https://orcid.org/0000-0002-7980-7098","institution":"Reddit","correspondingAuthor":false,"prefix":"","firstName":"Darío","middleName":"Metola","lastName":"Rodríguez","suffix":""}],"badges":[],"createdAt":"2025-04-17 15:57:21","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":false,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-6473289/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6473289/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":80994705,"identity":"6eda58d0-4601-4eab-a37d-bf75ae352ab3","added_by":"auto","created_at":"2025-04-21 04:48:02","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":378026,"visible":true,"origin":"","legend":"\u003cp\u003eScatter plot representation of Old English vectors.\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-6473289/v1/12e68a222592ca974503f3b5.png"},{"id":80994576,"identity":"8cb02dd3-e6d7-4371-8ab2-bc61bb7a1436","added_by":"auto","created_at":"2025-04-21 04:40:02","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":54323,"visible":true,"origin":"","legend":"\u003cp\u003eModel performance comparison.\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-6473289/v1/ef7872426411bbb02760a103.png"},{"id":80995257,"identity":"db66f899-8c9c-4d0e-9fb1-66c01623b53a","added_by":"auto","created_at":"2025-04-21 04:56:03","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":830016,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6473289/v1/b9d2422d-81db-45cf-823f-425a59bc93dd.pdf"},{"id":80994575,"identity":"71ef0c28-0c99-490b-a4e0-251a104f4be7","added_by":"auto","created_at":"2025-04-21 04:40:02","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":26523,"visible":true,"origin":"","legend":"","description":"","filename":"Tables.docx","url":"https://assets-eu.researchsquare.com/files/rs-6473289/v1/68fb0b4fc02ee472967fdbb8.docx"}],"financialInterests":"The authors declare no competing interests.","formattedTitle":"\u003cp\u003e\u003cstrong\u003eLanguage-specific embeddings of Old English with character-level processing\u003c/strong\u003e\u003c/p\u003e","fulltext":[{"header":"1. Aims and scope","content":"\u003cp\u003eThe aim of this study is to present and evaluate a comprehensive neural network model for Old English linguistic analysis which has been designed to provide researchers with an effective computational tool adhering to principles of availability, scalability, and high performance. Building upon previous approaches to Old English language modeling, our work takes a new line: the incorporation of language-specific word embeddings derived directly from \u003cem\u003eThe Dictionary of Old English Corpus\u003c/em\u003e (DOEC). This is a significant departure from previous studies based on the idea that low-resource historical languages like Old English must rely primarily on cross-lingual transfer learning from related modern languages.[1]\u003c/p\u003e\n\u003cp\u003eOur research addresses three questions: (i) Can language-specific word embeddings derived from historical corpora improve parsing accuracy compared to multilingual transfer learning? (ii) Do character-level models offer significant advantages over word-level models for processing morphologically complex historical languages? (iii) How do the different components of the neural pipeline perform on specific linguistic tasks of Old English processing? Through model comparison and assessment of performance metrics, the article demonstrates that a neural architecture combining language-specific embeddings with character-level processing outperforms previous approaches to Old English parsing. Our model achieves state-of-the-art results while remaining computationally efficient and readily adaptable for research applications in historical linguistics and digital humanities.\u003c/p\u003e\n\u003cp\u003eThe article is structured as follows. Section 2 reviews embedding technologies in NLP libraries. Section 3 surveys recent advances in the computational processing of historical languages, with particular focus on Old English. Section 4 details our methodology and presents comprehensive performance results for each component of our pipeline, while Section 5 shows the performance by component. Section 6 discusses the results by comparing our approach with previous work on Old English parsing. Section 7 summarises the main conclusions and makes an assessment of our contribution for future research in the computational processing of historical languages.\u003c/p\u003e\n\u003cp\u003e[1] British English spelling is kept throughout the article except when referring to components and tasks of PLN, such as \u003cem\u003etokenizer\u003c/em\u003e or \u003cem\u003enormalization\u003c/em\u003e, where the -z- orthography, far more frequent in NLP studies, has been preferred.\u003c/p\u003e"},{"header":"2. Embeddings in NLP libraries","content":"\u003cp\u003eThe NLP libraries NLTK, spaCy, and Stanza represent three different generations and designs, in such a way that each has advantages for different applications. NLTK, the oldest library, serves primarily educational purposes with extensive algorithmic implementations and corpus access but has performance limitations in production environments, as pointed out by Bird et al. (2019). spaCy, developed by Explosion AI, addresses these shortcomings through Cython-optimised code and an intuitive object-oriented design, which makes it the preferred choice for industry applications requiring high-performance English text processing (Honnibal et al., 2020). Stanza, created by Stanford NLP Group, takes a different approach by implementing state-of-the-art neural network models with an emphasis on linguistic accuracy and multilingual support (Qi et al., 2020). Stanza currently covers 66 languages across 112 datasets and provides a Python interface to Stanford CoreNLP for advanced functionalities like coreference resolution. The main difference among the three NLP libraries lies in their priorities: spaCy optimises speed and developer experience, whereas Stanza prioritises linguistic accuracy and language coverage, which leads many users to adopt hybrid approaches that leverage the strengths of each library.\u003c/p\u003e\n\u003cp\u003eThese libraries perform a set of text analysis tasks, typically beginning with fundamental operations like tokenization, part-of-speech tagging, and lemmatization; moving to dependency parsing and named entity recognition; and finishing up in advanced semantic analysis including sentiment analysis, coreference resolution, and relation extraction. While basic tasks like tokenization and simple rule-based tagging can function without embeddings, most modern approaches to NLP rely on embeddings for optimal performance. NLP embeddings are based on the hypothesis that words that occur in similar contexts tend to have similar meanings (Turney \u0026amp; Pantel, 2010). An embedding is a dense vector representation of a linguistic unit (usually a word) in a continuous, low-dimensional space where the semantic and syntactic properties of linguistic units are captured by vector proximity (Mikolov et al., 2013). \u0026nbsp; While traditional vector representations like one-hot encoding treat words as isolated, discrete symbols in high-dimensional sparse vectors (Nadkarni et al., 2011), embeddings encode semantic relations in dense, lower-dimensional vectors where similar words cluster together in the vector space (Goldberg, 2016). Embeddings, therefore, enable generalisation across meanings and provide computational efficiency (Jurafsky \u0026amp; Martin, 2023).\u003c/p\u003e\n\u003cp\u003eWord embeddings have undergone rapid evolution over the past decade, with several breakthrough papers establishing the foundations of modern approaches: Mikolov et al. (2013a) revolutionised the field with Word2Vec, which introduced efficient models for learning high-quality word vectors from large datasets. Their follow-up work (Mikolov et al., 2013b) demonstrated the ability of these vectors to capture semantic and syntactic regularities. Pennington et al. (2014) advanced the field further with GloVe (Global Vectors). GloVe combines global matrix factorisation with local context window methods to efficiently leverage statistical information from entire corpora. The paradigm shift toward contextual embeddings began with Peters et al. (2018), who introduced ELMo (Embeddings from Language Models). Peters et al. (2018) proved that deep bidirectional language models could generate context-sensitive representations that significantly improve performance across various NLP tasks. This evolution led to Devlin et al.´s (2019) introduction of BERT (Bidirectional Encoder Representations from Transformers), which established the state-of-the-art of pre-training and fine-tuning that has transformed NLP approaches and performance standards.\u003c/p\u003e\n\u003cp\u003eWord embeddings fall into three main categories, namely, static word embeddings, contextual embeddings and sub-lexical embeddings.\u003c/p\u003e\n\u003cp\u003eStatic word embeddings include Word2Vec (Mikolov et al.,, 2013a), which employs CBOW and Skip-gram architectures to learn single-vector word representations by predicting words from context or vice versa; GloVe (Pennington et al., 2014), which combines global matrix factorisation with local context window methods to capture both statistical and contextual information; and FastText (Bojanowski et al., 2017), which extends Word2Vec by representing words as bags of character n-grams to handle morphologically rich languages and unseen words.\u003c/p\u003e\n\u003cp\u003eContextual embeddings generate dynamic representations based on surrounding text: ELMo (Peters et al., 2018) uses bidirectional LSTMs to produce context-dependent vectors; BERT (Devlin et al., 2019) employs transformer architecture with masked language modeling for deeply bidirectional representations; and GPT models (Radford et al., 2018, 2019; Brown et al., 2020) resort to unidirectional transformers trained on next-word prediction. This evolution from static to contextual representations marks a fundamental shift in NLP because it reconceptualises words from isolated units with fixed meanings to dynamic entities whose representation varies with linguistic context. Against this backdrop, specialised embedding types include cross-lingual embeddings (Conneau et al., 2018) that align vector spaces across languages; domain-specific embeddings trained on specialised corpora; and multi-modal embeddings that integrate text with other data types like images (CLIP by Radford et al., 2021) or audio\u003c/p\u003e\n\u003cp\u003eSub-lexical embedding approaches operate at multiple granularity levels, including character-level, subword level and hierarchical approaches.\u003c/p\u003e\n\u003cp\u003eCharacter-level embeddings provide fundamental sub-word information. For example, Kim et al. (2016) demonstrate effective character-level CNNs for language modeling and text classification by applying convolutional filters to character embeddings to capture morphological patterns without explicit linguistic knowledge. In the same line, Ma \u0026amp; Hovy (2016) show significant improvements in sequence labeling tasks by combining character-level CNNs with word embeddings.\u003c/p\u003e\n\u003cp\u003eSubword tokenization occupies the middle ground with three major approaches: Byte-Pair Encoding, which merges frequent character pairs to create subword units and is now used by many transformer models (Sennrich et al., 2016); WordPiece (Schuster \u0026amp; Nakajima, 2012), adopted in BERT, which uses a likelihood maximisation criterion for merging; and Unigram Language Model (Kudo, 2018), which treats segmentation as a probabilistic problem.\u003c/p\u003e\n\u003cp\u003eHierarchical approaches combine multiple representation levels and are contextualised. Ling et al. (2015), for instance, propose character-to-word models using bidirectional LSTMs (Long Short-Term Memory recurrent neural networks), and Peters et al. (2018) implement multi-level representations derived from LSTMs in ELMo (Embeddings from Language Models).\u003c/p\u003e\n\u003cp\u003eThe picture that emerges from a brief review of recent reseach is that sub-lexical approaches offer substantial advantages: vocabulary efficiency (Mielke et al., 2021), morphological awareness for morphologically rich languages (Çetinoğlu \u0026amp; Çöltekin, 2022; Shareghi et al., 2023), efficient handling of out-of-vocabulary words (Hofmann, 2022), sensitivity to spelling variations (Clark et al., 2022), and compatibility with cross-lingual transfer learning (Chung et al., 2023). As Rust et al. (2024) remark, modern transformer models typically employ subword tokenization, while specialised NLP applications with high morphological demands often benefit from dealing with character, subword, and word levels simultaneously (Shareghi et al., 2023).\u003c/p\u003e\n\u003cp\u003eMajor NLP libraries incorporate embeddings as core components. spaCy integrates pre-trained word embeddings directly into its processing pipeline. As described by Honnibal et al. (2020), spaCy offers multiple embedding options, including static GloVe vectors and transformers-based contextual embeddings. These embeddings serve as input features for its neural network models that handle tasks like parsing and named entity recognition. The architecture of spaCy allows computational linguists to swap embedding models based on their specific requirements and resource constraints. Stanza, according to Qi et al. (2020), takes a more comprehensive approach to embeddings. It implements a multi-level embedding strategy that combines word-level, character-level, and when available, BERT-based contextual embeddings. This approach guarantees the handling of morphologically rich languages and out-of-vocabulary words. The neural models of Stanza rely on these embeddings for all analysis tasks. Unlike spaCy’s more modular approach, the embedding strategy of Stanza is more deeply integrated within its neural architecture, thus prioritising linguistic accuracy over customisability (Qi et al., 2020).\u003c/p\u003e"},{"header":"3. The computational parsing of Old English","content":"\u003cp\u003eRecent advances in the computational processing of historical languages have addressed the questions raised by ancient texts through different techniques and methodologies. For diachronic semantics, Hamilton et al. (2018) apply diachronic word embeddings to detect linguistic shifts, while Dubossarsky et al. (2019) develop a temporal referencing framework that distinguishes genuine semantic change from random sampling fluctuations. In text normalization, Bollmann (2019) demonstrates that character-based neural machine translation approaches outperform rule-based systems for handling non-standardised orthography. Manjavacas et al. (2019) develop a joint-learning approach for lemmatization that incorporates character-level embeddings and contextual information.\u003c/p\u003e\n\u003cp\u003eIn the field of Old English studies, a relatively wide variety of digital resources can be used for computational analysis. \u003cem\u003eThe Dictionary of Old English Corpus\u003c/em\u003e (DOEC) contains approximately 3.1 million words with minimal textual annotation. \u003cem\u003eThe York-Toronto-Helsinki Parsed Corpus of Old English Prose\u003c/em\u003e (YCOE) offers approximately 1.5 million words with POS tagging and constituent-oriented syntactic annotation, while its poetic counterpart comprises 71,000 words of verse annotated with the same tagset. \u003cem\u003eThe Parallel Corpus of Old English Prose\u003c/em\u003e provides sentence-level and word-level alignment between the Old and Present-Day English versions of selected texts comprising 300,000 words in the source language. These corpora are complemented by lexicographical sources that include \u003cem\u003eThe Dictionary of Old English\u003c/em\u003e (DOE), covering the headwords A-I with exhaustive morphological, grammatical and semantic information, and the digital version of \u003cem\u003eBosworth-Toller Anglo-Saxon Dictionary\u003c/em\u003e, with approximately 35,000 machine-readable headwords.\u003c/p\u003e\n\u003cp\u003eAlthough these resources constitute a comparatively large dataset if compared with other old Germanic languages, they do not suffice to meet the requirements of up-to-date computational resources like transformer-based models (Mart\u0026iacute;n Arista et al., 2025). Beyond resource scarcity, the computational parsing of Old English presents other difficulties like morphological complexity, non-standardised spelling, decaying inflections and flexible word order. Recent work has made some progress in applying modern NLP techniques to this diachronic stage of the English language. Faulkner (2023), for example, introduces a corpus philology approach that extracts large datasets for analysing spelling variations, effectively challenging traditional philological claims about Old English spelling through the analysis of over 216,000 spellings for words beginning with \u0026lt;h\u0026gt; followed by a vowel.\u003c/p\u003e\n\u003cp\u003eBrigada Villa \u0026amp; Giarda (2023a) test multilingual parsing performance on Old English using different language sets for training. Their findings show that models incorporating Old English data significantly outperform those without it and that Icelandic and German combinations perform best due to their retention of morphosyntactic features or, in the case of Icelandic, certain graphemes. In a follow-up study, Brigada Villa \u0026amp; Giarda (2023b) develop a rule-based approach for root identification in Old English which achieves 89.49% accuracy. This significantly outperforms their previous multilingual parser approach, although it is limited to the adaptation of the YCOE constituent parsing to dependency parsing.\u003c/p\u003e\n\u003cp\u003eMart\u0026iacute;n Arista (2022) establishes the foundations for applying Universal Dependencies to Old English. The thrust of the argument is that standard UD annotation lacks sufficient representation of the associative lexicon of Old English. This author adds gloss and morphological relatedness fields to the annotation scheme. Building on this framework, Mart\u0026iacute;n Arista (2024) demonstrates that adding a MORPHREL field in the CoNLL-U Plus format can capture both short-distance and long-distance morphological relations among \u003cem\u003eun\u003c/em\u003e-derivatives in Old English.\u003c/p\u003e\n\u003cp\u003eMart\u0026iacute;n Arista et al. (2025) evaluate different training procedures for automatic UD annotation of Old English using spaCy and MobileBERT. By testing various corpus sizes, these authors find that larger training corpora improve performance across all pipeline stages and that pre-training the tok2vec stage yields better results than the default pipeline. Mart\u0026iacute;n Arista et al.\u0026rsquo;s (2025) model achieves precision rates of approximately 95% for lexical categories and 80% for dependency relations, thus outperforming previous multilingual approaches.\u003c/p\u003e\n\u003cp\u003eThese advances in the computational processing of Old English are compared with the proposal made in this article in Section 6.\u003c/p\u003e"},{"header":"4. Methodology","content":"\u003cp\u003eThe main innovation of our approach is the development of language-specific word embeddings for Old English. Previous approaches to Old English language model building either omitted this aspect entirely or attempted to leverage word vectors from typologically related languages (Brigada Villa \u0026amp; Giarda, 2023). Our model, by contrast, incorporates word embeddings constructed directly from the DOEC, comprising approximately 57,000 tokens distributed across 100 dimensions.\u003c/p\u003e\n\u003cp\u003eWord embeddings represent a dense, low-dimensional projection of the sparse high-dimensional word vector space. While traditional word vectors are typically sparse with most entries being zero counts, embeddings compress this information into dense vectors with continuous real values (both positive and negative), ranging in dimensionality from 50 to 1000 (Jurafsky et al., 2023). The dimensionality selection represents a critical balance between computational efficiency and semantic expressiveness. Lower-dimensional embeddings (50-100 dimensions) consume less memory and process more quickly but may lose subtle semantic distinctions. Higher-dimensional embeddings (300-1000 dimensions) capture fine-grain relations but require more computational resources.\u003c/p\u003e\n\u003cp\u003eFor Old English, we empirically determined that 100-dimensional embeddings provide an optimal balance, as they capture sufficient semantic information while being compatible with the relatively small corpus size. This dimensionality allows the model to encode various linguistic aspects such as grammatical gender, verbal tense, and number; and maintains the distributional semantics necessary for contextual prediction. Figure 1 illustrates these embeddings with randomly selected words positioned according to their contextual similarity: words appearing at the center of the scatter plot exhibit stronger contextual relations than those at the periphery.\u003c/p\u003e\n\u003cp\u003eThe embeddings were created using word2vec, which used the continuous bag-of-words (CBOW) architecture with a context window of 5 words. This approach allows the model to learn word representations based on their distributional properties in the corpus, in such a way that words appearing in similar contexts tend to have similar meanings (Chaubard et al., 2019). The resulting embeddings serve as the point of departure for the POS tagger and dependency parser processors. This substantially improves their performance compared to models without language-specific embeddings.\u003c/p\u003e\n\u003cp\u003eOur Old English language model was implemented using the Stanza Python library (Qi et al., 2020), which provides a neural network pipeline built on PyTorch. The complete model comprises four core processors: tokenizer, lemmatizer, POS tagger, and dependency parser. A distinctive feature of our approach is the comparison between traditional word-level tokenization and character-level models, which operate on individual characters rather than complete words.\u003c/p\u003e\n\u003cp\u003eCharacter-level models have gained prominence for their effectiveness in handling languages with complex morphology (Adel et al., 2018). Unlike word-level models that treat each word as an atomic unit, character-level models process text character by character, thus allowing them to capture subword information and morphological patterns. This approach is particularly valuable for historical languages like Old English that exhibit rich inflectional morphology and orthographic variation.\u003c/p\u003e\n\u003cp\u003eThe Stanza library supports both character-level and non-character-level training options. To evaluate their relative effectiveness for Old English, we trained each processor (except the tokenizer, which cannot be implemented as a character-level model in Stanza) in both configurations. This comparative approach allowed us to quantify the performance benefits of character-level processing specifically for Old English.\u003c/p\u003e\n\u003cp\u003eFor training and evaluation, we used a dataset of 25,000 annotated Old English words (Mart\u0026iacute;n Arista et al., 2023), divided into training (80%), development (10%), and test (10%) sets. This three-way split helps prevent overfitting by providing separate evaluation criteria during and after training. Each processor was trained sequentially on the training set. The fine-tuning of parameters was based on development set performance before final evaluation on the test set.\u003c/p\u003e"},{"header":"5. Results","content":"\u003cp\u003eThe tokenizer demonstrated excellent performance in word segmentation but faced difficulties with sentence boundaries. Table 1 tabulates the metrics of the performance of the tokenizer.\u003c/p\u003e\n\u003cp\u003eThe relatively poor performance in sentence segmentation (64.06% on the test set) can be attributed to the inconsistent punctuation in Old English texts. Unlike contemporary English, which follows standardised punctuation conventions, Old English manuscripts exhibit considerable variation in punctuation usage (see, for instance, Scragg 1974; Gneuss \u0026amp; Lapidge, 2014; Parkes, 2016). The 14-percentage-point drop between development and test set sentence segmentation (78.08% vs. 64.06%) indicates that the test set contains more complex or ambiguous sentence structures that pose additional difficulties.\u003c/p\u003e\n\u003cp\u003eThe lemmatizer was the first processor for which we could implement and compare character-level and non-character-level approaches. Table 2 provides a performance comparison between these implementations.\u003c/p\u003e\n\u003cp\u003eThe character-level model demonstrated greater training efficiency because it achieved higher accuracy (72.49% vs. 71.61% on the development set) and, moreover, converged twice as quickly (5 epochs vs. 10 epochs).\u0026nbsp;The superior results of the character-level model is related to its ability to grasp morphological features at the sub-word level, which are widespread through the inflectional morphology of Old English. The efficiency advantage is particularly significant given the computational constraints often faced when processing historical languages like Old English. In this respect,\u0026nbsp;the modest 1.08 percentage point improvement in accuracy represents hundreds of additional correctly lemmatized words in a typical corpus section, which improves downstream task performance.\u003c/p\u003e\n\u003cp\u003eThe POS tagger was the first processor to incorporate both character-level processing and the word embeddings that we developed. Table 3 presents a detailed performance comparison across different tagging categories.\u003c/p\u003e\n\u003cp\u003eAs shown in Table, 3, the character-level model consistently outperformed the non-character-level approach across all tagging categories.\u0026nbsp;The high UPOS accuracy (94.21%) shows that basic part-of-speech categorisation is reliable and can provide a solid foundation for downstream syntactic analysis.\u0026nbsp;The 1.95 percentage point advantage in UFeats accuracy (90.12% vs. 88.17%) highlights the ability of the character-level model to capture inflectional morphology, which is essential for identifying agreement patterns at the noun phrase and sentence level.\u003c/p\u003e\n\u003cp\u003eThe dependency parser represents the most complex component of our pipeline and the one that most directly benefits from both character-level processing and word embeddings. Table 4 presents the performance across various parsing metrics.\u003c/p\u003e\n\u003cp\u003eAs can be seen in Table 4, the character-level model achieved superior scores across all metrics (UAS: 88.92% vs. 85.01%; LAS: 79.65% vs. 74.78% on the test set) and converged in half the time (18 batches vs. 36 batches). This efficiency advantage results in significant time and resource savings when working with large corpora or when retraining the model with additional data.\u0026nbsp;The consistent gap between UAS and LAS scores (approximately 9-10 percentage points) evidences that the model is more effective at identifying syntactic relationships than correctly labeling them. The lower MLAS (68.23%) and BLEX (66.72%) scores further demonstrate that performance decreases when morphological features and lemmatization are incorporated into evaluation metrics, which, in turn, highlights the difficulties of the combined analysis of Old English morphology and syntax.\u0026nbsp;Figure 2 presents an overall representation of the performance of each model, evaluated agains the dev and test sets after being trained.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;To gain a deeper understanding of the performance of the dependency parser, we analysed precision, recall, and F1-scores for specific dependency relations. Table 5 highlights performance for selected relations using the character-level model on the test set.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;Three levels of performance emerge in Table 5. High-performing relations (F1 \u0026gt; 0.80) typically involve limited structural variation and occur frequently in the training data. They include coordinating conjunctions (cc, 0.9060), numeral modifiers (nummod, 0.9442), and adverbial modifiers (advmod, 0.8221). Average-performing relations (F1 0.60-0.80) comprise core arguments like nominal subjects (nsubj, 0.7716) and root relations (root, 0.8357). They show reasonably good performance but present shortcomings in complex sentences. The relative clause relation (acl:relcl, 0.6667) performs adequately despite involving subordination. Low-performing relations (F1 \u0026lt; 0.40) include complex clausal relations (ccomp, 0.2609; xcomp, 0.0800), auxiliaries (aux, 0.2222), and orphaned elements (orphan, 0.0889). These configurations often involve long-distance dependencies, discontinuous constituents, or ambiguous syntactic relations. The particularly poor performance on auxiliary relations (aux, F1=0.2222) may reflect the difficulty in distinguishing auxiliary verbs from main verbs in Old English, where the grammaticalisation of auxiliaries was still in progress (Mart\u0026iacute;n Arista \u0026amp; Ojanguren L\u0026oacute;pez, 2018; Mart\u0026iacute;n Arista, 2020). Similarly, the issues with oblique nominals (obl, F1=0.4000) may result from the case-marking system, where the function of noun phrases is indicated morphologically rather than positionally, which creates ambiguity for the parser. The extremely low performance on open clausal complements (xcomp, F1=0.0800) and orphaned constituents (orphan, F1=0.0889) points to specific syntactic constructions that require special attention in the future development of the model. These constructions often involve ellipsis or discontinuous dependencies that defy the sequential processing approach of neural models.\u003c/p\u003e"},{"header":"6. Discussion: innovations and performance","content":"\u003cp\u003eThis section compares the approaches to computational parsing and language modeling of Old English taken by Brigada Villa \u0026amp; Giarda (2023) and Mart\u0026iacute;n Arista et al. (2025) with this study. The focus in on methodological innovations and performance metrics.\u003c/p\u003e\n\u003cp\u003eThe three papers represent distinct but complementary approaches to the computational analysis of Old English. Brigada Villa \u0026amp; Giarda (2023) rely on a multilingual parser to evaluate cross-lingual transfer learning approaches for parsing Old English. These authors train UUParser v2.4 by using various combinations of modern Germanic languages (Swedish, German, and Icelandic) both with and without Old English data. Then, they examine how related languages might enhance parsing performance for this low-resource historical language. Mart\u0026iacute;n Arista et al. (2025) train a model from scratch on Old English data exclusively, testing different configurations for a spaCy pipeline to automatically annotate Old English with Universal Dependencies. They compare three approaches (default pipeline, pre-trained tok2vec, and transformer-based) with training corpora of varying sizes (1,000-20,000 words). The present study implements a comprehensive Old English language model built using the Stanza Python library. It incorporates word embeddings derived from the DOEC. This study contrasts character-level models with traditional word tokenization approaches across a 25,000-word dataset, with detailed performance metrics for each processor in the pipeline.\u003c/p\u003e\n\u003cp\u003eA crucial difference among these studies is their approach to training data and model architecture. Brigada Villa \u0026amp; Giarda (2023) worked with a small dataset (292 sentences, 5,315 tokens) from religious prose texts, employing cross-lingual transfer learning with modern Germanic languages (Icelandic, German, and Swedish). These languages were chosen for their typological similarities to Old English, such as the case inflection of Icelandic and the flexible word order of German. The authors intentionally limited the support language treebanks to 60,000 tokens each to prevent size-related bias. Their model training involved a 30-epoch process, selecting the best performing epoch based on LAS scores on the development set. Mart\u0026iacute;n Arista et al. (2025) used a larger dataset (25,000 words) from various Old English texts, including homilies, chronicles, and history, as well as biblical and legal documents. They carried out monolingual training using spaCy pipelines, testing three approaches: a baseline tok2vec model, a pretrained model initialised on unannotated Old English text, and a MobileBERT transformer. For the pre-trained model, they used the full Old English corpus (3 million words) to initialise the tok2vec stage, and for the transformer model, they trained a custom tokenizer to handle Old English-specific graphemes (\u0026aelig;, \u0026AElig;, \u0026eth;, \u0026ETH;, \u0026thorn;, \u0026THORN;). This study utilises the Stanza neural network pipeline with both character-level and non-character-level models. The 25,000-word dataset is divided into training (80%), development (10%), and test (10%) sets. A distinctive feature of this approach is the creation of language-specific word embeddings for Old English using word2vec on the DOEC, representing approximately 57,000 tokens across 100 dimensions.\u003c/p\u003e\n\u003cp\u003eFocusing to the accuracy metrics, the monolingual approach adopted in Mart\u0026iacute;n Arista et al. (2025) and in this study turns out more accurate results, in line with Meechan-Maddon \u0026amp; Nivre (2019), who demonstrate that in the parsing of low-resource languages, annotation of the target language yields more accurate results than cross-lingual transfer models.\u003c/p\u003e\n\u003cp\u003eBrigada Villa \u0026amp; Giarda (2023) report the following key metrics for their best models: Old English monolingual model: 60.79% UAS, 64.39% LA, 47.23% LAS; OE + Icelandic: 68.44% UAS, 73.76% LA, 58.70% LAS (best overall UAS and LAS); OE + German + Icelandic: 66.34% UAS, 74.29% LA, 57.42% LAS (best LA). Their best model, combining Old English with Icelandic, achieved 68.44% UAS and 58.70% LAS. Their approach demonstrated that related languages could improve parsing accuracy, though adding German or Swedish showed diminishing returns.\u003c/p\u003e\n\u003cp\u003eMart\u0026iacute;n Arista et al. (2025) show performance increasing with larger training datasets across all three of their model configurations. The pretrained tok2vec model achieves superior results (83.24% UAS, 74.23% LAS) with 20k words of training data. The findings emphasise the importance of dataset size and domain-specific pretraining over cross-linguistic transfer. These authors achieve the following results: TAG_ACC (XPOS): 75-95% (improving with larger datasets); POS_ACC (UPOS): 75-95% (improving with larger datasets); MORPH_ACC: 75-88% (improving with larger datasets); LEMMA_ACC: 75-87% (improving with larger datasets); DEP_UAS: 70-82% (improving with larger datasets); DEP_LAS: 65-73% (improving with larger datasets); SENTS_F: 75-78% (relatively stable across all dataset sizes).\u003c/p\u003e\n\u003cp\u003eThe present study shows metrics for specific processors. It also provides a detailed description of precision, recall, and F1 scores for every dependency relation type: Tokenizer: 99.63% (dev)/99.84% (test) accuracy for word tokenization; 78.08% (dev)/64.06% (test) for sentence segmentation; Lemmatizer: 72.49% with character model (best score at epoch 5) vs. 71.61% with no-character model (best score at epoch 10); POS Tagger: Character models show consistent advantages across all tag categories; Dependency Parser: UAS: 84-89% (character model), 81-85% (no-character model); LAS: 75-80% (character model), 71-75% (no-character model); Root: 83.57% accuracy (test charlm).\u003c/p\u003e\n\u003cp\u003eWhen directly comparing performance metrics, it turns out that with respect to overall dependency parsing accuracy Brigada Villa \u0026amp; Giarda\u0026acute;s (2023) best model achieved 68.44% UAS and 58.70% LAS. The best model by Mart\u0026iacute;n Arista et al. (2025) achieved 83.24% UAS and 74.23% LAS, while the best model presented in this study achieves ~84-89% UAS and ~75-80% LAS. \u0026nbsp;The persistent gap between UAS and LAS scores evidenced by all studies indicates the difficulty of correctly labeling dependency relations compared to identifying head-dependent attachments. Nevertheless, the clear progression in performance demonstrates that both the specific Old English word embeddings and the character-level modeling in the later studies improve parsing accuracy. Table 6 compares the performance metrics across the three Old English parsing studies.\u003c/p\u003e\n\u003cp\u003eAll three studies identify problematic structures and functions in parsing Old English. Brigada Villa \u0026amp; Giarda (2023) highlight several key error sources: words with multiple grammatical functions (e.g., \u003cem\u003ene\u003c/em\u003e \u0026lsquo;not\u0026rsquo; as both adverb and conjunction), postpositions, which none of their models could correctly handle, relative clauses with varying pronouns (only correctly identifying those with \u003cem\u003e\u0026thorn;e\u003c/em\u003e \u0026lsquo;that\u0026rsquo;), and discontinuous relative clauses, where the models incorrectly attached clauses to the nearest noun rather than their true antecedent. Mart\u0026iacute;n Arista et al. (2025) find more structural challenges: negative contractions of verbs, pronouns, and adverbs, noun and prepositional phrases, especially flat multiword expressions, roots in copulative, existential, and conditional sentences, oblique nominals without prepositional marking, foreign words, complex sentences with orphans, clausal complements, and adverbial modifiers. This study provides the most detailed error analysis, with specific dependency relations showing particularly low performance, like auxiliary relations (aux/aux:pass, 18-33% precision), clausal relations (ccomp/xcomp, very low precision) and oblique relations (obl, 34-40% precision in some contexts).\u003c/p\u003e\n\u003cp\u003eEach of the three studies in computational approaches to Old English contributes some methodological innovations that collectively advance historical language processing. Brigada Villa \u0026amp; Giarda (2023) demonstrate that cross-lingual transfer learning can effectively leverage knowledge from modern Germanic languages for Old English parsing. Mart\u0026iacute;n Arista et al. (2025) contribute through monolingual training approaches optimised for historical language processing. The present study introduces two basic innovations: word embeddings derived from the entire DOEC and systematic comparison between character-level and token-level models. The accuracy metrics underscore the importance of the specific embedding and the superiority of the character-level model. The present study also demonstrates that the Stanza-based pipeline can provide an integrated solution and identifies additional challenging dependency relations for future research.\u003c/p\u003e"},{"header":"7. Conclusion","content":"\u003cp\u003eOur study makes several contributions to computational linguistics for historical languages. We addressed three research questions, with results that demonstrate some advantages of our approach.\u003c/p\u003e\n\u003cp\u003eFirst, we show that language-specific word embeddings substantially improve parsing accuracy for Old English, which parts company with the multilingual metod based on the idea that transfer from related languages is the only viable approach for low-resource historical languages. Our 100-dimensional embeddings derived from the DOEC capture semantic relations specific to Old English and enables more accurate contextual prediction than would be possible with embeddings from typologically or areally related languages.\u003c/p\u003e\n\u003cp\u003eSecond, our comparative analysis provides empirical evidence for the superiority of character-level approaches in languages with complex morphology and orthographic variation. For Old English, character-level models consistently outperformed their word-level counterparts across all processors, with accuracy improvements of 1.08% for lemmatization, 1.97% for POS tagging, and 4.87% for dependency parsing (LAS). Moreover, character-level models exhibited greater training efficiency, converging in half the time and requiring fewer computational resources. The performance of our model represents a substantial improvement over previous approaches. While Brigada Villa \u0026amp; Giarda (2023) reported a maximum Labeled Attachment Score of 58.70% using a multilingual parser combining Old English with Icelandic, our character-level model achieves 79.65% LAS on the test set. This 20.95 percentage point improvement underscores the value of our combined approach compared to previous methods.\u003c/p\u003e\n\u003cp\u003eThird, our analysis of component-specific performance reveals that while common syntactic relations are modeled with high accuracy (\u0026gt;80% F1 for determiners, subjects, and case marking), complex structures remain challenging (\u0026lt;30% F1 for clausal complements and orphans). This understanding of where the model does not completely succeed can guide future research.\u003c/p\u003e\n\u003cp\u003eThe scalability of the model constitutes another advantage. It can be continuously improved through additional training data, integrated with annotation interfaces to facilitate human workflows, and combined with other NLP tools through the spaCy-stanza interface. This interoperability enhances its practical utility for researchers.\u003c/p\u003e\n\u003cp\u003eDespite these advances, our research has several limitations that should be acknowledged. Our training dataset, while larger than those used in previous studies, still represents a small fraction of the extant Old English corpus and is biased toward prose texts, potentially limiting performance on poetic texts with their distinctive syntax and vocabulary. The model also struggles with sentence boundary detection (64.06% accuracy), which affects downstream processing, particularly for complex multi-clause sentences. Additionally, our approach does not address dialectal variation within Old English, treating it as a homogeneous language despite documented regional and temporal differences that impact morphology and syntax. Finally, our evaluation metrics, while standard for computational linguistics, may not fully capture the nuanced linguistic phenomena of interest to historical linguists.\u003c/p\u003e\n\u003cp\u003eFuture work should focus on expanding the Old English treebank to enhance performance for sentence segmentation and less common dependency relations, on normalising punctuation to improve sentence boundary detection, and on developing named entity recognition capabilities. Experimenting with different embedding dimensionalities and architectures -particularly larger dimensions (300-500) for capturing subtle semantic distinctions- could improve performance on complex relations like clausal complements.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eAcknowledgement:\u003c/h2\u003e \u003cp\u003eWe gratefully acknowledge the grant PID2023-149762NB-100, funded by MCIN / AEI / \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.13039/501100011033\u003c/span\u003e\u003cspan address=\"10.13039/501100011033\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eAdel, H., Asgari, E., \u0026amp; Sch\u0026uuml;tze, H. (2018). Overview of character-based models for natural language processing. In A. Gelbukh (Ed.), \u003cem\u003eComputational Linguistics and Intelligent Text Processing. CICLing 2017\u003c/em\u003e. Lecture Notes in Computer Science (Vol. 10761). Springer. https://doi.org/10.1007/978-3-319-77113-7_1\u003c/li\u003e\n\u003cli\u003eBird, S., Klein, E., \u0026amp; Loper, E. (2019). \u003cem\u003eNatural language processing with Python: Analyzing text with the Natural Language Toolkit\u003c/em\u003e (2nd ed.). O\u0026apos;Reilly Media.\u003c/li\u003e\n\u003cli\u003eBojanowski, P., Grave, E., Joulin, A., \u0026amp; Mikolov, T. (2017). Enriching word vectors with subword information. \u003cem\u003eTransactions of the Association for Computational Linguistics\u003c/em\u003e, 5, 135-146. https://doi.org/10.1162/tacl_a_00051\u003c/li\u003e\n\u003cli\u003eBollmann, M. (2019). A large-scale comparison of historical text normalization systems. In \u003cem\u003eProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies\u003c/em\u003e (pp. 3885-3898). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1389\u003c/li\u003e\n\u003cli\u003eBosworth, J., \u0026amp; Toller, T. N. (2018). \u003cem\u003eAn Anglo-Saxon Dictionary Online\u003c/em\u003e (O. Tich\u0026yacute;, \u0026amp; A. Boer, Eds.). Faculty of Arts, Charles University. http://bosworth.V.cuni.cz\u003c/li\u003e\n\u003cli\u003eBrigada Villa, L., \u0026amp; Giarda, M. (2023a). Using modern languages to parse ancient ones: a test on Old English. In \u003cem\u003eProceedings of the 5th Workshop on Research in Computational Linguistic Typology and Multilingual NLP\u003c/em\u003e (pp. 30-41). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.sigtyp-1.4\u003c/li\u003e\n\u003cli\u003eBrigada Villa, L., \u0026amp; Giarda, M. (2023b). From YCOE to UD: rule-based root identification in Old English. In \u003cem\u003eProceedings of the Third Workshop on Language Technology for Digital Historical Archives (LT4HALA 2024)\u003c/em\u003e (pp. 22-29). European Language Resources Association.\u003c/li\u003e\n\u003cli\u003eBrown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., \u0026amp; Amodei, D. (2020). Language models are few-shot learners. \u003cem\u003eAdvances in Neural Information Processing Systems\u003c/em\u003e, 33, 1877-1901. https://doi.org/10.48550/arXiv.2005.14165\u003c/li\u003e\n\u003cli\u003eCameron, A., Amos, A. C., Healey, A. D., Holland, J., McDougall, D., McDougall, I., Speirs, N., \u0026amp; Thompson, P. (Eds.). (2018). \u003cem\u003eDictionary of Old English: A to I online\u003c/em\u003e. Dictionary of Old English Project, University of Toronto. https://www.doe.utoronto.ca\u003c/li\u003e\n\u003cli\u003e\u0026Ccedil;etinoğlu, \u0026Ouml;., \u0026amp; \u0026Ccedil;\u0026ouml;ltekin, \u0026Ccedil;. (2022). A thorough evaluation of character-level models for Turkish natural language processing. \u003cem\u003eNatural Language Engineering\u003c/em\u003e, 28(3), 275-301. https://doi.org/10.1017/S1351324921000218\u003c/li\u003e\n\u003cli\u003eChaubard, F., Fang, M., Genthial, G., Mundra, R., \u0026amp; Socher, R. (2019). \u003cem\u003eCS224n: Natural language processing with deep learning, lecture notes: Part I, Winter 2019\u003c/em\u003e. Retrieved January 26, 2024, from https://web.stanford.edu/class/cs224n/\u003c/li\u003e\n\u003cli\u003eChung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S. S., Dai, Z., Suzgun, M., Chen, X. V., Chowdhery, A., Castro-Ros, A., Pellat, M., Robinson, K., Rose, T., Krakover, I., Braz, P. E., Roberts, A., Dyer, J., Ser, D., Fernando, J., Zoph, B., Welinder, P., Baljekar, P., Firat, O., Zettlemoyer, L., Hughes, M., Austin, J., Norouzi, M., Tan, M., Sohl-Dickstein, J., Dean, J., \u0026amp; Wei, J. (2023). Scaling instruction-finetuned language models. Computational Linguistics, 49(4), 965-985. https://doi.org/10.1162/coli_a_00474\u003c/li\u003e\n\u003cli\u003eClark, J. H., Garrette, D., Turc, I., \u0026amp; Wieting, J. (2022). Canine: Pre-training an efficient tokenization-free encoder for language representation. \u003cem\u003eTransactions of the Association for Computational Linguistics\u003c/em\u003e, 10, 73-91. https://doi.org/10.1162/tacl_a_00448\u003c/li\u003e\n\u003cli\u003eConneau, A., Lample, G., Ranzato, M., Denoyer, L., \u0026amp; J\u0026eacute;gou, H. (2018). Word translation without parallel data. In \u003cem\u003eInternational Conference on Learning Representations\u003c/em\u003e. https://doi.org/10.48550/arXiv.1710.04087\u003c/li\u003e\n\u003cli\u003ede Marneffe, M. C., Manning, C. D., Nivre, J., \u0026amp; Zeman, D. (2021). Universal Dependencies. \u003cem\u003eComputational Linguistics\u003c/em\u003e, 47(2), 255-308. https://doi.org/10.1162/coli_a_00402\u003c/li\u003e\n\u003cli\u003eDevlin, J., Chang, M. W., Lee, K., \u0026amp; Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In \u003cem\u003eProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics\u003c/em\u003e (pp. 4171-4186). https://doi.org/10.18653/v1/N19-1423\u003c/li\u003e\n\u003cli\u003eDubossarsky, H., Hengchen, S., Tahmasebi, N., \u0026amp; Schlechtweg, D. (2019). Time-out: Temporal referencing for robust modeling of lexical semantic change. In \u003cem\u003eProceedings of the 57th Annual Meeting of the Association for Computational Linguistics\u003c/em\u003e (pp. 457-470). Association for Computational Linguistics. https://doi.org/10.18653/v1/P19-1044\u003c/li\u003e\n\u003cli\u003eFaulkner, M. (2023). Corpus philology: Using the Dictionary of Old English to get bigger data for Old English spelling variation. \u003cem\u003eDigital Scholarship in the Humanities\u003c/em\u003e, 38(4), 1508-1521. https://doi.org/10.1093/llc/fqad023\u003c/li\u003e\n\u003cli\u003eGneuss, H., \u0026amp; Lapidge, M. (2014). \u003cem\u003eAnglo-Saxon Manuscripts: A Bibliographical Handbook\u003c/em\u003e. University of Toronto Press.\u003c/li\u003e\n\u003cli\u003eGoldberg, Y. (2016). A primer on neural network models for natural language processing. \u003cem\u003eJournal of Artificial Intelligence Research\u003c/em\u003e, 57, 345-420. https://doi.org/10.1613/jair.4992\u003c/li\u003e\n\u003cli\u003eHamilton, W. L., Leskovec, J., \u0026amp; Jurafsky, D. (2018). Diachronic word embeddings reveal statistical laws of semantic change. In \u003cem\u003eProceedings of the 54th Annual Meeting of the Association for Computational Linguistics\u003c/em\u003e (pp. 1489-1501). Association for Computational Linguistics. https://doi.org/10.18653/v1/P16-1141\u003c/li\u003e\n\u003cli\u003eHaug, D. T. T., \u0026amp; J\u0026oslash;hndal, M. (2021). Creating a parallel corpus of Old and Modern French. \u003cem\u003eJournal of Data Mining and Digital Humanities\u003c/em\u003e, 2021. https://doi.org/10.46298/jdmdh.6566\u003c/li\u003e\n\u003cli\u003eHealey, A. diPaolo. (Ed.). (2018\u003cem\u003e). The Dictionary of Old English Corpus in Electronic Form\u003c/em\u003e. Dictionary of Old English Project, University of Toronto. https://www.doe.utoronto.ca/pages/pub/web-corpus.html\u003c/li\u003e\n\u003cli\u003eHofmann, V., Pierrehumbert, J. B., \u0026amp; Sch\u0026uuml;tze, H. (2022). Modeling morphological processing in human language understanding. \u003cem\u003eComputational Linguistics\u003c/em\u003e, 48(2), 219-271. https://doi.org/10.1162/coli_a_00437\u003c/li\u003e\n\u003cli\u003eHonnibal, M., Montani, I., Van Landeghem, S., \u0026amp; Boyd, A. (2020). \u003cem\u003espaCy: Industrial-strength Natural Language Processing in Python\u003c/em\u003e. https://doi.org/10.5281/zenodo.1212303\u003c/li\u003e\n\u003cli\u003eJurafsky, D., \u0026amp; Martin, J. H. (2023). \u003cem\u003eSpeech and language processing\u003c/em\u003e (3rd ed.). Pearson.\u003c/li\u003e\n\u003cli\u003eKim, Y., Jernite, Y., Sontag, D., \u0026amp; Rush, A. M. (2016). Character-aware neural language models. In \u003cem\u003eProceedings of the Thirtieth AAAI Conference on Artificial Intelligence\u003c/em\u003e (pp. 2741-2749). https://doi.org/10.5555/3016100.3016285\u003c/li\u003e\n\u003cli\u003eKudo, T. (2018). Subword regularization: Improving neural network translation models with multiple subword candidates. In \u003cem\u003eProceedings of the 56th Annual Meeting of the Association for Computational Linguistics\u003c/em\u003e (pp. 66-75). https://doi.org/10.18653/v1/P18-1007\u003c/li\u003e\n\u003cli\u003eLing, W., Lu\u0026iacute;s, T., Marujo, L., Astudillo, R. F., Amir, S., Dyer, C., Black, A. W., \u0026amp; Trancoso, I. (2015). Finding function in form: Compositional character models for open vocabulary word representation. In \u003cem\u003eProceedings of the 2015 Conference on Empirical Methods in Natural Language Processing\u003c/em\u003e (pp. 1520-1530). https://doi.org/10.18653/v1/D15-1176\u003c/li\u003e\n\u003cli\u003eMa, X., \u0026amp; Hovy, E. (2016). End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In \u003cem\u003eProceedings of the 54th Annual Meeting of the Association for Computational Linguistics\u003c/em\u003e (pp. 1064-1074). https://doi.org/10.18653/v1/P16-1101\u003c/li\u003e\n\u003cli\u003eManjavacas, E., K\u0026aacute;d\u0026aacute;r, \u0026Aacute;., \u0026amp; Kestemont, M. (2019). Improving lemmatization of non-standard languages with joint learning. In \u003cem\u003eProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies\u003c/em\u003e (pp. 1493-1503). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1153\u003c/li\u003e\n\u003cli\u003eMart\u0026iacute;n Arista, J. (2020). Further remarks on the deflexion and grammaticalization of the Old English past participle with habban. \u003cem\u003eInternational Journal of English Studies\u003c/em\u003e, 20(1): 51-71. https://doi.org/10.6018/ijes.404881\u003c/li\u003e\n\u003cli\u003eMart\u0026iacute;n Arista, J. (2022). Old English Universal Dependencies: Categories, Functions and Specific Fields. In \u003cem\u003eProceedings of the 14th International Conference on Agents and Artificial Intelligence - Volume 3: ICAART\u003c/em\u003e (pp. 945-951). SCITEPRESS. https://doi.org/10.5220/0010896700003116\u003c/li\u003e\n\u003cli\u003eMart\u0026iacute;n Arista, J. (2024). Toward a Universal Dependencies Treebank of Old English: Representing the Morphological Relatedness of Un-Derivatives. \u003cem\u003eLanguages\u003c/em\u003e, 9(3), 76. https://doi.org/10.3390/languages9030076\u003c/li\u003e\n\u003cli\u003eMart\u0026iacute;n Arista, J., \u0026amp; Ojanguren L\u0026oacute;pez, A. E. (2018). Grammaticalization and deflexion in progress. The past participle in the Old English passive. \u003cem\u003eStudia Neophilologica\u003c/em\u003e, 90(2), 155-175. https://doi.org/10.1080/00393274.2018.1444421\u003c/li\u003e\n\u003cli\u003eMart\u0026iacute;n Arista, J., Dom\u0026iacute;nguez Barrag\u0026aacute;n, S., Fidalgo Allo, L., Garc\u0026iacute;a Fern\u0026aacute;ndez, L., Hamdoun Bghiyel, Y., Lacalle Palacios, M., Mateo Mendaza, R., Novo Urraca, C., Ojanguren L\u0026oacute;pez, A. E., Ru\u0026iacute;z Narbona, E., Torre Alonso, R., \u0026amp; Vea Escarza, R. (2023). \u003cem\u003eParCorOEv3. An open access annotated parallel corpus Old English-English\u003c/em\u003e. Nerthus Project, Universidad de La Rioja. www.nerthusproject.com\u003c/li\u003e\n\u003cli\u003eMart\u0026iacute;n Arista, J., Ojanguren L\u0026oacute;pez, A. E., \u0026amp; Dom\u0026iacute;nguez Barrag\u0026aacute;n, S. (2025). Universal Dependencies annotation of Old English with spaCy and MobileBERT. Evaluation and perspectives. \u003cem\u003eProcesamiento del Lenguaje Natural\u003c/em\u003e (forthcoming).\u003c/li\u003e\n\u003cli\u003eMeechan-Maddon, A., \u0026amp; Nivre, J. (2019). How to parse low-resource languages: Cross-lingual parsing, target language annotation, or both? In \u003cem\u003eProceedings of the Fifth International Conference on Dependency Linguistics (DepLing 2019)\u003c/em\u003e (pp. 112-120). Association for Computational Linguistics. https://doi.org/10.48550/arXiv.1908.10626\u003c/li\u003e\n\u003cli\u003eMielke, S. J., Barrault, L., Liu, F., \u0026amp; Bapna, A. (2021). Between words and characters: A Brief history of open-vocabulary modeling and tokenization in NLP. \u003cem\u003eTransactions of the Association for Computational Linguistics\u003c/em\u003e, 9, 1153-1175. https://doi.org/10.1162/tacl_a_00416\u003c/li\u003e\n\u003cli\u003eMikolov, T., Chen, K., Corrado, G., \u0026amp; Dean, J. (2013a). Efficient estimation of word representations in vector space. In \u003cem\u003eInternational Conference on Learning Representations\u003c/em\u003e. https://doi.org/10.48550/arXiv.1301.3781\u003c/li\u003e\n\u003cli\u003eMikolov, T., Sutskever, I., Chen, K., Corrado, G. S., \u0026amp; Dean, J. (2013b). Distributed representations of words and phrases and their compositionality. In \u003cem\u003eAdvances in Neural Information Processing Systems\u003c/em\u003e (pp. 3111-3119). https://doi.org/10.48550/arXiv.1310.4546\u003c/li\u003e\n\u003cli\u003eNadkarni, P. M., Ohno-Machado, L., \u0026amp; Chapman, W. W. (2011). Natural language processing: An introduction. \u003cem\u003eJournal of the American Medical Informatics Association\u003c/em\u003e, 18(5), 544-551. https://doi.org/10.1136/amiajnl-2011-000464\u003c/li\u003e\n\u003cli\u003eParkes, M. B. (2016). \u003cem\u003ePause and Effect: An Introduction to the History of Punctuation in the West\u003c/em\u003e. Routledge.\u003c/li\u003e\n\u003cli\u003ePennington, J., Socher, R., \u0026amp; Manning, C. D. (2014). GloVe: Global vectors for word representation. In \u003cem\u003eProceedings of the 2014 Conference on Empirical Methods in Natural Language Processing\u003c/em\u003e (pp. 1532-1543). https://doi.org/10.3115/v1/D14-1162\u003c/li\u003e\n\u003cli\u003ePeters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., \u0026amp; Zettlemoyer, L. (2018). Deep contextualized word representations. In \u003cem\u003eProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics\u003c/em\u003e (pp. 2227-2237). https://doi.org/10.18653/v1/N18-1202\u003c/li\u003e\n\u003cli\u003ePintzuk, S., \u0026amp; Plug, L. (Eds.). (2001). \u003cem\u003eThe York-Helsinki Parsed Corpus of Old English Poetry\u003c/em\u003e. Department of Language and Linguistic Science, University of York. http://www-users.york.ac.uk/~lang18/pcorpus.html\u003c/li\u003e\n\u003cli\u003eQi, P., Zhang, Y., Zhang, Y., Bolton, J., \u0026amp; Manning, C. D. (2020). Stanza: A Python natural language processing toolkit for many human languages. In \u003cem\u003eProceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations\u003c/em\u003e (pp. 101-108). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-demos.14\u003c/li\u003e\n\u003cli\u003eRadford, A., Narasimhan, K., Salimans, T., \u0026amp; Sutskever, I. (2018). \u003cem\u003eImproving language understanding by generative pre-training\u003c/em\u003e. OpenAI.\u003c/li\u003e\n\u003cli\u003eRadford, A., Wu, J., Child, R., Luan, D., Amodei, D., \u0026amp; Sutskever, I. (2019). Language models are unsupervised multitask learners. \u003cem\u003eOpenAI Blog\u003c/em\u003e, 1(8).\u003c/li\u003e\n\u003cli\u003eRadford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., \u0026amp; Sutskever, I. (2021). Learning transferable visual models from natural language supervision. In \u003cem\u003eInternational Conference on Machine Learning\u003c/em\u003e (pp. 8748-8763). https://doi.org/10.48550/arXiv.2103.00020\u003c/li\u003e\n\u003cli\u003eRust, P., Pfeifer, J., Vulić, I., Ruder, S., \u0026amp; Gurevych, I. (2024). How does vocabulary design impact the utility of large language models? \u003cem\u003eTransactions of the Association for Computational Linguistics\u003c/em\u003e, 12, 167-188. https://doi.org/10.1162/tacl_a_00609\u003c/li\u003e\n\u003cli\u003eSchuster, M., \u0026amp; Nakajima, K. (2012). Japanese and Korean voice search. In \u003cem\u003e2012 IEEE International Conference on Acoustics, Speech and Signal Processing\u003c/em\u003e (pp. 5149-5152). https://doi.org/10.1109/ICASSP.2012.6289079\u003c/li\u003e\n\u003cli\u003eScragg, D. G. (1974). \u003cem\u003eA History of English Spelling\u003c/em\u003e. Manchester University Press.\u003c/li\u003e\n\u003cli\u003eSennrich, R., Haddow, B., \u0026amp; Birch, A. (2016). Neural machine translation of rare words with subword units. In \u003cem\u003eProceedings of the 54th Annual Meeting of the Association for Computational Linguistics\u003c/em\u003e (pp. 1715-1725). https://doi.org/10.18653/v1/P16-1162\u003c/li\u003e\n\u003cli\u003eShareghi, E., Bollegala, D., Kann, K., \u0026amp; Bisk, Y. (2023). Advancements and challenges in morphologically rich languages. \u003cem\u003eAnnual Review of Linguistics\u003c/em\u003e, 9, 251-274. https://doi.org/10.1146/annurev-linguistics-031720-123720\u003c/li\u003e\n\u003cli\u003eTaylor, A., Warner, A., Pintzuk, S., \u0026amp; Beths, F. (2003). \u003cem\u003eThe York-Toronto-Helsinki Parsed Corpus of Old English Prose\u003c/em\u003e. Department of Language and Linguistic Science, University of York. http://www-users.york.ac.uk/~lang22/YCOE/YcoeHome.htm\u003c/li\u003e\n\u003cli\u003eTurney, P. D., \u0026amp; Pantel, P. (2010). From frequency to meaning: Vector space models of semantics. \u003cem\u003eJournal of Artificial Intelligence Research\u003c/em\u003e, 37, 141-188. https://doi.org/10.1613/jair.2934\u003c/li\u003e\n\u003c/ol\u003e"},{"header":"Tables","content":"\u003cp\u003eTables 1 to 6 are available in the Supplementary Files section\u003c/p\u003e"}],"fulltextSource":"","fullText":"","funders":[{"identity":"1ef7e204-883f-4ee2-9378-c10eb4fd5d44","identifier":"10.13039/501100011033","name":"Agencia Estatal de Investigación","awardNumber":"PID2023-149762NB-100","order_by":0}],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Neural network models, Language-specific word embeddings, Character-level embeddings, Old English processing, Dependency parsing","lastPublishedDoi":"10.21203/rs.3.rs-6473289/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6473289/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThis article seeks to contribute to narrowing the gap between philological research and computational linguistics by providing a neural network model for Old English analysis. The article makes three key innovations: (i) the development of language-specific word embeddings derived directly from \u003cem\u003eThe Dictionary of Old English Corpus\u003c/em\u003e; (ii) a comparative analysis of character-level versus word-level models that demonstrates the superior performance of character-level processing for morphologically rich historical languages; and (iii) a comprehensive Stanza-based pipeline that outperforms previous approaches to Old English parsing. Our model achieves 88.92% Unlabeled Attachment Score and 79.65% Labeled Attachment Score on dependency parsing tasks, which represents approximately 20 percentage point improvement over previous state-of-the-art multilingual approaches. The main conclusion of this work is that language-specific resources and character-level modeling are more effective for Old English processing than cross-linguistic transfer learning. This opens new avenues for computational research in historical linguistics and digital humanities.\u003c/p\u003e","manuscriptTitle":"Language-specific embeddings of Old English with character-level processing","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-04-21 04:39:58","doi":"10.21203/rs.3.rs-6473289/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"d777bb63-a514-4819-807e-98f7f5949c70","owner":[],"postedDate":"April 21st, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":47325721,"name":"Linguistics"}],"tags":[],"updatedAt":"2025-04-21T04:39:58+00:00","versionOfRecord":[],"versionCreatedAt":"2025-04-21 04:39:58","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6473289","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6473289","identity":"rs-6473289","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}