Transformers and large language models are efficient feature extractors for electronic health record studies

Transformers and large language models are efficient feature extractors for electronic health record studies | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Transformers and large language models are efficient feature extractors for electronic health record studies Kevin Yuan, Chang Ho Yoon, Qingze Gu, Henry Munby, Sarah Walker, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4651377/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 21 Mar, 2025 Read the published version in Communications Medicine → Version 1 posted You are reading this latest preprint version Abstract While unstructured free-text data is abundant in electronic health records, challenges in accurate and scalable information extraction often result in a preference for incorporating information from less specific clinical codes instead. We investigate the efficacy of modern natural language processing methods (NLP) and large language models (LLMs) in supervised and few-shot scenarios to extract features from 938,150 hospital antibiotic prescriptions from Oxfordshire, UK. A subset of 4000 most frequent indications for antibiotic use was labelled by clinical researchers into 11 categories, describing the infection source/clinical syndrome, for model training. On separate internal (n = 2000) and external test datasets (n = 2000), the fine-tuned domain-specific Bio + Clinical BERT model averaged an F1 score of 0.97 and 0.98 respectively across the classes and outperformed traditional regex (F1 = 0.71 and 0.74) and n-grams/XGBoost (F1 = 0.86 and 0.84). A few-shot OpenAI GPT4 model achieved F1 scores of 0.71 and 0.86 without using labelled training data and a fine-tuned GPT3.5 model F1 scores of 0.95 and 0.97. Finetuned BERT-based transformer models currently outperform LLMs for structured tasks, while few shot LLMs match the performance of traditional NLP without the need for labelling. Comparing infection sources extracted from ICD10 codes to those parsed from free-text indications, free-text indications revealed 31% more specific infection sources. With their high accuracy, modern transformer-based models have the potential to be used widely throughout medicine to structure free-text records, providing more granular information than clinical codes, facilitating better research and patient care. Figures Figure 1 Figure 2 Figure 3 Figure 4 Introduction Electronic health records (EHRs) offer unprecedented quantities of structured and unstructured data for driving research and improving care delivery. Manually extracting relevant information from unstructured free-text EHRs is costly and laborious. Recent developments in natural language processing (NLP) and the advent of large language models (LLMs) offer promising and potentially transformational alternatives that can accurately acquire relevant information from unstructured text for patient diagnosis 1–3 , as well as perform several routine tasks from medical records 4,5 . As in other medical domains, studies of antibiotic resistance, use, and stewardship have traditionally relied on manual review of clinical notes and prescriptions 6–8 or mapping of International Classification of Diseases (ICD) diagnostic codes to identify acute infection diagnoses and chronic comorbidities 9,10 . However, in studies of sepsis, ICD codes were less sensitive than clinical data for detecting cases 11,12 , particularly for less common infections like meningitis 13 , and had variable validity 14 . Additionally, since codes are recorded only after patient discharge, assigned infection sources may not align with individual antibiotic prescriptions, particularly during the more uncertain initial phase of inpatient care. Conversely, manual chart review has higher sensitivity and can detect indications evolving over time, but time and cost constraints mean that case numbers are often limited. For research, applying LLMs to entire medical records can effectively make predictions and generate new medical content 4,5 . However, there is also a clear need for research and service applications to be able to extract specific individual features from EHRs reliably and efficiently whilst also meeting information governance requirements. As an example of this targeted approach, we investigated several methods using electronic prescribing data to classify the source of an infection/infectious syndrome being treated, analysing free-text indications documented by clinicians 15 justifying the reason for antibiotic prescriptions. We compared state-of-the-art NLP models, Bidirectional Encoder Representations from Transformers (BERT) 16 , LLMs, and Generative Pretrained Transformer (GPT) 17 , to traditional ICD code-based approaches, classical NLP methods and regular expression-based text searches. We show modern approaches have potential to accurately extract key information from medical records at scale, potentially opening opportunities for new epidemiological and intervention studies across all of medicine, as well as possibilities for improving care delivery. Methods Study design and population We used EHRs from two distinct hospital sites, Oxford and Banbury, with Oxford serving as our training and internal test set, and Banbury as our external test set. These sites collectively provide 1100 beds, serving 750,000 residents in Oxfordshire, ~ 1% of the UK population. Deidentified data were obtained from Infections in Oxfordshire Research Database (IORD), which has approvals from the National Research Ethics Service South Central – Oxford C Research Ethics Committee (19/SC/0403), the Health Research Authority and the Confidentiality Advisory Group (19/CAG/0144) as a deidentified database without individual consent. All patients aged ≥ 16 years who had antibiotic prescriptions were included. ‘Ground Truth’ Labelling Two clinical researchers reviewed each antibiotic indication text string used for training and testing (described below) to establish a reference or ‘ground truth’ label for the clinical syndrome being treated. Any discrepancies were resolved by a third researcher, a clinical infection specialist. Antibiotic indications were labelled using 11 categories representing infection source: Urinary; Respiratory; Abdominal; Neurological; Skin and Soft Tissue; Ear, Nose and Throat (ENT); Orthopaedic; Other specific (i.e. another body site); Non-specific (i.e. no body site provided, e.g. “sepsis”), Prophylaxis, Not informative (i.e. text unrelated to the source of infection, e.g. “as instructed by Dr X”). Each category was recorded as a binary variable, such that more than one potential source could be recorded, e.g. the input string “urinary/chest” would be labelled as both urinary and respiratory. An additional variable was used to document the presence of uncertainty expressed by the prescriber, e.g. “urinary/chest” or “? UTI”. Comparator Classification by ICD10 Codes As a comparator, we mapped primary and secondary ICD10 diagnosis codes from the same admission as the antibiotic prescription to the 11 infection sources using CCSR classifications 18 as an intermediate step (see Supplement and Appendix S1 ). Traditional Classification Methods Regex Rules The most intuitive and deterministic method for classifying free-text is searching for specific keywords from a list of predefined words for a given category. We employed fuzzy regular expression (regex) matching patterns with term-specific word boundaries and variable fuzziness to allow for misspellings and variations (see Supplement and Appendix S2 ). n-grams & XGBoost A second approach used a separate tokeniser, embedding and classifier structure; specifically, Scikit-learn’s n-grams & count vectorisation and the gradient boosting model architecture XGBoost 19,20 . Each free-text indication term was broken up into overlapping subwords of length $ n $ and then count-vectorised, with the count representing the frequency of each subword’s occurrence. The vectors of dimension $ vocabulary size $ were then fed as input features to the classification model. We determined the optimal n-gram size ( $ n $ ) and hyperparameters for XGBoost during model training by maximising the receiver operator curve area under the curve (ROC-AUC) (details below). BERT Classifier Current state-of-the-art NLP tasks employ models built on Transformer architectures, with the Bidirectional Encoder Representations from Transformers (BERT) model family well suited for many tasks requiring semantic understanding. We finetuned 21 pre-trained BERT models on a single GPU instance and used BERT for both encoding and classification. We evaluated the original generic “uncased base BERT” model, pre-trained on the BooksCorpus and English Wikipedia and a domain-specific “Bio + Clinical BERT”, pre-trained on biomedical and clinical text 22,23 . Few-Shot and finetuned LLM Classifier Compared to BERT, the GPT family enables zero- or few-shot learning, i.e. there is potentially minimal need for labelled data for task-specific training. We developed prompts for GPT4, comprised of instructions and the target categories, asking the model to complete the categories (see Appendix S3 for specific prompts, and Appendix S4 for details of batch sizes) 24 . Similarly, a GPT3.5 model was finetuned on the training dataset and instructed to classify the free-text. Finetuning was achieved by presenting the desired output alongside the training input data. We used the same system prompt as for the few-shot model, while providing the training examples which were fed in batches of ten. Additional model hyperparameters, such as learning rate, and epochs, were chosen through grid search. Training, Test and External Evaluation We divided the Oxford data with a 90/10 train/test split, resulting in a raw training set and internal test set. From the training data, we labelled and used the 4000 most frequently occurring unique indications. To make labelling tractable we discarded the remaining unlabelled data from the training set. From the internal test data, we randomly selected and exhaustively labelled indications present in 2000 prescriptions. For the external test set from Banbury, we also labelled 2000 randomly selected entries. All models were trained on the training dataset with grid-search based hyperparameter tuning based on cross validation and tested on both the internal and external test sets. The multi-label performance of each method was evaluated using weighted F1 scores, precision-recall (PR-AUC) and ROC-AUC. Weighted averages take into account the varied distributions of the classes, such that more common classes contribute more to the overall average, producing estimates that reflect the original data source and represent real-world performance. Results We obtained antibiotic prescribing indication data from 826,533 prescriptions from 171,460 adult inpatients, ≥ 16 years, between 01-October-2014 and 30-June-2021 from three hospitals in Oxford, UK. The most commonly prescribed antibiotics were co-amoxiclav (n = 269,945, 33%), gentamicin (n = 70,002, 8%), and metronidazole (n = 65,094, 8%) ( Appendix S5 ), and the most common specialities were General Surgery (n = 146,719, 18%), Acute General Medicine (n = 98,687, 12%), and Trauma and Orthopaedics (n = 90,719, 11%) ( Appendix S6 ). Patients were a median 56 years old (IQR 36–73), and 94,721 (55%) were female. We also used an independent external test dataset to assess classifier performance further, from the Horton Hospital, Banbury (~ 30 miles from Oxford). This dataset comprised 111,617 prescriptions from 25,924 patients between 01-December-2014 and 30-June-2021, with 13,650 unique free-text indications. Antibiotics prescribed ( Appendix S5 ) and specialities ( Appendix S6 ) were broadly similar to the Oxford training set. Patients were a median 67 years old (IQR 47–80), and 13,853 (53%) were female. Prescription indications From the 826,533 Oxford prescriptions, 86,611 unique free-text indications were recorded. The top 10 accounted for 41% of all prescriptions; these included “Perioperative Prophylaxis” (20%), “UTI” (4%), “LRTI” (3%), “Sepsis” (3%), and “CAP” (3%). The most commonly occurring 4000 unique indications, used for model training, accounted for 84% (692,310) of prescriptions ( Appendix S7 ). As expected, different wording was used to reflect similar concepts, e.g. “CAP [community acquired pneumonia]”, “LRTI [lower respiratory tract infection]”, “chest infection”, and “pneumonia”. Additionally, misspellings were common, e.g. “infction”, “c. dififcile”. Multiple examples expressed uncertainty, or multiple potential sources of infection, e.g. “sepsis ?source”, “UTI/Chest”, etc. Reflecting the complexity of prescribing, there were multiple potentially informative, but rarely occurring indications, e.g., “transplant pyelonephritis”, “Ludwig’s angina”, and “deep neck infection”, which were only seen 51 (< 1%), 27 (< 1%), and 13 (< 1%) times respectively ( Appendix S8 ). ‘Ground truth’ labels Following labelling by clinical experts, the 4000 most commonly occurring free-text indications were classified into 11 categories, with a separate variable capturing the presence of uncertainty. The most commonly assigned sources were “Prophylaxis” (267,788/692,310 prescriptions, 39%), “Respiratory” (125,744, 18%) and “Abdominal” (61,670, 9%). 50% (n = 344,773) prescriptions had “No Specific Source”. The most uncertainty was expressed in “Neurological” and ENT cases at 38% and 33%, respectively (Fig. 1 A). Although “Respiratory” was the most common category overall after “Prophylaxis”, there were more distinct text strings associated with “Abdominal” infections, with “Skin and Soft Tissue” infection also having a disproportionately larger number of unique text strings (Fig. 1 B). Most ‘multi-source’ prescriptions were a combination of “Prophylaxis” and a source (> 90%). Excluding prophylaxis, the most common combinations of sources were “No Specific source” and “Not Informative”, “Urinary” and “Respiratory”, and “Skin & Soft Tissue” and ENT, in 1.6%, 0.58%, 0.41% prescriptions, respectively (Fig. 1 C-D). The former two reflected diagnostic uncertainty and the latter reflected infections of the face, head and neck frequently involving skin/soft tissue. Classifier performance We trained classifiers using the labelled training data from three Oxford hospitals (Fig. 2 ). Compared to clinician-assigned labels, within the internal Oxford test dataset (n = 2000), the weight-averaged F1 score across classes was highest using Bio + Clinical BERT (Average F1 = 0.97 [worst performing class F1 = 0.84, best performing F1 = 0.98]) followed by finetuned GPT3.5 (F1 = 0.95 [0.77–0.99]), base-BERT (F1 = 0.93 [0.23–0.98]) and tokenisation + XGBoost (F1 = 0.86 [0.64–0.96]). Nearly all approaches exceeded traditional regular expression-based matching (F1 = 0.71 [0.00-0.93]). The few-shot GPT4 model which did not require labelled data performed similarly to this baseline (F1 = 0.71 [0.30–0.98]). Similar performance characteristics were achieved on the external validation dataset from Banbury (n = 2000; weight-averaged F1 scores: Bio + Clinical BERT 0.98 [0.87-1.00], finetuned GPT3.5 0.97 [0.70-1.00], Base BERT 0.97 [0.63–0.99], XGBoost 0.84 [0.63-1.00], Regex 0.74 [0.00-0.96], GTP4 0.86 [0.25-1.00]) (Table 1 , also shows classification run times). Table 1 Model performance metrics for the internal (Oxford) and external (Banbury) test sets . Each score is listed with the weighted average across the classes (sources), with the lowest and highest performing class. Overall Accuracy refers to the score calculated for a sample treated as a whole. The finetuned Bio + Clinical BERT outperforms all other methods on both internal and external test sets. Internal Oxford test set Model F1 Score ROC AUC PR AUC Per Class Accuracy Accuracy Training Runtime Test Runtime Aggregation Average Lowest Highest Average Lowest Highest Average Lowest Highest Average Lowest Highest Overall Per 4k Per 10k Regex 0.71 0.00 0.93 - - - - - - 0.82 0.32 0.99 0.14 - 6.4s XGBoost 0.86 0.64 0.96 0.96 0.87 0.99 0.9 0.62 0.99 0.95 0.92 1.00 0.72 6s 1.2s Base BERT 0.93 0.23 0.98 0.99 0.91 1.00 0.97 0.69 0.99 0.98 0.97 0.99 0.88 282s 1 82.2s Bio + Clinical BERT 0.97 0.84 0.98 0.99 0.96 1.00 0.98 0.88 1.00 0.99 0.98 1.00 0.94 279s 2 83.1s Fine-Tuned OpenAI GPT3.5 0.95 0.77 0.99 - - - - - - 0.98 0.97 1.00 0.91 ~ 3500s 1 ~ 3000s 2 Few-Shot OpenAI GPT4 0.71 0.30 0.98 - - - - - - 0.87 0.64 1.00 0.50 - ~ 3000s 1 External Banbury test set Model F1 Score ROC AUC PR AUC Per Class Accuracy Accuracy Aggregation Average Lowest Highest Average Lowest Highest Average Lowest Highest Average Lowest Highest Overall Regex 0.74 0.00 0.96 - - - - - - 0.82 0.41 0.99 0.24 XGBoost 0.84 0.63 1.00 0.94 0.86 1.00 0.87 0.57 1.00 0.94 0.88 1.00 0.68 Base BERT 0.97 0.63 0.99 0.99 0.95 1.00 0.98 0.75 1.00 0.99 0.99 1.00 0.95 Bio+Clinical BERT 0.98 0.87 1.00 0.99 0.97 1.00 0.98 0.87 1.00 0.99 0.99 1.00 0.97 Fine-Tuned OpenAI GPT3.5 0.97 0.70 1.00 - - - - - - 0.99 0.98 1.00 0.95 Few-Shot OpenAI GPT4 0.86 0.25 1.00 - - - - - - 0.95 0.81 1.00 0.73 [1] Using one Nvidia V100 GPU [2] OpenAI’s cloud service Classifier Performance by Class Using the best performing classifier, Bio + Clinical BERT, we assessed performance within each category. The best-performing categories within our internal test set were “Respiratory”, “No Specific Source” and “Prophylaxis” (F1 score = 0.98), followed by “Urinary” (0.97), “Abdominal” (0.96), “Orthopaedic” (0.90), “Not Informative” (0.89) and “Neurological” (0.88). The worst performing category was Orthopaedic (0.84), likely due to the high variety of terms used and low number of training samples (n = 14, Appendix S9 ). Uncertainty was also well detected (0.96) (Fig. 3 A, Appendix S10 ). In the external test data, scores varied slightly, with all source categories except for “Not Informative” having F1 scores on average 0.02 higher compared to the internal test set. These small differences likely arose from different compositions of categories and the amount of shared vocabulary between the training and test datasets. Misclassifications Most misclassifications were spread evenly across classes for single indications. The two most common misclassifications occurred for “Orthopaedic” and “Other Specific” cases, with 12% being misclassified as “Prophylaxis” and 8% as “Skin and Soft Tissue”, respectively on the internal test set. On the external test set, most misclassifications were predicted to be “Other Specific” or “Prophylaxis” (Fig. 3 C). Training Dataset Size We examined the effect of training size on model performance using randomly selected training dataset subsets of 250, 500, 750, 1000, 1500, 2000, 3000, and 4000 unique indications, tested using both internal/external test sets. There was a notable increase in performance (AUC-ROC and F1 scores) when the training size increased from 250 to 1000 samples, suggesting a minimum of 1000 training samples for adequate performance. However, we saw limited improvement as training dataset size rose to 4000, indicating there may be only marginal gains to expanding the training data beyond 4000 samples ( Appendix S11 ). Comparing free-text indications to ICD10 codes We also compared infection sources from manually labelled ‘ground truth’ free-text indications to sources inferred from ICD10 diagnostic codes. 31% of sources classified as “unspecific” using diagnostic codes could be resolved into specific sources using free-text. Rarer infection sources such as “CNS” and “ENT” (< 1% and no occurrence in diagnostic codes) were represented better by sources extracted from free-text (4% and 4% respectively). Overall, where defined, sources listed in clinical codes generally concurred with the ‘ground truth’ free-text sources (Fig. 4 ). Discussion We show that modern NLP methods can extract clinically relevant details from semi-structured free-text fields. In our example application, a finetuned Bio + Clincal BERT transformer model classified the infection source being treated using clinician-written antibiotic indication text with an F1 score of 0.97 (harmonic mean of sensitivity and positive predictive value). Although this required manual labelling of several thousand text strings, exhaustive labelling of all possible prior strings was not required to achieve this performance. The accuracy of the Bio + Clinical BERT model was substantially better than a sophisticated regular expression-based approach, despite the latter being the solution that many healthcare institutions and researchers might choose at present. We also explored what performance might be possible using LLMs without having to label data, e.g. where this is not possible or too resource intensive. However, few-shot learning with GPT4 only achieved modest performance, but it was still similar to the baseline regex method and more consistent across classes. Using LLMs with labelled training data, i.e. a fine-tuned GPT3.5 variant achieved results comparable to the Bio + Clinical BERT approach when correctly specified and tuned but showed more limited potential for deployment as responses can vary greatly in formatting, making it difficult to parse correctly into a rigid format required for most downstream tasks or EHR systems. In environments with limited computing resources where the deployment of deep-learning models is not feasible, regex and XGBoost-based models provide possible alternatives with a reduced runtime of 6.4 and 1.2 seconds/10k indications vs 82.2 seconds/10k for Bio + Clinial BERT. Currently, research or clinical use of free-text may be limited by concerns that personal data may be included. Here, by homogenising and categorising sensitive free-text data, we present a privacy-aware solution that enables researchers to utilise the depth of free-text data without direct access or the possibility of identifying specific patients. Our study has several limitations, including that we only used a subset of the available training data, through the non-exhaustive labelling of a subset of antibiotic indication text strings. However, labelling the 4000 most common unique terms, accounting for 84% of the data, achieved very high performance, with sensitivity analyses suggesting that labelling more examples would not have improved performance substantially. This is likely possible because the underlying Bio + Clinical BERT model is already pre-trained on medical terms and capable of inferring similar words. Of note, many of the remaining 17% of text strings were different combinations of already labelled words, suggesting fewer “new” or unseen keywords than might be expected. Not fully labelling the training data also makes it more difficult to compare class distributions with the test data sets. We also only used a subset of the test data to evaluate performance; however the 2000 randomly selected samples are likely representative. Although the labelling process was somewhat subjective, independent labelling by two clinical researchers with a third adjudicating any discrepancies was designed to minimise this. Future enhancements could optimise computational efficiency while maintaining comparable performance. While XGBoost offers faster training and inference times, its performance was poorer than Bio + Clinical BERT. Therefore, smaller, more efficient NLP models might balance computational demands and performance. Techniques such as model pruning, quantisation, and knowledge distillation could reduce model size and computational requirements while preserving performance 25–27 . While GPT4 deployments can comply with data governance requirements, its use presents challenges in some settings, as it is usually accessed via third-party cloud compute providers rather than healthcare institutions. Where data need to remain on site, open-source, locally-deployed language models, such as LLAMA, ALPACA or Mistral 7B, may be alternatives that could be further investigated 28–30 . Our approach has several possible applications; for example, it could be used to monitor and evaluate prescribing practice across different conditions, it provides classification of possible infection sources for epidemiological research 31 , and is also a mechanism for extracting standardised features from medical records for use in predictive algorithms being developed to improve patient care. Although we demonstrate excellent performance for antibiotic indications, it could also be applied to other short strings of free-text, for example descriptions of surgical procedures, patient functional states, or presenting complaints in emergency department and hospital admission data. Across all these domains, refined patient stratification could improve both research and care delivery. In summary, we show that state-of-the-art NLP can be used to efficiently and accurately categorise semi-structured free-text in medical records. This has the potential to be applied widely to analyse medical records more accurately. Declarations Author contributions DWE, TZ, ASW, KY and CHY conceived the study. KY, CHY, QG, HM and DWE analysed the data. KY and DWE wrote the first draft of the manuscript. All authors contributed to revising the manuscript. Acknowledgements and funding This work was supported by the National Institute for Health Research Health Protection Research Unit (NIHR HPRU) in Healthcare Associated Infections and Antimicrobial Resistance at Oxford University in partnership with the UK Health Security Agency (NIHR200915), and the NIHR Biomedical Research Centre, Oxford. DWE is a Big Data Institute Robertson Fellow. ASW is an NIHR Senior Investigator. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, the Department of Health or the UK Health Security Agency. KY is supported by the EPSRC Centre for Doctoral Training in Health Data Science (EP/S02428X/1). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Declaration of interests No other author has a conflict of interest to declare. Data sharing The data analysed are available from the Infections in Oxfordshire Research Database (https://oxfordbrc.nihr.ac.uk/research-themes/modernising-medical-microbiology-and-big-infection-diagnostics/infections-in-oxfordshire-research-database-iord/), subject to an application and research proposal meeting on the ethical and governance requirements of the Database. Labelled training and test datasets and the pre-trained BERT model are also available via an application to the Database. Code availability All tools developed for this study (Regex builder, Bio+Clinical BERT pipeline, GPT3.5 finetuning and GPT4 few-shot request tools), pre-trained models, as well as the evaluation framework, are available on GitHub: https://github.com/kevihiiin/EHR-Indication-Processing/ References Kirby JC, Speltz P, Rasmussen LV, et al. PheKB: a catalog and workflow for creating electronic phenotype algorithms for transportability. J Am Med Inform Assoc 2016;23(6):1046–52. Jamian L, Wheless L, Crofford LJ, Barnado A. Rule-based and machine learning algorithms identify patients with systemic sclerosis accurately in the electronic health record. Arthritis Res Ther 2019;21(1):305. Maarseveen TD, Maurits MP, Niemantsverdriet E, van der Helm-van Mil AHM, Huizinga TWJ, Knevel R. Handwork vs machine: a comparison of rheumatoid arthritis patient populations as identified from EHR free-text by diagnosis extraction through machine-learning or traditional criteria-based chart review. Arthritis Res Ther 2021;23(1):174. Singhal K, Azizi S, Tu T, et al. Large language models encode clinical knowledge. Nature 2023;620(7972):172–80. Jiang LY, Liu XC, Nejatian NP, et al. Health system-scale language models are all-purpose prediction engines. Nature 2023;619(7969):357–62. McCoy D, Sebti R, Kuyumjian AG. An evaluation of selected indications and appropriateness of Ampicillin/Sulbactam, an unrestricted antimicrobial, at a single center. Pharm Ther 2017;42(3):189. Timmons V, Townsend J, McKenzie R, Burdalski C, Adams-Sommer V. An evaluation of provider-chosen antibiotic indications as a targeted antimicrobial stewardship intervention. Am J Infect Control 2018;46(10):1174–9. Javaid W, Cavanaugh S, Lupone C, Stewart T, Fazili T, White B. Fixed vs. Free-text Documentation of Indication for Antibiotic Orders. Open Forum Infect Dis 2017;4(Suppl 1):S325–6. Kadri SS, Lai YL, Warner S, et al. Inappropriate empirical antibiotic therapy for bloodstream infections based on discordant in-vitro susceptibilities: a retrospective cohort analysis of prevalence, predictors, and mortality risk in US hospitals. Lancet Infect Dis 2021;21(2):241–51. Yoon CH, Bartlett S, Stoesser N, et al. Mortality risks associated with empirical antibiotic activity in Escherichia coli bacteraemia: an analysis of electronic health records. J Antimicrob Chemother 2022;77(9):2536–45. Gaieski DF, Edwards JM, Kallan MJ, Carr BG. Benchmarking the Incidence and Mortality of Severe Sepsis in the United States*. Crit Care Med [Internet] 2013;41(5). Available from: https://journals.lww.com/ccmjournal/fulltext/2013/05000/benchmarking_the_incidence_and_mortality_of_severe.2.aspx Rhee C, Dantes R, Epstein L, et al. Incidence and Trends of Sepsis in US Hospitals Using Clinical vs Claims Data, 2009-2014. JAMA 2017;318(13):1241–9. Wiese AD, Griffin MR, Stein CM, et al. Validation of discharge diagnosis codes to identify serious infections among middle age and older adults. BMJ Open 2018;8(6):e020857. Jolley RJ, Sawka KJ, Yergens DW, Quan H, Jetté N, Doig CJ. Validity of administrative data in recording sepsis: a systematic review. Crit Care 2015;19(1):139. Saini S, Leung V, Si E, et al. Documenting the indication for antimicrobial prescribing: a scoping review. BMJ Qual Saf 2022;31(11):787–99. Devlin J, Chang M-W, Lee K, Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding [Internet]. 2019 [cited 2024 Mar 14];Available from: http://arxiv.org/abs/1810.04805 Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I. Language models are unsupervised multitask learners. OpenAI Blog 2019;1(8):9. Clinical Classifications Software Refined (CCSR) [Internet]. [cited 2022 May 12];Available from: https://hcup-us.ahrq.gov/toolssoftware/ccsr/ccs_refined.jsp Pedregosa F, Varoquaux G, Gramfort A, et al. Scikit-learn: Machine learning in Python. J Mach Learn Res 2011;12:2825–30. Chen T, Guestrin C. XGBoost: A Scalable Tree Boosting System [Internet]. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, NY, USA: Association for Computing Machinery; 2016 [cited 2024 Mar 14]. p. 785–94.Available from: https://dl.acm.org/doi/10.1145/2939672.2939785 Wolf T, Debut L, Sanh V, et al. HuggingFace’s Transformers: State-of-the-art Natural Language Processing [Internet]. 2020 [cited 2024 Mar 14];Available from: http://arxiv.org/abs/1910.03771 Alsentzer E, Murphy J, Boag W, et al. Publicly Available Clinical BERT Embeddings [Internet]. In: Proceedings of the 2nd Clinical Natural Language Processing Workshop. Minneapolis, Minnesota, USA: Association for Computational Linguistics; 2019. p. 72–8.Available from: https://www.aclweb.org/anthology/W19-1909 Johnson AEW, Pollard TJ, Shen L, et al. MIMIC-III, a freely accessible critical care database. Sci Data 2016;3(1):160035. OpenAI (2023). Gpt-4 technical report. ArXiv Prepr ArXiv230308774 2023; Michel P, Levy O, Neubig G. Are Sixteen Heads Really Better than One? [Internet]. In: Advances in Neural Information Processing Systems. Curran Associates, Inc.; 2019 [cited 2024 Mar 18]. Available from: https://proceedings.neurips.cc/paper_files/paper/2019/hash/2c601ad9d2ff9bc8b282670cdd54f69f-Abstract.html Zafrir O, Boudoukh G, Izsak P, Wasserblat M. Q8BERT: Quantized 8Bit BERT [Internet]. In: 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS Edition (EMC2-NIPS). 2019 [cited 2024 Mar 18]. p. 36–9.Available from: https://ieeexplore.ieee.org/abstract/document/9463531 Sanh V, Debut L, Chaumond J, Wolf T. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. ArXiv Prepr ArXiv191001108 2019; Touvron H, Lavril T, Izacard G, et al. LLaMA: Open and Efficient Foundation Language Models [Internet]. 2023 [cited 2024 Mar 14];Available from: http://arxiv.org/abs/2302.13971 Taori R, Gulrajani I, Zhang T, et al. Stanford Alpaca: An Instruction-following LLaMA model [Internet]. GitHub Repos. 2023;Available from: https://github.com/tatsu-lab/stanford_alpaca Jiang AQ, Sablayrolles A, Mensch A, et al. Mistral 7B [Internet]. 2023 [cited 2024 Mar 14];Available from: http://arxiv.org/abs/2310.06825 Gu Q, Wei J, Yoon CH, et al. Distinct patterns of vital sign and inflammatory marker responses in adults with suspected bloodstream infection. J Infect 2024;106156. OpenAI Chat API Reference [Internet]. [cited 2024 Jan 23];Available from: https://platform.openai.com/docs/api-reference/chat/create Additional Declarations There is NO Competing Interest. Supplementary Files Supplementaryappendix.docx Cite Share Download PDF Status: Published Journal Publication published 21 Mar, 2025 Read the published version in Communications Medicine → Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4651377","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":323738513,"identity":"3c72cc8f-837c-46be-849c-b64071db2f0c","order_by":0,"name":"Kevin Yuan","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABEklEQVRIiWNgGAWjYBACCSA+AGEyH0CT4yGohS0BSBgQpwWmwoA4LZLtPYaHCxgOJ/bP7vn4mKfij5x5A/MxCYYaOwaDM+guhQBpnjMGh2cAtcy4c3az4YwzBsYyB9iSDRiOJTMYnG3AqkVOIi3hMA9QS8ON3G0SH9sMEmcw8Bg+YGA7wGBwHrvD5OSfQbTMv5Hz/EfiP4N6oBaDAwz/cGuRlmA+ANay4UYOG8PHBoMECZAtjG0HcDpMsicZqMUg3XjjjTRjyRnHjA1nMAP9ktiXzCOJw/sSxw82f+apsJaddyP54WeeGjl5CfbmYxIfvtnJ8Z1JwO4yMDBoRuIwA3ECzliBgzoC8qNgFIyCUTCiAQAnPltSuFC6zwAAAABJRU5ErkJggg==","orcid":"https://orcid.org/0000-0003-1283-2712","institution":"University of Oxford","correspondingAuthor":true,"prefix":"","firstName":"Kevin","middleName":"","lastName":"Yuan","suffix":""},{"id":323738514,"identity":"59d1810e-f4c2-43d9-8d5c-21be64cee6f7","order_by":1,"name":"Chang Ho Yoon","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Chang","middleName":"Ho","lastName":"Yoon","suffix":""},{"id":323738515,"identity":"fd00d506-aa8f-4748-9d22-6da47bb0159d","order_by":2,"name":"Qingze Gu","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Qingze","middleName":"","lastName":"Gu","suffix":""},{"id":323738516,"identity":"9722472f-409d-4b61-bef5-772470f15b73","order_by":3,"name":"Henry Munby","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Henry","middleName":"","lastName":"Munby","suffix":""},{"id":323738517,"identity":"3d0856fe-0e86-4e65-9ba6-e0730997a921","order_by":4,"name":"Sarah Walker","email":"","orcid":"","institution":"University of Oxford","correspondingAuthor":false,"prefix":"","firstName":"Sarah","middleName":"","lastName":"Walker","suffix":""},{"id":323738518,"identity":"59df8975-4906-4743-8b3f-7fe75a6a400d","order_by":5,"name":"Tingting Zhu","email":"","orcid":"https://orcid.org/0000-0002-1552-5630","institution":"University of Oxford","correspondingAuthor":false,"prefix":"","firstName":"Tingting","middleName":"","lastName":"Zhu","suffix":""},{"id":323738519,"identity":"cc85f915-87b4-425a-bb51-2bbdb94e5c2a","order_by":6,"name":"David Eyre","email":"","orcid":"https://orcid.org/0000-0001-5095-6367","institution":"University of Oxford","correspondingAuthor":false,"prefix":"","firstName":"David","middleName":"","lastName":"Eyre","suffix":""}],"badges":[],"createdAt":"2024-06-28 01:40:06","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4651377/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4651377/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s43856-025-00790-1","type":"published","date":"2025-03-21T04:00:00+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":62144794,"identity":"d03c7a4e-9b0e-44e3-a417-66bc07abcb76","added_by":"auto","created_at":"2024-08-09 18:08:26","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":362880,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eInfection source distributions within labelled training data from three Oxford hospitals\u003c/strong\u003e. Bar charts A) and B) show the distribution of the sources and the uncertainty relative to the infection source. The up-set plots C) and D) show the occurrence of multiple sources within the same prescription. Panels A) \u0026amp; C) show distributions across the entire labelled indications training set, panels B) \u0026amp; D) across a distinct set of 4000 most common indications. *Indications falling into ENT such as “neck abscess” were often also labelled with Skin \u0026amp; Soft Tissue.\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-4651377/v1/619950001701115b60259c6b.png"},{"id":62143751,"identity":"d09094a7-f86f-4b1c-a5db-18abc2792191","added_by":"auto","created_at":"2024-08-09 18:00:26","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":123097,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eData processing flow chart for training and internal and external test data sets\u003c/strong\u003e. Prescribing data was fetched from EHR databases and filtered for complete data. The 4000 most frequent indications within the training split were labelled, all remaining training data was discarded. 2000 entries were randomly sampled from both the internal and external test datasets and exhaustively labelled, resulting in a total of three datasets (training set, internal test set, external test set).\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-4651377/v1/257c8e6d0726cf3e558f0d10.png"},{"id":62143755,"identity":"93aa7d90-9762-4aa9-ad10-94d2c9e2ac52","added_by":"auto","created_at":"2024-08-09 18:00:26","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":77412,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003e\u003cstrong\u003ePerformance metrics for Bio+Clinical BERT on both internal and external test sets.\u003c/strong\u003e\u003c/em\u003e\u003cem\u003e Bar charts A) and B) show the per-class prediction performance. Confusion matrices C) and D) are single indication test cases and show model prediction errors across the sources for given ground truths (clinician assigned sources). Panels A) and C) show evaluations performed on the internal test set from three Oxford hospitals, B) and D) on the external test set from the Banbury hospital.\u003c/em\u003e\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-4651377/v1/2bf4d4036fe826d6809895e1.png"},{"id":62144796,"identity":"45075041-583a-4ae4-be1f-ca2ad8345fc9","added_by":"auto","created_at":"2024-08-09 18:08:26","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":62163,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eComparing infection sources between clinician assigned free-text indications (left) and diagnostic codes (right) in the training and internal and external test sets\u003c/strong\u003e. Clinician assigned categories were extracted from prescribing data and manually labelled, diagnostic codes sources calculated from procedure and discharge codes.\u003c/p\u003e","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-4651377/v1/b22885a19bc29ee4d7477fb2.png"},{"id":79004516,"identity":"d80b3b99-8ef4-4b7f-9718-57c045f37137","added_by":"auto","created_at":"2025-03-22 07:10:02","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1476381,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4651377/v1/637e255e-a1b7-4568-90d9-5534b766e64f.pdf"},{"id":62143753,"identity":"63d9d2ba-c045-42a2-bc40-2b51ccb75c54","added_by":"auto","created_at":"2024-08-09 18:00:26","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":369314,"visible":true,"origin":"","legend":"","description":"","filename":"Supplementaryappendix.docx","url":"https://assets-eu.researchsquare.com/files/rs-4651377/v1/2f06a362893dc828c5df9258.docx"}],"financialInterests":"There is \u003cb\u003eNO\u003c/b\u003e Competing Interest.","formattedTitle":"Transformers and large language models are efficient feature extractors for electronic health record studies","fulltext":[{"header":"Introduction","content":"\u003cp\u003eElectronic health records (EHRs) offer unprecedented quantities of structured and unstructured data for driving research and improving care delivery. Manually extracting relevant information from unstructured free-text EHRs is costly and laborious. Recent developments in natural language processing (NLP) and the advent of large language models (LLMs) offer promising and potentially transformational alternatives that can accurately acquire relevant information from unstructured text for patient diagnosis\u003csup\u003e1–3\u003c/sup\u003e, as well as perform several routine tasks from medical records\u003csup\u003e4,5\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eAs in other medical domains, studies of antibiotic resistance, use, and stewardship have traditionally relied on manual review of clinical notes and prescriptions\u003csup\u003e6–8\u003c/sup\u003e or mapping of International Classification of Diseases (ICD) diagnostic codes to identify acute infection diagnoses and chronic comorbidities\u003csup\u003e9,10\u003c/sup\u003e. However, in studies of sepsis, ICD codes were less sensitive than clinical data for detecting cases\u003csup\u003e11,12\u003c/sup\u003e, particularly for less common infections like meningitis\u003csup\u003e13\u003c/sup\u003e, and had variable validity\u003csup\u003e14\u003c/sup\u003e. Additionally, since codes are recorded only after patient discharge, assigned infection sources may not align with individual antibiotic prescriptions, particularly during the more uncertain initial phase of inpatient care. Conversely, manual chart review has higher sensitivity and can detect indications evolving over time, but time and cost constraints mean that case numbers are often limited.\u003c/p\u003e \u003cp\u003eFor research, applying LLMs to entire medical records can effectively make predictions and generate new medical content\u003csup\u003e4,5\u003c/sup\u003e. However, there is also a clear need for research and service applications to be able to extract specific individual features from EHRs reliably and efficiently whilst also meeting information governance requirements. As an example of this targeted approach, we investigated several methods using electronic prescribing data to classify the source of an infection/infectious syndrome being treated, analysing free-text indications documented by clinicians\u003csup\u003e15\u003c/sup\u003e justifying the reason for antibiotic prescriptions. We compared state-of-the-art NLP models, Bidirectional Encoder Representations from Transformers (BERT)\u003csup\u003e16\u003c/sup\u003e, LLMs, and Generative Pretrained Transformer (GPT)\u003csup\u003e17\u003c/sup\u003e, to traditional ICD code-based approaches, classical NLP methods and regular expression-based text searches. We show modern approaches have potential to accurately extract key information from medical records at scale, potentially opening opportunities for new epidemiological and intervention studies across all of medicine, as well as possibilities for improving care delivery.\u003c/p\u003e "},{"header":"Methods","content":"\u003cp\u003eStudy design and population\u003c/p\u003e\u003cp\u003eWe used EHRs from two distinct hospital sites, Oxford and Banbury, with Oxford serving as our training and internal test set, and Banbury as our external test set. These sites collectively provide 1100 beds, serving 750,000 residents in Oxfordshire, ~ 1% of the UK population. Deidentified data were obtained from Infections in Oxfordshire Research Database (IORD), which has approvals from the National Research Ethics Service South Central – Oxford C Research Ethics Committee (19/SC/0403), the Health Research Authority and the Confidentiality Advisory Group (19/CAG/0144) as a deidentified database without individual consent. All patients aged ≥ 16 years who had antibiotic prescriptions were included.\u003c/p\u003e\u003cp\u003e‘Ground Truth’ Labelling\u003c/p\u003e\u003cp\u003eTwo clinical researchers reviewed each antibiotic indication text string used for training and testing (described below) to establish a reference or ‘ground truth’ label for the clinical syndrome being treated. Any discrepancies were resolved by a third researcher, a clinical infection specialist. Antibiotic indications were labelled using 11 categories representing infection source: Urinary; Respiratory; Abdominal; Neurological; Skin and Soft Tissue; Ear, Nose and Throat (ENT); Orthopaedic; Other specific (i.e. another body site); Non-specific (i.e. no body site provided, e.g. “sepsis”), Prophylaxis, Not informative (i.e. text unrelated to the source of infection, e.g. “as instructed by Dr X”). Each category was recorded as a binary variable, such that more than one potential source could be recorded, e.g. the input string “urinary/chest” would be labelled as both urinary and respiratory. An additional variable was used to document the presence of uncertainty expressed by the prescriber, e.g. “urinary/chest” or “? UTI”.\u003c/p\u003e\u003cp\u003eComparator Classification by ICD10 Codes\u003c/p\u003e\u003cp\u003eAs a comparator, we mapped primary and secondary ICD10 diagnosis codes from the same admission as the antibiotic prescription to the 11 infection sources using CCSR classifications\u003csup\u003e18\u003c/sup\u003e as an intermediate step (see Supplement and \u003cem\u003eAppendix S1\u003c/em\u003e).\u003c/p\u003e\u003cp\u003eTraditional Classification Methods\u003c/p\u003e\n\u003ch3\u003eRegex Rules\u003c/h3\u003e\n\u003cp\u003eThe most intuitive and deterministic method for classifying free-text is searching for specific keywords from a list of predefined words for a given category. We employed fuzzy regular expression (regex) matching patterns with term-specific word boundaries and variable fuzziness to allow for misspellings and variations (see Supplement and \u003cem\u003eAppendix S2\u003c/em\u003e).\u003c/p\u003e\n\u003ch3\u003en-grams \u0026 XGBoost\u003c/h3\u003e\n\u003cp\u003eA second approach used a separate tokeniser, embedding and classifier structure; specifically, Scikit-learn\u0026rsquo;s n-grams \u0026amp; count vectorisation and the gradient boosting model architecture XGBoost\u003csup\u003e19,20\u003c/sup\u003e. Each free-text indication term was broken up into overlapping subwords of length \u003cspan\u003e$\u003c/span\u003en\u003cspan\u003e$\u003c/span\u003e and then count-vectorised, with the count representing the frequency of each subword\u0026rsquo;s occurrence. The vectors of dimension \u003cspan\u003e$\u003c/span\u003evocabulary size\u003cspan\u003e$\u003c/span\u003e were then fed as input features to the classification model. We determined the optimal n-gram size (\u003cspan\u003e$\u003c/span\u003en\u003cspan\u003e$\u003c/span\u003e) and hyperparameters for XGBoost during model training by maximising the receiver operator curve area under the curve (ROC-AUC) (details below).\u003c/p\u003e \u003cp\u003eBERT Classifier\u003c/p\u003e \u003cp\u003eCurrent state-of-the-art NLP tasks employ models built on Transformer architectures, with the Bidirectional Encoder Representations from Transformers (BERT) model family well suited for many tasks requiring semantic understanding. We finetuned\u003csup\u003e21\u003c/sup\u003e pre-trained BERT models on a single GPU instance and used BERT for both encoding and classification. We evaluated the original generic \u0026ldquo;uncased base BERT\u0026rdquo; model, pre-trained on the BooksCorpus and English Wikipedia and a domain-specific \u0026ldquo;Bio\u0026thinsp;+\u0026thinsp;Clinical BERT\u0026rdquo;, pre-trained on biomedical and clinical text\u003csup\u003e22,23\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eFew-Shot and finetuned LLM Classifier\u003c/p\u003e \u003cp\u003eCompared to BERT, the GPT family enables zero- or few-shot learning, i.e. there is potentially minimal need for labelled data for task-specific training. We developed prompts for GPT4, comprised of instructions and the target categories, asking the model to complete the categories (see \u003cem\u003eAppendix S3\u003c/em\u003e for specific prompts, and \u003cem\u003eAppendix S4\u003c/em\u003e for details of batch sizes)\u003csup\u003e24\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eSimilarly, a GPT3.5 model was finetuned on the training dataset and instructed to classify the free-text. Finetuning was achieved by presenting the desired output alongside the training input data. We used the same system prompt as for the few-shot model, while providing the training examples which were fed in batches of ten. Additional model hyperparameters, such as learning rate, and epochs, were chosen through grid search.\u003c/p\u003e \u003cp\u003eTraining, Test and External Evaluation\u003c/p\u003e \u003cp\u003eWe divided the Oxford data with a 90/10 train/test split, resulting in a raw training set and internal test set. From the training data, we labelled and used the 4000 most frequently occurring unique indications. To make labelling tractable we discarded the remaining unlabelled data from the training set. From the internal test data, we randomly selected and exhaustively labelled indications present in 2000 prescriptions. For the external test set from Banbury, we also labelled 2000 randomly selected entries. All models were trained on the training dataset with grid-search based hyperparameter tuning based on cross validation and tested on both the internal and external test sets.\u003c/p\u003e \u003cp\u003eThe multi-label performance of each method was evaluated using weighted F1 scores, precision-recall (PR-AUC) and ROC-AUC. Weighted averages take into account the varied distributions of the classes, such that more common classes contribute more to the overall average, producing estimates that reflect the original data source and represent real-world performance.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003eWe obtained antibiotic prescribing indication data from 826,533 prescriptions from 171,460 adult inpatients, \u0026ge;\u0026thinsp;16 years, between 01-October-2014 and 30-June-2021 from three hospitals in Oxford, UK. The most commonly prescribed antibiotics were co-amoxiclav (n\u0026thinsp;=\u0026thinsp;269,945, 33%), gentamicin (n\u0026thinsp;=\u0026thinsp;70,002, 8%), and metronidazole (n\u0026thinsp;=\u0026thinsp;65,094, 8%) (\u003cem\u003eAppendix S5\u003c/em\u003e), and the most common specialities were General Surgery (n\u0026thinsp;=\u0026thinsp;146,719, 18%), Acute General Medicine (n\u0026thinsp;=\u0026thinsp;98,687, 12%), and Trauma and Orthopaedics (n\u0026thinsp;=\u0026thinsp;90,719, 11%) (\u003cem\u003eAppendix S6\u003c/em\u003e). Patients were a median 56 years old (IQR 36\u0026ndash;73), and 94,721 (55%) were female.\u003c/p\u003e\n\u003cp\u003eWe also used an independent external test dataset to assess classifier performance further, from the Horton Hospital, Banbury (~\u0026thinsp;30 miles from Oxford). This dataset comprised 111,617 prescriptions from 25,924 patients between 01-December-2014 and 30-June-2021, with 13,650 unique free-text indications. Antibiotics prescribed (\u003cem\u003eAppendix S5\u003c/em\u003e) and specialities (\u003cem\u003eAppendix S6\u003c/em\u003e) were broadly similar to the Oxford training set. Patients were a median 67 years old (IQR 47\u0026ndash;80), and 13,853 (53%) were female.\u003c/p\u003e\n\u003cp\u003ePrescription indications\u003c/p\u003e\n\u003cp\u003eFrom the 826,533 Oxford prescriptions, 86,611 unique free-text indications were recorded. The top 10 accounted for 41% of all prescriptions; these included \u0026ldquo;Perioperative Prophylaxis\u0026rdquo; (20%), \u0026ldquo;UTI\u0026rdquo; (4%), \u0026ldquo;LRTI\u0026rdquo; (3%), \u0026ldquo;Sepsis\u0026rdquo; (3%), and \u0026ldquo;CAP\u0026rdquo; (3%). The most commonly occurring 4000 unique indications, used for model training, accounted for 84% (692,310) of prescriptions (\u003cem\u003eAppendix S7\u003c/em\u003e).\u003c/p\u003e\n\u003cp\u003eAs expected, different wording was used to reflect similar concepts, e.g. \u0026ldquo;CAP [community acquired pneumonia]\u0026rdquo;, \u0026ldquo;LRTI [lower respiratory tract infection]\u0026rdquo;, \u0026ldquo;chest infection\u0026rdquo;, and \u0026ldquo;pneumonia\u0026rdquo;. Additionally, misspellings were common, e.g. \u0026ldquo;infction\u0026rdquo;, \u0026ldquo;c. dififcile\u0026rdquo;. Multiple examples expressed uncertainty, or multiple potential sources of infection, e.g. \u0026ldquo;sepsis ?source\u0026rdquo;, \u0026ldquo;UTI/Chest\u0026rdquo;, etc. Reflecting the complexity of prescribing, there were multiple potentially informative, but rarely occurring indications, e.g., \u0026ldquo;transplant pyelonephritis\u0026rdquo;, \u0026ldquo;Ludwig\u0026rsquo;s angina\u0026rdquo;, and \u0026ldquo;deep neck infection\u0026rdquo;, which were only seen 51 (\u0026lt;\u0026thinsp;1%), 27 (\u0026lt;\u0026thinsp;1%), and 13 (\u0026lt;\u0026thinsp;1%) times respectively (\u003cem\u003eAppendix S8\u003c/em\u003e).\u003c/p\u003e\n\u003cp\u003e\u0026lsquo;Ground truth\u0026rsquo; labels\u003c/p\u003e\n\u003cp\u003eFollowing labelling by clinical experts, the 4000 most commonly occurring free-text indications were classified into 11 categories, with a separate variable capturing the presence of uncertainty. The most commonly assigned sources were \u0026ldquo;Prophylaxis\u0026rdquo; (267,788/692,310 prescriptions, 39%), \u0026ldquo;Respiratory\u0026rdquo; (125,744, 18%) and \u0026ldquo;Abdominal\u0026rdquo; (61,670, 9%). 50% (n\u0026thinsp;=\u0026thinsp;344,773) prescriptions had \u0026ldquo;No Specific Source\u0026rdquo;. The most uncertainty was expressed in \u0026ldquo;Neurological\u0026rdquo; and ENT cases at 38% and 33%, respectively (Fig. \u003cspan\u003e1\u003c/span\u003eA). Although \u0026ldquo;Respiratory\u0026rdquo; was the most common category overall after \u0026ldquo;Prophylaxis\u0026rdquo;, there were more distinct text strings associated with \u0026ldquo;Abdominal\u0026rdquo; infections, with \u0026ldquo;Skin and Soft Tissue\u0026rdquo; infection also having a disproportionately larger number of unique text strings (Fig. \u003cspan\u003e1\u003c/span\u003eB). Most \u0026lsquo;multi-source\u0026rsquo; prescriptions were a combination of \u0026ldquo;Prophylaxis\u0026rdquo; and a source (\u0026gt;\u0026thinsp;90%). Excluding prophylaxis, the most common combinations of sources were \u0026ldquo;No Specific source\u0026rdquo; and \u0026ldquo;Not Informative\u0026rdquo;, \u0026ldquo;Urinary\u0026rdquo; and \u0026ldquo;Respiratory\u0026rdquo;, and \u0026ldquo;Skin \u0026amp; Soft Tissue\u0026rdquo; and ENT, in 1.6%, 0.58%, 0.41% prescriptions, respectively (Fig. \u003cspan\u003e1\u003c/span\u003eC-D). The former two reflected diagnostic uncertainty and the latter reflected infections of the face, head and neck frequently involving skin/soft tissue.\u003c/p\u003e\n\u003cp\u003eClassifier performance\u003c/p\u003e\n\u003cp\u003eWe trained classifiers using the labelled training data from three Oxford hospitals (Fig. \u003cspan\u003e2\u003c/span\u003e). Compared to clinician-assigned labels, within the internal Oxford test dataset (n\u0026thinsp;=\u0026thinsp;2000), the weight-averaged F1 score across classes was highest using Bio\u0026thinsp;+\u0026thinsp;Clinical BERT (Average F1\u0026thinsp;=\u0026thinsp;0.97 [worst performing class F1\u0026thinsp;=\u0026thinsp;0.84, best performing F1\u0026thinsp;=\u0026thinsp;0.98]) followed by finetuned GPT3.5 (F1\u0026thinsp;=\u0026thinsp;0.95 [0.77\u0026ndash;0.99]), base-BERT (F1\u0026thinsp;=\u0026thinsp;0.93 [0.23\u0026ndash;0.98]) and tokenisation\u0026thinsp;+\u0026thinsp;XGBoost (F1\u0026thinsp;=\u0026thinsp;0.86 [0.64\u0026ndash;0.96]). Nearly all approaches exceeded traditional regular expression-based matching (F1\u0026thinsp;=\u0026thinsp;0.71 [0.00-0.93]). The few-shot GPT4 model which did not require labelled data performed similarly to this baseline (F1\u0026thinsp;=\u0026thinsp;0.71 [0.30\u0026ndash;0.98]). Similar performance characteristics were achieved on the external validation dataset from Banbury (n\u0026thinsp;=\u0026thinsp;2000; weight-averaged F1 scores: Bio\u0026thinsp;+\u0026thinsp;Clinical BERT 0.98 [0.87-1.00], finetuned GPT3.5 0.97 [0.70-1.00], Base BERT 0.97 [0.63\u0026ndash;0.99], XGBoost 0.84 [0.63-1.00], Regex 0.74 [0.00-0.96], GTP4 0.86 [0.25-1.00]) (Table \u003cspan\u003e1\u003c/span\u003e, also shows classification run times).\u003c/p\u003e\n\u003cdiv\u003e\n \u003ctable id=\"Tab1\" border=\"1\"\u003e\n \u003ccaption language=\"En\"\u003e\n \u003cdiv\u003eTable 1\u003c/div\u003e\n \u003cdiv\u003e\n \u003cp\u003e\u003cstrong\u003eModel performance metrics for the internal (Oxford) and external (Banbury) test sets\u003c/strong\u003e. Each score is listed with the weighted average across the classes\u003c/p\u003e\n \u003cdiv\u003e\n \u003cp\u003e(sources), with the lowest and highest performing class. Overall Accuracy refers to the score calculated for a sample treated as a whole. The finetuned Bio\u0026thinsp;+\u0026thinsp;Clinical BERT outperforms all other methods on both internal and external test sets. Internal Oxford test set\u003c/p\u003e\n \u003c/div\u003e\n \u003c/div\u003e\n \u003c/caption\u003e\n \u003cthead\u003e\n \u003ctr\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eModel\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colspan=\"3\"\u003e\n \u003cp\u003eF1 Score\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\" colspan=\"3\"\u003e\n \u003cp\u003eROC AUC\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\u0026nbsp;\u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003ePR AUC\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\u0026nbsp;\u003c/th\u003e\n \u003cth align=\"left\" colspan=\"3\"\u003e\n \u003cp\u003ePer Class Accuracy\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eAccuracy\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eTraining Runtime\u003c/p\u003e\n \u003c/th\u003e\n \u003cth align=\"left\"\u003e\n \u003cp\u003eTest Runtime\u003c/p\u003e\n \u003c/th\u003e\n \u003c/tr\u003e\n \u003c/thead\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cem\u003eAggregation\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cem\u003eAverage\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cem\u003eLowest\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cem\u003eHighest\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cem\u003eAverage\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cem\u003eLowest\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cem\u003eHighest\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cem\u003eAverage\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cem\u003eLowest\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cem\u003eHighest\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cem\u003eAverage\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cem\u003eLowest\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cem\u003eHighest\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cem\u003eOverall\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cem\u003ePer 4k\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cem\u003ePer 10k\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cem\u003eRegex\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.71\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.00\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.93\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.82\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.32\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.99\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.14\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e-\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e6.4s\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cem\u003eXGBoost\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.86\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.64\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.96\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.96\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.87\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.99\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.9\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.62\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.99\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.95\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.92\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.00\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.72\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e6s\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.2s\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cem\u003eBase BERT\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.93\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.23\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.98\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.99\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.91\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.00\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.97\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.69\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.99\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.98\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.97\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.99\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.88\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e282s\u003csup\u003e1\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e82.2s\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003eBio\u0026thinsp;+\u0026thinsp;Clinical BERT\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.97\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.84\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.98\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.99\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.96\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.00\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.98\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.88\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.00\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.99\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.98\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.00\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.94\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e279s\u003csup\u003e2\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e83.1s\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cem\u003eFine-Tuned OpenAI GPT3.5\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.95\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.77\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.99\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.98\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.97\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.00\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.91\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e~\u0026thinsp;3500s\u003csup\u003e1\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e~\u0026thinsp;3000s\u003csup\u003e2\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cem\u003eFew-Shot OpenAI GPT4\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.71\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.30\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.98\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.87\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.64\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.00\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e0.50\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e\u003cstrong\u003e-\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd align=\"left\"\u003e\n \u003cp\u003e~\u0026thinsp;3000s\u003csup\u003e1\u003c/sup\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n \u003ctfoot\u003e\n \u003ctr\u003e\n \u003ctd colspan=\"16\"\u003e\u003cem\u003eExternal Banbury test set\u003c/em\u003e\u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tfoot\u003e\n \u003c/table\u003e\n\u003c/div\u003e\n\u003cp\u003e\u003cbr\u003e\u003c/p\u003e\n\u003ctable border=\"0\" cellspacing=\"0\" cellpadding=\"0\" width=\"888\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd width=\"17.097862767154105%\"\u003e\n \u003cp\u003e\u003cstrong\u003eModel\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"18.785151856017997%\" colspan=\"3\"\u003e\n \u003cp\u003e\u003cstrong\u003eF1 Score\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"19.34758155230596%\" colspan=\"3\"\u003e\n \u003cp\u003e\u003cstrong\u003eROC AUC\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.299212598425197%\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd width=\"6.299212598425197%\"\u003e\n \u003cp\u003e\u003cstrong\u003ePR AUC\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.186726659167604%\"\u003e\u003cbr\u003e\u003c/td\u003e\n \u003ctd width=\"18.44769403824522%\" colspan=\"3\"\u003e\n \u003cp\u003e\u003cstrong\u003ePer Class Accuracy\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.536557930258717%\"\u003e\n \u003cp\u003e\u003cstrong\u003eAccuracy\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"17.059483726150393%\"\u003e\n \u003cp\u003e\u003cem\u003eAggregation\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e\u003cem\u003eAverage\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e\u003cem\u003eLowest\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e\u003cem\u003eHighest\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e\u003cem\u003eAverage\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e\u003cem\u003eLowest\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.734006734006734%\"\u003e\n \u003cp\u003e\u003cem\u003eHighest\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e\u003cem\u003eAverage\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e\u003cem\u003eLowest\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e\u003cem\u003eHighest\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e\u003cem\u003eAverage\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e\u003cem\u003eLowest\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e\u003cem\u003eHighest\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.519640852974186%\"\u003e\n \u003cp\u003e\u003cem\u003eOverall\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"17.059483726150393%\"\u003e\n \u003cp\u003e\u003cem\u003eRegex\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e0.74\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e0.00\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e0.96\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.734006734006734%\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e0.82\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e0.41\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e0.99\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.519640852974186%\"\u003e\n \u003cp\u003e0.24\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"17.059483726150393%\"\u003e\n \u003cp\u003e\u003cem\u003eXGBoost\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e0.84\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e0.63\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.00\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e0.94\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e0.86\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.734006734006734%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.00\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e0.87\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e0.57\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.00\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e0.94\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e0.88\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.00\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.519640852974186%\"\u003e\n \u003cp\u003e0.68\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"17.059483726150393%\"\u003e\n \u003cp\u003e\u003cem\u003eBase BERT\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e0.97\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e0.63\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e0.99\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.99\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e0.95\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.734006734006734%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.00\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.98\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e0.75\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.00\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.99\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.99\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.00\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.519640852974186%\"\u003e\n \u003cp\u003e0.95\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"17.059483726150393%\"\u003e\n \u003cp\u003e\u003cstrong\u003e\u003cem\u003eBio+Clinical BERT\u003c/em\u003e\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.98\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.87\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.00\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.99\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e0.97\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.734006734006734%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.00\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.98\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.87\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.00\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.99\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.99\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.00\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.519640852974186%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.97\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"17.059483726150393%\"\u003e\n \u003cp\u003e\u003cem\u003eFine-Tuned OpenAI GPT3.5\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e0.97\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e0.70\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.00\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.734006734006734%\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e\u003cstrong\u003e0.99\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e0.98\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.00\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.519640852974186%\"\u003e\n \u003cp\u003e0.95\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"17.059483726150393%\"\u003e\n \u003cp\u003e\u003cem\u003eFew-Shot OpenAI GPT4\u003c/em\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e0.86\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e0.25\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.00\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.734006734006734%\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.285072951739618%\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e-\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e0.95\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e0.81\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"6.172839506172839%\"\u003e\n \u003cp\u003e\u003cstrong\u003e1.00\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.519640852974186%\"\u003e\n \u003cp\u003e0.73\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e[1]\u0026nbsp;Using one Nvidia V100 GPU\u003c/p\u003e\n\u003cp\u003e[2] OpenAI\u0026rsquo;s cloud service\u003c/p\u003e\n\u003cp\u003eClassifier Performance by Class\u003c/p\u003e\n\u003cp\u003eUsing the best performing classifier, Bio\u0026thinsp;+\u0026thinsp;Clinical BERT, we assessed performance within each category. The best-performing categories within our internal test set were \u0026ldquo;Respiratory\u0026rdquo;, \u0026ldquo;No Specific Source\u0026rdquo; and \u0026ldquo;Prophylaxis\u0026rdquo; (F1 score\u0026thinsp;=\u0026thinsp;0.98), followed by \u0026ldquo;Urinary\u0026rdquo; (0.97), \u0026ldquo;Abdominal\u0026rdquo; (0.96), \u0026ldquo;Orthopaedic\u0026rdquo; (0.90), \u0026ldquo;Not Informative\u0026rdquo; (0.89) and \u0026ldquo;Neurological\u0026rdquo; (0.88). The worst performing category was Orthopaedic (0.84), likely due to the high variety of terms used and low number of training samples (n\u0026thinsp;=\u0026thinsp;14, \u003cem\u003eAppendix S9\u003c/em\u003e). Uncertainty was also well detected (0.96) (Fig. \u003cspan\u003e3\u003c/span\u003eA, \u003cem\u003eAppendix S10\u003c/em\u003e).\u003c/p\u003e\n\u003cp\u003eIn the external test data, scores varied slightly, with all source categories except for \u0026ldquo;Not Informative\u0026rdquo; having F1 scores on average 0.02 higher compared to the internal test set. These small differences likely arose from different compositions of categories and the amount of shared vocabulary between the training and test datasets.\u003c/p\u003e\n\u003cp\u003eMisclassifications\u003c/p\u003e\n\u003cp\u003eMost misclassifications were spread evenly across classes for single indications. The two most common misclassifications occurred for \u0026ldquo;Orthopaedic\u0026rdquo; and \u0026ldquo;Other Specific\u0026rdquo; cases, with 12% being misclassified as \u0026ldquo;Prophylaxis\u0026rdquo; and 8% as \u0026ldquo;Skin and Soft Tissue\u0026rdquo;, respectively on the internal test set. On the external test set, most misclassifications were predicted to be \u0026ldquo;Other Specific\u0026rdquo; or \u0026ldquo;Prophylaxis\u0026rdquo; (Fig. \u003cspan\u003e3\u003c/span\u003eC).\u003c/p\u003e\n\u003cp\u003eTraining Dataset Size\u003c/p\u003e\n\u003cp\u003eWe examined the effect of training size on model performance using randomly selected training dataset subsets of 250, 500, 750, 1000, 1500, 2000, 3000, and 4000 unique indications, tested using both internal/external test sets. There was a notable increase in performance (AUC-ROC and F1 scores) when the training size increased from 250 to 1000 samples, suggesting a minimum of 1000 training samples for adequate performance. However, we saw limited improvement as training dataset size rose to 4000, indicating there may be only marginal gains to expanding the training data beyond 4000 samples (\u003cem\u003eAppendix S11\u003c/em\u003e).\u003c/p\u003e\n\u003cp\u003eComparing free-text indications to ICD10 codes\u003c/p\u003e\n\u003cp\u003eWe also compared infection sources from manually labelled \u0026lsquo;ground truth\u0026rsquo; free-text indications to sources inferred from ICD10 diagnostic codes. 31% of sources classified as \u0026ldquo;unspecific\u0026rdquo; using diagnostic codes could be resolved into specific sources using free-text. Rarer infection sources such as \u0026ldquo;CNS\u0026rdquo; and \u0026ldquo;ENT\u0026rdquo; (\u0026lt;\u0026thinsp;1% and no occurrence in diagnostic codes) were represented better by sources extracted from free-text (4% and 4% respectively). Overall, where defined, sources listed in clinical codes generally concurred with the \u0026lsquo;ground truth\u0026rsquo; free-text sources (Fig. \u003cspan\u003e4\u003c/span\u003e).\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eWe show that modern NLP methods can extract clinically relevant details from semi-structured free-text fields. In our example application, a finetuned Bio\u0026thinsp;+\u0026thinsp;Clincal BERT transformer model classified the infection source being treated using clinician-written antibiotic indication text with an F1 score of 0.97 (harmonic mean of sensitivity and positive predictive value). Although this required manual labelling of several thousand text strings, exhaustive labelling of all possible prior strings was not required to achieve this performance. The accuracy of the Bio\u0026thinsp;+\u0026thinsp;Clinical BERT model was substantially better than a sophisticated regular expression-based approach, despite the latter being the solution that many healthcare institutions and researchers might choose at present.\u003c/p\u003e \u003cp\u003eWe also explored what performance might be possible using LLMs without having to label data, e.g. where this is not possible or too resource intensive. However, few-shot learning with GPT4 only achieved modest performance, but it was still similar to the baseline regex method and more consistent across classes. Using LLMs with labelled training data, i.e. a fine-tuned GPT3.5 variant achieved results comparable to the Bio\u0026thinsp;+\u0026thinsp;Clinical BERT approach when correctly specified and tuned but showed more limited potential for deployment as responses can vary greatly in formatting, making it difficult to parse correctly into a rigid format required for most downstream tasks or EHR systems. In environments with limited computing resources where the deployment of deep-learning models is not feasible, regex and XGBoost-based models provide possible alternatives with a reduced runtime of 6.4 and 1.2 seconds/10k indications vs 82.2 seconds/10k for Bio\u0026thinsp;+\u0026thinsp;Clinial BERT.\u003c/p\u003e \u003cp\u003eCurrently, research or clinical use of free-text may be limited by concerns that personal data may be included. Here, by homogenising and categorising sensitive free-text data, we present a privacy-aware solution that enables researchers to utilise the depth of free-text data without direct access or the possibility of identifying specific patients.\u003c/p\u003e \u003cp\u003eOur study has several limitations, including that we only used a subset of the available training data, through the non-exhaustive labelling of a subset of antibiotic indication text strings. However, labelling the 4000 most common unique terms, accounting for 84% of the data, achieved very high performance, with sensitivity analyses suggesting that labelling more examples would not have improved performance substantially. This is likely possible because the underlying Bio\u0026thinsp;+\u0026thinsp;Clinical BERT model is already pre-trained on medical terms and capable of inferring similar words. Of note, many of the remaining 17% of text strings were different combinations of already labelled words, suggesting fewer \u0026ldquo;new\u0026rdquo; or unseen keywords than might be expected. Not fully labelling the training data also makes it more difficult to compare class distributions with the test data sets. We also only used a subset of the test data to evaluate performance; however the 2000 randomly selected samples are likely representative. Although the labelling process was somewhat subjective, independent labelling by two clinical researchers with a third adjudicating any discrepancies was designed to minimise this.\u003c/p\u003e \u003cp\u003eFuture enhancements could optimise computational efficiency while maintaining comparable performance. While XGBoost offers faster training and inference times, its performance was poorer than Bio\u0026thinsp;+\u0026thinsp;Clinical BERT. Therefore, smaller, more efficient NLP models might balance computational demands and performance. Techniques such as model pruning, quantisation, and knowledge distillation could reduce model size and computational requirements while preserving performance\u003csup\u003e25\u0026ndash;27\u003c/sup\u003e. While GPT4 deployments can comply with data governance requirements, its use presents challenges in some settings, as it is usually accessed via third-party cloud compute providers rather than healthcare institutions. Where data need to remain on site, open-source, locally-deployed language models, such as LLAMA, ALPACA or Mistral 7B, may be alternatives that could be further investigated\u003csup\u003e28\u0026ndash;30\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eOur approach has several possible applications; for example, it could be used to monitor and evaluate prescribing practice across different conditions, it provides classification of possible infection sources for epidemiological research\u003csup\u003e31\u003c/sup\u003e, and is also a mechanism for extracting standardised features from medical records for use in predictive algorithms being developed to improve patient care. Although we demonstrate excellent performance for antibiotic indications, it could also be applied to other short strings of free-text, for example descriptions of surgical procedures, patient functional states, or presenting complaints in emergency department and hospital admission data. Across all these domains, refined patient stratification could improve both research and care delivery.\u003c/p\u003e \u003cp\u003eIn summary, we show that state-of-the-art NLP can be used to efficiently and accurately categorise semi-structured free-text in medical records. This has the potential to be applied widely to analyse medical records more accurately.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eAuthor contributions\u003c/h2\u003e\n\u003cp\u003eDWE, TZ, ASW, KY and CHY conceived the study. KY, CHY, QG, HM and DWE analysed the data. KY and DWE wrote the first draft of the manuscript. All authors contributed to revising the manuscript.\u003c/p\u003e\n\u003ch2\u003eAcknowledgements and funding\u003c/h2\u003e\n\u003cp\u003eThis work was supported by the\u0026nbsp;National Institute for Health Research Health Protection Research Unit\u0026nbsp;(NIHR HPRU)\u0026nbsp;in Healthcare Associated Infections and Antimicrobial Resistance at Oxford University in partnership with the UK Health Security Agency (NIHR200915), and the\u0026nbsp;NIHR Biomedical Research Centre, Oxford. DWE is a Big Data Institute Robertson Fellow. ASW is an NIHR Senior Investigator. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, the Department of Health or the UK Health Security Agency. KY is supported by the EPSRC Centre for Doctoral Training in Health Data Science (EP/S02428X/1). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.\u003c/p\u003e\n\u003ch2\u003eDeclaration of interests\u003c/h2\u003e\n\u003cp\u003eNo other author has a conflict of interest to declare.\u003c/p\u003e\n\u003ch2\u003eData sharing\u003c/h2\u003e\n\u003cp\u003eThe data analysed are available from the Infections in Oxfordshire Research Database (https://oxfordbrc.nihr.ac.uk/research-themes/modernising-medical-microbiology-and-big-infection-diagnostics/infections-in-oxfordshire-research-database-iord/), subject to an application and research proposal meeting on the ethical and governance requirements of the Database. Labelled training and test datasets and the pre-trained BERT model are also available via an application to the Database.\u003c/p\u003e\n\u003ch2\u003eCode availability\u003c/h2\u003e\n\u003cp\u003eAll tools developed for this study (Regex builder, Bio+Clinical BERT pipeline, GPT3.5 finetuning and GPT4 few-shot request tools), pre-trained models, as well as the evaluation framework, are available on GitHub:\u003cbr\u003ehttps://github.com/kevihiiin/EHR-Indication-Processing/\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e"},{"header":"References ","content":"\u003col\u003e\n\u003cli\u003eKirby JC, Speltz P, Rasmussen LV, et al. PheKB: a catalog and workflow for creating electronic phenotype algorithms for transportability. J Am Med Inform Assoc 2016;23(6):1046\u0026ndash;52. \u003c/li\u003e\n\u003cli\u003eJamian L, Wheless L, Crofford LJ, Barnado A. Rule-based and machine learning algorithms identify patients with systemic sclerosis accurately in the electronic health record. Arthritis Res Ther 2019;21(1):305. \u003c/li\u003e\n\u003cli\u003eMaarseveen TD, Maurits MP, Niemantsverdriet E, van der Helm-van Mil AHM, Huizinga TWJ, Knevel R. Handwork vs machine: a comparison of rheumatoid arthritis patient populations as identified from EHR free-text by diagnosis extraction through machine-learning or traditional criteria-based chart review. Arthritis Res Ther 2021;23(1):174. \u003c/li\u003e\n\u003cli\u003eSinghal K, Azizi S, Tu T, et al. Large language models encode clinical knowledge. Nature 2023;620(7972):172\u0026ndash;80. \u003c/li\u003e\n\u003cli\u003eJiang LY, Liu XC, Nejatian NP, et al. Health system-scale language models are all-purpose prediction engines. Nature 2023;619(7969):357\u0026ndash;62. \u003c/li\u003e\n\u003cli\u003eMcCoy D, Sebti R, Kuyumjian AG. An evaluation of selected indications and appropriateness of Ampicillin/Sulbactam, an unrestricted antimicrobial, at a single center. Pharm Ther 2017;42(3):189. \u003c/li\u003e\n\u003cli\u003eTimmons V, Townsend J, McKenzie R, Burdalski C, Adams-Sommer V. An evaluation of provider-chosen antibiotic indications as a targeted antimicrobial stewardship intervention. Am J Infect Control 2018;46(10):1174\u0026ndash;9. \u003c/li\u003e\n\u003cli\u003eJavaid W, Cavanaugh S, Lupone C, Stewart T, Fazili T, White B. Fixed vs. Free-text Documentation of Indication for Antibiotic Orders. Open Forum Infect Dis 2017;4(Suppl 1):S325\u0026ndash;6. \u003c/li\u003e\n\u003cli\u003eKadri SS, Lai YL, Warner S, et al. Inappropriate empirical antibiotic therapy for bloodstream infections based on discordant in-vitro susceptibilities: a retrospective cohort analysis of prevalence, predictors, and mortality risk in US hospitals. Lancet Infect Dis 2021;21(2):241\u0026ndash;51. \u003c/li\u003e\n\u003cli\u003eYoon CH, Bartlett S, Stoesser N, et al. Mortality risks associated with empirical antibiotic activity in Escherichia coli bacteraemia: an analysis of electronic health records. J Antimicrob Chemother 2022;77(9):2536\u0026ndash;45. \u003c/li\u003e\n\u003cli\u003eGaieski DF, Edwards JM, Kallan MJ, Carr BG. Benchmarking the Incidence and Mortality of Severe Sepsis in the United States*. Crit Care Med [Internet] 2013;41(5). Available from: https://journals.lww.com/ccmjournal/fulltext/2013/05000/benchmarking_the_incidence_and_mortality_of_severe.2.aspx\u003c/li\u003e\n\u003cli\u003eRhee C, Dantes R, Epstein L, et al. Incidence and Trends of Sepsis in US Hospitals Using Clinical vs Claims Data, 2009-2014. JAMA 2017;318(13):1241\u0026ndash;9. \u003c/li\u003e\n\u003cli\u003eWiese AD, Griffin MR, Stein CM, et al. Validation of discharge diagnosis codes to identify serious infections among middle age and older adults. BMJ Open 2018;8(6):e020857. \u003c/li\u003e\n\u003cli\u003eJolley RJ, Sawka KJ, Yergens DW, Quan H, Jett\u0026eacute; N, Doig CJ. Validity of administrative data in recording sepsis: a systematic review. Crit Care 2015;19(1):139. \u003c/li\u003e\n\u003cli\u003eSaini S, Leung V, Si E, et al. Documenting the indication for antimicrobial prescribing: a scoping review. BMJ Qual Saf 2022;31(11):787\u0026ndash;99. \u003c/li\u003e\n\u003cli\u003eDevlin J, Chang M-W, Lee K, Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding [Internet]. 2019 [cited 2024 Mar 14];Available from: http://arxiv.org/abs/1810.04805\u003c/li\u003e\n\u003cli\u003eRadford A, Wu J, Child R, Luan D, Amodei D, Sutskever I. Language models are unsupervised multitask learners. OpenAI Blog 2019;1(8):9. \u003c/li\u003e\n\u003cli\u003eClinical Classifications Software Refined (CCSR) [Internet]. [cited 2022 May 12];Available from: https://hcup-us.ahrq.gov/toolssoftware/ccsr/ccs_refined.jsp\u003c/li\u003e\n\u003cli\u003ePedregosa F, Varoquaux G, Gramfort A, et al. Scikit-learn: Machine learning in Python. J Mach Learn Res 2011;12:2825\u0026ndash;30. \u003c/li\u003e\n\u003cli\u003eChen T, Guestrin C. XGBoost: A Scalable Tree Boosting System [Internet]. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, NY, USA: Association for Computing Machinery; 2016 [cited 2024 Mar 14]. p. 785\u0026ndash;94.Available from: https://dl.acm.org/doi/10.1145/2939672.2939785\u003c/li\u003e\n\u003cli\u003eWolf T, Debut L, Sanh V, et al. HuggingFace\u0026rsquo;s Transformers: State-of-the-art Natural Language Processing [Internet]. 2020 [cited 2024 Mar 14];Available from: http://arxiv.org/abs/1910.03771\u003c/li\u003e\n\u003cli\u003eAlsentzer E, Murphy J, Boag W, et al. Publicly Available Clinical BERT Embeddings [Internet]. In: Proceedings of the 2nd Clinical Natural Language Processing Workshop. Minneapolis, Minnesota, USA: Association for Computational Linguistics; 2019. p. 72\u0026ndash;8.Available from: https://www.aclweb.org/anthology/W19-1909\u003c/li\u003e\n\u003cli\u003eJohnson AEW, Pollard TJ, Shen L, et al. MIMIC-III, a freely accessible critical care database. Sci Data 2016;3(1):160035. \u003c/li\u003e\n\u003cli\u003eOpenAI (2023). Gpt-4 technical report. ArXiv Prepr ArXiv230308774 2023;\u003c/li\u003e\n\u003cli\u003eMichel P, Levy O, Neubig G. Are Sixteen Heads Really Better than One? [Internet]. In: Advances in Neural Information Processing Systems. Curran Associates, Inc.; 2019 [cited 2024 Mar 18]. Available from: https://proceedings.neurips.cc/paper_files/paper/2019/hash/2c601ad9d2ff9bc8b282670cdd54f69f-Abstract.html\u003c/li\u003e\n\u003cli\u003eZafrir O, Boudoukh G, Izsak P, Wasserblat M. Q8BERT: Quantized 8Bit BERT [Internet]. In: 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing - NeurIPS Edition (EMC2-NIPS). 2019 [cited 2024 Mar 18]. p. 36\u0026ndash;9.Available from: https://ieeexplore.ieee.org/abstract/document/9463531\u003c/li\u003e\n\u003cli\u003eSanh V, Debut L, Chaumond J, Wolf T. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. ArXiv Prepr ArXiv191001108 2019;\u003c/li\u003e\n\u003cli\u003eTouvron H, Lavril T, Izacard G, et al. LLaMA: Open and Efficient Foundation Language Models [Internet]. 2023 [cited 2024 Mar 14];Available from: http://arxiv.org/abs/2302.13971\u003c/li\u003e\n\u003cli\u003eTaori R, Gulrajani I, Zhang T, et al. Stanford Alpaca: An Instruction-following LLaMA model [Internet]. GitHub Repos. 2023;Available from: https://github.com/tatsu-lab/stanford_alpaca\u003c/li\u003e\n\u003cli\u003eJiang AQ, Sablayrolles A, Mensch A, et al. Mistral 7B [Internet]. 2023 [cited 2024 Mar 14];Available from: http://arxiv.org/abs/2310.06825\u003c/li\u003e\n\u003cli\u003eGu Q, Wei J, Yoon CH, et al. Distinct patterns of vital sign and inflammatory marker responses in adults with suspected bloodstream infection. J Infect 2024;106156. \u003c/li\u003e\n\u003cli\u003eOpenAI Chat API Reference [Internet]. [cited 2024 Jan 23];Available from: https://platform.openai.com/docs/api-reference/chat/create \u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"nature-portfolio","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"","title":"Nature Portfolio","twitterHandle":"","acdcEnabled":false,"dfaEnabled":false,"editorialSystem":"ejp","reportingPortfolio":"","inReviewEnabled":true,"inReviewRevisionsEnabled":false},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-4651377/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4651377/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eWhile unstructured free-text data is abundant in electronic health records, challenges in accurate and scalable information extraction often result in a preference for incorporating information from less specific clinical codes instead. We investigate the efficacy of modern natural language processing methods (NLP) and large language models (LLMs) in supervised and few-shot scenarios to extract features from 938,150 hospital antibiotic prescriptions from Oxfordshire, UK. A subset of 4000 most frequent indications for antibiotic use was labelled by clinical researchers into 11 categories, describing the infection source/clinical syndrome, for model training. On separate internal (n\u0026thinsp;=\u0026thinsp;2000) and external test datasets (n\u0026thinsp;=\u0026thinsp;2000), the fine-tuned domain-specific Bio\u0026thinsp;+\u0026thinsp;Clinical BERT model averaged an F1 score of 0.97 and 0.98 respectively across the classes and outperformed traditional regex (F1\u0026thinsp;=\u0026thinsp;0.71 and 0.74) and n-grams/XGBoost (F1\u0026thinsp;=\u0026thinsp;0.86 and 0.84). A few-shot OpenAI GPT4 model achieved F1 scores of 0.71 and 0.86 without using labelled training data and a fine-tuned GPT3.5 model F1 scores of 0.95 and 0.97. Finetuned BERT-based transformer models currently outperform LLMs for structured tasks, while few shot LLMs match the performance of traditional NLP without the need for labelling. Comparing infection sources extracted from ICD10 codes to those parsed from free-text indications, free-text indications revealed 31% more specific infection sources. With their high accuracy, modern transformer-based models have the potential to be used widely throughout medicine to structure free-text records, providing more granular information than clinical codes, facilitating better research and patient care.\u003c/p\u003e","manuscriptTitle":"Transformers and large language models are efficient feature extractors for electronic health record studies","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-08-09 18:00:21","doi":"10.21203/rs.3.rs-4651377/v1","editorialEvents":[],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"communications-medicine","isNatureJournal":true,"hasQc":false,"allowDirectSubmit":false,"externalIdentity":"commsmed","sideBox":"Learn more about [Communications Medicine](http://www.nature.com/commsmed)","snPcode":"43856","submissionUrl":"https://mts-commsmed.nature.com/cgi-bin/main.plex","title":"Communications Medicine","twitterHandle":"@commsmedicine","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"ejp","reportingPortfolio":"Communications Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"6325a641-09ba-46b5-be28-246e4cdc69e4","owner":[],"postedDate":"August 9th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2025-03-22T07:09:56+00:00","versionOfRecord":{"articleIdentity":"rs-4651377","link":"https://doi.org/10.1038/s43856-025-00790-1","journal":{"identity":"communications-medicine","isVorOnly":false,"title":"Communications Medicine"},"publishedOn":"2025-03-21 04:00:00","publishedOnDateReadable":"March 21st, 2025"},"versionCreatedAt":"2024-08-09 18:00:21","video":"","vorDoi":"10.1038/s43856-025-00790-1","vorDoiUrl":"https://doi.org/10.1038/s43856-025-00790-1","workflowStages":[]},"version":"v1","identity":"rs-4651377","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4651377","identity":"rs-4651377","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}