Developing and Testing a Framework for Coding General Practitioners' Free-Text Diagnoses in Electronic Medical Records - A Reliability Study for Generating Training Data in Natural Language Processing

Developing and Testing a Framework for Coding General Practitioners' Free-Text Diagnoses in Electronic Medical Records - A Reliability Study for Generating Training Data in Natural Language Processing | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Developing and Testing a Framework for Coding General Practitioners' Free-Text Diagnoses in Electronic Medical Records - A Reliability Study for Generating Training Data in Natural Language Processing Audrey Wallnöfer, Jakob M. Burgstaller, Katja Weiss, Thomas Rosemann, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4131283/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 16 Jul, 2024 Read the published version in BMC Primary Care → Version 1 posted 10 You are reading this latest preprint version Abstract Background Diagnoses entered by general practitioners into electronic medical records have great potential for research and practice, but unfortunately, diagnoses are often in uncoded format, making them of little use. Natural language processing (NLP) could assist in coding free-text diagnoses, but NLP models require local training data to unlock their potential. The aim of this study was to develop a framework of research-relevant diagnostic codes, to test the framework using free-text diagnoses from a Swiss primary care database and to generate training data for NLP modelling. Methods The framework of diagnostic codes was developed based on input from local stakeholders and consideration of epidemiological data. After pre-testing, the framework contained 105 diagnostic codes, which were then applied by two raters who independently coded randomly drawn lines of free text (LoFT) from diagnosis lists extracted from the electronic medical records of 3000 patients of 27 general practitioners. Coding frequency and mean occurrence rates (n and %) and inter-rater reliability (IRR) of coding were calculated using Cohen's kappa (Κ). Results The sample consisted of 26,980 LoFTs and in 56.3% no code could be assigned because it was not a specific diagnosis. The most common diagnostic codes were, 'dorsopathies' (3.9%) and 'other diseases of the circulatory system' (3.1%). Raters were in almost perfect agreement (Κ ≥0.81) for 69 of the 105 diagnostic codes, and 28 codes showed a substantial agreement (K between 0.61 and 0.80). Both high coding frequency and almost perfect agreement was found in 37 codes, including codes that are particularly difficult to identify from components of the electronic medical record, such as musculoskeletal conditions, cancer or tobacco use. Conclusion The coding framework was characterised by a subset of very frequent and highly reliable diagnostic codes, which will be the most valuable targets for training NLP models for automated disease classification based on free-text diagnoses from Swiss general practice. General Practitioners Electronic Medical Records Diagnostic Coding Reliability Training data Background Routine data from primary care services can importantly contribute to health services research and other monitoring activities. In Switzerland, primary care is predominantly delivered by general practitioners (GPs), and 70% of the population visits a GP at least once a year (1). Importantly for research and monitoring, the majority of healthcare contacts take place in this setting of care (2, 3). Diagnostic data compiled by GPs is therefore a potential ressource for research and monitoring (4-8). However, for statistical synthesis, diagnostic data requires coding (9). Unfortunately, due to time pressure and the complexity of coding frameworks, diagnostic coding is very difficult to implement properly by GPs and there is no financial incentive for diagnostic coding in outpatients in Switzerland (10-12). Thus, coded diagnoses are scarce for reasearch and monitoring in Swiss primary care. The increasing use of electronic medical records by GPs makes data increasingly accessible for research, with even greater potential if coded diagnoses were readily available (13-16). As a result, there is a need to advance the diagnostic coding of diagnoses obtained from GPs. Various methods can be used to achieve this, including purpose-built classification systems for primary care, such as the ICPC-2 code ( International Classification of Primary Care, 2nd edition ) (10, 12, 17-19). However, the ICPC-2 code classifies reasons for encounters on a consultation level, which does not necessarily correspond to all diagnoses present, potentially leading to corresponding underestimation in epidemiological studies. The most widely used system for diagnostic coding is the ICD-10 system ( 10th revision of the International Statistical Classification of Diseases and Related Health Problems ) (20). ICD-10 is a classification system introduced by the World Health Organisation and serves as a global standard for identifying and reporting diseases and health conditions. It allows methodical documentation of disorders and diseases, injuries and other related health conditions. The ICD-10 system, however, differentiates almost 70,000 diagnoses in a highly granulated fashion, making the system very precise but also very difficult to apply for inexperienced raters and it is therefore hardly suitable for coding by GPs (10, 17, 21). Artificial intelligence applications from the domain of natural language processing (NLP) have substantially improved in recent years, are increasingly available and have great potential to support diagnostic coding in medicine (22-24). However, to maximize their effectiveness, NLP models require training ideally on local and sufficiently sized and accurately labelled data, which may be scarce depending on healthcare setting (25). In Swiss general practice, this challenge is particularly difficult for reasons explained above. In addition, even if GPs were to code their diagnoses, the accuracy of coding would still be highly uncertain, given the pacity of training and lack of incentives GPs have in this domain. In order to face this challenge of lacking training data from Swiss general practice, we aimed to develop a framework of relevant diagnostic codes, apply it to a dataset and measure the frequency of codes as well as the reliability of coding, which will be relevant for further using the data for NLP training. Methods Study design, setting and ethics statement This was a study of frequency and inter-rater reliability (IRR) in diagnostic coding using a purposely-developed coding framework in a large primary care database. To select the diagnostic codes, we harvested opinions from local stakeholders as well as epidemiological data to emphasize both the local relevance of codes and expected prevalence of diagnoses in this setting. The large primary care database involved was the FIRE database (FIRE stands for “Family Medicine Research using Electronic Medical Records”), which contains anonymized patient data from Swiss GPs’ electronic medical records (26). The local Ethics Committee of the Canton of Zurich waived approval for research with the FIRE database because patient data is fully anonymized and therefore outside the scope of the Swiss Human Research Act (BASEC-Nr. Req2017–00797). The study was conducted in accordance with the Declaration of Helsinki and good clinical practice guidelines. Diagnostic codes We pre-specified that the number of different diagnostic codes should be limited to approximately 100 in order to prevent over-dispersion. To take relevance for local stakeholders into account, 4 stakeholders (JB, LJ, OS, AP) independently compiled a list of diagnostic codes they deemed relevant to their research. To complete these tasks, the experts used the ICD-10 framework as a template and performed up-coding to the highest level of the code that still was meaningful to them. Unused codes from each ICD-10 chapter were grouped together into a code range containing the remaining diseases for the respective chapter. To consider the expected prevalence of diagnoses in general practice, we used four previously published lists of the 100 most frequent ICD-10 diagnoses in general practice from Nordrhein-Westfalen (NRW-lists), each list covering consecutive three-month periods ranging between the second quarter of 2021 and the first quarter of 2022 (27-30). Diagnostic codes were directly selected for the subsequent coding process if at least three out of four stakeholders independently proposed to include them. Additionally, we included codes proposed by two stakeholders if additionally appearing on each NRW-list. Codes that were proposed by only one or two stakeholders and also appeared on each of the four NRW-lists were subjected to a second committee of stakeholders (SM, AP, AW, KW) who rated the importance of each code to their research on a scale from 1 (lowest importance) to 3 (highest importance). Codes achieving at least 5 points were added to the selection diagnostic codes used in the subsequent coding process ultimately consisting of 115 different codes. Data selection, coding process and analysis For this study, we used data from 27 GPs nested in 10 different general practices. Specifically, from each practice, we randomly drew 300 patients with at least one consultation in the year 2019. From these patients, we exported the patient ID and diagnosis lists in free-text format from their last consultation in 2019, as imputed by the GPs. This data was transferred into a spreadsheet where each line of free-text (LoFT) was assigned to an individual cell. A pre-testing subset containing 10% of the LoFT was drawn to test the intended coding process and refine the coding framework where necessary. Pre-testing revealed redundancies and very low occurrence of specific codes, which were subsequently unified or removed from the selection and thus, the final coding framework consisted of 105 different codes (see Additional File 1 ). The coding process involved two trained physicians (AW and DB) who were tasked to independently assign the diagnostic codes to each LoFT. Raters were tasked to code every LoFT, which reflected an unambiguous diagnosis. In the event of ambiguity or information insufficient to code a diagnosis (such as LoFT describing mere symptoms, laboratory test results or low certainty differential diagnostic considerations) the code for “no diagnosis” was assigned, so that every LoFT in the dataset was coded. In all of the LoFT, we determined for each diagnostic code: 1) frequency by rater, 2) average occurrence rate (as percentage) using the total count of LoFT as denominator and the respective code as numerator, 3) inter-rater agreement (IRA) using the total count of LoFT as denominator and the count of LoFT with concordant coding (absence or presence of the respective code) of the respective code as numerator and 4) inter-rater reliability (IRR) using Cohen’s kappa as measure (31). We used counts and proportions (n and %) for descriptive statistics. We interpreted Κ ≥0.81 as almost perfect agreement K between 0.61 and 0.80 as substantial agreement. For data analysis, we used the software R (Version 4.2.0) (32). Results Sample and frequency analyses From the random sample of 3000 patients, we obtained 26,980 LoFT (of which 2,800 were used for pre-testing). To the 26,980 LoFT, raters 1 and 2 assigned 31,672 and 31,864 codes respectively (the number of codes exceeded the number of LoFT because of cases where multiple codes were assigned to a single LoFT). Taken together, raters most frequently assigned diagnostic codes: “no diagnosis” (56.3%), “dorsopathies” (3.9%), “other diseases of the circulatory system” (3.1%,) and “other diseases of the musculoskeletal system and connective tissue” (2.8%). A frequency of at least 200 (0.7% of LoFT) by at least one rater was encountered in 30 codes (see Table 1) and a frequency of at least 100 (0.4%) was encountered in 51 codes. Eleven codes were assigned with a frequency below 30 (0.1%) by either rater (see Additional File 2 for the complete frequency analysis). Table 1: The thirty most frequently assigned codes or code ranges ICD-Origin Code Rater 1 Rater 2 Avg. of LoFT% Kappa none no diagnosis 15300 15091 56.3% 0.856 M40-M54 dorsopathies 1056 1066 3.9% 0.932 I00-I99 other diseases of the circulatory system 824 865 3.1% 0.848 M00-M99 other diseases of the musculoskeletal system and connective tissue 769 758 2.8% 0.743 I10 primary hypertension 713 704 2.6% 0.972 S00-T98 injury, poisoning and certain other consequences of external causes 654 690 2.5% 0.853 D00-D48 other neoplasms 581 588 2.2% 0.852 E78 disorders of lipoprotein metabolism and other lipidaemias 545 539 2.0% 0.985 E00-E90 other endocrine, nutritional and metabolic diseases 489 501 1.8% 0.876 M60-M79 soft tissue disorders 415 463 1.6% 0.734 K00-K93 other diseases of the digestive system 414 449 1.6% 0.786 L00-L99 other diseases of the skin and subcutaneous tissue 401 458 1.6% 0.833 H00-H59 diseases of the eye and adnexa 344 350 1.3% 0.900 C00-C99 malignant neoplasms 333 359 1.3% 0.839 F17 mental and behavioural disorders due to use of tobacco 312 315 1.2% 0.969 I20-I25 ischaemic heart diseases 297 305 1.1% 0.925 K57 diverticular disease of intestine 284 281 1.0% 0.973 N00-N99 other diseases of the genitourinary system 252 302 1.0% 0.780 K21 gastro-oesophageal reflux disease 262 260 1.0% 0.957 E65-E68 obesity and other hyperalimentation 260 260 1.0% 0.961 G00-G99 other diseases of the nervous system 234 263 0.9% 0.778 F32-F33 depressive episode and recurrent depressive disorder 238 249 0.9% 0.96 E00-E07 disorders of thyroid gland 223 238 0.9% 0.883 J00-J99 other diseases of the respiratory system 212 247 0.9% 0.751 A00-B99 intestinal infectious diseases 236 218 0.8% 0.785 K40-K46 hernia 221 221 0.8% 0.950 H60-H95 other diseases of the ear and mastoid process 226 207 0.8% 0.862 E11 type 2 diabetes mellitus 217 211 0.8% 0.906 I83 varicose veins of lower extremities 195 217 0.8% 0.882 D50-D90 other diseases of the blood and blood-forming organs and certain disorders involving the immune mechanism 183 207 0.7% 0.782 Agreement and reliability With respect to measures of coding agreement, we found IRA of >0.98 in all assigned codes except “no diagnosis” (IRA = 0.93). With respect to IRR, we found Kappa values ≥0.810 in 69 of all the 105 diagnostic codes and 28 codes showed Kappa between 0.610 and <0.810. Simultaneously a frequency of 100 by at least one rater and a Kappa value ≥0.81 was found in 37 codes (see Table 2). Among these frequently assigned diagnostic codes, we found the highest IRR in “disorders of lipoprotein metabolism and other lipidaemias” (Kappa = 0.985), “diverticular disease of intestine” (Kappa = 0.973) and “primary hypertension” (Kappa = 0.972). Table 2: Codes that were both frequently and reliably assigned ICD-10 Code Code Rater 1 Rater 2 Avg. of LoFT% Kappa E78 disorders of lipoprotein metabolism and other lipidaemias 545 539 2.0% 0.985 K57 diverticular disease of intestine 284 281 1.0% 0.973 I10 primary hypertension 713 704 2.6% 0.972 F17 mental and behavioural disorders due to use of tobacco 312 315 1.2% 0.969 I11-I14 hypertension with end organ damage 163 159 0.6% 0.962 I48 atrial fibrillation and flutter 147 140 0.5% 0.961 E65-E68 obesity and other hyperalimentation 260 260 1.0% 0.961 E55 vitamin D deficiency 112 113 0.4% 0.960 F32-F33 depressive episode and recurrent depressive disorder 238 249 0.9% 0.960 K21 gastro-oesophageal reflux disease 262 260 1.0% 0.957 N18 chronic kidney disease 110 108 0.4% 0.954 M17 arthritis of the knee 183 176 0.7% 0.952 N40 hyperplasia of prostate 106 115 0.4% 0.950 K40-K46 intestinal hernia 221 221 0.8% 0.950 G47 sleep disorders 150 147 0.6% 0.949 J45 asthma 174 177 0.7% 0.945 M40-M54 dorsopathies 1056 1066 3.9% 0.932 I20-I25 ischaemic heart diseases 297 305 1.1% 0.925 K64 haemorrhoids and perianal venous thrombosis 122 119 0.4% 0.921 K29 gastritis and duodenitis 143 150 0.5% 0.914 E11 type 2 diabetes mellitus 217 211 0.8% 0.906 H00-H59 diseases of the eye and adnexa 344 350 1.3% 0.900 N80-N98 noninflammatory disorders of female genital tract 133 128 0.5% 0.896 E00-E07 disorders of thyroid gland 223 238 0.9% 0.883 I83 varicose veins of lower extremities 195 217 0.8% 0.882 E00-E90 other endocrine, nutritional and metabolic diseases 489 501 1.8% 0.876 I60-I69 cerebrovascular diseases 135 130 0.5% 0.874 H60-H95 other diseases of the ear and mastoid process 226 207 0.8% 0.862 no diagnosis no diagnosis 15300 15091 56.3% 0.856 S00-T98 injury, poisoning and certain other consequences of external causes 654 690 2.5% 0.853 D00-D48 other neoplasms 581 588 2.2% 0.852 I00-I99 other diseases of the circulatory system 824 865 3.1% 0.848 F40-F48 neurotic, stress-related and somatoform disorders 162 197 0.7% 0.839 C00-C99 malignant neoplasms 333 359 1.3% 0.839 L00-L99 other diseases of the skin and subcutaneous tissue 401 458 1.6% 0.833 Discussion Coded diagnoses from Swiss GP are difficult to obtain but necessary for training NLP models. In this study, we developed a set of 105 diagnostic codes, applied them to a moderately sized dataset of only about 26,000 LoFT and measured frequencies as well as reliability of codes. Over a third of the codes achieved both a frequency above 100 and an almost perfect IRR and are thus suitable for training NLP models using this dataset. The most promising codes in this regard are those that are not easily identified by methods using other data from the electronic medical record (such as laboratory tests or disease-specific medications) and LoFT are the only data source, such as musculoskeletal conditions, cancer or tobacco use. We developed diagnostic codes with the a priori intention of generating training data for NLP models. To do this, we attempted to limit the granularity of the diagnostic codes to around 100 items in order to avoid over-dispersion, where rarely occurring codes would have insufficient frequency to train NLP models on moderately sized datasets. Within the set of coded LoFT, 51 codes were assigned at least 100 times by both raters and are therefore potential candidates for exploring the feasibility of NLP. Interestingly, however, more than half of the LoFT were coded as 'no diagnosis', suggesting that GPs use this space for additional information that does not amount to a specific diagnosis. This is consistent with findings from other studies that have analysed the content of LoFT, showing that non-specific or insufficient information is common in medical documentation (33-36) but substantially reduced the yield of LoFT for obtaining coded diagnostic data in our study. Specifically, ambiguous acronyms or abbreviations (37-39), unstructured information (39-41), as well as physicians’ and institutional stylistic preferences contribute to non-diagnostic information in free-text diagnoses (42). Raters in our study were notably challenged by non-diagnostic information in LoFT, which manifested itself in an IRA of only 93%, whereas all other codes had IRA ≥98%. We strongly expect that these difficulties will be transferred to the NLP modelling process and methods will be needed to deal not only with false positive identifications but also with ambiguity within the LoFT itself. Third party review and arbitration can be used to further process the training data, but such human arbitration is arguably not a perfect gold standard and may inevitably introduce bias in addition to that introduced when the LoFT was created. This chain of fundamental validity issues highlights important future limitations of NLP-identified diagnoses and foreseeably questions the feasibility of fully automated coding in cases where very high accuracy is required. Unsurprisingly, the most frequently assigned diagnostic codes were those for the most common chronic or recurrent conditions, particularly those of the musculoskeletal and cardiovascular systems (43). Several of these diagnoses were already identifiable in the FIRE database based on algorithms applied to routine data such as prescribed medications (e.g., antidiabetic drugs to identify diabetes) or results of clinical or laboratory tests (e.g., body mass index for obesity) (44). However, there are several important and common diagnoses for which sufficiently specific identification criteria based on routine data are lacking, including musculoskeletal conditions, cancer, tobacco use, depression, sleep disorders and many others, which are important targets of research in general practice. These diagnoses represent the area where we expect NLP to add the most value for research using the FIRE database. With regard to the plausibility of the code frequency, the rankings of the codes were plausible when taking into account the ranking of the corresponding disease prevalence estimates in the Swiss population. Specifically, dorsopathies, followed by essential hypertension and hyperlipidemia, are the most frequently appearing chronic diseases in this setting according to external studies (45-50). Moreover, frequencies in our study are also very similar to a study measuring reasons for encounters in general practice where diseases of the musculoskeletal and cardio-circulatory systems were by far the most prevalent, thus adding to the plausibility of our results (51-53). With regard to IRR, we observed almost perfect agreement (Kappa ≥0.810) in two thirds of the codes and substantial agreement in another quarter. Taken together, more than 90% of codes had at least substantial agreement when rated by raters having completed medical school without further training. These findings are comparatively favorable when similar studies with inexperienced raters are considered (21, 54, 55) and equal to studies with experienced raters (56). Depending on the research question and the target diseases to be coded, Kappa values ≥0.500 are generally deemed sufficient (31, 54, 57) and thus, the codes we developed appeared to perform sufficiently. Previous studies have shown that code frequency is associated with IRR (58, 59). This finding was replicated in our study, where all of the 20 most frequent codes reached either an almost perfect or substantial IRR, while the 20 least frequent codes had a Kappa ≤ 0.600. Strengths and limitations: This research project describes the design and reliability testing of a custom coding framework to be used for training NLP models. The project can serve as a template for similar research, which will become increasingly important given the growing role of AI in medicine and the associated need for local training data tailored to local factors such as languages and use cases. The use of LoFT from general practice-based medical diagnosis lists is a very prominent use case in this regard, and our study provides estimates of code frequencies based on a moderately sized dataset, which can be achieved with a small investment in manual coding labour. The methods used are highly feasible and provide transparent metrics that help in further interpretation of NLP modelling results, especially when considering the IRR of coding by human raters labelling the training data. The moderate size and locality of the dataset may be a major limitation. We tried to include LoFT data from a representative sample of Swiss GPs, but this sample still only included 27 of them, and these were nested in 10 different medical practices. The local jargon of these GPs may limit the applicability of NLP models based on these training data, and NLP models need to be tested within, but more importantly outside, this dataset. Conclusion We developed and tested a framework of research-relevant diagnostic codes in a primary care research database to train NLP models based on free text data. We have identified a subset of very frequent and highly reliable diagnostic codes, and the next step in the research agenda is to train NLP models with the obtained data and evaluate their performance in automated disease classification. Abbreviations FIRE Family Medicine ICPC Research using Electronic Medical Records LoFT Lines of free-text GP General Practitioner ICD-10 International Statistical Classification of Diseases and Related Health Problems, Tenth Revision IRA Interrater agreement IRR Interrater reliability Declarations Ethics approval and consent to participate: The local Ethics Committee of the Canton of Zurich waived approval for the present study because the FIRE project is outside the scope of the law on human research and studies utilizing data from the FIRE project are thus exempt from ethics review (BASEC-Nr. Req2017–00797). Consent for publication: Consent for publication was waived by the ethics committee due to the anonymization of the data at the practice level Consent to participate declaration : not applicable due to the anonymization of the data at the practice level Availability of data and materials: The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request. Competing interests: The authors declare that no conflicts of interest are relevant to any aspects of this work. Funding: This study received no external funding. Authors' contributions: SM and JB conceived and designed the study; SM and JB performed data acquisition; AW performed data analysis and drafted the original draft; SM, KW, TR and OS revised the original draft of the manuscript and the version to be published. All the authors revised and approved the final manuscript for publication. Acknowledgements: We thank Levy Jäger (LJ) and Andreas Plate (AP) for their contribution in the code selection, as well as Donika Balaj (DB), Adriana Keller (AK) and Gino Bopp (GB) for their contribution to the coding procedure (DB) pre-testing the coding framework (AK and GB). Authors' information (optional): Not applicable. References Statistik Bf. Konsultationen bei Generalistinnen und Generalisten nach Geschlecht, Alter, Bildungsniveau, Sprachgebiet. In: Statistik Bf, editor. 30.10.2018. Green LA, Fryer GE, Jr., Yawn BP, Lanier D, Dovey SM. The ecology of medical care revisited. N Engl J Med. 2001;344(26):2021-5. Senn N, Tiaré Ebert S, Cohidon C. Die Hausarztmedizin in der Schweiz – Perspektiven. Analyse basierend auf den Indikatoren des Programm SPAM (Swiss Primary Care Active Monitoring). Obsan Bulletin 2016;11/2016:4. Meci A, Du Breuil F, Vilcu A, Pitel T, Guerrisi C, Robard Q, et al. The Sentiworld project: global mapping of sentinel surveillance networks in general practice. BMC Prim Care. 2022;23(1):173. Clothier HJ, Fielding JE, Kelly HA. An evaluation of the Australian Sentinel Practice Research Network (ASPREN) surveillance for influenza-like illness. Commun Dis Intell Q Rep. 2005;29(3):231-47. Liljeqvist GT, Staff M, Puech M, Blom H, Torvaldsen S. Automated data extraction from general practice records in an Australian setting: trends in influenza-like illness in sentinel general practices and emergency departments. BMC public health. 2011;11:435. de Lusignan S, Hague N, van Vlymen J, Kumarapeli P. Routinely-collected general practice data are complex, but with systematic processing can be used for quality improvement and research. Inform Prim Care. 2006;14(1):59-66. de Lusignan S, van Weel C. The use of routinely collected computer data for research in primary care: opportunities and challenges. Fam Pract. 2006;23(2):253-63. Nicholls SG, Langan SM, Benchimol EI. Routinely collected data: the importance of high-quality diagnostic coding to research. Cmaj. 2017;189(33):E1054-e5. Kühlein T, Virtanen M, Claus C, Popert U, van Boven K. [Coding in general practice-Will the ICD-11 be a step forward?]. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz. 2018;61(7):828-35. Lasker RD, Marquis MS. The intensity of physicians' work in patient visits--implications for the coding of patient evaluation and management services. N Engl J Med. 1999;341(5):337-41. Letrilliart L, Gelas-Dore B, Ortolan B, Colin C. Prometheus: the implementation of clinical coding schemes in French routine general practice. Inform Prim Care. 2006;14(3):157-65. Biro SC, Barber DT, Kotecha JA. Trends in the use of electronic medical records. Canadian family physician Medecin de famille canadien. 2012;58(1):e21. Chang F, Gupta N. Progress in electronic medical record adoption in Canada. Canadian family physician Medecin de famille canadien. 2015;61(12):1076-84. Statistik Bf. Führung der Krankengeschichten an den Standorten der Arztpraxen und ambulanten Zentren. 2022. Djalali S. Wer eHealth sucht, findet einen Haufen Papier. Schweizerische Ärztezeitung (SÄZ). 2015;96(43):1575-8. Frese T, Herrmann K, Bungert-Kahl P, Sandholzer H. Inter-rater reliability of the ICPC-2 in a German general practice setting. Swiss Med Wkly. 2012;142:w13621. Lamberts H, Wood M, Hofmans-Okkes IM. International primary care classifications: the effect of fifteen years of evolution. Fam Pract. 1992;9(3):330-9. Verbeke M, Schrans D, Deroose S, De Maeseneer J. The International Classification of Primary Care (ICPC-2): an essential tool in the EPR of the GP. Stud Health Technol Inform. 2006;124:809-14. WHO. International Statistical Classification of Diseases and Related Health Problems 10th Revision 2019 [Available from: https://icd.who.int/browse10/2019/en. Stausberg J, Lehmann N, Kaczmarek D, Stein M. Reliability of diagnoses coding with ICD-10. Int J Med Inform. 2008;77(1):50-7. Jiang F, Jiang Y, Zhi H, Dong Y, Li H, Ma S, et al. Artificial intelligence in healthcare: past, present and future. Stroke and vascular neurology. 2017;2(4). Yu K-H, Beam AL, Kohane IS. Artificial intelligence in healthcare. Nature biomedical engineering. 2018;2(10):719-31. Chen M, Hao Y, Hwang K, Wang L, Wang L. Disease prediction by machine learning over big data from healthcare communities. Ieee Access. 2017;5:8869-79. Althnian A, AlSaeed D, Al-Baity H, Samha A, Dris AB, Alzakari N, et al. Impact of Dataset Size on Classification Performance: An Empirical Evaluation in the Medical Domain. Applied Sciences. 2021;11(2):796. Chmiel C, Bhend H, Senn O, Zoller M, Rosemann T. The FIRE project: a milestone for research in primary care in Switzerland. Swiss Med Wkly. 2011;140:w13142. Nordrhein KV. Die 100 häufigsten ICD-10-Schlüssel und Kurztexte (nach Fachgruppen). In: Nordrhein KV, editor. 2 Quartal 20212021. Nordrhein KV. Die 100 häufigsten ICD-10-Schlüssel und Kurztexte (nach Fachgruppen) In: Nordrhein KV, editor. 3 Quartal 20212021. Nordrhein KV. Die 100 häufigsten ICD-10-Schlüssel und Kurztexte (nach Fachgruppen). In: Nordrhein KV, editor. 4 Quartal 20212021. Nordrhein KV. Die 100 häufigsten ICD-10-Schlüssel und Kurztexte (nach Fachgruppen). In: Nordrhein KV, editor. 1 Quartal 20222022. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-74. Team RC. The R Project for Statistical Computing 2022 [4.2.0:[Available from: https://www.R-project.org/. Tsai CH, Eghdam A, Davoody N, Wright G, Flowerday S, Koch S. Effects of Electronic Health Record Implementation and Barriers to Adoption and Use: A Scoping Review and Qualitative Analysis of the Content. Life (Basel). 2020;10(12). Nguyen L, Bellucci E, Nguyen LT. Electronic health records implementation: an evaluation of information system impact and contingency factors. Int J Med Inform. 2014;83(11):779-96. Lium JT, Tjora A, Faxvaag A. No paper, but the same routines: a qualitative exploration of experiences in two Norwegian hospitals deprived of the paper based medical record. BMC medical informatics and decision making. 2008;8:2. Whittaker AA, Aufdenkamp M, Tinley S. Barriers and facilitators to electronic documentation in a rural hospital. J Nurs Scholarsh. 2009;41(3):293-300. Hanauer DA, Mei Q, Law J, Khanna R, Zheng K. Supporting information retrieval from electronic health records: A report of University of Michigan's nine-year experience in developing and using the Electronic Medical Record Search Engine (EMERSE). J Biomed Inform. 2015;55:290-300. Barrows Jr RC, Busuioc M, Friedman C. Limited parsing of notational text visit notes: ad-hoc vs. NLP approaches. Proc AMIA Symp. 2000:51-5. Tang KL, Lucyk K, Quan H. Coder perspectives on physician-related barriers to producing high-quality administrative data: a qualitative study. CMAJ open. 2017;5(3):E617-e22. Brown T, Zelch B, Lee JY, Doctor JN, Linder JA, Sullivan MD, et al. A Qualitative Description of Clinician Free-Text Rationales Entered within Accountable Justification Interventions. Appl Clin Inform. 2022;13(4):820-7. Rubio-López I, Costumero R, Ambit H, Gonzalo-Martín C, Menasalvas E, Rodríguez González A. Acronym Disambiguation in Spanish Electronic Health Narratives Using Machine Learning Techniques. Stud Health Technol Inform. 2017;235:251-5. Feder SL. Data Quality in Electronic Health Records Research: Quality Domains and Assessment Methods. West J Nurs Res. 2018;40(5):753-66. Excoffier S, Herzig L, N'Goran AA, Déruaz-Luyet A, Haller DM. Prevalence of multimorbidity in general practice: a cross-sectional study within the Swiss Sentinel Surveillance System (Sentinella). BMJ open. 2018;8(3):e019616. Meier R, Grischott T, Rachamin Y, Jäger L, Senn O, Rosemann T, et al. Importance of different electronic medical record components for chronic disease identification in a Swiss primary care database: a cross-sectional study. Swiss Med Wkly. 2023;153:40107. (SGB) BSG. Häufigkeit von Rücken- oder Kopfschmerzen. In: 2023 O, editor. 2023. Danon-Hersch N, Marques-Vidal P, Bovet P, Chiolero A, Paccaud F, Pécoud A, et al. Prevalence, awareness, treatment and control of high blood pressure in a Swiss city general population: the CoLaus study. Eur J Cardiovasc Prev Rehabil. 2009;16(1):66-72. Walther D, Curjuric I, Dratva J, Schaffner E, Quinto C, Rochat T, et al. High blood pressure: prevalence and adherence to guidelines in a population-based cohort. Swiss Med Wkly. 2016;146:w14323. Statistik Bf. Personen mit Bluthochdruck nach Geschlecht, Alter, Bildungsniveau, Sprachgebiet. In: Statistik Bf, editor.: BFS; 2017. Marco Storni RL, ­Kaeser M. Schweizerische Gesundheitsbefragung 2017. In: (BFS) BfS, editor.: Bundesamt für Statistik; 2018. Estoppey D, Paccaud F, Vollenweider P, Marques-Vidal P. Trends in self-reported prevalence and management of hypertension, hypercholesterolemia and diabetes in Swiss adults, 1997-2007. BMC public health. 2011;11:114. Tandjung R, Hanhart A, Bärtschi F, Keller R, Steinhauer A, Rosemann T, Senn O. Referral rates in Swiss primary care with a special emphasis on reasons for encounter. Swiss Med Wkly. 2015;145:w14244. Lurquin B, Kellou N, Colin C, Letrilliart L. Comparison of rural and urban French GPs' activity: a cross-sectional study. Rural Remote Health. 2021;21(3):5865. Schäfer I, Hansen H, Ruppel T, Lühmann D, Wagner HO, Kazek A, Scherer M. Regional differences in reasons for consultation and general practitioners' spectrum of services in northern Germany - results of a cross-sectional observational study. BMC Fam Pract. 2020;21(1):22. Wockenfuss R, Frese T, Herrmann K, Claussnitzer M, Sandholzer H. Three- and four-digit ICD-10 is not a reliable classification system in primary care. Scand J Prim Health Care. 2009;27(3):131-6. Asadi F, Hosseini MA, Almasi S. Reliability of trauma coding with ICD-10. Chin J Traumatol. 2022;25(2):102-6. Peng M, Eastwood C, Boxill A, Jolley RJ, Rutherford L, Carlson K, et al. Coding reliability and agreement of International Classification of Disease, 10(th) revision (ICD-10) codes in emergency department data. International journal of population data science. 2018;3(1):445. Cheniaux E, Landeira-Fernandez J, Versiani M. The diagnoses of schizophrenia, schizoaffective disorder, bipolar disorder and unipolar depression: interrater reliability and congruence between DSM-IV and ICD-10. Psychopathology. 2009;42(5):293-8. Koopman B, Karimi S, Nguyen A, McGuire R, Muscatello D, Kemp M, et al. Automatic classification of diseases from free-text death certificates for real-time surveillance. BMC medical informatics and decision making. 2015;15:53. Mandrekar JN. Measures of interrater agreement. J Thorac Oncol. 2011;6(1):6-7. Additional Declarations No competing interests reported. Supplementary Files Additionalfile1.docx Additionalfile2.docx Cite Share Download PDF Status: Published Journal Publication published 16 Jul, 2024 Read the published version in BMC Primary Care → Version 1 posted Editorial decision: Revision requested 21 May, 2024 Reviews received at journal 20 May, 2024 Reviews received at journal 01 May, 2024 Reviewers agreed at journal 18 Apr, 2024 Reviewers agreed at journal 18 Apr, 2024 Reviewers invited by journal 18 Apr, 2024 Editor invited by journal 12 Apr, 2024 Submission checks completed at journal 09 Apr, 2024 Editor assigned by journal 09 Apr, 2024 First submitted to journal 19 Mar, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4131283","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":289463026,"identity":"d02ba7e3-b15b-4db3-a99b-ca2f2c9a6e09","order_by":0,"name":"Audrey Wallnöfer","email":"","orcid":"","institution":"University and University Hospital Zurich","correspondingAuthor":false,"prefix":"","firstName":"Audrey","middleName":"","lastName":"Wallnöfer","suffix":""},{"id":289463028,"identity":"b6c84541-c71f-4b3d-911a-dd87a409fffb","order_by":1,"name":"Jakob M. Burgstaller","email":"","orcid":"","institution":"University and University Hospital Zurich","correspondingAuthor":false,"prefix":"","firstName":"Jakob","middleName":"M.","lastName":"Burgstaller","suffix":""},{"id":289463030,"identity":"b032096e-4ba3-4ac8-82ff-798e4ab84e1b","order_by":2,"name":"Katja Weiss","email":"","orcid":"","institution":"University and University Hospital Zurich","correspondingAuthor":false,"prefix":"","firstName":"Katja","middleName":"","lastName":"Weiss","suffix":""},{"id":289463034,"identity":"a547ef59-7689-4987-9a6d-ed03c3bec8f9","order_by":3,"name":"Thomas Rosemann","email":"","orcid":"","institution":"University and University Hospital Zurich","correspondingAuthor":false,"prefix":"","firstName":"Thomas","middleName":"","lastName":"Rosemann","suffix":""},{"id":289463038,"identity":"01e814ba-3412-4c1a-b8c9-2ed7118165d4","order_by":4,"name":"Oliver Senn","email":"","orcid":"","institution":"University and University Hospital Zurich","correspondingAuthor":false,"prefix":"","firstName":"Oliver","middleName":"","lastName":"Senn","suffix":""},{"id":289463042,"identity":"9d1732f6-d518-486f-ad31-6546e391670f","order_by":5,"name":"Stefan Markun","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA3ElEQVRIiWNgGAWjYJCCDxCKDYgr4Cw8gI2BcQZCyxmStTC2EaFFfn7zwYafOxii+WcfS5MunHfYXr6B+dkDfFoMjrElNvaeYcidcS7tmPTMbYcTGxvYzA3wamHjMX/A28aQ23CGvU2ad9vhBGYGHjYJvA5r4//Y+BeoZT5Yy5zD9myEtDAc42FsBtmy4QzbMWnehsOMPYS0GBxLM2yWbZPI3XiGLdma51h64gxmNjP8Dms+/LDxbZtN7rwzbIa3eWqs7eXbm5/hdxgEIKthJkL9KBgFo2AUjAL8AABIkkDxFNy4SAAAAABJRU5ErkJggg==","orcid":"","institution":"University and University Hospital Zurich","correspondingAuthor":true,"prefix":"","firstName":"Stefan","middleName":"","lastName":"Markun","suffix":""}],"badges":[],"createdAt":"2024-03-19 14:50:19","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4131283/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4131283/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1186/s12875-024-02514-1","type":"published","date":"2024-07-16T16:13:38+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":61596753,"identity":"a3dafa49-6b07-4de7-8070-30d59f198657","added_by":"auto","created_at":"2024-08-01 17:29:42","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":645510,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4131283/v1/8a7f39cb-0d8d-414b-b30c-39037cd64923.pdf"},{"id":54572444,"identity":"22c23be6-af27-4787-a4ec-fb7bab76aea0","added_by":"auto","created_at":"2024-04-12 13:03:05","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":25,"visible":true,"origin":"","legend":"","description":"","filename":"Additionalfile1.docx","url":"https://assets-eu.researchsquare.com/files/rs-4131283/v1/63fba2dedb5285876ebed20c.docx"},{"id":54572445,"identity":"af43931d-b839-4148-b05d-6f96a831fc97","added_by":"auto","created_at":"2024-04-12 13:03:05","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":23872,"visible":true,"origin":"","legend":"","description":"","filename":"Additionalfile2.docx","url":"https://assets-eu.researchsquare.com/files/rs-4131283/v1/73bb8e87a14feb918c19582d.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Developing and Testing a Framework for Coding General Practitioners' Free-Text Diagnoses in Electronic Medical Records - A Reliability Study for Generating Training Data in Natural Language Processing","fulltext":[{"header":"Background","content":"\u003cp\u003eRoutine data from primary care services can importantly contribute to health services research and other monitoring activities. In Switzerland, primary care is predominantly delivered by general practitioners (GPs), and 70% of the population visits a GP at least once a year (1). Importantly for research and monitoring, the majority of healthcare contacts take place in this setting of care (2, 3). Diagnostic data compiled by GPs is therefore a potential ressource for research and monitoring (4-8). However, for statistical synthesis, diagnostic data requires coding (9). Unfortunately, due to time pressure and the complexity of coding frameworks, diagnostic coding is very difficult to implement properly by GPs and there is no financial incentive for diagnostic coding in outpatients in Switzerland (10-12). Thus, coded diagnoses are scarce for reasearch and monitoring in Swiss primary care.\u003c/p\u003e\n\u003cp\u003eThe increasing use of electronic medical records by GPs makes data increasingly accessible for research, with even greater potential if coded diagnoses were readily available (13-16). As a result, there is a need to advance the diagnostic coding of diagnoses obtained from GPs. Various methods can be used to achieve this, including purpose-built classification systems for primary care, such as the ICPC-2 code (\u003cem\u003eInternational Classification of Primary Care, 2nd edition\u003c/em\u003e) (10, 12, 17-19). However, the ICPC-2 code classifies reasons for encounters on a consultation level, which does not necessarily correspond to all diagnoses present, potentially leading to corresponding underestimation in epidemiological studies. The most widely used system for diagnostic coding is the ICD-10 system (\u003cem\u003e10th revision of the International Statistical Classification of Diseases and Related Health Problems\u003c/em\u003e) (20). ICD-10 is a classification system introduced by the World Health Organisation and serves as a global standard for identifying and reporting diseases and health conditions. It allows methodical documentation of disorders and diseases, injuries and other related health conditions. The ICD-10 system, however, differentiates almost 70,000 diagnoses in a highly granulated fashion, making the system very precise but also very difficult to apply for inexperienced raters and it is therefore hardly suitable for coding by GPs (10, 17, 21).\u003c/p\u003e\n\u003cp\u003eArtificial intelligence applications from the domain of natural language processing (NLP) have substantially improved in recent years, are increasingly available and have great potential to support diagnostic coding in medicine (22-24). However, to maximize their effectiveness, NLP models require training ideally on local and sufficiently sized and accurately labelled data, which may be scarce depending on healthcare setting (25). In Swiss general practice, this challenge is particularly difficult for reasons explained above. In addition, even if GPs were to code their diagnoses, the accuracy of coding would still be highly uncertain, given the pacity of training and lack of incentives GPs have in this domain. In order to face this challenge of lacking training data from Swiss general practice, we aimed to develop a framework of relevant diagnostic codes, apply it to a dataset and measure the frequency of codes as well as the reliability of coding, which will be relevant for further using the data for NLP training.\u003c/p\u003e"},{"header":"Methods","content":"\u003ch2\u003eStudy design, setting and ethics statement\u003c/h2\u003e\n\u003cp\u003eThis was a study of frequency and inter-rater reliability (IRR) in diagnostic coding using a purposely-developed coding framework in a large primary care database. To select the diagnostic codes, we harvested opinions from local stakeholders as well as epidemiological data to emphasize both the local relevance of codes and expected prevalence of diagnoses in this setting. The large primary care database involved was the FIRE database (FIRE stands for “Family Medicine Research using Electronic Medical Records”), which contains anonymized patient data from Swiss GPs’ electronic medical records (26). The local Ethics Committee of the Canton of Zurich waived approval for research with the FIRE database because patient data is fully anonymized and therefore outside the scope of the Swiss Human Research Act (BASEC-Nr. Req2017–00797). The study was conducted in accordance with the Declaration of Helsinki and good clinical practice guidelines.\u003c/p\u003e\n\u003ch2\u003eDiagnostic codes\u003c/h2\u003e\n\u003cp\u003eWe pre-specified that the number of different diagnostic codes should be limited to approximately 100 in order to prevent over-dispersion. To take relevance for local stakeholders into account, 4 stakeholders (JB, LJ, OS, AP) independently compiled a list of diagnostic codes they deemed relevant to their research. To complete these tasks, the experts used the ICD-10 framework as a template and performed up-coding to the highest level of the code that still was meaningful to them. Unused codes from each ICD-10 chapter were grouped together into a code range containing the remaining diseases for the respective chapter. To consider the expected prevalence of diagnoses in general practice, we used four previously published lists of the 100 most frequent ICD-10 diagnoses in general practice from Nordrhein-Westfalen (NRW-lists), each list covering consecutive three-month periods ranging between the second quarter of 2021 and the first quarter of 2022 (27-30). Diagnostic codes were directly selected for the subsequent coding process if at least three out of four stakeholders independently proposed to include them. Additionally, we included codes proposed by two stakeholders if additionally appearing on each NRW-list. Codes that were proposed by only one or two stakeholders and also appeared on each of the four NRW-lists were subjected to a second committee of stakeholders (SM, AP, AW, KW) who rated the importance of each code to their research on a scale from 1 (lowest importance) to 3 (highest importance). Codes achieving at least 5 points were added to the selection diagnostic codes used in the subsequent coding process ultimately consisting of 115 different codes.\u003c/p\u003e\n\u003ch2\u003eData selection, coding process and analysis\u003c/h2\u003e\n\u003cp\u003eFor this study, we used data from 27 GPs nested in 10 different general practices. Specifically, from each practice, we randomly drew 300 patients with at least one consultation in the year 2019. From these patients, we exported the patient ID and diagnosis lists in free-text format from their last consultation in 2019, as imputed by the GPs. This data was transferred into a spreadsheet where each line of free-text (LoFT) was assigned to an individual cell. A pre-testing subset containing 10% of the LoFT was drawn to test the intended coding process and refine the coding framework where necessary. Pre-testing revealed redundancies and very low occurrence of specific codes, which were subsequently unified or removed from the selection and thus, the final coding framework consisted of 105 different codes (see \u003cstrong\u003eAdditional File 1\u003c/strong\u003e).\u003c/p\u003e\n\u003cp\u003eThe coding process involved two trained physicians (AW and DB) who were tasked to independently assign the diagnostic codes to each LoFT. Raters were tasked to code every LoFT, which reflected an unambiguous diagnosis. In the event of ambiguity or information insufficient to code a diagnosis (such as LoFT describing mere symptoms, laboratory test results or low certainty differential diagnostic considerations) the code for “no diagnosis” was assigned, so that every LoFT in the dataset was coded.\u003c/p\u003e\n\u003cp\u003eIn all of the LoFT, we determined for each diagnostic code: 1) frequency by rater, 2) average occurrence rate (as percentage) using the total count of LoFT as denominator and the respective code as numerator, 3) inter-rater agreement (IRA) using the total count of LoFT as denominator and the count of LoFT with concordant coding (absence or presence of the respective code) of the respective code as numerator and 4) inter-rater reliability (IRR) using Cohen’s kappa as measure (31). We used counts and proportions (n and %) for descriptive statistics. We interpreted Κ ≥0.81 as almost perfect agreement K between 0.61 and 0.80 as substantial agreement. For data analysis, we used the software R (Version 4.2.0) (32).\u003c/p\u003e"},{"header":"Results","content":"\u003ch2\u003eSample and frequency analyses\u003c/h2\u003e\n\u003cp\u003eFrom the random sample of 3000 patients, we obtained 26,980 LoFT (of which 2,800 were used for pre-testing). To the 26,980 LoFT, raters 1 and 2 assigned 31,672 and 31,864 codes respectively (the number of codes exceeded the number of LoFT because of cases where multiple codes were assigned to a single LoFT). Taken together, raters most frequently assigned diagnostic codes: \u0026ldquo;no diagnosis\u0026rdquo; (56.3%), \u0026ldquo;dorsopathies\u0026rdquo; (3.9%), \u0026ldquo;other diseases of the circulatory system\u0026rdquo; (3.1%,) and \u0026ldquo;other diseases of the musculoskeletal system and connective tissue\u0026rdquo; (2.8%). A frequency of at least 200 (0.7% of LoFT) by at least one rater was encountered in 30 codes (see Table 1) and a frequency of at least 100 (0.4%) was encountered in 51 codes. Eleven codes were assigned with a frequency below 30 (0.1%) by either rater (see \u003cstrong\u003eAdditional File 2\u003c/strong\u003e for the complete frequency analysis).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eTable 1: The thirty most frequently assigned codes or code ranges\u003c/p\u003e\n\u003ctable border=\"0\" cellspacing=\"0\" cellpadding=\"0\" width=\"623\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eICD-Origin\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003eCode\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003eRater 1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003eRater 2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003eAvg. of LoFT%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003eKappa\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003enone\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003eno diagnosis\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e15300\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e15091\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e56.3%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.856\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eM40-M54\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003edorsopathies\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e1056\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e1066\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e3.9%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.932\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eI00-I99\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003eother diseases of the circulatory system\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e824\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e865\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e3.1%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.848\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eM00-M99\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003eother diseases of the musculoskeletal system and connective tissue\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e769\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e758\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e2.8%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.743\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eI10\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003eprimary hypertension\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e713\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e704\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e2.6%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.972\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eS00-T98\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003einjury, poisoning and certain other consequences of external causes\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e654\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e690\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e2.5%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.853\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eD00-D48\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003eother neoplasms\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e581\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e588\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e2.2%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.852\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eE78\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003edisorders of lipoprotein metabolism and other lipidaemias\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e545\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e539\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e2.0%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.985\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eE00-E90\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003eother endocrine, nutritional and metabolic diseases\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e489\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e501\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e1.8%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.876\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eM60-M79\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003esoft tissue disorders\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e415\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e463\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e1.6%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.734\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eK00-K93\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003eother diseases of the digestive system\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e414\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e449\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e1.6%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.786\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eL00-L99\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003eother diseases of the skin and subcutaneous tissue\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e401\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e458\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e1.6%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.833\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eH00-H59\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003ediseases of the eye and adnexa\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e344\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e350\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e1.3%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.900\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eC00-C99\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003emalignant neoplasms\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e333\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e359\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e1.3%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.839\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eF17\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003emental and behavioural disorders due to use of tobacco\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e312\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e315\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e1.2%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.969\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eI20-I25\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003eischaemic heart diseases\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e297\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e305\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e1.1%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.925\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eK57\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003ediverticular disease of intestine\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e284\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e281\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e1.0%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.973\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eN00-N99\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003eother diseases of the genitourinary system\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e252\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e302\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e1.0%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.780\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eK21\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003egastro-oesophageal reflux disease\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e262\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e260\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e1.0%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.957\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eE65-E68\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003eobesity and other hyperalimentation\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e260\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e260\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e1.0%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.961\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eG00-G99\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003eother diseases of the nervous system\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e234\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e263\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.9%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.778\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eF32-F33\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003edepressive episode and recurrent depressive disorder\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e238\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e249\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.9%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.96\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eE00-E07\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003edisorders of thyroid gland\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e223\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e238\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.9%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.883\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eJ00-J99\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003eother diseases of the respiratory system\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e212\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e247\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.9%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.751\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eA00-B99\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003eintestinal infectious diseases\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e236\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e218\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.8%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.785\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eK40-K46\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003ehernia\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e221\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e221\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.8%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.950\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eH60-H95\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003eother diseases of the ear and mastoid process\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e226\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e207\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.8%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.862\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eE11\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003etype 2 diabetes mellitus\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e217\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e211\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.8%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.906\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eI83\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003evaricose veins of lower extremities\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e195\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e217\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.8%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.882\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"12%\"\u003e\n \u003cp\u003eD50-D90\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"56.64%\"\u003e\n \u003cp\u003eother diseases of the blood and blood-forming organs and certain disorders involving the immune mechanism\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e183\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e207\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.7%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"7.84%\"\u003e\n \u003cp\u003e0.782\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003ch2\u003eAgreement and reliability\u003c/h2\u003e\n\u003cp\u003eWith respect to measures of coding agreement, we found IRA of \u0026gt;0.98 in all assigned codes except \u0026ldquo;no diagnosis\u0026rdquo; (IRA = 0.93). With respect to IRR, we found Kappa values \u0026ge;0.810 in 69 of all the 105 diagnostic codes and 28 codes showed Kappa between 0.610 and \u0026lt;0.810. Simultaneously a frequency of 100 by at least one rater and a Kappa value \u0026ge;0.81 was found in 37 codes (see Table 2). Among these frequently assigned diagnostic codes, we found the highest IRR in \u0026ldquo;disorders of lipoprotein metabolism and other lipidaemias\u0026rdquo; (Kappa = 0.985), \u0026ldquo;diverticular disease of intestine\u0026rdquo; (Kappa = 0.973) and \u0026ldquo;primary hypertension\u0026rdquo; (Kappa = 0.972).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eTable 2: Codes that were both frequently and reliably assigned\u003c/p\u003e\n\u003ctable border=\"0\" cellspacing=\"0\" cellpadding=\"0\" width=\"623\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eICD-10 Code\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003eCode\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eRater 1\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eRater 2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eAvg. of LoFT%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eKappa\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eE78\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003edisorders of lipoprotein metabolism and other lipidaemias\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e545\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e539\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e2.0%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.985\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eK57\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003ediverticular disease of intestine\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e284\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e281\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e1.0%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.973\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eI10\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003eprimary hypertension\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e713\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e704\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e2.6%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.972\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eF17\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003emental and behavioural disorders due to use of tobacco\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e312\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e315\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e1.2%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.969\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eI11-I14\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003ehypertension with end organ damage\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e163\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e159\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.6%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.962\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eI48\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003eatrial fibrillation and flutter\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e147\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e140\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.5%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.961\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eE65-E68\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003eobesity and other hyperalimentation\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e260\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e260\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e1.0%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.961\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eE55\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003evitamin D deficiency\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e112\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e113\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.4%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.960\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eF32-F33\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003edepressive episode and recurrent depressive disorder\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e238\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e249\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.9%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.960\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eK21\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003egastro-oesophageal reflux disease\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e262\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e260\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e1.0%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.957\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eN18\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003echronic kidney disease\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e110\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e108\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.4%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.954\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eM17\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003earthritis of the knee\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e183\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e176\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.7%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.952\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eN40\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003ehyperplasia of prostate\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e106\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e115\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.4%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.950\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eK40-K46\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003eintestinal hernia\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e221\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e221\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.8%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.950\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eG47\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003esleep disorders\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e150\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e147\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.6%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.949\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eJ45\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003easthma\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e174\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e177\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.7%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.945\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eM40-M54\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003edorsopathies\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e1056\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e1066\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e3.9%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.932\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eI20-I25\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003eischaemic heart diseases\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e297\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e305\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e1.1%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.925\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eK64\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003ehaemorrhoids and perianal venous thrombosis\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e122\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e119\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.4%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.921\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eK29\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003egastritis and duodenitis\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e143\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e150\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.5%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.914\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eE11\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003etype 2 diabetes mellitus\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e217\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e211\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.8%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.906\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eH00-H59\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003ediseases of the eye and adnexa\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e344\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e350\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e1.3%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.900\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eN80-N98\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003enoninflammatory disorders of female genital tract\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e133\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e128\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.5%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.896\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eE00-E07\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003edisorders of thyroid gland\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e223\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e238\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.9%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.883\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eI83\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003evaricose veins of lower extremities\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e195\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e217\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.8%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.882\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eE00-E90\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003eother endocrine, nutritional and metabolic diseases\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e489\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e501\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e1.8%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.876\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eI60-I69\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003ecerebrovascular diseases\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e135\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e130\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.5%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.874\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eH60-H95\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003eother diseases of the ear and mastoid process\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e226\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e207\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.8%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.862\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eno diagnosis\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003eno diagnosis\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e15300\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e15091\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e56.3%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.856\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eS00-T98\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003einjury, poisoning and certain other consequences of external causes\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e654\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e690\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e2.5%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.853\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eD00-D48\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003eother neoplasms\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e581\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e588\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e2.2%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.852\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eI00-I99\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003eother diseases of the circulatory system\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e824\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e865\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e3.1%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.848\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eF40-F48\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003eneurotic, stress-related and somatoform disorders\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e162\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e197\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.7%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.839\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eC00-C99\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003emalignant neoplasms\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e333\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e359\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e1.3%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.839\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003eL00-L99\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"49.438202247191015%\"\u003e\n \u003cp\u003eother diseases of the skin and subcutaneous tissue\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e401\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e458\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e1.6%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"10.112359550561798%\"\u003e\n \u003cp\u003e0.833\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e"},{"header":"Discussion","content":"\u003cp\u003eCoded diagnoses from Swiss GP are difficult to obtain but necessary for training NLP models. In this study, we developed a set of 105 diagnostic codes, applied them to a moderately sized dataset of only about 26,000 LoFT and measured frequencies as well as reliability of codes. Over a third of the codes achieved both a frequency above 100 and an almost perfect IRR and are thus suitable for training NLP models using this dataset. The most promising codes in this regard are those that are not easily identified by methods using other data from the electronic medical record (such as laboratory tests or disease-specific medications) and LoFT are the only data source, such as musculoskeletal conditions, cancer or tobacco use.\u003c/p\u003e\n\u003cp\u003eWe developed diagnostic codes with the a priori intention of generating training data for NLP models. To do this, we attempted to limit the granularity of the diagnostic codes to around 100 items in order to avoid over-dispersion, where rarely occurring codes would have insufficient frequency to train NLP models on moderately sized datasets. Within the set of coded LoFT, 51 codes were assigned at least 100 times by both raters and are therefore potential candidates for exploring the feasibility of NLP. Interestingly, however, more than half of the LoFT were coded as 'no diagnosis', suggesting that GPs use this space for additional information that does not amount to a specific diagnosis. This is consistent with findings from other studies that have analysed the content of LoFT, showing that non-specific or insufficient information is common in medical documentation (33-36) but substantially reduced the yield of LoFT for obtaining coded diagnostic data in our study. Specifically, ambiguous acronyms or abbreviations (37-39), unstructured information (39-41), as well as physicians’ and institutional stylistic preferences contribute to non-diagnostic information in free-text diagnoses (42). Raters in our study were notably challenged by non-diagnostic information in LoFT, which manifested itself in an IRA of only 93%, whereas all other codes had IRA ≥98%. We strongly expect that these difficulties will be transferred to the NLP modelling process and methods will be needed to deal not only with false positive identifications but also with ambiguity within the LoFT itself. Third party review and arbitration can be used to further process the training data, but such human arbitration is arguably not a perfect gold standard and may inevitably introduce bias in addition to that introduced when the LoFT was created. This chain of fundamental validity issues highlights important future limitations of NLP-identified diagnoses and foreseeably questions the feasibility of fully automated coding in cases where very high accuracy is required.\u003c/p\u003e\n\u003cp\u003eUnsurprisingly, the most frequently assigned diagnostic codes were those for the most common chronic or recurrent conditions, particularly those of the musculoskeletal and cardiovascular systems (43). Several of these diagnoses were already identifiable in the FIRE database based on algorithms applied to routine data such as prescribed medications (e.g., antidiabetic drugs to identify diabetes) or results of clinical or laboratory tests (e.g., body mass index for obesity) (44). However, there are several important and common diagnoses for which sufficiently specific identification criteria based on routine data are lacking, including musculoskeletal conditions, cancer, tobacco use, depression, sleep disorders and many others, which are important targets of research in general practice. These diagnoses represent the area where we expect NLP to add the most value for research using the FIRE database.\u003c/p\u003e\n\u003cp\u003eWith regard to the plausibility of the code frequency, the rankings of the codes were plausible when taking into account the ranking of the corresponding disease prevalence estimates in the Swiss population. Specifically, dorsopathies, followed by essential hypertension and hyperlipidemia, are the most frequently appearing chronic diseases in this setting according to external studies (45-50). Moreover, frequencies in our study are also very similar to a study measuring reasons for encounters in general practice where diseases of the musculoskeletal and cardio-circulatory systems were by far the most prevalent, thus adding to the plausibility of our results (51-53).\u003c/p\u003e\n\u003cp\u003eWith regard to IRR, we observed almost perfect agreement (Kappa ≥0.810) in two thirds of the codes and substantial agreement in another quarter. Taken together, more than 90% of codes had at least substantial agreement when rated by raters having completed medical school without further training. These findings are comparatively favorable when similar studies with inexperienced raters are considered (21, 54, 55) and equal to studies with experienced raters (56). Depending on the research question and the target diseases to be coded, Kappa values ≥0.500 are generally deemed sufficient (31, 54, 57) and thus, the codes we developed appeared to perform sufficiently. Previous studies have shown that code frequency is associated with IRR (58, 59). This finding was replicated in our study, where all of the 20 most frequent codes reached either an almost perfect or substantial IRR, while the 20 least frequent codes had a Kappa ≤ 0.600.\u003c/p\u003e\n\u003cp\u003eStrengths and limitations:\u003c/p\u003e\n\u003cp\u003eThis research project describes the design and reliability testing of a custom coding framework to be used for training NLP models. The project can serve as a template for similar research, which will become increasingly important given the growing role of AI in medicine and the associated need for local training data tailored to local factors such as languages and use cases. The use of LoFT from general practice-based medical diagnosis lists is a very prominent use case in this regard, and our study provides estimates of code frequencies based on a moderately sized dataset, which can be achieved with a small investment in manual coding labour. The methods used are highly feasible and provide transparent metrics that help in further interpretation of NLP modelling results, especially when considering the IRR of coding by human raters labelling the training data.\u003c/p\u003e\n\u003cp\u003eThe moderate size and locality of the dataset may be a major limitation. We tried to include LoFT data from a representative sample of Swiss GPs, but this sample still only included 27 of them, and these were nested in 10 different medical practices. The local jargon of these GPs may limit the applicability of NLP models based on these training data, and NLP models need to be tested within, but more importantly outside, this dataset.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eWe developed and tested a framework of research-relevant diagnostic codes in a primary care research database to train NLP models based on free text data. We have identified a subset of very frequent and highly reliable diagnostic codes, and the next step in the research agenda is to train NLP models with the obtained data and evaluate their performance in automated disease classification.\u0026nbsp;\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cp\u003eFIRE\u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp;Family Medicine ICPC Research using Electronic Medical Records\u003c/p\u003e\n\u003cp\u003eLoFT\u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp;Lines of free-text\u003c/p\u003e\n\u003cp\u003eGP\u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp;General Practitioner\u003c/p\u003e\n\u003cp\u003eICD-10\u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;International Statistical Classification of Diseases and Related Health Problems, Tenth Revision\u003c/p\u003e\n\u003cp\u003eIRA\u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp;Interrater agreement\u003c/p\u003e\n\u003cp\u003eIRR \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; \u0026nbsp; Interrater reliability\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003e\u003cstrong\u003eEthics approval and consent to participate:\u003c/strong\u003e The local Ethics Committee of the Canton of Zurich waived approval for the present study because the FIRE project is outside the scope of the law on human research and\u0026nbsp;studies utilizing data from the FIRE project are thus exempt from ethics review (BASEC-Nr. Req2017\u0026ndash;00797).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsent for publication:\u003c/strong\u003e Consent for publication was waived by the ethics committee due to the anonymization of the data at the practice level\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConsent to participate declaration\u003c/strong\u003e: not applicable due to the anonymization of the data at the practice level\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAvailability of data and materials:\u003c/strong\u003e The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCompeting interests:\u003c/strong\u003e The authors declare that no conflicts of interest are relevant to any aspects of this work.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eFunding:\u003c/strong\u003e This study received no external funding.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthors\u0026apos; contributions:\u003c/strong\u003e SM and JB conceived and designed the study; SM and JB performed data acquisition; AW performed data analysis and drafted the original draft; SM, KW, TR and OS revised the original draft of the manuscript and the version to be published. All the authors revised and approved the final manuscript for publication.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAcknowledgements:\u003c/strong\u003e We thank Levy J\u0026auml;ger (LJ) and Andreas Plate (AP) for their contribution in the code selection, as well as Donika Balaj (DB), Adriana Keller (AK) and Gino Bopp (GB) for their contribution to the coding procedure (DB) pre-testing the coding framework (AK and GB).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAuthors\u0026apos; information (optional):\u003c/strong\u003e Not applicable.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eStatistik Bf. Konsultationen bei Generalistinnen und Generalisten nach Geschlecht, Alter, Bildungsniveau, Sprachgebiet. In: Statistik Bf, editor. 30.10.2018.\u003c/li\u003e\n\u003cli\u003eGreen LA, Fryer GE, Jr., Yawn BP, Lanier D, Dovey SM. The ecology of medical care revisited. N Engl J Med. 2001;344(26):2021-5.\u003c/li\u003e\n\u003cli\u003eSenn N, Tiar\u0026eacute; Ebert S, Cohidon C. Die Hausarztmedizin in der Schweiz \u0026ndash; Perspektiven. Analyse basierend auf den Indikatoren des Programm SPAM (Swiss Primary Care Active Monitoring). Obsan Bulletin 2016;11/2016:4.\u003c/li\u003e\n\u003cli\u003eMeci A, Du Breuil F, Vilcu A, Pitel T, Guerrisi C, Robard Q, et al. The Sentiworld project: global mapping of sentinel surveillance networks in general practice. BMC Prim Care. 2022;23(1):173.\u003c/li\u003e\n\u003cli\u003eClothier HJ, Fielding JE, Kelly HA. An evaluation of the Australian Sentinel Practice Research Network (ASPREN) surveillance for influenza-like illness. Commun Dis Intell Q Rep. 2005;29(3):231-47.\u003c/li\u003e\n\u003cli\u003eLiljeqvist GT, Staff M, Puech M, Blom H, Torvaldsen S. Automated data extraction from general practice records in an Australian setting: trends in influenza-like illness in sentinel general practices and emergency departments. BMC public health. 2011;11:435.\u003c/li\u003e\n\u003cli\u003ede Lusignan S, Hague N, van Vlymen J, Kumarapeli P. Routinely-collected general practice data are complex, but with systematic processing can be used for quality improvement and research. Inform Prim Care. 2006;14(1):59-66.\u003c/li\u003e\n\u003cli\u003ede Lusignan S, van Weel C. The use of routinely collected computer data for research in primary care: opportunities and challenges. Fam Pract. 2006;23(2):253-63.\u003c/li\u003e\n\u003cli\u003eNicholls SG, Langan SM, Benchimol EI. Routinely collected data: the importance of high-quality diagnostic coding to research. Cmaj. 2017;189(33):E1054-e5.\u003c/li\u003e\n\u003cli\u003eK\u0026uuml;hlein T, Virtanen M, Claus C, Popert U, van Boven K. [Coding in general practice-Will the ICD-11 be a step forward?]. Bundesgesundheitsblatt Gesundheitsforschung Gesundheitsschutz. 2018;61(7):828-35.\u003c/li\u003e\n\u003cli\u003eLasker RD, Marquis MS. The intensity of physicians\u0026apos; work in patient visits--implications for the coding of patient evaluation and management services. N Engl J Med. 1999;341(5):337-41.\u003c/li\u003e\n\u003cli\u003eLetrilliart L, Gelas-Dore B, Ortolan B, Colin C. Prometheus: the implementation of clinical coding schemes in French routine general practice. Inform Prim Care. 2006;14(3):157-65.\u003c/li\u003e\n\u003cli\u003eBiro SC, Barber DT, Kotecha JA. Trends in the use of electronic medical records. Canadian family physician Medecin de famille canadien. 2012;58(1):e21.\u003c/li\u003e\n\u003cli\u003eChang F, Gupta N. Progress in electronic medical record adoption in Canada. Canadian family physician Medecin de famille canadien. 2015;61(12):1076-84.\u003c/li\u003e\n\u003cli\u003eStatistik Bf. F\u0026uuml;hrung der Krankengeschichten an den Standorten der Arztpraxen und ambulanten Zentren. 2022.\u003c/li\u003e\n\u003cli\u003eDjalali S. Wer eHealth sucht, findet einen Haufen Papier. Schweizerische \u0026Auml;rztezeitung (S\u0026Auml;Z). 2015;96(43):1575-8.\u003c/li\u003e\n\u003cli\u003eFrese T, Herrmann K, Bungert-Kahl P, Sandholzer H. Inter-rater reliability of the ICPC-2 in a German general practice setting. Swiss Med Wkly. 2012;142:w13621.\u003c/li\u003e\n\u003cli\u003eLamberts H, Wood M, Hofmans-Okkes IM. International primary care classifications: the effect of fifteen years of evolution. Fam Pract. 1992;9(3):330-9.\u003c/li\u003e\n\u003cli\u003eVerbeke M, Schrans D, Deroose S, De Maeseneer J. The International Classification of Primary Care (ICPC-2): an essential tool in the EPR of the GP. Stud Health Technol Inform. 2006;124:809-14.\u003c/li\u003e\n\u003cli\u003eWHO. International Statistical Classification of Diseases and Related Health Problems 10th Revision 2019 [Available from: https://icd.who.int/browse10/2019/en.\u003c/li\u003e\n\u003cli\u003eStausberg J, Lehmann N, Kaczmarek D, Stein M. Reliability of diagnoses coding with ICD-10. Int J Med Inform. 2008;77(1):50-7.\u003c/li\u003e\n\u003cli\u003eJiang F, Jiang Y, Zhi H, Dong Y, Li H, Ma S, et al. Artificial intelligence in healthcare: past, present and future. Stroke and vascular neurology. 2017;2(4).\u003c/li\u003e\n\u003cli\u003eYu K-H, Beam AL, Kohane IS. Artificial intelligence in healthcare. Nature biomedical engineering. 2018;2(10):719-31.\u003c/li\u003e\n\u003cli\u003eChen M, Hao Y, Hwang K, Wang L, Wang L. Disease prediction by machine learning over big data from healthcare communities. Ieee Access. 2017;5:8869-79.\u003c/li\u003e\n\u003cli\u003eAlthnian A, AlSaeed D, Al-Baity H, Samha A, Dris AB, Alzakari N, et al. Impact of Dataset Size on Classification Performance: An Empirical Evaluation in the Medical Domain. Applied Sciences. 2021;11(2):796.\u003c/li\u003e\n\u003cli\u003eChmiel C, Bhend H, Senn O, Zoller M, Rosemann T. The FIRE project: a milestone for research in primary care in Switzerland. Swiss Med Wkly. 2011;140:w13142.\u003c/li\u003e\n\u003cli\u003eNordrhein KV. Die 100 h\u0026auml;ufigsten ICD-10-Schl\u0026uuml;ssel und Kurztexte (nach Fachgruppen). In: Nordrhein KV, editor. 2 Quartal 20212021.\u003c/li\u003e\n\u003cli\u003eNordrhein KV. Die 100 h\u0026auml;ufigsten ICD-10-Schl\u0026uuml;ssel und Kurztexte (nach Fachgruppen) In: Nordrhein KV, editor. 3 Quartal 20212021.\u003c/li\u003e\n\u003cli\u003eNordrhein KV. Die 100 h\u0026auml;ufigsten ICD-10-Schl\u0026uuml;ssel und Kurztexte (nach Fachgruppen). In: Nordrhein KV, editor. 4 Quartal 20212021.\u003c/li\u003e\n\u003cli\u003eNordrhein KV. Die 100 h\u0026auml;ufigsten ICD-10-Schl\u0026uuml;ssel und Kurztexte (nach Fachgruppen). In: Nordrhein KV, editor. 1 Quartal 20222022.\u003c/li\u003e\n\u003cli\u003eLandis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-74.\u003c/li\u003e\n\u003cli\u003eTeam RC. The R Project for Statistical Computing 2022 [4.2.0:[Available from: https://www.R-project.org/.\u003c/li\u003e\n\u003cli\u003eTsai CH, Eghdam A, Davoody N, Wright G, Flowerday S, Koch S. Effects of Electronic Health Record Implementation and Barriers to Adoption and Use: A Scoping Review and Qualitative Analysis of the Content. Life (Basel). 2020;10(12).\u003c/li\u003e\n\u003cli\u003eNguyen L, Bellucci E, Nguyen LT. Electronic health records implementation: an evaluation of information system impact and contingency factors. Int J Med Inform. 2014;83(11):779-96.\u003c/li\u003e\n\u003cli\u003eLium JT, Tjora A, Faxvaag A. No paper, but the same routines: a qualitative exploration of experiences in two Norwegian hospitals deprived of the paper based medical record. BMC medical informatics and decision making. 2008;8:2.\u003c/li\u003e\n\u003cli\u003eWhittaker AA, Aufdenkamp M, Tinley S. Barriers and facilitators to electronic documentation in a rural hospital. J Nurs Scholarsh. 2009;41(3):293-300.\u003c/li\u003e\n\u003cli\u003eHanauer DA, Mei Q, Law J, Khanna R, Zheng K. Supporting information retrieval from electronic health records: A report of University of Michigan\u0026apos;s nine-year experience in developing and using the Electronic Medical Record Search Engine (EMERSE). J Biomed Inform. 2015;55:290-300.\u003c/li\u003e\n\u003cli\u003eBarrows Jr RC, Busuioc M, Friedman C. Limited parsing of notational text visit notes: ad-hoc vs. NLP approaches. Proc AMIA Symp. 2000:51-5.\u003c/li\u003e\n\u003cli\u003eTang KL, Lucyk K, Quan H. Coder perspectives on physician-related barriers to producing high-quality administrative data: a qualitative study. CMAJ open. 2017;5(3):E617-e22.\u003c/li\u003e\n\u003cli\u003eBrown T, Zelch B, Lee JY, Doctor JN, Linder JA, Sullivan MD, et al. A Qualitative Description of Clinician Free-Text Rationales Entered within Accountable Justification Interventions. Appl Clin Inform. 2022;13(4):820-7.\u003c/li\u003e\n\u003cli\u003eRubio-L\u0026oacute;pez I, Costumero R, Ambit H, Gonzalo-Mart\u0026iacute;n C, Menasalvas E, Rodr\u0026iacute;guez Gonz\u0026aacute;lez A. Acronym Disambiguation in Spanish Electronic Health Narratives Using Machine Learning Techniques. Stud Health Technol Inform. 2017;235:251-5.\u003c/li\u003e\n\u003cli\u003eFeder SL. Data Quality in Electronic Health Records Research: Quality Domains and Assessment Methods. West J Nurs Res. 2018;40(5):753-66.\u003c/li\u003e\n\u003cli\u003eExcoffier S, Herzig L, N\u0026apos;Goran AA, D\u0026eacute;ruaz-Luyet A, Haller DM. Prevalence of multimorbidity in general practice: a cross-sectional study within the Swiss Sentinel Surveillance System (Sentinella). BMJ open. 2018;8(3):e019616.\u003c/li\u003e\n\u003cli\u003eMeier R, Grischott T, Rachamin Y, J\u0026auml;ger L, Senn O, Rosemann T, et al. Importance of different electronic medical record components for chronic disease identification in a Swiss primary care database: a cross-sectional study. Swiss Med Wkly. 2023;153:40107.\u003c/li\u003e\n\u003cli\u003e(SGB) BSG. H\u0026auml;ufigkeit von R\u0026uuml;cken- oder Kopfschmerzen. In: 2023 O, editor. 2023.\u003c/li\u003e\n\u003cli\u003eDanon-Hersch N, Marques-Vidal P, Bovet P, Chiolero A, Paccaud F, P\u0026eacute;coud A, et al. Prevalence, awareness, treatment and control of high blood pressure in a Swiss city general population: the CoLaus study. Eur J Cardiovasc Prev Rehabil. 2009;16(1):66-72.\u003c/li\u003e\n\u003cli\u003eWalther D, Curjuric I, Dratva J, Schaffner E, Quinto C, Rochat T, et al. High blood pressure: prevalence and adherence to guidelines in a population-based cohort. Swiss Med Wkly. 2016;146:w14323.\u003c/li\u003e\n\u003cli\u003eStatistik Bf. Personen mit Bluthochdruck nach Geschlecht, Alter, Bildungsniveau, Sprachgebiet. In: Statistik Bf, editor.: BFS; 2017.\u003c/li\u003e\n\u003cli\u003eMarco Storni RL, \u0026shy;Kaeser M. Schweizerische Gesundheitsbefragung 2017. In: (BFS) BfS, editor.: Bundesamt f\u0026uuml;r Statistik; 2018.\u003c/li\u003e\n\u003cli\u003eEstoppey D, Paccaud F, Vollenweider P, Marques-Vidal P. Trends in self-reported prevalence and management of hypertension, hypercholesterolemia and diabetes in Swiss adults, 1997-2007. BMC public health. 2011;11:114.\u003c/li\u003e\n\u003cli\u003eTandjung R, Hanhart A, B\u0026auml;rtschi F, Keller R, Steinhauer A, Rosemann T, Senn O. Referral rates in Swiss primary care with a special emphasis on reasons for encounter. Swiss Med Wkly. 2015;145:w14244.\u003c/li\u003e\n\u003cli\u003eLurquin B, Kellou N, Colin C, Letrilliart L. Comparison of rural and urban French GPs\u0026apos; activity: a cross-sectional study. Rural Remote Health. 2021;21(3):5865.\u003c/li\u003e\n\u003cli\u003eSch\u0026auml;fer I, Hansen H, Ruppel T, L\u0026uuml;hmann D, Wagner HO, Kazek A, Scherer M. Regional differences in reasons for consultation and general practitioners\u0026apos; spectrum of services in northern Germany - results of a cross-sectional observational study. BMC Fam Pract. 2020;21(1):22.\u003c/li\u003e\n\u003cli\u003eWockenfuss R, Frese T, Herrmann K, Claussnitzer M, Sandholzer H. Three- and four-digit ICD-10 is not a reliable classification system in primary care. Scand J Prim Health Care. 2009;27(3):131-6.\u003c/li\u003e\n\u003cli\u003eAsadi F, Hosseini MA, Almasi S. Reliability of trauma coding with ICD-10. Chin J Traumatol. 2022;25(2):102-6.\u003c/li\u003e\n\u003cli\u003ePeng M, Eastwood C, Boxill A, Jolley RJ, Rutherford L, Carlson K, et al. Coding reliability and agreement of International Classification of Disease, 10(th) revision (ICD-10) codes in emergency department data. International journal of population data science. 2018;3(1):445.\u003c/li\u003e\n\u003cli\u003eCheniaux E, Landeira-Fernandez J, Versiani M. The diagnoses of schizophrenia, schizoaffective disorder, bipolar disorder and unipolar depression: interrater reliability and congruence between DSM-IV and ICD-10. Psychopathology. 2009;42(5):293-8.\u003c/li\u003e\n\u003cli\u003eKoopman B, Karimi S, Nguyen A, McGuire R, Muscatello D, Kemp M, et al. Automatic classification of diseases from free-text death certificates for real-time surveillance. BMC medical informatics and decision making. 2015;15:53.\u003c/li\u003e\n\u003cli\u003eMandrekar JN. Measures of interrater agreement. J Thorac Oncol. 2011;6(1):6-7.\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"bmc-primary-care","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"famp","sideBox":"Learn more about [BMC Primary Care](https://bmcprimcare.biomedcentral.com/)","snPcode":"","submissionUrl":"https://author-welcome.nature.com/12875","title":"BMC Primary Care","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"General Practitioners, Electronic Medical Records, Diagnostic Coding, Reliability, Training data","lastPublishedDoi":"10.21203/rs.3.rs-4131283/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4131283/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eBackground\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eDiagnoses entered by general practitioners into electronic medical records have great potential for research and practice, but unfortunately, diagnoses are often in uncoded format, making them of little use. Natural language processing (NLP) could assist in coding free-text diagnoses, but NLP models require local training data to unlock their potential. The aim of this study was to develop a framework of research-relevant diagnostic codes, to test the framework using free-text diagnoses from a Swiss primary care database and to generate training data for NLP modelling.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMethods\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe framework of diagnostic codes was developed based on input from local stakeholders and consideration of epidemiological data. After pre-testing, the framework contained 105 diagnostic codes, which were then applied by two raters who independently coded randomly drawn lines of free text (LoFT) from diagnosis lists extracted from the electronic medical records of 3000 patients of 27 general practitioners. Coding frequency and mean occurrence rates (n and %) and inter-rater reliability (IRR) of coding were calculated using Cohen's kappa (Κ).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe sample consisted of 26,980 LoFTs and in 56.3% no code could be assigned because it was not a specific diagnosis. The most common diagnostic codes were, 'dorsopathies' (3.9%) and 'other diseases of the circulatory system' (3.1%). Raters were in almost perfect agreement (Κ ≥0.81) for 69 of the 105 diagnostic codes, and 28 codes showed a substantial agreement (K between 0.61 and 0.80). Both high coding frequency and almost perfect agreement was found in 37 codes, including codes that are particularly difficult to identify from components of the electronic medical record, such as musculoskeletal conditions, cancer or tobacco use.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusion\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe coding framework was characterised by a subset of very frequent and highly reliable diagnostic codes, which will be the most valuable targets for training NLP models for automated disease classification based on free-text diagnoses from Swiss general practice.\u003c/p\u003e","manuscriptTitle":"Developing and Testing a Framework for Coding General Practitioners' Free-Text Diagnoses in Electronic Medical Records - A Reliability Study for Generating Training Data in Natural Language Processing","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-04-12 13:02:58","doi":"10.21203/rs.3.rs-4131283/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-05-21T09:29:07+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-05-21T02:58:40+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-05-01T05:41:32+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"12ce2d0c-0a1b-48b7-92c4-a92d370ad1ea","date":"2024-04-19T01:54:18+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"b175170e-31f9-4dfb-95f3-780032bb4c91","date":"2024-04-19T01:27:33+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-04-19T00:07:48+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2024-04-12T11:31:10+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-04-10T00:20:08+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-04-10T00:20:08+00:00","index":"","fulltext":""},{"type":"submitted","content":"BMC Primary Care","date":"2024-03-19T14:49:01+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"bmc-primary-care","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"famp","sideBox":"Learn more about [BMC Primary Care](https://bmcprimcare.biomedcentral.com/)","snPcode":"","submissionUrl":"https://author-welcome.nature.com/12875","title":"BMC Primary Care","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"922ed0e7-680c-4aea-96cb-fa3d93d2cc23","owner":[],"postedDate":"April 12th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2024-08-01T17:14:49+00:00","versionOfRecord":{"articleIdentity":"rs-4131283","link":"https://doi.org/10.1186/s12875-024-02514-1","journal":{"identity":"bmc-primary-care","isVorOnly":false,"title":"BMC Primary Care"},"publishedOn":"2024-07-16 16:13:38","publishedOnDateReadable":"July 16th, 2024"},"versionCreatedAt":"2024-04-12 13:02:58","video":"","vorDoi":"10.1186/s12875-024-02514-1","vorDoiUrl":"https://doi.org/10.1186/s12875-024-02514-1","workflowStages":[]},"version":"v1","identity":"rs-4131283","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4131283","identity":"rs-4131283","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}