ICF coding automated: a validation study for self-supervised architecture in electronic health records

ICF coding automated: a validation study for self-supervised architecture in electronic health records | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article ICF coding automated: a validation study for self-supervised architecture in electronic health records Nieminen Linda, Ketamo Harri, Kankaanpää Markku This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6221473/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background The International Classification of Functioning, Disability and Health (ICF) provides a comprehensive framework for assessing health beyond disease-centric models, yet its integration into clinical practice remains limited. Key challenges include the complexity of ICF coding, the lack of standardized implementation in electronic health records, and the prevalence of unstructured health data. This study aims to validate an algorithm for automatically converting unstructured health record texts into ICF codes. Using self-supervised learning methods, this research aims to improve ICF utilization while ensuring compliance with data regulations. Results The analysis dataset included 151 electronic healthcare documents from different healthcare professionals, including physicians, nurses, therapists, social workers and rehabilitation counsellors. The algorithm performed equally well on texts from different professionals. The overall performance on the analysis dataset was 0.94 for precision and 0.88 for recall, resulting in an F1 score of 0.91. Conclusions The results of this validity study demonstrate the algorithm's strong performance in automatically generating ICF codes from free-text clinical notes. Integration of the algorithm into electronic health records systems has the potential to have a significant impact on the effectiveness and efficiency of health care systems. By automating the extraction and assignment of ICF codes, the algorithm can facilitate several important improvements within the health care system. Future research should investigate the scalability, interoperability, and cross-cultural validation of the algorithm. International classification of functioning disability and health electronic health records health information systems artificial intelligence unsupervised machine learning Figures Figure 1 Figure 2 Introduction The International Classification of Functioning, Disability and Health (ICF)[ 1 ], while offering a comprehensive framework for describing health beyond disease-focused models, has faced challenges in its practical application within the health system[ 2 , 3 ]. The slow implementation challenges the widespread adoption of the ICF and limits its potential to improve patient care, research and knowledge-based governance[ 3 , 4 ]. Several interrelated factors contribute to this underutilization. First, the complexity of the ICF presents a significant barrier to its adoption. Despite the development of instruments since the introduction of the ICF in 2001, including the ICF Checklist[ 5 ], the ICF Core Sets[ 6 ], and the WHO Disability Assessment Schedule 2.0 (WHODAS 2.0)[ 7 ], these resources have had limited impact on daily clinical routines. The complexity and time-consuming nature of these methods often make them unsuitable for fast-paced clinical environments. This, in turn, prevents the full realization of the ICF's potential benefits, such as understanding service needs through a holistic understanding of patients’ functioning and disability. Second, current electronic health record (EHR) systems often lack standardized structures for storing ICF codes, creating difficulties in integrating and retrieving this information[ 8 – 10 ]. Healthcare professionals do not routinely use the ICF to code patient information, partly owing to the absence of clear mandates for its implementation. A major obstacle lies in the nature of EHR data itself. The majority (approximately 80%) of health data in EHRs are unstructured, making automated extraction and conversion to ICF codes difficult[ 11 ]. Manual coding is impractical because of the volume of data and the complexity of the ICF itself. With over 1,600 codes, each with detailed and nuanced definitions [ 12 ], the ICF requires substantial effort from professionals to understand and apply it correctly. Furthermore, the language used in ICF definitions often differs from the natural language used in clinical practice, further complicating the coding process and leading to potential inconsistencies and errors. The large amount of unstructured data in EHRs makes manual ICF coding impractical, which makes automated ICF coding the only option. Traditional text analysis or natural language processing (NLP) methods are not optimal for handling this type of data because of i) the narrow understanding of real-world professional language use, ii) the demand for training data for deep learning models is colossal for 1600 codes, iii) the trustworthiness of the models is not good enough from privacy point of view, or iv) for some of the known models, the computational resources are out of reach of real-world use. Because of these known limitations, the approach suggested in this study is based on combining unsupervised /self-supervised learning methods to bypass the limitations of supervised learning, deep learning and the use of large language models. Automated ICF coding has the potential to revolutionize the use of EHR data, enabling population-level monitoring of health-related functioning and disability, and personalized medicine with effective service choices fit for individuals’ purposes, to name a few. This current validity study builds upon previous research. Prior work involved two key phases: (1) algorithm optimization, which focused on identifying the most effective definition sets[ 13 ], and (2) a feasibility study, which evaluated the algorithm's performance on EHR text from patients with chronic low back pain[ 14 ]. The feasibility study demonstrated the algorithm's sufficient specificity and sensitivity for reliable ICF code extraction. Earlier research also validated a three-layered mechanism (grammar, morphology, and semantic similarity) combined with the definition set and focused on expanding the algorithm's capabilities while enhancing data security. This study aims to further validate the algorithm's performance in a real-world clinical context. The primary aim of this study was to validate an algorithm for automatically converting unstructured EHR text into ICF codes in a complex group of patients in need of broad support from the health service network. An essential part of this study was to ensure compliance with relevant data regulations. The research questions were as follows: 1) What is the performance (precision, recall, and F1 score) of the text-to-ICF algorithm in this population dataset? 2) How does the algorithm's performance vary across different health professional groups? 3) How does the algorithm's performance compare to the results of the previous feasibility study? Materials and methods 2.1 Population data This study utilized longitudinal data from the Wellbeing Services County of Pirkanmaa patient register, Finland, related to patients with spinal cord injury (SCI). The study was approved by the Wellbeing Services County of Pirkanmaa. As this was a registry-based study maintaining patient anonymity, the regional medical research ethics committee of the Wellbeing Services County of Pirkanmaa waived the need for ethical approval and informed consent. Electronic health record (EHR) texts were retrospectively collected from the Uranus (CGI) EHR system. The target population consisted of adult (18 years and older) SCI patients with EHR documents related to both a neurological rehabilitation ward stay and a subsequent SCI outpatient clinic visit approximately one year after discharge. Rehabilitation episodes were required to be a direct consequence of acute SCI-related injury, illness, or disease. The data collected included texts drawn by different healthcare professionals during the inpatient rehabilitation period and a collectively written text from the multidisciplinary team at the outpatient visit. The outpatient clinic visit texts were limited to those finalized by the end of December 2022, and the inpatient rehabilitation texts were limited to those starting in January 2020. The initial search yielded 64 patients and associated EHR documents. Exclusion criteria were applied to remove duplicates, rehabilitation episodes not related to acute SCI, outpatient visits that were not the first postdischarge visit, consultations for different specialties, and texts outside the specified timeframes. Following these criteria, the final dataset comprised 49 patients. A computer-assisted randomization process was used to select 10 patients' documents to form the dataset for this study. These documents included contributions from professionals directly involved in multidisciplinary rehabilitation and outpatient care. Two health professionals with a thorough understanding of the ICF annotated functioning data from the free text, adhering to the ICF framework (see Fig. 1 ) and linking rules[ 15 ]. Text related to body structures (S codes) was excluded. The annotated data of 10 patients were randomly divided into two equal sets (5/5). One set served as the analysis dataset for evaluating the algorithm, and the other constituted the definition set used to provide examples for the algorithm. This definition dataset was supplemented with data from previous studies, resulting in a final definition dataset of approximately 12,000 code‒phrase pairs. 2.2 Algorithm Description The HeadAI's text-to-ICF algorithm uses an architecture designed for efficient and accurate translation of free-text clinical notes into ICF codes. This approach addresses the challenges posed by limited training data and computational resources while still achieving robust performance. The overall goal of the algorithm is to find semantic similarities between the EHR texts/notes and the given ICF definition set. The operation consists of three layers: grammar, morphology and semantics (language model). Earlier versions of the algorithm have been published in previous publications [ 13 , 16 – 18 ] The first layer, the grammar layer , performs grammatical analysis, converting each word into its base form. This normalization process creates a standardized representation of the text, simplifying subsequent analysis and ensuring consistent interpretation regardless of surface-level variations in word form. The second layer, the morphology layer , builds upon grammatical analysis by considering morphology. This allows the algorithm to understand different word forms, sentence structures, and word order variations. By incorporating morphological understanding, the algorithm can accurately interpret the meaning of words and phrases within their grammatical context, even when phrasing or syntax varies. The third layer, the small language model layer , employs a specialized language model tailored for ICF coding. This model focuses on recognizing and interpreting clinical jargon, synonyms, and related concepts. Unlike large, general-purpose language models, this model is specifically designed to understand the semantic nuances relevant to ICF codes within clinical text. This targeted approach enhances morphological analysis, enabling the identification of relevant concepts even when expressed via diverse terminology. This three-layered architecture facilitates efficient ICF code identification. The algorithm uses the processed text to identify meanings and definitions corresponding to specific ICF codes, assigning the appropriate code upon a match. This structure is computationally efficient, enabling the translation of clinical notes into ICF codes with relatively low overhead. Critically, it does not require the extensive training data typically associated with large language models. The decision to employ a self-supervised learning approach was driven by several key factors. First, access to a large, diverse corpus of labeled patient records was limited, precluding the use of supervised learning methods. Second, the computational resources necessary for training large language models were not available. Finally, the ICF coding task demands energy-efficient computation. 2.3 Data Security This study prioritized data security and ethical considerations throughout all stages of the research process. The study was conducted in accordance with the Declaration of Helsinki. Data processing was conducted within a secure, controlled research environment maintained by the registry holder. Only authorized researchers were allowed to upload the data onto the computing unit, start the computing processes and finally see the results in the secured environments. The results were not removed from the secure environment, and all the data in the computing unit (raw data and results) were deleted after the research was finalized. This physical isolation and controlled access minimized the risk of unauthorized access or data breaches. This study complied with European data protection regulations, including the General Data Protection Regulation (GDPR), the AI Act, and the Data Act. Recognizing the computational process as a potentially high-risk activity under the AI Act, all operations were subject to human oversight. 2.4 Statistical methods This study employed precision, recall (sensitivity), and the F1 score as primary evaluation metrics. While the previous feasibility study used sensitivity and specificity, these metrics are less informative when true negatives are difficult to ascertain, as is often the case with free-text clinical notes. It also challenges the use of AUC- ROC-curves (the area under the receiver operating characteristics curve). The precision, recall, and F1 score provide a more balanced assessment of the algorithm's performance in such scenarios. The algorithm's output was compared against manual annotation performed by two health professionals with ICF expertise. Health professionals (HP) were considered the gold standard. The findings were classified as true positive (if the algorithm and HP identified the same ICF code), false positive (if the algorithm identified a code not found by the HP and confirmed as incorrect after the validation process), false negative (if HP identified a code missed by the algorithm), or reappraisal true positive (if the algorithm identified a code initially missed by the HP but deemed correct after the validation process). This validation process allowed for the correction of potential oversights by the human annotators, enhancing the accuracy of the performance evaluation. The results The data included documents from various healthcare professionals involved in SCI patient care: physicians (neurologists and physiatrists), nurses, therapists (physiotherapists, occupational therapists, speech therapists, nutritional therapists), social workers, and rehabilitation counselors. Additionally, a multiprofessional team drew collaborative texts related to outpatient clinic visits. Among the ten patient sample, 447 EHR texts were collected. After two health professionals annotated the free text to third- and fourth-level ICF codes deductively, 9349 findings of ICF-related data were obtained. Interrater reliability was measured on a dataset formed by one of the patients’ texts. For the 814 annotated words or phrases, the percentage of agreement was 97% (95% CI 96%-98%). From the annotated dataset, the annotated data of 5 patients were used to provide examples for the algorithm (extension to the definition set), and another 5 were used for the analysis. The analysis dataset was compressed on an individual document level so that recurring codes were condensed into a single finding. In the analysis, the findings of the algorithm (text-to-ICF) were analyzed against the findings of the health professional annotators (HPs). The results of the analysis are presented in Table 1 . Table 1 The results of the performance of the text-to-ICF algorithm. PT Physician SW OT Nurse RC Nutr. MPT ALL Documents 13 74 37 5 5 8 4 5 151 True codes total 288 302 156 124 137 70 16 242 1335 HP first round findings 201 237 120 95 113 59 12 199 1036 Algorithm extra findings 87 65 36 29 24 11 4 43 299 True positives (GM) 253 257 139 104 121 57 12 207 1150 False negatives (GM) 35 45 17 20 16 13 4 35 185 False positives (GM) 18 21 9 12 6 5 0 17 88 Precision 0,94 0,93 0,96 0,91 0,96 0,93 1 0,93 0,94 Recall 0,89 0,87 0,9 0,86 0,9 0,84 0,8 0,87 0,88 F1 score 0,91 0,9 0,93 0,88 0,93 0,88 0,89 0,9 0,91 HP = health professional, PT = physiotherapist, SW = social worker, OT = occupational therapist, RC = rehabilitation counsellor, Nutr.= nutritionist, MPT = multiprofessional team. True codes total = how many true positive codes were verified by the professional (the professional being the gold standard); HP first round findings = the codes found by the health professionals at first annotation round; GM extra findings = the algorithm identified a code initially missed by the health professionals but deemed correct after reappraisal. The previous feasibility study used different evaluation metrics (sensitivity and specificity). To compare the results reliably, we used the main results of the feasibility study to calculate the precision (0,95), recall (0,83), and F1 score (0,89). There were no significant differences in how well text-to-ICF performed between professional groups in terms of true positives, false negatives, false positives, precision, recall and F1 score. Discussion The results of this validity study demonstrate the algorithm's strong performance in automatically generating ICF codes from free-text clinical notes in a new dataset. The primary aim of this study was to evaluate the algorithm's validity by applying it to a different patient population (persons with SCI) and a more diverse set of authors (various healthcare professionals) than the previous feasibility study, which focused on chronic low back pain patients and physician-authored texts. Persons with SCI were chosen because of the complex nature of their condition, characterized by major changes in their functioning[ 19 ]. These patients typically require extensive multidisciplinary care, making them an ideal population for evaluating the algorithm's performance on complex, multiprofessional data. Furthermore, the significant societal cost associated with SCI[ 20 ] underscores the potential benefits of automated ICF coding in this context. The algorithm successfully generated ICF codes with good performance, requiring some additions to the definition dataset to be adapted to a new context. This suggests the potential for scalability across diverse healthcare settings. There were no significant differences in performance between different health professional groups, which may indicate that the working culture of EHR text writing is similar across all professional groups. This result underlines the ability of the algorithm to be used in different health and social care environments and professional groups. A comparison with a previous feasibility study[ 14 ] revealed an improvement in the F1 score in the current study. While the precision was slightly lower, the recall was notably higher. This suggests that the algorithm's ability to identify relevant ICF concepts within the text has improved. The lower precision may indicate a broader capture of relevant codes, potentially at the cost of including some false positives. However, the overall improvement in the F1 score, which harmonizes precision and recall, indicates a net positive change in the algorithm's performance. A key factor contributing to this improved performance is likely the substantial increase in the volume of the definition dataset. The current study utilized approximately 12,000 examples, whereas approximately 4,000 examples were used in the previous study. A larger and more diverse training dataset generally[ 21 ] allows machine learning models to perform more accurately with unseen data, which is crucial for real-world applications. In terms of performance compared with hardware requirements, unsupervised learning algorithms offer a viable solution in such resource-constrained environments. By leveraging the inherent structure and patterns within the data, the algorithm can make predictions without relying on explicit labels. This approach is similar to the self-supervised learning techniques used to improve the efficiency of large language models, demonstrating its effectiveness in handling complex language-related tasks with limited resources. Throughout the research, attention was given to upholding the principles and requirements of the GDPR, the AI Act, and the Data Act, demonstrating a commitment to responsible data handling and ethical research practices. This study had certain limitations. First, the sample size was relatively small, with data collected from only ten patients. However, even within this limited sample, the analysis included 151 documents and involved seven different professional groups. This suggests that the data, while limited in patient numbers, were comprehensive in terms of the range of clinical documentation and professionals represented. Second, the focus on persons with SCI may limit the generalizability of the findings to other patient populations. While this group presents complex care needs and multiprofessional involvement, the specific ICF codes and clinical language used may differ from those encountered in other specialties. Third, the definition dataset used to provide examples to the algorithm was developed by healthcare professionals from the same region as the data source. This may introduce cultural biases that could affect the algorithm's performance when applied to data from different regions or healthcare systems. Despite these limitations, the study provides valuable insights into the potential of automated ICF coding in a real-world clinical setting. Future research with larger and more diverse datasets is needed to further validate the algorithm's performance and generalizability across different patient populations, clinical contexts, and cultural settings. More importantly, studies should be conducted on the effectiveness of using such an algorithm on a generic problem in a clinical or administrative setting. In contrast to disease-focused classifications such as the International Classification of Diseases (ICD), the ICF offers a broader perspective on health. With increasing life expectancy and the increasing prevalence of chronic conditions, assessing everyday functioning and its challenges has become increasingly important. The COVID-19 pandemic underscored this need, revealing that even after recovering from acute illness, many individuals experience persistent functional limitations[ 22 ]. Consequently, functioning has been proposed as a third indicator for measuring healthcare outcomes, alongside morbidity and mortality[ 4 ]. The ICF provides a comprehensive framework for understanding an individual's functioning and disability across various life domains, enabling a more holistic assessment of health that considers both the impact of disease and the individual's capacity to participate in society. Integrating the algorithm into EHR systems has the potential to significantly impact the effectiveness and efficiency of health systems. By automating the extraction and assignment of ICF codes, the algorithm can facilitate several key improvements, including but not limited to personalized care, risk factor identification, understanding disease-specific or population-level functioning and disability, resource allocation, and service choices. A significant advantage of this algorithm and its associated training data is that it is potentially ready for direct implementation in a production environment without requiring further training rounds. However, extensive validation remains essential before widespread deployment. Critically, as medical device and software certification requires a clear definition of the product's intended use and context, medical device certification has not been sought for the algorithm. This step will need to be addressed once the specific system and intended application of the algorithm are determined. This process ensures that the algorithm is deployed responsibly in compliance with requirements within the healthcare system. The European Health Data Space (EHDS) initiative presents an exciting opportunity for large-scale, pan-European health data analysis. Future research should explore the applicability of the automated ICF coding methodology within the EHDS framework. Integrating such a system from the outset is crucial for the EHDS infrastructure, facilitating the translation of unstructured text into structured data at the European level. This is particularly relevant given the inclusion of the ICF standard in the EHDS initiative[ 23 ], which aims to standardize patient summaries shared across Europe. With this in mind, future research should investigate the scalability, interoperability, and cross-cultural validation of the algorithm. Ethical considerations associated with large-scale data analysis and automated coding should be made. Furthermore, future studies should expand the scope to broader patient groups and diverse clinical, research, and administrative settings in health and social care. The continuous refinement of the algorithm should be organized to ensure robustness, and the impact on the effectiveness and efficiency of the coded ICF data within different settings should be assessed. By addressing these future research directions, the potential of automated ICF coding can be fully realized, ultimately contributing to better patient care and more informed healthcare systems. Conclusion The self-supervised architecture of the text-to-ICF algorithm presented in this study achieved high performance in the automation of ICF codes. The acquisition of structured data on the functioning, disability, and health of individuals holds the promise of delivering the most appropriate health services to everyone. Declarations Ethics approval and consent to participate As this was a registry-based study maintaining patient anonymity, the regional medical research ethics committee of the Wellbeing Services County of Pirkanmaa waived the need for ethical approval and informed consent. Consent for publication Not applicable. Availability of data and materials The datasets supporting the conclusions of this article were used under a data transfer contract and are not publicly available due to general data protection regulations. However, anonymized data are available from the corresponding author upon reasonable request and with the permission of the Wellbeing Services of Pirkanmaa, Finland. Headai’s text-to-ICF algorithm is a commercial semantic computing infrastructure. It can be licensed and run in Linux and Azure clouds and servers in isolated mode (as in this study). Furthermore, Headai’s text-to-ICF algorithm’s REST-API is available for cases where data can be transferred to the internet. Competing interests Author H.K. is employed by HeadAI Ltd. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. Funding This study was financially supported by the Tampere University Hospital Support Foundation, Tampere University Hospital, Finland (project number MK367), and Finnish State Research Funding (project number 9AC067). Authors’ contributions All authors contributed to the design of the study, which shaped the research and the manuscript. L.N. was responsible for data collection, data analysis and interpretation of the analysis data. H.K. was responsible for the methodology of the study. The manuscript was drafted by L.N. and H.K., with critical feedback from M.K. All authors read and approved the final manuscript. Acknowledgements The authors thank MHSc Jaana Leivo for participating as a second health professional in the interpretation of the analysis data. References World Health Organisation WHO. ICF Beginner’s Guide: Towards a Common Language for Functioning, Disability and Health [Internet]. 2002 [cited 2022 Jan 22]. Available from: https://www.who.int/publications/m/item/icf-beginner-s-guide-towards-a-common-language-for-functioning-disability-and-health Wiegand NM, Belting J, Fekete C, Gutenbrunner C, Reinhardt JD. All talk, no action?: the global diffusion and clinical implementation of the international classification of functioning, disability, and health. Am J Phys Med Rehabil [Internet]. 2012 [cited 2022 Aug 31];91:550–60. Available from: https://pubmed.ncbi.nlm.nih.gov/22561387/ Maritz R, Aronsky D, Prodinger B. The International Classification of Functioning, Disability and Health (ICF) in Electronic Health Records. A Systematic Literature Review. Appl Clin Inform [Internet]. 2017 [cited 2022 Oct 26];8:964–80. Available from: https://pubmed.ncbi.nlm.nih.gov/28933506/ Stucki G, Bickenbach J. Functioning: the third health indicator in the health system and the key indicator for rehabilitation. Eur J Phys Rehabil Med [Internet]. 2017 [cited 2022 Sep 2];53:134–8. Available from: https://pubmed.ncbi.nlm.nih.gov/28118696/ ICF Checklist [Internet]. [cited 2022 Aug 31]. Available from: https://www.who.int/publications/m/item/icf-checklist Bickenbach J, Cieza A, Selb M, Stucki G (eds). ICF Core Sets - Manual for clinical practice [Internet]. Göttingen; 2020. Available from: https://www.icf-core-sets.org/ Üstün TB, Chatterji S, Kostanjsek N, Rehm J, Kennedy C, Epping-Jordan J, et al. Developing the World Health Organization Disability Assessment Schedule 2.0. Bull World Health Organ. 2010;88:815–23. Frattura L, Simoncello A, Bassi G, Soranzio A, Terreni S, Sbroiavacca F. The FBE development project: toward flexible electronic standards-based bio-psycho-social individual records. Stud Health Technol Inform. 2012;180:651–5. Newman-Griffis D, Camacho Maldonado J, Ho P-S, Sacco M, Jimenez Silva R, Porcino J, et al. Linking Free Text Documentation of Functioning and Disability to the ICF With Natural Language Processing. Frontiers in Rehabilitation Sciences. 2021;2:742702. Meskers CGM, van der Veen S, Kim J, Meskers CJW, Smit QTS, Verkijk S, et al. Automated recognition of functioning, activity and participation in COVID-19 from electronic patient records by natural language processing: a proof- of- concept. Ann Med [Internet]. 2022 [cited 2022 Sep 9];54:235–43. Available from: https://www.tandfonline.com/doi/abs/10.1080/07853890.2021.2025418 Kreimeyer K, Foster M, Pandey A, Arya N, Halford G, Jones SF, et al. Natural language processing systems for capturing and standardizing unstructured clinical information: A systematic review. J Biomed Inform. 2017;73:14–29. WHO classifications. International Classification of Functioning, Disability and Health (ICF) [Internet]. [cited 2021 Dec 17]. Available from: https://www.who.int/standards/classifications/international-classification-of-functioning-disability-and-health Nieminen L, Vuori J, Ketamo H, Kankaanpää Markku. Applying semantic computing for health care professionals: the timing of intervention is the key for successful rehabilitation. Proceedings of 31st conference of open innovations association FRUCT. Helsinki, Finland: FRUCT Association; 2022. p. 201–6. Nieminen L. Decision Support for Tailored Biopsychosocial Rehabilitation In Non-specific Low Back Pain [Internet]. [Tampere]: Tampere University; 2023 [cited 2025 Feb 24]. Available from: https://urn.fi/URN:ISBN:978-952-03-2851-1 Cieza A, Geyh S, Chatterji S, Kostanjsek N, Üstün B, Stucki G. ICF linking rules: An update based on lessons learned. J Rehabil Med. 2005;37:212–8. Ketamo H. Taking Printed Books into Internet of Things. In: Berntzen, Böhm, editors. The proceedings of the Eighth International Conference on Advances in Human oriented and Personalized Mechanisms, Technologies, and Services, CENTRIC. Barcelona; 2015. p. 5–11. Ketamo H, Moisio M, Passi-Rauste A, Alamäki A. Mapping the Future Curriculum: Adopting Artifical Intelligence and Analytics in Forecasting Competence Needs. Proceedings of the 10th European Conference on Intangibels and Intellectual Capital ECIIC 2019. Chieti-Pescara, Italy: Sargiacomo, M.; 2019. p. 144–53. Alamäki A, Aunimo L, Ketamo H, Parvinen L. Interactive Machine Learning: Managing Information Richness in Highly Anonymized Conversation Data. In: Camarinha-Matos LM, Afsarmanesh H, Antonelli D, editors. The Proceeding of 20th IFIP WG 55 Working Conference on Virtual Enterprises, PRO-VE. Turin, Italy; 2019. p. 173–83. Krysa JA, Gregorio MP, Pohar Manhas K, MacIsaac R, Papathanassoglou E, Ho CH. Empowerment, Communication, and Navigating Care: The Experience of Persons With Spinal Cord Injury From Acute Hospitalization to Inpatient Rehabilitation. Frontiers in rehabilitation sciences. 2022;3:904716. Diop M, Epstein D, Gaggero A. Quality of life, health and social costs of patients with spinal cord injury: A systematic review. Eur J Public Health. 2021;31. Gröger C. There is no AI without data. Commun ACM. 2021;64:98–108. Boggs D, Polack S, Kuper H, Foster A. Shifting the focus to functioning: essential for achieving Sustainable Development Goal 3, inclusive Universal Health Coverage and supporting COVID-19 survivors. https://doi.org/101080/1654971620211903214 [Internet]. 2021 [cited 2022 Nov 30];14. Available from: https://www.tandfonline.com/doi/abs/10.1080/16549716.2021.1903214 eHealth Network. Guideline on the electronic exchange of health data under Cross-Border Directive 2011/24/EU [Internet]. Budapest; 2024 Nov. Available from: https://health.ec.europa.eu/system/files/2023-10/ehn_guidelines_patientsummary_en.pdf Additional Declarations Competing interest reported. Author H.K. is employed by HeadAI Ltd. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6221473","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":430635611,"identity":"167f98b0-ae78-4a2c-be72-244a881fce62","order_by":0,"name":"Nieminen Linda","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABCklEQVRIiWNgGAWjYBACPmYGBgkGhgQGBiADRDHwMzA2gNj8bDi0sGFokWyAaJFsw6WFAaYFBgwOQLRLNuDSws578MYPhjQGc3bmoxse7rDJM77d3Pa5oIZBgg+nw/iSLXsYchgsm9nSbiSeSSs2u3OwefaMYwwSuP3CYybBw1DBYHCYx+xGYtvhxG03EpuZedgY6vBpkfwD1sL/Dajlf+LmGSAt//DbIs0DdBjQFjaglgOJGySAWnjb8GoxtpYxSOMxOMwGclhy4gygX5h5+yRwauHnP2N4801FspzB+cPPbv5ss0vsn93+mJnnm42EfAMOPWBgwMCD4EggkUQCkhSPglEwCkbBSAAAUgZMAMR4/18AAAAASUVORK5CYII=","orcid":"","institution":"Wellbeing services county of Pirkanmaa","correspondingAuthor":true,"prefix":"","firstName":"Nieminen","middleName":"","lastName":"Linda","suffix":""},{"id":430635613,"identity":"cf547699-d28b-4a3d-b562-83880626575e","order_by":1,"name":"Ketamo Harri","email":"","orcid":"","institution":"HeadAI Ltd","correspondingAuthor":false,"prefix":"","firstName":"Ketamo","middleName":"","lastName":"Harri","suffix":""},{"id":430635614,"identity":"c7939b23-43f0-4f9e-a7c6-81689ec8a3d1","order_by":2,"name":"Kankaanpää Markku","email":"","orcid":"","institution":"Tampere university Hospital","correspondingAuthor":false,"prefix":"","firstName":"Kankaanpää","middleName":"","lastName":"Markku","suffix":""}],"badges":[],"createdAt":"2025-03-13 15:38:31","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6221473/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6221473/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":78828138,"identity":"58627354-f8f6-409f-8b3e-ec34a06572be","added_by":"auto","created_at":"2025-03-19 13:03:39","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":45136,"visible":true,"origin":"","legend":"\u003cp\u003eThe structure of the ICF.\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-6221473/v1/db086d4c9a0d1e3f1d978ad9.png"},{"id":78828139,"identity":"902ccd06-6f98-42e2-80da-7115cc03dae3","added_by":"auto","created_at":"2025-03-19 13:03:39","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":134397,"visible":true,"origin":"","legend":"\u003cp\u003eArchitecture view: HeadAI's text-to-ICF algorithm connected to EHR, the ICF definition dataset, and data dashboards.\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-6221473/v1/a7df60abc8311a96609fb48a.png"},{"id":78830482,"identity":"da87b742-e364-4c8a-83f8-c908db17f78a","added_by":"auto","created_at":"2025-03-19 13:27:42","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":702831,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6221473/v1/8460396b-5f00-4ca7-98a1-76f13dca850b.pdf"}],"financialInterests":"Competing interest reported. Author H.K. is employed by HeadAI Ltd. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.","formattedTitle":"ICF coding automated: a validation study for self-supervised architecture in electronic health records","fulltext":[{"header":"Introduction","content":"\u003cp\u003eThe International Classification of Functioning, Disability and Health (ICF)[\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e], while offering a comprehensive framework for describing health beyond disease-focused models, has faced challenges in its practical application within the health system[\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. The slow implementation challenges the widespread adoption of the ICF and limits its potential to improve patient care, research and knowledge-based governance[\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. Several interrelated factors contribute to this underutilization.\u003c/p\u003e \u003cp\u003eFirst, the complexity of the ICF presents a significant barrier to its adoption. Despite the development of instruments since the introduction of the ICF in 2001, including the ICF Checklist[\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e], the ICF Core Sets[\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e], and the WHO Disability Assessment Schedule 2.0 (WHODAS 2.0)[\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e], these resources have had limited impact on daily clinical routines. The complexity and time-consuming nature of these methods often make them unsuitable for fast-paced clinical environments. This, in turn, prevents the full realization of the ICF's potential benefits, such as understanding service needs through a holistic understanding of patients’ functioning and disability.\u003c/p\u003e \u003cp\u003eSecond, current electronic health record (EHR) systems often lack standardized structures for storing ICF codes, creating difficulties in integrating and retrieving this information[\u003cspan additionalcitationids=\"CR9\" citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e–\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. Healthcare professionals do not routinely use the ICF to code patient information, partly owing to the absence of clear mandates for its implementation.\u003c/p\u003e \u003cp\u003eA major obstacle lies in the nature of EHR data itself. The majority (approximately 80%) of health data in EHRs are unstructured, making automated extraction and conversion to ICF codes difficult[\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. Manual coding is impractical because of the volume of data and the complexity of the ICF itself. With over 1,600 codes, each with detailed and nuanced definitions [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e], the ICF requires substantial effort from professionals to understand and apply it correctly. Furthermore, the language used in ICF definitions often differs from the natural language used in clinical practice, further complicating the coding process and leading to potential inconsistencies and errors.\u003c/p\u003e \u003cp\u003eThe large amount of unstructured data in EHRs makes manual ICF coding impractical, which makes automated ICF coding the only option. Traditional text analysis or natural language processing (NLP) methods are not optimal for handling this type of data because of i) the narrow understanding of real-world professional language use, ii) the demand for training data for deep learning models is colossal for 1600 codes, iii) the trustworthiness of the models is not good enough from privacy point of view, or iv) for some of the known models, the computational resources are out of reach of real-world use. Because of these known limitations, the approach suggested in this study is based on combining unsupervised /self-supervised learning methods to bypass the limitations of supervised learning, deep learning and the use of large language models.\u003c/p\u003e \u003cp\u003eAutomated ICF coding has the potential to revolutionize the use of EHR data, enabling population-level monitoring of health-related functioning and disability, and personalized medicine with effective service choices fit for individuals’ purposes, to name a few.\u003c/p\u003e \u003cp\u003eThis current validity study builds upon previous research. Prior work involved two key phases: (1) algorithm optimization, which focused on identifying the most effective definition sets[\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e], and (2) a feasibility study, which evaluated the algorithm's performance on EHR text from patients with chronic low back pain[\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e]. The feasibility study demonstrated the algorithm's sufficient specificity and sensitivity for reliable ICF code extraction. Earlier research also validated a three-layered mechanism (grammar, morphology, and semantic similarity) combined with the definition set and focused on expanding the algorithm's capabilities while enhancing data security. This study aims to further validate the algorithm's performance in a real-world clinical context.\u003c/p\u003e \u003cp\u003eThe primary aim of this study was to validate an algorithm for automatically converting unstructured EHR text into ICF codes in a complex group of patients in need of broad support from the health service network. An essential part of this study was to ensure compliance with relevant data regulations. The research questions were as follows:\u003c/p\u003e \u003cp\u003e1) What is the performance (precision, recall, and F1 score) of the text-to-ICF algorithm in this population dataset?\u003c/p\u003e \u003cp\u003e2) How does the algorithm's performance vary across different health professional groups?\u003c/p\u003e \u003cp\u003e3) How does the algorithm's performance compare to the results of the previous feasibility study?\u003c/p\u003e "},{"header":"Materials and methods","content":"\u003cp\u003e\u003cspan\u003e\u003cstrong\u003e2.1 Population data\u003c/strong\u003e\u003cbr\u003e\u003c/span\u003e\u003c/p\u003e\n\u003cp\u003eThis study utilized longitudinal data from the Wellbeing Services County of Pirkanmaa patient register, Finland, related to patients with spinal cord injury (SCI). The study was approved by the Wellbeing Services County of Pirkanmaa. As this was a registry-based study maintaining patient anonymity, the regional medical research ethics committee of the Wellbeing Services County of Pirkanmaa waived the need for ethical approval and informed consent.\u003c/p\u003e\n\u003cp\u003eElectronic health record (EHR) texts were retrospectively collected from the Uranus (CGI) EHR system. The target population consisted of adult (18 years and older) SCI patients with EHR documents related to both a neurological rehabilitation ward stay and a subsequent SCI outpatient clinic visit approximately one year after discharge. Rehabilitation episodes were required to be a direct consequence of acute SCI-related injury, illness, or disease. The data collected included texts drawn by different healthcare professionals during the inpatient rehabilitation period and a collectively written text from the multidisciplinary team at the outpatient visit. The outpatient clinic visit texts were limited to those finalized by the end of December 2022, and the inpatient rehabilitation texts were limited to those starting in January 2020.\u003c/p\u003e\n\u003cp\u003eThe initial search yielded 64 patients and associated EHR documents. Exclusion criteria were applied to remove duplicates, rehabilitation episodes not related to acute SCI, outpatient visits that were not the first postdischarge visit, consultations for different specialties, and texts outside the specified timeframes. Following these criteria, the final dataset comprised 49 patients. A computer-assisted randomization process was used to select 10 patients\u0026apos; documents to form the dataset for this study. These documents included contributions from professionals directly involved in multidisciplinary rehabilitation and outpatient care.\u003c/p\u003e\n\u003cp\u003eTwo health professionals with a thorough understanding of the ICF annotated functioning data from the free text, adhering to the ICF framework (see Fig.\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e) and linking rules[\u003cspan class=\"CitationRef\"\u003e15\u003c/span\u003e]. Text related to body structures (S codes) was excluded.\u003c/p\u003e\n\u003cp\u003eThe annotated data of 10 patients were randomly divided into two equal sets (5/5). One set served as the analysis dataset for evaluating the algorithm, and the other constituted the definition set used to provide examples for the algorithm. This definition dataset was supplemented with data from previous studies, resulting in a final definition dataset of approximately 12,000 code‒phrase pairs.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2.2 Algorithm Description\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe HeadAI\u0026apos;s text-to-ICF algorithm uses an architecture designed for efficient and accurate translation of free-text clinical notes into ICF codes. This approach addresses the challenges posed by limited training data and computational resources while still achieving robust performance. The overall goal of the algorithm is to find semantic similarities between the EHR texts/notes and the given ICF definition set. The operation consists of three layers: grammar, morphology and semantics (language model). Earlier versions of the algorithm have been published in previous publications [\u003cspan class=\"CitationRef\"\u003e13\u003c/span\u003e, \u003cspan class=\"CitationRef\"\u003e16\u003c/span\u003e\u0026ndash;\u003cspan class=\"CitationRef\"\u003e18\u003c/span\u003e]\u003c/p\u003e\n\u003cp\u003eThe first layer, the \u003cstrong\u003egrammar layer\u003c/strong\u003e, performs grammatical analysis, converting each word into its base form. This normalization process creates a standardized representation of the text, simplifying subsequent analysis and ensuring consistent interpretation regardless of surface-level variations in word form.\u003c/p\u003e\n\u003cp\u003eThe second layer, the \u003cstrong\u003emorphology layer\u003c/strong\u003e, builds upon grammatical analysis by considering morphology. This allows the algorithm to understand different word forms, sentence structures, and word order variations. By incorporating morphological understanding, the algorithm can accurately interpret the meaning of words and phrases within their grammatical context, even when phrasing or syntax varies.\u003c/p\u003e\n\u003cp\u003eThe third layer, the \u003cstrong\u003esmall language model layer\u003c/strong\u003e, employs a specialized language model tailored for ICF coding. This model focuses on recognizing and interpreting clinical jargon, synonyms, and related concepts. Unlike large, general-purpose language models, this model is specifically designed to understand the semantic nuances relevant to ICF codes within clinical text. This targeted approach enhances morphological analysis, enabling the identification of relevant concepts even when expressed via diverse terminology.\u003c/p\u003e\n\u003cp\u003eThis three-layered architecture facilitates efficient ICF code identification. The algorithm uses the processed text to identify meanings and definitions corresponding to specific ICF codes, assigning the appropriate code upon a match. This structure is computationally efficient, enabling the translation of clinical notes into ICF codes with relatively low overhead. Critically, it does not require the extensive training data typically associated with large language models.\u003c/p\u003e\n\u003cp\u003eThe decision to employ a self-supervised learning approach was driven by several key factors. First, access to a large, diverse corpus of labeled patient records was limited, precluding the use of supervised learning methods. Second, the computational resources necessary for training large language models were not available. Finally, the ICF coding task demands energy-efficient computation.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2.3 Data Security\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis study prioritized data security and ethical considerations throughout all stages of the research process. The study was conducted in accordance with the Declaration of Helsinki. Data processing was conducted within a secure, controlled research environment maintained by the registry holder. Only authorized researchers were allowed to upload the data onto the computing unit, start the computing processes and finally see the results in the secured environments. The results were not removed from the secure environment, and all the data in the computing unit (raw data and results) were deleted after the research was finalized. This physical isolation and controlled access minimized the risk of unauthorized access or data breaches.\u003c/p\u003e\n\u003cp\u003eThis study complied with European data protection regulations, including the General Data Protection Regulation (GDPR), the AI Act, and the Data Act. Recognizing the computational process as a potentially high-risk activity under the AI Act, all operations were subject to human oversight.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003e2.4 Statistical methods\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThis study employed precision, recall (sensitivity), and the F1 score as primary evaluation metrics. While the previous feasibility study used sensitivity and specificity, these metrics are less informative when true negatives are difficult to ascertain, as is often the case with free-text clinical notes. It also challenges the use of AUC- ROC-curves (the area under the receiver operating characteristics curve). The precision, recall, and F1 score provide a more balanced assessment of the algorithm\u0026apos;s performance in such scenarios.\u003c/p\u003e\n\u003cp\u003eThe algorithm\u0026apos;s output was compared against manual annotation performed by two health professionals with ICF expertise. Health professionals (HP) were considered the gold standard. The findings were classified as true positive (if the algorithm and HP identified the same ICF code), false positive (if the algorithm identified a code not found by the HP and confirmed as incorrect after the validation process), false negative (if HP identified a code missed by the algorithm), or reappraisal true positive (if the algorithm identified a code initially missed by the HP but deemed correct after the validation process). This validation process allowed for the correction of potential oversights by the human annotators, enhancing the accuracy of the performance evaluation.\u003c/p\u003e"},{"header":"The results","content":"\u003cp\u003eThe data included documents from various healthcare professionals involved in SCI patient care: physicians (neurologists and physiatrists), nurses, therapists (physiotherapists, occupational therapists, speech therapists, nutritional therapists), social workers, and rehabilitation counselors. Additionally, a multiprofessional team drew collaborative texts related to outpatient clinic visits.\u003c/p\u003e \u003cp\u003eAmong the ten patient sample, 447 EHR texts were collected. After two health professionals annotated the free text to third- and fourth-level ICF codes deductively, 9349 findings of ICF-related data were obtained. Interrater reliability was measured on a dataset formed by one of the patients\u0026rsquo; texts. For the 814 annotated words or phrases, the percentage of agreement was 97% (95% CI 96%-98%). From the annotated dataset, the annotated data of 5 patients were used to provide examples for the algorithm (extension to the definition set), and another 5 were used for the analysis. The analysis dataset was compressed on an individual document level so that recurring codes were condensed into a single finding. In the analysis, the findings of the algorithm (text-to-ICF) were analyzed against the findings of the health professional annotators (HPs). The results of the analysis are presented in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eThe results of the performance of the text-to-ICF algorithm.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"10\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c8\" colnum=\"8\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c9\" colnum=\"9\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c10\" colnum=\"10\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e\u0026nbsp;\u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePT\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003ePhysician\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSW\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eOT\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eNurse\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eRC\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c8\"\u003e \u003cp\u003eNutr.\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c9\"\u003e \u003cp\u003eMPT\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c10\"\u003e \u003cp\u003eALL\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDocuments\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e13\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e74\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e37\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e151\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTrue codes total\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e288\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e302\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e156\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e124\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e137\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e70\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e16\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e242\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e1335\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHP first round findings\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e201\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e237\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e120\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e95\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e113\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e59\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e199\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e1036\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAlgorithm extra findings\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e87\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e65\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e36\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e29\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e24\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e43\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e299\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTrue positives (GM)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e253\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e257\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e139\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e104\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e121\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e57\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e207\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e1150\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFalse negatives (GM)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e35\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e45\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e20\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e16\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e13\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e35\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e185\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFalse positives (GM)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e18\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e21\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e12\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e88\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePrecision\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0,94\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0,93\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0,96\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0,91\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0,96\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0,93\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e0,93\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e\u003cb\u003e0,94\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eRecall\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0,89\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0,87\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0,9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0,86\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0,9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0,84\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e0,8\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e0,87\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e\u003cb\u003e0,88\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eF1 score\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e0,91\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0,9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0,93\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c5\"\u003e \u003cp\u003e0,88\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c6\"\u003e \u003cp\u003e0,93\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c7\"\u003e \u003cp\u003e0,88\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c8\"\u003e \u003cp\u003e0,89\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c9\"\u003e \u003cp\u003e0,9\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c10\"\u003e \u003cp\u003e\u003cb\u003e0,91\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eHP\u0026thinsp;=\u0026thinsp;health professional, PT\u0026thinsp;=\u0026thinsp;physiotherapist, SW\u0026thinsp;=\u0026thinsp;social worker, OT\u0026thinsp;=\u0026thinsp;occupational therapist, RC\u0026thinsp;=\u0026thinsp;rehabilitation counsellor, Nutr.= nutritionist, MPT\u0026thinsp;=\u0026thinsp;multiprofessional team. True codes total\u0026thinsp;=\u0026thinsp;how many true positive codes were verified by the professional (the professional being the gold standard); HP first round findings\u0026thinsp;=\u0026thinsp;the codes found by the health professionals at first annotation round; GM extra findings\u0026thinsp;=\u0026thinsp;the algorithm identified a code initially missed by the health professionals but deemed correct after reappraisal.\u003c/p\u003e \u003cp\u003eThe previous feasibility study used different evaluation metrics (sensitivity and specificity). To compare the results reliably, we used the main results of the feasibility study to calculate the precision (0,95), recall (0,83), and F1 score (0,89). There were no significant differences in how well text-to-ICF performed between professional groups in terms of true positives, false negatives, false positives, precision, recall and F1 score.\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eThe results of this validity study demonstrate the algorithm's strong performance in automatically generating ICF codes from free-text clinical notes in a new dataset. The primary aim of this study was to evaluate the algorithm's validity by applying it to a different patient population (persons with SCI) and a more diverse set of authors (various healthcare professionals) than the previous feasibility study, which focused on chronic low back pain patients and physician-authored texts.\u003c/p\u003e \u003cp\u003ePersons with SCI were chosen because of the complex nature of their condition, characterized by major changes in their functioning[\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e]. These patients typically require extensive multidisciplinary care, making them an ideal population for evaluating the algorithm's performance on complex, multiprofessional data. Furthermore, the significant societal cost associated with SCI[\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e] underscores the potential benefits of automated ICF coding in this context.\u003c/p\u003e \u003cp\u003eThe algorithm successfully generated ICF codes with good performance, requiring some additions to the definition dataset to be adapted to a new context. This suggests the potential for scalability across diverse healthcare settings. There were no significant differences in performance between different health professional groups, which may indicate that the working culture of EHR text writing is similar across all professional groups. This result underlines the ability of the algorithm to be used in different health and social care environments and professional groups.\u003c/p\u003e \u003cp\u003eA comparison with a previous feasibility study[\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e] revealed an improvement in the F1 score in the current study. While the precision was slightly lower, the recall was notably higher. This suggests that the algorithm's ability to identify relevant ICF concepts within the text has improved. The lower precision may indicate a broader capture of relevant codes, potentially at the cost of including some false positives. However, the overall improvement in the F1 score, which harmonizes precision and recall, indicates a net positive change in the algorithm's performance.\u003c/p\u003e \u003cp\u003eA key factor contributing to this improved performance is likely the substantial increase in the volume of the definition dataset. The current study utilized approximately 12,000 examples, whereas approximately 4,000 examples were used in the previous study. A larger and more diverse training dataset generally[\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e] allows machine learning models to perform more accurately with unseen data, which is crucial for real-world applications.\u003c/p\u003e \u003cp\u003eIn terms of performance compared with hardware requirements, unsupervised learning algorithms offer a viable solution in such resource-constrained environments. By leveraging the inherent structure and patterns within the data, the algorithm can make predictions without relying on explicit labels. This approach is similar to the self-supervised learning techniques used to improve the efficiency of large language models, demonstrating its effectiveness in handling complex language-related tasks with limited resources. Throughout the research, attention was given to upholding the principles and requirements of the GDPR, the AI Act, and the Data Act, demonstrating a commitment to responsible data handling and ethical research practices.\u003c/p\u003e \u003cp\u003eThis study had certain limitations. First, the sample size was relatively small, with data collected from only ten patients. However, even within this limited sample, the analysis included 151 documents and involved seven different professional groups. This suggests that the data, while limited in patient numbers, were comprehensive in terms of the range of clinical documentation and professionals represented.\u003c/p\u003e \u003cp\u003eSecond, the focus on persons with SCI may limit the generalizability of the findings to other patient populations. While this group presents complex care needs and multiprofessional involvement, the specific ICF codes and clinical language used may differ from those encountered in other specialties.\u003c/p\u003e \u003cp\u003eThird, the definition dataset used to provide examples to the algorithm was developed by healthcare professionals from the same region as the data source. This may introduce cultural biases that could affect the algorithm's performance when applied to data from different regions or healthcare systems.\u003c/p\u003e \u003cp\u003eDespite these limitations, the study provides valuable insights into the potential of automated ICF coding in a real-world clinical setting. Future research with larger and more diverse datasets is needed to further validate the algorithm's performance and generalizability across different patient populations, clinical contexts, and cultural settings. More importantly, studies should be conducted on the effectiveness of using such an algorithm on a generic problem in a clinical or administrative setting.\u003c/p\u003e \u003cp\u003eIn contrast to disease-focused classifications such as the International Classification of Diseases (ICD), the ICF offers a broader perspective on health. With increasing life expectancy and the increasing prevalence of chronic conditions, assessing everyday functioning and its challenges has become increasingly important. The COVID-19 pandemic underscored this need, revealing that even after recovering from acute illness, many individuals experience persistent functional limitations[\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]. Consequently, functioning has been proposed as a third indicator for measuring healthcare outcomes, alongside morbidity and mortality[\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. The ICF provides a comprehensive framework for understanding an individual's functioning and disability across various life domains, enabling a more holistic assessment of health that considers both the impact of disease and the individual's capacity to participate in society.\u003c/p\u003e \u003cp\u003eIntegrating the algorithm into EHR systems has the potential to significantly impact the effectiveness and efficiency of health systems. By automating the extraction and assignment of ICF codes, the algorithm can facilitate several key improvements, including but not limited to personalized care, risk factor identification, understanding disease-specific or population-level functioning and disability, resource allocation, and service choices.\u003c/p\u003e \u003cp\u003eA significant advantage of this algorithm and its associated training data is that it is potentially ready for direct implementation in a production environment without requiring further training rounds. However, extensive validation remains essential before widespread deployment. Critically, as medical device and software certification requires a clear definition of the product's intended use and context, medical device certification has not been sought for the algorithm. This step will need to be addressed once the specific system and intended application of the algorithm are determined. This process ensures that the algorithm is deployed responsibly in compliance with requirements within the healthcare system.\u003c/p\u003e \u003cp\u003eThe European Health Data Space (EHDS) initiative presents an exciting opportunity for large-scale, pan-European health data analysis. Future research should explore the applicability of the automated ICF coding methodology within the EHDS framework. Integrating such a system from the outset is crucial for the EHDS infrastructure, facilitating the translation of unstructured text into structured data at the European level. This is particularly relevant given the inclusion of the ICF standard in the EHDS initiative[\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e], which aims to standardize patient summaries shared across Europe.\u003c/p\u003e \u003cp\u003eWith this in mind, future research should investigate the scalability, interoperability, and cross-cultural validation of the algorithm. Ethical considerations associated with large-scale data analysis and automated coding should be made.\u003c/p\u003e \u003cp\u003eFurthermore, future studies should expand the scope to broader patient groups and diverse clinical, research, and administrative settings in health and social care. The continuous refinement of the algorithm should be organized to ensure robustness, and the impact on the effectiveness and efficiency of the coded ICF data within different settings should be assessed.\u003c/p\u003e \u003cp\u003eBy addressing these future research directions, the potential of automated ICF coding can be fully realized, ultimately contributing to better patient care and more informed healthcare systems.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eThe self-supervised architecture of the text-to-ICF algorithm presented in this study achieved high performance in the automation of ICF codes. The acquisition of structured data on the functioning, disability, and health of individuals holds the promise of delivering the most appropriate health services to everyone.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch5\u003eEthics approval and consent to participate\u003c/h5\u003e\n\u003cp\u003eAs this was a registry-based study maintaining patient anonymity, the regional medical research ethics committee of the Wellbeing Services County of Pirkanmaa waived the need for ethical approval and informed consent.\u003c/p\u003e\n\u003ch5\u003eConsent for publication\u003c/h5\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003ch5\u003eAvailability of data and materials\u003c/h5\u003e\n\u003cp\u003eThe datasets supporting the conclusions of this article were used under a data transfer contract and are not publicly available due to general data protection regulations. However, anonymized data are available from the corresponding author upon reasonable request and with the permission of the Wellbeing Services of Pirkanmaa, Finland. Headai\u0026rsquo;s text-to-ICF algorithm is a commercial semantic computing infrastructure. It can be licensed and run in Linux and Azure clouds and servers in isolated mode (as in this study). Furthermore, Headai\u0026rsquo;s text-to-ICF algorithm\u0026rsquo;s REST-API is available for cases where data can be transferred to the internet.\u003c/p\u003e\n\u003ch5\u003eCompeting interests\u003c/h5\u003e\n\u003cp\u003eAuthor H.K. is employed by HeadAI Ltd. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.\u003c/p\u003e\n\u003ch5\u003eFunding\u003c/h5\u003e\n\u003cp\u003eThis study was financially supported by the Tampere University Hospital Support Foundation, Tampere University Hospital, Finland (project number MK367), and Finnish State Research Funding (project number 9AC067).\u003c/p\u003e\n\u003ch5\u003eAuthors\u0026rsquo; contributions\u003c/h5\u003e\n\u003cp\u003eAll authors contributed to the design of the study, which shaped the research and the manuscript. L.N. was responsible for data collection, data analysis and interpretation of the analysis data. H.K. was responsible for the methodology of the study. The manuscript was drafted by L.N. and H.K., with critical feedback from M.K. All authors read and approved the final manuscript.\u003c/p\u003e\n\u003ch5\u003eAcknowledgements\u003c/h5\u003e\n\u003cp\u003eThe authors thank MHSc Jaana Leivo for participating as a second health professional in the interpretation of the analysis data.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eWorld Health Organisation WHO. ICF Beginner\u0026rsquo;s Guide: Towards a Common Language for Functioning, Disability and Health [Internet]. 2002 [cited 2022 Jan 22]. Available from: https://www.who.int/publications/m/item/icf-beginner-s-guide-towards-a-common-language-for-functioning-disability-and-health\u003c/li\u003e\n\u003cli\u003eWiegand NM, Belting J, Fekete C, Gutenbrunner C, Reinhardt JD. All talk, no action?: the global diffusion and clinical implementation of the international classification of functioning, disability, and health. Am J Phys Med Rehabil [Internet]. 2012 [cited 2022 Aug 31];91:550\u0026ndash;60. Available from: https://pubmed.ncbi.nlm.nih.gov/22561387/\u003c/li\u003e\n\u003cli\u003eMaritz R, Aronsky D, Prodinger B. The International Classification of Functioning, Disability and Health (ICF) in Electronic Health Records. A Systematic Literature Review. Appl Clin Inform [Internet]. 2017 [cited 2022 Oct 26];8:964\u0026ndash;80. Available from: https://pubmed.ncbi.nlm.nih.gov/28933506/\u003c/li\u003e\n\u003cli\u003eStucki G, Bickenbach J. Functioning: the third health indicator in the health system and the key indicator for rehabilitation. Eur J Phys Rehabil Med [Internet]. 2017 [cited 2022 Sep 2];53:134\u0026ndash;8. Available from: https://pubmed.ncbi.nlm.nih.gov/28118696/\u003c/li\u003e\n\u003cli\u003eICF Checklist [Internet]. [cited 2022 Aug 31]. Available from: https://www.who.int/publications/m/item/icf-checklist\u003c/li\u003e\n\u003cli\u003eBickenbach J, Cieza A, Selb M, Stucki G (eds). ICF Core Sets - Manual for clinical practice [Internet]. G\u0026ouml;ttingen; 2020. Available from: https://www.icf-core-sets.org/\u003c/li\u003e\n\u003cli\u003e\u0026Uuml;st\u0026uuml;n TB, Chatterji S, Kostanjsek N, Rehm J, Kennedy C, Epping-Jordan J, et al. Developing the World Health Organization Disability Assessment Schedule 2.0. Bull World Health Organ. 2010;88:815\u0026ndash;23. \u003c/li\u003e\n\u003cli\u003eFrattura L, Simoncello A, Bassi G, Soranzio A, Terreni S, Sbroiavacca F. The FBE development project: toward flexible electronic standards-based bio-psycho-social individual records. Stud Health Technol Inform. 2012;180:651\u0026ndash;5. \u003c/li\u003e\n\u003cli\u003eNewman-Griffis D, Camacho Maldonado J, Ho P-S, Sacco M, Jimenez Silva R, Porcino J, et al. Linking Free Text Documentation of Functioning and Disability to the ICF With Natural Language Processing. Frontiers in Rehabilitation Sciences. 2021;2:742702. \u003c/li\u003e\n\u003cli\u003eMeskers CGM, van der Veen S, Kim J, Meskers CJW, Smit QTS, Verkijk S, et al. Automated recognition of functioning, activity and participation in COVID-19 from electronic patient records by natural language processing: a proof- of- concept. Ann Med [Internet]. 2022 [cited 2022 Sep 9];54:235\u0026ndash;43. Available from: https://www.tandfonline.com/doi/abs/10.1080/07853890.2021.2025418\u003c/li\u003e\n\u003cli\u003eKreimeyer K, Foster M, Pandey A, Arya N, Halford G, Jones SF, et al. Natural language processing systems for capturing and standardizing unstructured clinical information: A systematic review. J Biomed Inform. 2017;73:14\u0026ndash;29. \u003c/li\u003e\n\u003cli\u003eWHO classifications. International Classification of Functioning, Disability and Health (ICF) [Internet]. [cited 2021 Dec 17]. Available from: https://www.who.int/standards/classifications/international-classification-of-functioning-disability-and-health\u003c/li\u003e\n\u003cli\u003eNieminen L, Vuori J, Ketamo H, Kankaanp\u0026auml;\u0026auml; Markku. Applying semantic computing for health care professionals: the timing of intervention is the key for successful rehabilitation. Proceedings of 31st conference of open innovations association FRUCT. Helsinki, Finland: FRUCT Association; 2022. p. 201\u0026ndash;6. \u003c/li\u003e\n\u003cli\u003eNieminen L. Decision Support for Tailored Biopsychosocial Rehabilitation In Non-specific Low Back Pain [Internet]. [Tampere]: Tampere University; 2023 [cited 2025 Feb 24]. Available from: https://urn.fi/URN:ISBN:978-952-03-2851-1\u003c/li\u003e\n\u003cli\u003eCieza A, Geyh S, Chatterji S, Kostanjsek N, \u0026Uuml;st\u0026uuml;n B, Stucki G. ICF linking rules: An update based on lessons learned. J Rehabil Med. 2005;37:212\u0026ndash;8. \u003c/li\u003e\n\u003cli\u003eKetamo H. Taking Printed Books into Internet of Things. In: Berntzen, B\u0026ouml;hm, editors. The proceedings of the Eighth International Conference on Advances in Human oriented and Personalized Mechanisms, Technologies, and Services, CENTRIC. Barcelona; 2015. p. 5\u0026ndash;11. \u003c/li\u003e\n\u003cli\u003eKetamo H, Moisio M, Passi-Rauste A, Alam\u0026auml;ki A. Mapping the Future Curriculum: Adopting Artifical Intelligence and Analytics in Forecasting Competence Needs. Proceedings of the 10th European Conference on Intangibels and Intellectual Capital ECIIC 2019. Chieti-Pescara, Italy: Sargiacomo, M.; 2019. p. 144\u0026ndash;53. \u003c/li\u003e\n\u003cli\u003eAlam\u0026auml;ki A, Aunimo L, Ketamo H, Parvinen L. Interactive Machine Learning: Managing Information Richness in Highly Anonymized Conversation Data. In: Camarinha-Matos LM, Afsarmanesh H, Antonelli D, editors. The Proceeding of 20th IFIP WG 55 Working Conference on Virtual Enterprises, PRO-VE. Turin, Italy; 2019. p. 173\u0026ndash;83. \u003c/li\u003e\n\u003cli\u003eKrysa JA, Gregorio MP, Pohar Manhas K, MacIsaac R, Papathanassoglou E, Ho CH. Empowerment, Communication, and Navigating Care: The Experience of Persons With Spinal Cord Injury From Acute Hospitalization to Inpatient Rehabilitation. Frontiers in rehabilitation sciences. 2022;3:904716. \u003c/li\u003e\n\u003cli\u003eDiop M, Epstein D, Gaggero A. Quality of life, health and social costs of patients with spinal cord injury: A systematic review. Eur J Public Health. 2021;31. \u003c/li\u003e\n\u003cli\u003eGr\u0026ouml;ger C. There is no AI without data. Commun ACM. 2021;64:98\u0026ndash;108. \u003c/li\u003e\n\u003cli\u003eBoggs D, Polack S, Kuper H, Foster A. Shifting the focus to functioning: essential for achieving Sustainable Development Goal 3, inclusive Universal Health Coverage and supporting COVID-19 survivors. https://doi.org/101080/1654971620211903214 [Internet]. 2021 [cited 2022 Nov 30];14. Available from: https://www.tandfonline.com/doi/abs/10.1080/16549716.2021.1903214\u003c/li\u003e\n\u003cli\u003eeHealth Network. Guideline on the electronic exchange of health data under Cross-Border Directive 2011/24/EU [Internet]. Budapest; 2024 Nov. Available from: https://health.ec.europa.eu/system/files/2023-10/ehn_guidelines_patientsummary_en.pdf\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"International classification of functioning, disability and health, electronic health records, health information systems, artificial intelligence, unsupervised machine learning","lastPublishedDoi":"10.21203/rs.3.rs-6221473/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6221473/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground\u003c/h2\u003e \u003cp\u003eThe International Classification of Functioning, Disability and Health (ICF) provides a comprehensive framework for assessing health beyond disease-centric models, yet its integration into clinical practice remains limited. Key challenges include the complexity of ICF coding, the lack of standardized implementation in electronic health records, and the prevalence of unstructured health data. This study aims to validate an algorithm for automatically converting unstructured health record texts into ICF codes. Using self-supervised learning methods, this research aims to improve ICF utilization while ensuring compliance with data regulations.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003eThe analysis dataset included 151 electronic healthcare documents from different healthcare professionals, including physicians, nurses, therapists, social workers and rehabilitation counsellors. The algorithm performed equally well on texts from different professionals. The overall performance on the analysis dataset was 0.94 for precision and 0.88 for recall, resulting in an F1 score of 0.91.\u003c/p\u003e\u003ch2\u003eConclusions\u003c/h2\u003e \u003cp\u003eThe results of this validity study demonstrate the algorithm's strong performance in automatically generating ICF codes from free-text clinical notes. Integration of the algorithm into electronic health records systems has the potential to have a significant impact on the effectiveness and efficiency of health care systems. By automating the extraction and assignment of ICF codes, the algorithm can facilitate several important improvements within the health care system. Future research should investigate the scalability, interoperability, and cross-cultural validation of the algorithm.\u003c/p\u003e","manuscriptTitle":"ICF coding automated: a validation study for self-supervised architecture in electronic health records","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-03-19 13:03:35","doi":"10.21203/rs.3.rs-6221473/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"d5c76b95-6c30-4d0c-a8a5-479c86607839","owner":[],"postedDate":"March 19th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2025-03-19T13:11:37+00:00","versionOfRecord":[],"versionCreatedAt":"2025-03-19 13:03:35","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-6221473","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6221473","identity":"rs-6221473","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}