Data Quality in Clinical Coding: A Critical Analysis and Preliminary Study

Abstract Clinical coding is a vital yet complex part of healthcare practice. Most automated coding research relies on imperfect training data, which has an unavoidable negative impact on the quality of code prediction. A key contributing issue, vastly overlooked in current research, is the ubiquitous presence of undercoding and various types of coding errors in widely used coding datasets. From another angle, coding audit is as challenging as coding itself due to the lack of assistive tools, which is also under-studied compared to automated coding research. In this work, we uncover substantial undercoding and errors in commonly used datasets and present the first empirical study on their impacts on the performances of automated coding algorithms. To enable this, we develop an interpretable coding pipeline, using large language models (LLMs) for evidence extraction and code verification, and a multiclass classifier trained on a large-scale dataset of silver-standard evidence-code pairs for code prediction. Assisted by the pipeline, three professional coders systematically identify, categorise, and correct errors in two widely-used coding datasets that have human-annotated evidence texts for assigned codes. As an AI-assisted coding audit tool, the current study uncovers significant data quality issues, including a 76.3% undercoding rate in MDACE and a 29.7% error rate in CodiEsp. Re-evaluating existing models on error-corrected datasets results in consistent performance improvements. Additionally, the pipeline is a highly potential coding framework, which achieves superior or comparative performances to state-of-the-art LLM-based methods. The results underscore the necessity of shifting research focus from model-centric to data-centric solutions in clinical AI. Competing Interest Statement The authors have declared no competing interest. Funding Statement This study did not receive any funding. Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Yes Footnotes This paragraph of the first footnote will contain the date on which you submitted your paper for review, which is populated by IEEE. (e-mail: education.supriyakhadka{at}gmail.com). (e-mail: kvasile.palade{at}coventry.ac.uk). 1, Significantly extended the related work about clinical code quality analysis and coding audit 2, Significantly extended the manual review of code quality to the complete set of MDACE dataset and CodiEsp datasets 3, Totally rewrote the Results section with a the first subsection about "Inter-Annotator Agreement is High", the second and third subsections about "Undercoding Findings" and "Error Detection Findings" respectively, the fourth subsection about "Correcting Errors Benefits Automated Clinical Coding", and the fifth subsection about "Performance of the Code Prediction Pipeline" 4, In the subsection about "Performance of the Code Prediction Pipeline", the experiments were extended with quasi-ablation studies, comparisons agains state-of-the-art LLM-based approaches, and additional experiments on a third dataset MIMIV-IV-ICD10-Mini 5, Additionally, we extended the experiments to demonstrate that the LLM ensemble in our proposed approach can serve as a highly potential assisted coding audit tool, in the subsection about "Potential for Clinical Coding Audit" 6, We add a comprehensive section of discussions, in the subsection about "Discussions, Limitations and Future Work" 7, The abstract and conclusions sections, as well as the Introduction section, were heavily rewritten Data Availability All data produced in the present work are contained in the manuscript.