Latent Structure in EHR Data: Reconstruction of Diabetes Markers with Sparse NMF

ABSTRACT The dimensionality of electronic health record (EHR) data continues to grow as more clinical variables are recorded, often resulting in redundancy, sparsity, and analytical intractability. In this study, we apply non-negative matrix factorization (NMF) to a high-dimensional laboratory dataset of patients with type II diabetes to estimate the minimum latent dimensionality required to preserve clinically meaningful information. Using both within-patient imputation and across-patient generalization tasks, we evaluate the ability of the learned representations to reconstruct two key clinical lab values: blood glucose and HbA1c. Our findings show that clinically acceptable accuracy can be achieved with a dimensionality reduction of up to 80% and a dimensionality of 230 to 300, supporting the presence of a compact, low-dimensional latent structure underlying high-dimensional clinical data. Competing Interest Statement The authors have declared no competing interest. Funding Statement This project was funded in part by NIH grants R01 LM006910 and U01TR002062. Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes The details of the IRB/oversight body that provided approval or exemption for the research described are given below: The Institutional Review Board of Columbia University gave ethical approval for this work. I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Yes Footnotes ae2722{at}cumc.columbia.edu hripcsak{at}columbia.edu Data Availability All data produced in the present work are contained in the manuscript. Source medical record data are not available.