Predicting New York Heart Association (NYHA) Heart Failure Classification from medical student notes following simulated patient encounters

Predicting New York Heart Association (NYHA) Heart Failure Classification from medical student notes following simulated patient encounters | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Predicting New York Heart Association (NYHA) Heart Failure Classification from medical student notes following simulated patient encounters Ishan R. Perera, Taylor Daniels, Janella Looney, Kimberly Gittings, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6197186/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 15 Jul, 2025 Read the published version in Scientific Reports → Version 1 posted 10 You are reading this latest preprint version Abstract Random forest models have demonstrated utility in the determination of New York Heart Association (NYHA) Heart Failure Classifications. This study aims to determine the prediction accuracy of a random forest model to derive NYHA Classification from medical students’ free text history of present illness (HPI). NYHA Classifications established terminology for delineation of various heart failure presentations, this terminology was converted into keywords shared by standardized patients. 649 typed HPIs were de-identified, tokenized, cleaned, and assessed for number of correct keywords, incorrect keywords, and keyword usage. Models were trained using bootstrapped training data and assessed on test data. In testing, the model demonstrated a 0.775% error rate in identifying NYHA II, 26.3% for NYHA III, and 6.90% for NYHA IV. Overall reporting a 0.420% estimated error rate on the bootstrap sample training set and an 8.20% misclassification rate on the testing set. In future applications, developing a method of instantaneous feedback centered around keywords and their importance measures, specifically as determined by the variable importance plot (VIP), may aid students in their determination of NYHA Classifications and improve their lexical density. Health sciences/Cardiology/Cardiovascular biology/Cardiovascular diseases/Heart failure Physical sciences/Mathematics and computing/Computer science New York Heart Association (NYHA) Heart Failure Classification Random Forest Machine Learning Simulated Patient Encounters Medical Education Figures Figure 1 Figure 2 Figure 3 Figure 4 Introduction Electronic Health Records (EHR) are conglomerates of widely differing data formats including both structured and unstructured data [ 1 – 3 ]. The Edward Via College of Osteopathic Medicine (VCOM) Hospital Integrated Clinical Cases (HICC) course has adapted this EHR format in subjective, objective, assessment, and plan (SOAP) Note documentation, as a critical part of the student assessment. This standard note type is structured; however, each section may contain highly pertinent data entered in an unstructured format. Here, heuristics are important, textual analysis provides a solution to uncovering the hidden data, namely the timely diagnosis of a patient’s ailment [ 3 ]. Textual analysis has been used for a multitude of applications, such as discovering language patterns using sentiment analysis, identifying document similarity and dissimilarity, classifying and sorting using clustering methods, and locating unusual behavior or anomalies in textual data [ 1 – 3 , 5 – 7 ]. Neural networks have outperformed current prediction systems for mortality risk in the Intensive Care Unit (ICU), risk levels in patients with atrial fibrillation, and risk of possible cardiac arrest in patients with heart disease [ 5 , 8 ]. Diagnostically, neural networks have surpassed current systems for identifying and evaluating cardiac arrhythmias, as well as genotypes and phenotypes of cardiovascular diseases [ 4 , 9 , 10 ]. Hospitals have been using text analytics and natural language processing for improved efficiency by reducing the amount of time physicians spend documenting cases [ 3 ]. Advanced systems provide medication and treatment recommendations based on textual input directly typed or vocalized into an EMR, known as a Computerized Clinical Decision Support System (CDSS) [ 11 ]. Analytical systems have been developed to assist physicians in extracting relevant information from unstructured modalities like free text, demographics, imaging, and disease trends in high-risk populations like seafarers [ 2 , 10 ]. By having easier access to a more inclusive picture of a patient, these systems can help physicians in their differential diagnosis, while better understanding their patient’s needs [ 2 , 10 ]. The New York Heart Association (NYHA) Functional Classification for Heart Failure stratifies patients by severity of subjective symptoms and eligibility determination for clinical trials [ 12 ]. This data is usually determined from the unstructured textual data within the EMR [ 1 , 13 , 14 ]. To date, the literature demonstrates the utility of machine-learning methodologies, such as random forests and decision trees in the identification of NYHA Classification from the unstructured data within the EMR [ 13 – 15 ]. The primary goal of this study is to predict NYHA Classification in the unstructured history of present illness (HPI) section of first year medical students’ standardized Block 4 HICC Cardiopulmonary Testing utilizing a random forest model. Methods This study received exemption status from the Edward Via College of Osteopathic Medicine Institutional Review Board on September 22, 2022 [1948897-1]. The data utilized for our analysis was generated by first year Osteopathic Medical Students (OMS1) from all four VCOM campuses (Blacksburg, VA; Spartanburg, SC; Auburn, AL; Monroe, LA) during their Block 4 HICC Cardiopulmonary Testing. Specifically, following their standardized patient examinations on one of four cases: Myocarditis (NYHA II), Familial Hypertrophic Cardiomyopathy (NYHA II), Sarcoidosis Cardiomyopathy (NYHA III), and Ischemic Cardiomyopathy (NYHA IV), students completed their unstructured content entry into a simulated EMR as part of their typical Block 4 testing process. This sampling method was of convenience, due to the pre-existing integration of these processes within our institution. Importantly, standardized patients were instructed on the case background, case presentation, physical exam, and level of clinical impairment based on NYHA Classification. Patients learned the appropriate descriptors to employ in their portrayal of the assigned case for the appropriate NYHA Classification. Post-examination, raw student data was de-identified and overall performance data was excluded. NYHA I Classification descriptors were not used in this series of testing; therefore, the study will only discuss NYHA II through IV. 649 free-text HPIs and assigned standardized cases were collected. Lexical analysis began with tokenization of responses by splitting paragraphs, sentences, and phrases into individual words or terms. These responses were then cleaned by the elimination of capitalization, punctuation, abbreviations, and symbols. Inverse Document Frequency was applied to find both important and frequently used words. Keyword lists were generated for each standardized case, these were then validated by an attending physician. Next, the amount of correct and incorrect keywords used by each student were calculated for their assigned case. The algorithm returned each keyword with a use count, percentage based on the number of correct keywords, and a percentage based on incorrect keywords specific to each individual case. A partial empty matrix was created with select data from responses consisting of the keyword count total, incorrect keyword total, correct keyword percentage, incorrect keyword percentage, and overall keyword usage denoted by either a 0 (did not) or a 1 (did use). The data was organized into a train-test split ⅝ - ⅜ prepared at random using 405 entries for training (203 NYHA II, 106 NYHA III, 96 NYHA IV) and 243 entries for testing (129 NYHA II, 57 NYHA III, 57 NYHA IV). To further expand our training set, bootstrap sampling or sample selection with replacement was conducted 10 times, once merged a total of 4,050 entries were generated (2080 NYHA II, 1065 NYHA III, 905 NYHA IV). Cross validation was also used, this resampling method utilizes repeated training set splits. Training data was split into 5 groups, 4 were used for training and 1 was withheld for testing. Each model was then evaluated on the testing group, the model’s score was maintained, but the model was discarded to allow for continuous improvement. Each data sample was assigned a group and remained in that group for the process. Random Forest Model Parameters The random forest was set to a node size of 1, the minimum size of terminal nodes. Importance, a metric assessment of the contribution of a variable to the overall model, was set to true. Proximity, a measure of closeness between two similar cases, was set to true. The model was set to generate 150 trees. The number of bootstrap replications was set to 150. The minimum number of variables required for a node to attempt a split (minsplit) was set to 2 and any split that did not improve the fit was set to be pruned (complexity parameter). Cross validation was set to 5-fold with a minsplit of 2 and a complexity parameter of 0. Statistical Analysis Discussed data falls under the category of parametric data. Specifically, Poisson Distributed Variables, such as the correct word count range and wrong word count range; Binary Distributed Variables, such as the presence or lack of keywords; Categorical Nominal Variables, such as NYHA Classification. As appropriate, variables were summarized as mean and standard deviation (SD). Multiple comparisons were not made, thus corrections were not indicated. The calculation for misclassification rate is described as the sum of total false positives and total false negatives all divided by the total observations. Final model comparison was made by this misclassification metric alone. All percentages were rounded to three significant figures, while binary data was reported as assessed. Analyses were conducted using R-Studio 2024.12.0 + 467 and Python version 3.11. Results Keywords 57 initial keywords were identified, 25 had zero variability indicating a lack of use by any student regardless of the case or intended classification. These 25 variables were dropped to reduce dimensionality as they didn’t provide any additional information for the models to improve upon. Additionally, highly correlated variables, such as the phrase “six months” were included as complete phrases and split on white-space as “six” and “months”. 11 initial keywords became 14 after the split for Myocarditis (NYHA II), 6 became 9 after the split for Familial Hypertrophic Cardiomyopathy (NYHA II), 7 became 10 with the split for Sarcoidosis Cardiomyopathy (NYHA III), and 8 became 10 with the split for Ischemic Cardiomyopathy (NYHA IV). In total, 32 individual keywords were assembled, 19 of them unique (Table 1 ). Table 1 Keyword List. All 32 keywords are listed, 19 unique keywords indicated by (*). Keywords Column 2 Column 3 Column 4 *muscle aches uncomfortable six weeks no sob 6 two months *flutter *marked *unable *cold *chair *increased comfortable *cough *worse *gym *dull *pass out *rest *swelling 2 months two *most *some two weeks *tired *any six months *activity sob For those students who received a NYHA II case, the average correct keywords used were 8.05 out of the maximum 14 possible correct keywords, while the average incorrect were 1.24. In the NYHA III case, the average correct keywords were 8.53 out of 10, while the incorrect average was 0.816. For the NYHA IV case the average correct keywords were 7.73 out of 10, while the incorrect average was 0.662. Overall, the average correct keywords for all cases was 8.09 (SD +/- 2.84) and 0.995 (SD +/- .998) for incorrect words (Table 2 ). Table 2 Keyword Averages. The table demonstrates the average correct and incorrect keywords used by students by case. Totals are reported with a standard deviation. Case Average Correct Keywords Average Incorrect Keywords NYHA II 8.05 1.24 NYHA III 8.53 0.816 NYHA IV 7.73 0.662 Total 8.09 (+/- 2.84) .995 (+/- .998) A variable importance plot (VIP) was generated to demonstrate the importance of a given variable in regard to the overall integrity of the model; in other words, removing a variable from the model would result in an appreciable proportion of observations to be misclassified (Fig. 1 ). Figure 1 a demonstrates the Mean Decrease Accuracy plot, which expresses the accuracy loss suffered by the model by excluding a particular variable. The correct word variable is the most important factor for Mean Decrease Accuracy. Similarly, Fig. 1 b demonstrates the Mean Decrease Gini plot, which is a representation of the per variable contribution to node homogeneity, a measure of node purity. Here, the keyword “months” is most important for Mean Decrease Gini, likely due to the artificially assigned symptom duration within each of the set cases. These variables are presented in descending importance from top to bottom in each graph. Random Forests The generated Random Forest model consisted of 150 trees. Traditionally, increasing the trees in the forest imbues the model with more accuracy; however, the mean squared error in our model stabilizes around 50 and 150 (Fig. 2 ). The lines in the figure represent each of the NYHA Classifications: red represents NYHA II, green NYHA III, blue NYHA IV, and black represents the data used strictly for testing. During training the model reported a 0% error rate in the identification of NYHA II cases, 1.01% error rate for NYHA III, and 0.552% error rate for NYHA IV (Table 3 ). When the model was assessed on the test data we found a 0.775% error rate for NYHA II, 26.3% for NYHA III, and 6.90% for NYHA IV (Table 4 ). Table 3: Random Forests Training Set Output. The table demonstrates the expected counts for each case in comparison to the count determined by the Random Forests model using the training data set. NYHA Classification NYHA II Determined NYHA III Determined NYHA IV Determined Error NYHA II Expected 2080 0 0 0 NYHA III Expected 0 1054 11 0.0101 NYHA IV Expected 0 8 897 0.00552 Table 4: Random Forests Test Set Output. The table demonstrates the expected counts for each case in comparison to the count determined by the Random Forests model using the testing data set. NYHA Classification NYHA II Determined NYHA III Determined NYHA IV Determined Error NYHA II Expected 126 3 0 0.00775 NYHA III Expected 12 45 0 0.263 NYHA IV Expected 5 0 52 0.0690 Figure 3 demonstrates a multidimensional scaling proximities plot, while Fig. 4 is focused on the convergence point of these axes. The plots illustrate model performance regarding the testing data set. From the graph we can see that there are three distinct lines representing each of the three classification groups, spaced equally in the XYZ coordinate plane, thus equally spaced within three-dimensions. The spatial distance amongst the groups indicates that the Random Forest model has clearly identified the NYHA Classifications. This model performed accurately based solely on the keywords used by students, correct word percentage, and wrong word percentage. There is slight overlap between groups, which indicates misclassification, demonstrated in the zoomed Fig. 4 . There are observations from other classifications that bleed into the other group clusters; however, this is diminutive in scale. The final model demonstrated a 0.420% misclassification rate on the bootstrap training set and an 8.20% misclassification rate on the testing set. Discussion This study led to the development of a 150-tree random forest model trained and tested on the identification of NYHA Classifications II-IV in the unstructured textual data of medical student HPI entries as opposed to relying on structured data [ 15 ]. Overall the model performed well with an 8.20% misclassification rate. The study began with lexical analysis, which developed a set of keywords most associated with determining the correct NYHA Classification (Table 1 ). The significance here being the environment in which this model is used. In the medical education setting we develop the scenario in which students extract pre-written information regarding the presented case, these pre-written scenarios hinge on the use of very targeted keywords. The students then report their findings in a structured SOAP note format, with much of their effort being placed in the HPI. This unstructured section is meant to be a clinical summary portraying the relevant details for the current illness and may be comparable to the verbal presentation one may give to a medical team. The performance of our model is in-line with many of the similarly assessed random forest models in the literature; however, we do find the model to disproportionately struggle with the classification of NYHA III cases [ 13 – 15 ]. Interestingly, this difficulty in delineating NYHA II and III cases has been previously documented [ 16 ]. In addition to relying on similar keywords to delineate between NYHA II and III cases, the inadequacies of the model may be due to a paucity of training data, as noted previously 203 of the original cases were NYHA II, 106 NYHA III, and just 96 NYHA IV. Many of the studies that reported better accuracy either included a larger initial dataset or a larger fold cross validation [ 13 – 15 ]. Determining the optimal tree count is an important aspect of balancing the computational strain in utilizing a random forest model and directly affects accuracy. As demonstrated in Fig. 3 , a maximum default of 150 trees was used in our study; however, we clearly appreciated a similar result with a forest of 50–75 trees. This reduction to half or less trees would result in a significant diminution in the computational costs of the model [ 17 ]. Additionally, expanding the number of trees increases the amount of variables used to make determinations by the model [ 17 ]. This composited use of the available variables juxtaposes our goal of creating a model that effectively uses the most essential variables for determinations. Keywords are a critical aspect of creating random forest models targeted at deciphering unstructured textual data; however, in the traditional sense these are solely used to aid a random forest model in making its determinations. Our study begins to assess the use of these keywords in each case, potentially allowing for the assessment of efficient and accurate portrayals of medical information otherwise known as medical decision making [ 3 ]. Similar computational models have been used in the academic setting to support standardized scoring platforms for medical education [ 9 ]. However, none have been poised to provide lexical feedback to students. Here, we demonstrated an average of 8.09 correct keywords in comparison to 0.995 incorrect keywords used by students throughout the cases. This demonstrates a tendency for the students to accurately identify keywords (Table 1 ) during the case, while self-assigning importance to the keywords prior to typing their HPIs. The students are likely demonstrating an ability to surmise the veiled VIP (Fig. 1 ) through their studies, preparation materials, and the overall standardized patient visit. Notably, the development of a VIP could assist educators in developing a more lexically-based assessment system that gives weighted credit to the use of keywords with a higher mean decrease accuracy. As we know the inclusion of keywords with a high mean decrease accuracy increases our model accuracy, thus likely maximizing the effectiveness of the medical case presentation. The limitations of the study stem from the initially limited sample size, unbalanced distribution of NYHA Classification cases, and the lack of NYHA I cases. Additionally, the equipment available to our study team was rudimentary, limiting our bootstrap iterations to 10 and our fold cross validation to 5. Improvements in our available equipment would have further increased the training and testing sample sizes. Conclusion Random forest models can be applied with acceptable accuracy in the determination of NYHA Classifications based solely on the free text HPI from medical student standardized patient simulated cases. Our model demonstrated a comparably low 0.420% and 8.20% misclassification rate, for training data and testing data respectively. These scores would likely improve with an increased volume of training cases and utilization of more purpose-built computational equipment. Continued improvement of this model would comfortably supplement an educator in assessing medical students on their performance in simulated cases. Future implementations of these findings may lead to the development of an instantaneous feedback system for students to receive constructive critique regarding their medical decision making and overall medical lexical density. Declarations Additional Information Competing Interests The authors declare no competing interests. Author Contribution I.R.P. and T.D. conceived the study. I.R.P., T.D., F.A.R., J.L., and K.G. collected and produced the data. T.D. generated the model and performed the statistical analysis. I.R.P. wrote the manuscript. I.R.P. and T.D. prepared the figures and tables. I.R.P., T.D., F.A.R., J.L., and K.G. critically revised and approved the final manuscript. Acknowledgement The authors would like to show special gratitude to the Edward Via College of Osteopathic Medicine Center for Simulation and Technology for supporting the completion of this research with their equipment. Data Availability The textual data analysed during the study are available in the following GitHub repository: https://github.com/tDaniels1701/Random-Forest-HPI-Classification.git References Zhang, R. et al. Discovering and identifying new york heart association classification from electronic health records. BMC Medical Informatics and Decision Making. 18 , (2018). Amir, A. et al. AALIM: a cardiac clinical decision support system powered by advanced multi-modal analytics. Studies in health technology and informatics. 160 Pt 2 , 846-50 (2010). Elbattah, M., Arnaud, É., Gignon, M., & Dequen, G. The role of text analytics in healthcare: a review of recent developments and applications. International Conference on Health Informatics. (2021). Krittanawong, C., Zhang, H., Wang, Z., Aydar, M., & Kitai, T. Artificial intelligence in precision cardiovascular medicine. J Am Coll Cardiol. 69 21 , 2657-2664 (2017). Krishnan, G.S., & Kamath, S.S. A supervised learning approach for icu mortality prediction based on unstructured electrocardiogram text reports. International Conference on Applications of Natural Language to Data Bases. (2018). Sugamiya, Y., Otani, T., Nakadate, R., & Takanishi, A. Construction of automatic scoring system to support objective evaluation of clinical skills in medical education. 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). 4177-4181 (2019). Chintalapudi, N., Battineni, G., Canio, M.D., Sagaro, G.G., & Amenta, F. Text mining with sentiment analysis on seafarers' medical documents. Int. J. Inf. Manag. Data Insights. 1 , 100005 (2021). Hill, N.R. et al. Predicting atrial fibrillation in primary care using machine learning. PLoS ONE. 14 , (2019). Savalia, S., & Emamian, V. Cardiac arrhythmia classification by multi-layer perceptron and convolution neural networks. Bioengineering. 5 , (2018). Li, J., Si, Y., Xu, T., & Jiang, S. Deep convolutional neural network based ecg classification system using information fusion and one-hot encoding techniques. Mathematical Problems in Engineering. 2018 , 7354081 (2018). Sutton, R. et al. An overview of clinical decision support systems: benefits, risks, and strategies for success. npj Digit. Med. 3, 17 (2020). Caraballo, C. et al. Clinical implications of the new york heart association classification. J Am Heart Assoc. 8 23 , (2019). Adejumo, P. et al. Natural language processing of clinical documentation to assess functional status in patients with heart failure. JAMA Netw Open. 7 11 , (2024). Zhang, R. et al. Automatic methods to extract new york heart association classification from clinical notes. Proceedings (IEEE Int Conf Bioinformatics Biomed). 1296-1299 (2017). Jandy, K., & Weichbroth, P. A machine learning approach to classifying new york heart association (NYHA) heart failure. Sci Rep. 14 , (2024). Raphael, C. et al. Limitations of the new york heart association functional classification system and self-reported walking distances in chronic heart failure. Heart. 93 4 , 476-482 (2006). Oshiro, T.M., Perez, P.S., & Baranauskas, J.A. How many trees in a random forest? Machine Learning and Data Mining in Pattern Recognition (MLDM). 7376 , (2012). Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 15 Jul, 2025 Read the published version in Scientific Reports → Version 1 posted Editorial decision: Revision requested 09 Apr, 2025 Reviews received at journal 09 Apr, 2025 Reviews received at journal 06 Apr, 2025 Reviewers agreed at journal 19 Mar, 2025 Reviewers agreed at journal 18 Mar, 2025 Reviewers invited by journal 17 Mar, 2025 Editor assigned by journal 17 Mar, 2025 Editor invited by journal 17 Mar, 2025 Submission checks completed at journal 15 Mar, 2025 First submitted to journal 10 Mar, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6197186","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":431191736,"identity":"536e5e42-d9d5-408d-b81f-8e0b3fda30c7","order_by":0,"name":"Ishan R. Perera","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA0ElEQVRIiWNgGAWjYBACNigthySWQJwWYziLoBYYSGwgWgufdPOxDx/32KVvuN/AJvGjYhsDP3uOAX6HyRxLnjnjWXLuhmMMbJI9Z24zSPa8IaBFIseYmecAc+7MNgZmY8a22wwGNwjZIpH/mfnPgfp0SZgWe8JacpiZGQ4cTuBnY2B8DLZFgrBfjBl7Dhw37GdLbHwI9AuPxJlnBXi1yM9ufszw40C1PBvz4QMHflTcluNvT96AVwuDBJzF2AAiefArR9UyCkbBKBgFowAHAABHAz+1arIV8QAAAABJRU5ErkJggg==","orcid":"","institution":"Edward Via College of Osteopathic Medicine - Virginia Campus","correspondingAuthor":true,"prefix":"","firstName":"Ishan","middleName":"R.","lastName":"Perera","suffix":""},{"id":431191737,"identity":"60f191d2-5198-4d9f-a836-a8c4965877ed","order_by":1,"name":"Taylor Daniels","email":"","orcid":"","institution":"Edward Via College of Osteopathic Medicine - Virginia Campus","correspondingAuthor":false,"prefix":"","firstName":"Taylor","middleName":"","lastName":"Daniels","suffix":""},{"id":431191738,"identity":"ce366aa4-f17a-4333-8a01-f52c763fa75a","order_by":2,"name":"Janella Looney","email":"","orcid":"","institution":"Edward Via College of Osteopathic Medicine - Virginia Campus","correspondingAuthor":false,"prefix":"","firstName":"Janella","middleName":"","lastName":"Looney","suffix":""},{"id":431191739,"identity":"44496433-3cfa-47bb-8e47-2bc74835f22a","order_by":3,"name":"Kimberly Gittings","email":"","orcid":"","institution":"Edward Via College of Osteopathic Medicine - Virginia Campus","correspondingAuthor":false,"prefix":"","firstName":"Kimberly","middleName":"","lastName":"Gittings","suffix":""},{"id":431191740,"identity":"08c9b5da-f8c6-4eca-9a05-1d1cb1108f2d","order_by":4,"name":"Frederic A. Rawlins, II","email":"","orcid":"","institution":"Edward Via College of Osteopathic Medicine - Virginia Campus","correspondingAuthor":false,"prefix":"","firstName":"II","middleName":"Frederic A.","lastName":"Rawlins","suffix":""}],"badges":[],"createdAt":"2025-03-10 16:26:35","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6197186/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6197186/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s41598-025-10179-8","type":"published","date":"2025-07-15T15:57:27+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":79257632,"identity":"43405cd2-53e2-41db-bf91-f06db71a10bb","added_by":"auto","created_at":"2025-03-26 09:07:56","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":319400,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eVariable Importance Plot. \u003c/strong\u003e(a) Mean Decrease Accuracy variable importance plot and (b) Mean Decrease Gini variable importance plot.\u003c/p\u003e","description":"","filename":"Figure1.png","url":"https://assets-eu.researchsquare.com/files/rs-6197186/v1/967992b61d574a049e3acd61.png"},{"id":79257631,"identity":"0fa94d83-0013-4e4c-8025-62740bcdaec6","added_by":"auto","created_at":"2025-03-26 09:07:56","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":119084,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eTrees vs. Error. \u003c/strong\u003eDemonstrates the flux of mean squared error in respect to increasing the number of trees in the Random Forest model.\u003c/p\u003e","description":"","filename":"Figure2.png","url":"https://assets-eu.researchsquare.com/files/rs-6197186/v1/d375ab5c3bc528e18bd7f5e5.png"},{"id":79257637,"identity":"fecec1a8-1cbc-40e7-94ac-f71a7ade0e77","added_by":"auto","created_at":"2025-03-26 09:07:56","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":80878,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eMultidimensional Scaling Proximity Plot.\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"Figure3.png","url":"https://assets-eu.researchsquare.com/files/rs-6197186/v1/99b8aa8380e66b1552b24c27.png"},{"id":79260583,"identity":"f1f40ab8-a501-4acb-a0bf-cff3eeae4ed6","added_by":"auto","created_at":"2025-03-26 09:23:57","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":128045,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cstrong\u003eZoomed Multidimensional Scaling Proximity Plot.\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"Figure4.png","url":"https://assets-eu.researchsquare.com/files/rs-6197186/v1/c07e2e5eb75ac81ee9a8aa7b.png"},{"id":87219341,"identity":"7c5423e0-7d6e-435d-afe0-5e8de281c2e9","added_by":"auto","created_at":"2025-07-21 16:03:53","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1171963,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6197186/v1/e9ec32bc-6aaf-40d9-8517-7584376fc1ce.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Predicting New York Heart Association (NYHA) Heart Failure Classification from medical student notes following simulated patient encounters","fulltext":[{"header":"Introduction","content":"\u003cp\u003eElectronic Health Records (EHR) are conglomerates of widely differing data formats including both structured and unstructured data [\u003cspan additionalcitationids=\"CR2\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. The Edward Via College of Osteopathic Medicine (VCOM) Hospital Integrated Clinical Cases (HICC) course has adapted this EHR format in subjective, objective, assessment, and plan (SOAP) Note documentation, as a critical part of the student assessment. This standard note type is structured; however, each section may contain highly pertinent data entered in an unstructured format. Here, heuristics are important, textual analysis provides a solution to uncovering the hidden data, namely the timely diagnosis of a patient\u0026rsquo;s ailment [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eTextual analysis has been used for a multitude of applications, such as discovering language patterns using sentiment analysis, identifying document similarity and dissimilarity, classifying and sorting using clustering methods, and locating unusual behavior or anomalies in textual data [\u003cspan additionalcitationids=\"CR2\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan additionalcitationids=\"CR6\" citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. Neural networks have outperformed current prediction systems for mortality risk in the Intensive Care Unit (ICU), risk levels in patients with atrial fibrillation, and risk of possible cardiac arrest in patients with heart disease [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. Diagnostically, neural networks have surpassed current systems for identifying and evaluating cardiac arrhythmias, as well as genotypes and phenotypes of cardiovascular diseases [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e, \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. Hospitals have been using text analytics and natural language processing for improved efficiency by reducing the amount of time physicians spend documenting cases [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. Advanced systems provide medication and treatment recommendations based on textual input directly typed or vocalized into an EMR, known as a Computerized Clinical Decision Support System (CDSS) [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. Analytical systems have been developed to assist physicians in extracting relevant information from unstructured modalities like free text, demographics, imaging, and disease trends in high-risk populations like seafarers [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. By having easier access to a more inclusive picture of a patient, these systems can help physicians in their differential diagnosis, while better understanding their patient\u0026rsquo;s needs [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe New York Heart Association (NYHA) Functional Classification for Heart Failure stratifies patients by severity of subjective symptoms and eligibility determination for clinical trials [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. This data is usually determined from the unstructured textual data within the EMR [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e, \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e]. To date, the literature demonstrates the utility of machine-learning methodologies, such as random forests and decision trees in the identification of NYHA Classification from the unstructured data within the EMR [\u003cspan additionalcitationids=\"CR14\" citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. The primary goal of this study is to predict NYHA Classification in the unstructured history of present illness (HPI) section of first year medical students\u0026rsquo; standardized Block 4 HICC Cardiopulmonary Testing utilizing a random forest model.\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003e This study received exemption status from the Edward Via College of Osteopathic Medicine Institutional Review Board on September 22, 2022 [1948897-1].\u003c/p\u003e \u003cp\u003eThe data utilized for our analysis was generated by first year Osteopathic Medical Students (OMS1) from all four VCOM campuses (Blacksburg, VA; Spartanburg, SC; Auburn, AL; Monroe, LA) during their Block 4 HICC Cardiopulmonary Testing. Specifically, following their standardized patient examinations on one of four cases: Myocarditis (NYHA II), Familial Hypertrophic Cardiomyopathy (NYHA II), Sarcoidosis Cardiomyopathy (NYHA III), and Ischemic Cardiomyopathy (NYHA IV), students completed their unstructured content entry into a simulated EMR as part of their typical Block 4 testing process. This sampling method was of convenience, due to the pre-existing integration of these processes within our institution. Importantly, standardized patients were instructed on the case background, case presentation, physical exam, and level of clinical impairment based on NYHA Classification. Patients learned the appropriate descriptors to employ in their portrayal of the assigned case for the appropriate NYHA Classification. Post-examination, raw student data was de-identified and overall performance data was excluded. NYHA I Classification descriptors were not used in this series of testing; therefore, the study will only discuss NYHA II through IV. 649 free-text HPIs and assigned standardized cases were collected.\u003c/p\u003e \u003cp\u003eLexical analysis began with tokenization of responses by splitting paragraphs, sentences, and phrases into individual words or terms. These responses were then cleaned by the elimination of capitalization, punctuation, abbreviations, and symbols. Inverse Document Frequency was applied to find both important and frequently used words. Keyword lists were generated for each standardized case, these were then validated by an attending physician. Next, the amount of correct and incorrect keywords used by each student were calculated for their assigned case. The algorithm returned each keyword with a use count, percentage based on the number of correct keywords, and a percentage based on incorrect keywords specific to each individual case. A partial empty matrix was created with select data from responses consisting of the keyword count total, incorrect keyword total, correct keyword percentage, incorrect keyword percentage, and overall keyword usage denoted by either a 0 (did not) or a 1 (did use).\u003c/p\u003e \u003cp\u003eThe data was organized into a train-test split ⅝ - ⅜ prepared at random using 405 entries for training (203 NYHA II, 106 NYHA III, 96 NYHA IV) and 243 entries for testing (129 NYHA II, 57 NYHA III, 57 NYHA IV). To further expand our training set, bootstrap sampling or sample selection with replacement was conducted 10 times, once merged a total of 4,050 entries were generated (2080 NYHA II, 1065 NYHA III, 905 NYHA IV). Cross validation was also used, this resampling method utilizes repeated training set splits. Training data was split into 5 groups, 4 were used for training and 1 was withheld for testing. Each model was then evaluated on the testing group, the model\u0026rsquo;s score was maintained, but the model was discarded to allow for continuous improvement. Each data sample was assigned a group and remained in that group for the process.\u003c/p\u003e \u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eRandom Forest Model Parameters\u003c/h2\u003e \u003cp\u003eThe random forest was set to a node size of 1, the minimum size of terminal nodes. Importance, a metric assessment of the contribution of a variable to the overall model, was set to true. Proximity, a measure of closeness between two similar cases, was set to true. The model was set to generate 150 trees. The number of bootstrap replications was set to 150. The minimum number of variables required for a node to attempt a split (minsplit) was set to 2 and any split that did not improve the fit was set to be pruned (complexity parameter). Cross validation was set to 5-fold with a minsplit of 2 and a complexity parameter of 0.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003eStatistical Analysis\u003c/h2\u003e \u003cp\u003eDiscussed data falls under the category of parametric data. Specifically, Poisson Distributed Variables, such as the correct word count range and wrong word count range; Binary Distributed Variables, such as the presence or lack of keywords; Categorical Nominal Variables, such as NYHA Classification. As appropriate, variables were summarized as mean and standard deviation (SD). Multiple comparisons were not made, thus corrections were not indicated. The calculation for misclassification rate is described as the sum of total false positives and total false negatives all divided by the total observations. Final model comparison was made by this misclassification metric alone. All percentages were rounded to three significant figures, while binary data was reported as assessed.\u003c/p\u003e \u003cp\u003eAnalyses were conducted using R-Studio 2024.12.0\u0026thinsp;+\u0026thinsp;467 and Python version 3.11.\u003c/p\u003e \u003c/div\u003e"},{"header":"Results","content":"\u003cp\u003e \u003cb\u003eKeywords\u003c/b\u003e \u003c/p\u003e \u003cp\u003e57 initial keywords were identified, 25 had zero variability indicating a lack of use by any student regardless of the case or intended classification. These 25 variables were dropped to reduce dimensionality as they didn\u0026rsquo;t provide any additional information for the models to improve upon. Additionally, highly correlated variables, such as the phrase \u0026ldquo;six months\u0026rdquo; were included as complete phrases and split on white-space as \u0026ldquo;six\u0026rdquo; and \u0026ldquo;months\u0026rdquo;. 11 initial keywords became 14 after the split for Myocarditis (NYHA II), 6 became 9 after the split for Familial Hypertrophic Cardiomyopathy (NYHA II), 7 became 10 with the split for Sarcoidosis Cardiomyopathy (NYHA III), and 8 became 10 with the split for Ischemic Cardiomyopathy (NYHA IV). In total, 32 individual keywords were assembled, 19 of them unique (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003e\u003cb\u003eKeyword List.\u003c/b\u003e All 32 keywords are listed, 19 unique keywords indicated by (*). \u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003e Keywords\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eColumn 2\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eColumn 3\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eColumn 4\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e*muscle aches\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003euncomfortable\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003esix\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eweeks\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eno sob\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e6\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003etwo months\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e*flutter\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e*marked\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e*unable\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e*cold\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e*chair\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e*increased\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ecomfortable\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e*cough\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e*worse\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e*gym\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e*dull\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e*pass out\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e*rest\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e*swelling\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003emonths\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003etwo\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e*most\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e*some\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003etwo weeks\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e*tired\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e*any\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003esix months\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e*activity\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003esob\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eFor those students who received a NYHA II case, the average correct keywords used were 8.05 out of the maximum 14 possible correct keywords, while the average incorrect were 1.24. In the NYHA III case, the average correct keywords were 8.53 out of 10, while the incorrect average was 0.816. For the NYHA IV case the average correct keywords were 7.73 out of 10, while the incorrect average was 0.662. Overall, the average correct keywords for all cases was 8.09 (SD +/- 2.84) and 0.995 (SD +/- .998) for incorrect words (Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e).\u003c/p\u003e\u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003e\u003cb\u003eKeyword Averages.\u003c/b\u003e The table demonstrates the average correct and incorrect keywords used by students by case. Totals are reported with a standard deviation.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCase\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAverage Correct Keywords\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAverage Incorrect Keywords\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNYHA II\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e8.05\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e1.24\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNYHA III\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e8.53\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.816\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNYHA IV\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e7.73\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0.662\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTotal\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e8.09 (+/- 2.84)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e.995 (+/- .998)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eA variable importance plot (VIP) was generated to demonstrate the importance of a given variable in regard to the overall integrity of the model; in other words, removing a variable from the model would result in an appreciable proportion of observations to be misclassified (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). Figure\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003ea demonstrates the Mean Decrease Accuracy plot, which expresses the accuracy loss suffered by the model by excluding a particular variable. The correct word variable is the most important factor for Mean Decrease Accuracy. Similarly, Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003eb demonstrates the Mean Decrease Gini plot, which is a representation of the per variable contribution to node homogeneity, a measure of node purity. Here, the keyword \u0026ldquo;months\u0026rdquo; is most important for Mean Decrease Gini, likely due to the artificially assigned symptom duration within each of the set cases. These variables are presented in descending importance from top to bottom in each graph.\u003c/p\u003e\n\u003ch3\u003eRandom Forests\u003c/h3\u003e\n\u003cp\u003eThe generated Random Forest model consisted of 150 trees. Traditionally, increasing the trees in the forest imbues the model with more accuracy; however, the mean squared error in our model stabilizes around 50 and 150 (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). The lines in the figure represent each of the NYHA Classifications: red represents NYHA II, green NYHA III, blue NYHA IV, and black represents the data used strictly for testing. During training the model reported a 0% error rate in the identification of NYHA II cases, 1.01% error rate for NYHA III, and 0.552% error rate for NYHA IV (Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). When the model was assessed on the test data we found a 0.775% error rate for NYHA II, 26.3% for NYHA III, and 6.90% for NYHA IV (Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e).\u003c/p\u003e \n\u003cp\u003e\u003cstrong\u003eTable 3: Random Forests Training Set Output.\u0026nbsp;\u003c/strong\u003eThe table demonstrates the expected counts for each case in comparison to the count determined by the Random Forests model using the training data set.\u003c/p\u003e\n\u003ctable style=\"border-collapse: collapse;border: none;width: 625px;\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:1.3in;border:solid #284E3F 1.0pt;border-right: solid #356854 1.0pt;background:#356854;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style=\"color:black;\"\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#F6F8F9;'\u003eNYHA Classification\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:solid #284E3F 1.0pt;border-left: none;border-bottom:solid #284E3F 1.0pt;border-right:solid #356854 1.0pt;background:#356854;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:white;'\u003eNYHA II Determined\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:solid #284E3F 1.0pt;border-left: none;border-bottom:solid #284E3F 1.0pt;border-right:solid #356854 1.0pt;background:#356854;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:white;'\u003eNYHA III Determined\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:solid #284E3F 1.0pt;border-left: none;border-bottom:solid #284E3F 1.0pt;border-right:solid #356854 1.0pt;background:#356854;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:white;'\u003eNYHA IV Determined\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border:solid #284E3F 1.0pt;border-left: none;background:#356854;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:white;'\u003eError\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:solid #284E3F 1.0pt;border-bottom:solid #356854 1.0pt;border-right:solid #356854 1.0pt;background:#356854;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#F6F8F9;'\u003eNYHA II Expected\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:none;border-bottom:solid #F6F8F9 1.0pt;border-right:solid white 1.0pt;background:white;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#434343;'\u003e2080\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:none;border-bottom:solid #F6F8F9 1.0pt;border-right:solid white 1.0pt;background:white;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#434343;'\u003e0\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:none;border-bottom:solid #F6F8F9 1.0pt;border-right:solid white 1.0pt;background:white;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#434343;'\u003e0\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:none;border-bottom:solid #F6F8F9 1.0pt;border-right:solid #284E3F 1.0pt;background:white;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#434343;'\u003e0\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:solid #284E3F 1.0pt;border-bottom:solid #356854 1.0pt;border-right:solid #356854 1.0pt;background:#356854;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#F6F8F9;'\u003eNYHA III Expected\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:none;border-bottom:solid #F6F8F9 1.0pt;border-right:solid #F6F8F9 1.0pt;background:#F6F8F9;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#434343;'\u003e0\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:none;border-bottom:solid #F6F8F9 1.0pt;border-right:solid #F6F8F9 1.0pt;background:#F6F8F9;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#434343;'\u003e1054\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:none;border-bottom:solid #F6F8F9 1.0pt;border-right:solid #F6F8F9 1.0pt;background:#F6F8F9;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:red;'\u003e11\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:none;border-bottom:solid #F6F8F9 1.0pt;border-right:solid #284E3F 1.0pt;background:#F6F8F9;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#434343;'\u003e0.0101\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:solid #284E3F 1.0pt;border-bottom:solid #284E3F 1.0pt;border-right:solid #356854 1.0pt;background:#356854;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#F6F8F9;'\u003eNYHA IV Expected\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:none;border-bottom:solid #284E3F 1.0pt;border-right:solid white 1.0pt;background:white;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#434343;'\u003e0\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:none;border-bottom:solid #284E3F 1.0pt;border-right:solid white 1.0pt;background:white;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:red;'\u003e8\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:none;border-bottom:solid #284E3F 1.0pt;border-right:solid white 1.0pt;background:white;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#434343;'\u003e897\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:none;border-bottom: solid #284E3F 1.0pt;border-right:solid #284E3F 1.0pt;background:white;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:right;line-height:115%;'\u003e\u003cspan style=\"color:black;\"\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#434343;'\u003e0.00552\u003c/span\u003e\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u003cstrong\u003eTable 4: Random Forests Test Set Output.\u0026nbsp;\u003c/strong\u003eThe table demonstrates the expected counts for each case in comparison to the count determined by the Random Forests model using the testing data set.\u003c/p\u003e\n\u003ctable style=\"border-collapse: collapse;border: none;width: 625px;\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:1.3in;border:solid #284E3F 1.0pt;border-right: solid #356854 1.0pt;background:#356854;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#F6F8F9;'\u003eNYHA Classification\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:solid #284E3F 1.0pt;border-left: none;border-bottom:solid #284E3F 1.0pt;border-right:solid #356854 1.0pt;background: #356854;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:white;'\u003eNYHA II Determined\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:solid #284E3F 1.0pt;border-left: none;border-bottom:solid #284E3F 1.0pt;border-right:solid #356854 1.0pt;background: #356854;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:white;'\u003eNYHA III Determined\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:solid #284E3F 1.0pt;border-left: none;border-bottom:solid #284E3F 1.0pt;border-right:solid #356854 1.0pt;background: #356854;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:white;'\u003eNYHA IV Determined\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border:solid #284E3F 1.0pt;border-left: none;background:#356854;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:white;'\u003eError\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:solid #284E3F 1.0pt;border-bottom:solid #356854 1.0pt;border-right:solid #356854 1.0pt;background:#356854;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#F6F8F9;'\u003eNYHA II Expected\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:none;border-bottom:solid #F6F8F9 1.0pt;border-right:solid white 1.0pt;background:white;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#434343;'\u003e126\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:none;border-bottom:solid #F6F8F9 1.0pt;border-right:solid white 1.0pt;background:white;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:red;'\u003e3\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:none;border-bottom:solid #F6F8F9 1.0pt;border-right:solid white 1.0pt;background:white;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#434343;'\u003e0\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:none;border-bottom:solid #F6F8F9 1.0pt;border-right:solid #284E3F 1.0pt;background:white;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#434343;'\u003e0.00775\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:solid #284E3F 1.0pt;border-bottom:solid #356854 1.0pt;border-right:solid #356854 1.0pt;background:#356854;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#F6F8F9;'\u003eNYHA III Expected\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:none;border-bottom:solid #F6F8F9 1.0pt;border-right:solid #F6F8F9 1.0pt;background:#F6F8F9;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:red;'\u003e12\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:none;border-bottom:solid #F6F8F9 1.0pt;border-right:solid #F6F8F9 1.0pt;background:#F6F8F9;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#434343;'\u003e45\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:none;border-bottom:solid #F6F8F9 1.0pt;border-right:solid #F6F8F9 1.0pt;background:#F6F8F9;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#434343;'\u003e0\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:none;border-bottom:solid #F6F8F9 1.0pt;border-right:solid #284E3F 1.0pt;background:#F6F8F9;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#434343;'\u003e0.263\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:solid #284E3F 1.0pt;border-bottom:solid #284E3F 1.0pt;border-right:solid #356854 1.0pt;background:#356854;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#F6F8F9;'\u003eNYHA IV Expected\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:none;border-bottom:solid #284E3F 1.0pt;border-right:solid white 1.0pt;background:white;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:red;'\u003e5\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:none;border-bottom:solid #284E3F 1.0pt;border-right:solid white 1.0pt;background:white;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#434343;'\u003e0\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:none;border-bottom:solid #284E3F 1.0pt;border-right:solid white 1.0pt;background:white;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#434343;'\u003e52\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width:1.3in;border-top:none;border-left:none;border-bottom:solid #284E3F 1.0pt;border-right:solid #284E3F 1.0pt;background:white;padding:2.0pt 6.0pt 2.0pt 6.0pt;height:15.75pt;\"\u003e\n \u003cp style='margin-top:0in;margin-right:0in;margin-bottom:0in;margin-left:0in;font-size:11.0pt;font-family:\"Calibri\",sans-serif;text-align:center;line-height:115%;'\u003e\u003cspan style='font-size:13px;line-height:115%;font-family:\"Times New Roman\",serif;color:#434343;'\u003e0.0690\u003c/span\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e \u003cp\u003eFigure \u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e demonstrates a multidimensional scaling proximities plot, while Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e is focused on the convergence point of these axes. The plots illustrate model performance regarding the testing data set. From the graph we can see that there are three distinct lines representing each of the three classification groups, spaced equally in the XYZ coordinate plane, thus equally spaced within three-dimensions. The spatial distance amongst the groups indicates that the Random Forest model has clearly identified the NYHA Classifications. This model performed accurately based solely on the keywords used by students, correct word percentage, and wrong word percentage. There is slight overlap between groups, which indicates misclassification, demonstrated in the zoomed Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e. There are observations from other classifications that bleed into the other group clusters; however, this is diminutive in scale. The final model demonstrated a 0.420% misclassification rate on the bootstrap training set and an 8.20% misclassification rate on the testing set.\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eThis study led to the development of a 150-tree random forest model trained and tested on the identification of NYHA Classifications II-IV in the unstructured textual data of medical student HPI entries as opposed to relying on structured data [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. Overall the model performed well with an 8.20% misclassification rate. The study began with lexical analysis, which developed a set of keywords most associated with determining the correct NYHA Classification (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). The significance here being the environment in which this model is used. In the medical education setting we develop the scenario in which students extract pre-written information regarding the presented case, these pre-written scenarios hinge on the use of very targeted keywords. The students then report their findings in a structured SOAP note format, with much of their effort being placed in the HPI. This unstructured section is meant to be a clinical summary portraying the relevant details for the current illness and may be comparable to the verbal presentation one may give to a medical team.\u003c/p\u003e \u003cp\u003eThe performance of our model is in-line with many of the similarly assessed random forest models in the literature; however, we do find the model to disproportionately struggle with the classification of NYHA III cases [\u003cspan additionalcitationids=\"CR14\" citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. Interestingly, this difficulty in delineating NYHA II and III cases has been previously documented [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e]. In addition to relying on similar keywords to delineate between NYHA II and III cases, the inadequacies of the model may be due to a paucity of training data, as noted previously 203 of the original cases were NYHA II, 106 NYHA III, and just 96 NYHA IV. Many of the studies that reported better accuracy either included a larger initial dataset or a larger fold cross validation [\u003cspan additionalcitationids=\"CR14\" citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eDetermining the optimal tree count is an important aspect of balancing the computational strain in utilizing a random forest model and directly affects accuracy. As demonstrated in Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e, a maximum default of 150 trees was used in our study; however, we clearly appreciated a similar result with a forest of 50\u0026ndash;75 trees. This reduction to half or less trees would result in a significant diminution in the computational costs of the model [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. Additionally, expanding the number of trees increases the amount of variables used to make determinations by the model [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. This composited use of the available variables juxtaposes our goal of creating a model that effectively uses the most essential variables for determinations.\u003c/p\u003e \u003cp\u003eKeywords are a critical aspect of creating random forest models targeted at deciphering unstructured textual data; however, in the traditional sense these are solely used to aid a random forest model in making its determinations. Our study begins to assess the use of these keywords in each case, potentially allowing for the assessment of efficient and accurate portrayals of medical information otherwise known as medical decision making [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. Similar computational models have been used in the academic setting to support standardized scoring platforms for medical education [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. However, none have been poised to provide lexical feedback to students.\u003c/p\u003e \u003cp\u003eHere, we demonstrated an average of 8.09 correct keywords in comparison to 0.995 incorrect keywords used by students throughout the cases. This demonstrates a tendency for the students to accurately identify keywords (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e) during the case, while self-assigning importance to the keywords prior to typing their HPIs. The students are likely demonstrating an ability to surmise the veiled VIP (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e) through their studies, preparation materials, and the overall standardized patient visit. Notably, the development of a VIP could assist educators in developing a more lexically-based assessment system that gives weighted credit to the use of keywords with a higher mean decrease accuracy. As we know the inclusion of keywords with a high mean decrease accuracy increases our model accuracy, thus likely maximizing the effectiveness of the medical case presentation.\u003c/p\u003e \u003cp\u003eThe limitations of the study stem from the initially limited sample size, unbalanced distribution of NYHA Classification cases, and the lack of NYHA I cases. Additionally, the equipment available to our study team was rudimentary, limiting our bootstrap iterations to 10 and our fold cross validation to 5. Improvements in our available equipment would have further increased the training and testing sample sizes.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eRandom forest models can be applied with acceptable accuracy in the determination of NYHA Classifications based solely on the free text HPI from medical student standardized patient simulated cases. Our model demonstrated a comparably low 0.420% and 8.20% misclassification rate, for training data and testing data respectively. These scores would likely improve with an increased volume of training cases and utilization of more purpose-built computational equipment. Continued improvement of this model would comfortably supplement an educator in assessing medical students on their performance in simulated cases. Future implementations of these findings may lead to the development of an instantaneous feedback system for students to receive constructive critique regarding their medical decision making and overall medical lexical density.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003e \u003cb\u003eAdditional Information\u003c/b\u003e \u003c/h2\u003e \u003cp\u003e \u003cstrong\u003eCompeting Interests\u003c/strong\u003e \u003cp\u003eThe authors declare no competing interests.\u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eI.R.P. and T.D. conceived the study. I.R.P., T.D., F.A.R., J.L., and K.G. collected and produced the data. T.D. generated the model and performed the statistical analysis. I.R.P. wrote the manuscript. I.R.P. and T.D. prepared the figures and tables. I.R.P., T.D., F.A.R., J.L., and K.G. critically revised and approved the final manuscript.\u003c/p\u003e\u003ch2\u003eAcknowledgement\u003c/h2\u003e\u003cp\u003eThe authors would like to show special gratitude to the Edward Via College of Osteopathic Medicine Center for Simulation and Technology for supporting the completion of this research with their equipment.\u003c/p\u003e\u003ch2\u003eData Availability\u003c/h2\u003e\u003cp\u003eThe textual data analysed during the study are available in the following GitHub repository: https://github.com/tDaniels1701/Random-Forest-HPI-Classification.git\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eZhang, R. et al. Discovering and identifying new york heart association classification from electronic health records. \u003cem\u003eBMC Medical Informatics and Decision Making. \u003c/em\u003e\u003cstrong\u003e18\u003c/strong\u003e, (2018).\u003c/li\u003e\n\u003cli\u003eAmir, A. et al. AALIM: a cardiac clinical decision support system powered by advanced multi-modal analytics. \u003cem\u003eStudies in health technology and informatics.\u003c/em\u003e\u003cstrong\u003e160 Pt 2\u003c/strong\u003e, 846-50 (2010).\u003c/li\u003e\n\u003cli\u003eElbattah, M., Arnaud, \u0026Eacute;., Gignon, M., \u0026amp; Dequen, G. The role of text analytics in healthcare: a review of recent developments and applications. \u003cem\u003eInternational Conference on Health Informatics.\u003c/em\u003e (2021). \u003c/li\u003e\n\u003cli\u003eKrittanawong, C., Zhang, H., Wang, Z., Aydar, M., \u0026amp; Kitai, T. Artificial intelligence in precision cardiovascular medicine. \u003cem\u003eJ Am Coll Cardiol. \u003c/em\u003e\u003cstrong\u003e69 21\u003c/strong\u003e, 2657-2664 (2017).\u003c/li\u003e\n\u003cli\u003eKrishnan, G.S., \u0026amp; Kamath, S.S. A supervised learning approach for icu mortality prediction based on unstructured electrocardiogram text reports. \u003cem\u003eInternational Conference on Applications of Natural Language to Data Bases.\u003c/em\u003e (2018). \u003c/li\u003e\n\u003cli\u003eSugamiya, Y., Otani, T., Nakadate, R., \u0026amp; Takanishi, A. Construction of automatic scoring system to support objective evaluation of clinical skills in medical education. \u003cem\u003e2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).\u003c/em\u003e 4177-4181 (2019). \u003c/li\u003e\n\u003cli\u003eChintalapudi, N., Battineni, G., Canio, M.D., Sagaro, G.G., \u0026amp; Amenta, F. Text mining with sentiment analysis on seafarers\u0026apos; medical documents. \u003cem\u003eInt. J. Inf. Manag. Data Insights.\u003c/em\u003e\u003cstrong\u003e1\u003c/strong\u003e, 100005 (2021).\u003c/li\u003e\n\u003cli\u003eHill, N.R. et al. Predicting atrial fibrillation in primary care using machine learning. \u003cem\u003ePLoS ONE.\u003c/em\u003e\u003cstrong\u003e14\u003c/strong\u003e, (2019).\u003c/li\u003e\n\u003cli\u003eSavalia, S., \u0026amp; Emamian, V. Cardiac arrhythmia classification by multi-layer perceptron and convolution neural networks. \u003cem\u003eBioengineering.\u003c/em\u003e\u003cstrong\u003e5\u003c/strong\u003e, (2018).\u003c/li\u003e\n\u003cli\u003eLi, J., Si, Y., Xu, T., \u0026amp; Jiang, S. Deep convolutional neural network based ecg classification system using information fusion and one-hot encoding techniques. \u003cem\u003eMathematical Problems in Engineering.\u003c/em\u003e\u003cstrong\u003e2018\u003c/strong\u003e, 7354081 (2018).\u003c/li\u003e\n\u003cli\u003eSutton, R. et al. An overview of clinical decision support systems: benefits, risks, and strategies for success. \u003cem\u003enpj Digit. Med.\u003c/em\u003e\u003cstrong\u003e3, \u003c/strong\u003e17 (2020).\u003c/li\u003e\n\u003cli\u003eCaraballo, C. et al. Clinical implications of the new york heart association classification. \u003cem\u003eJ Am Heart Assoc.\u003c/em\u003e\u003cstrong\u003e8 23\u003c/strong\u003e, (2019).\u003c/li\u003e\n\u003cli\u003eAdejumo, P. et al. Natural language processing of clinical documentation to assess functional status in patients with heart failure. \u003cem\u003eJAMA Netw Open.\u003c/em\u003e\u003cstrong\u003e7 11\u003c/strong\u003e, (2024).\u003c/li\u003e\n\u003cli\u003eZhang, R. et al. Automatic methods to extract new york heart association classification from clinical notes. \u003cem\u003eProceedings (IEEE Int Conf Bioinformatics Biomed).\u003c/em\u003e 1296-1299 (2017).\u003c/li\u003e\n\u003cli\u003eJandy, K., \u0026amp; Weichbroth, P. A machine learning approach to classifying new york heart association (NYHA) heart failure. \u003cem\u003eSci Rep. \u003c/em\u003e\u003cstrong\u003e14\u003c/strong\u003e, (2024).\u003c/li\u003e\n\u003cli\u003eRaphael, C. et al. Limitations of the new york heart association functional classification system and self-reported walking distances in chronic heart failure. \u003cem\u003eHeart.\u003c/em\u003e\u003cstrong\u003e93 4\u003c/strong\u003e, 476-482 (2006). \u003c/li\u003e\n\u003cli\u003eOshiro, T.M., Perez, P.S., \u0026amp; Baranauskas, J.A. How many trees in a random forest? \u003cem\u003eMachine Learning and Data Mining in Pattern Recognition (MLDM).\u003c/em\u003e\u003cstrong\u003e7376\u003c/strong\u003e, (2012).\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"New York Heart Association (NYHA) Heart Failure Classification, Random Forest, Machine Learning, Simulated Patient Encounters, Medical Education","lastPublishedDoi":"10.21203/rs.3.rs-6197186/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6197186/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eRandom forest models have demonstrated utility in the determination of New York Heart Association (NYHA) Heart Failure Classifications. This study aims to determine the prediction accuracy of a random forest model to derive NYHA Classification from medical students\u0026rsquo; free text history of present illness (HPI). NYHA Classifications established terminology for delineation of various heart failure presentations, this terminology was converted into keywords shared by standardized patients. 649 typed HPIs were de-identified, tokenized, cleaned, and assessed for number of correct keywords, incorrect keywords, and keyword usage. Models were trained using bootstrapped training data and assessed on test data. In testing, the model demonstrated a 0.775% error rate in identifying NYHA II, 26.3% for NYHA III, and 6.90% for NYHA IV. Overall reporting a 0.420% estimated error rate on the bootstrap sample training set and an 8.20% misclassification rate on the testing set. In future applications, developing a method of instantaneous feedback centered around keywords and their importance measures, specifically as determined by the variable importance plot (VIP), may aid students in their determination of NYHA Classifications and improve their lexical density.\u003c/p\u003e","manuscriptTitle":"Predicting New York Heart Association (NYHA) Heart Failure Classification from medical student notes following simulated patient encounters","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-03-26 09:07:52","doi":"10.21203/rs.3.rs-6197186/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-04-09T09:52:15+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-04-09T05:01:55+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-04-06T09:38:20+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"38963582764414382249906724444354314653","date":"2025-03-19T17:32:44+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"254529129014926375645673247442937583622","date":"2025-03-18T13:01:47+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-03-17T20:24:10+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-03-17T20:11:54+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2025-03-17T18:33:44+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-03-15T05:14:57+00:00","index":"","fulltext":""},{"type":"submitted","content":"Scientific Reports","date":"2025-03-10T15:57:56+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"17ff649a-4edf-4852-8d2a-15ca3265d217","owner":[],"postedDate":"March 26th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":45927947,"name":"Health sciences/Cardiology/Cardiovascular biology/Cardiovascular diseases/Heart failure"},{"id":45927948,"name":"Physical sciences/Mathematics and computing/Computer science"}],"tags":[],"updatedAt":"2025-07-21T16:00:58+00:00","versionOfRecord":{"articleIdentity":"rs-6197186","link":"https://doi.org/10.1038/s41598-025-10179-8","journal":{"identity":"scientific-reports","isVorOnly":false,"title":"Scientific Reports"},"publishedOn":"2025-07-15 15:57:27","publishedOnDateReadable":"July 15th, 2025"},"versionCreatedAt":"2025-03-26 09:07:52","video":"","vorDoi":"10.1038/s41598-025-10179-8","vorDoiUrl":"https://doi.org/10.1038/s41598-025-10179-8","workflowStages":[]},"version":"v1","identity":"rs-6197186","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6197186","identity":"rs-6197186","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}