Predicting Annotation Yield in Artificial Intelligence-Ranked Electronic Health Record Cohorts: A Regression-Based Framework for Efficient Manual Review

Predicting Annotation Yield in Artificial Intelligence-Ranked Electronic Health Record Cohorts: A Regression-Based Framework for Efficient Manual Review | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Predicting Annotation Yield in Artificial Intelligence-Ranked Electronic Health Record Cohorts: A Regression-Based Framework for Efficient Manual Review Assaf Landschaft, Leena Abdelmoity, Fatemeh Mohammad Alizadeh Chafjiri, and 3 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-6966149/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 31 Oct, 2025 Read the published version in BMC Medical Informatics and Decision Making → Version 1 posted 12 You are reading this latest preprint version Abstract Background Painstaking manual chart review of EHRs is still the chief bottleneck in retrospective studies, especially when rare-disease cohorts demand high specificity. Automated NLP rankers help, yet when trained on dated data they leave teams guessing how long to keep reviewing charts. We therefore present a regression-based ‘screening-saturation’ model that predicts residual yield at every point along the ranked list. Methods Leveraging a previously validated SVM that ranks notes for pediatric status epilepticus, we trained four predictive models: linear, polynomial, and support-vector regressions plus a lightweight neural net, on notes from 2013 and tested them on data from 2020. Our target was the proportion of true positives (ESE or RSE) expected below any score threshold. Results Polynomial regression offered the best balance of generalizability and interpretability, which demonstrated a strong predictive performance even under temporal data shifts. Regression outputs were used to simulate stopping rules for manual review, which captured 80% of positives after reviewing just 16.6% of notes (an 83% workload cut). Conclusion Our scalable, model-agnostic framework turns AI scores into actionable staffing decisions in clinical workflows. This screening-saturation model integrates with clinician-in-the-loop tools and adapts readily across medical domains that need lean chart review. Electronic Health Records (EHRs) Natural Language Processing (NLP) Retrospective cohort studies Status epilepticus Machine learning Annotation efficiency Yield prediction Artificial Intelligence (AI) Clinician in the Loop (CITL) Figures Figure 1 Figure 2 Figure 3 Background Retrospective cohort studies that analyze patient charts from electronic health records (EHRs) become increasingly critical to generate real-world evidence in neurology and other medical fields [ 1 – 3 ]. Since these patient charts often contain detailed clinical reasoning and symptom descriptions that are not captured in structured fields, they are crucial for the identification of complex or rare patient phenotypes. However, an accurate identification of the cases that meet strict inclusion criteria frequently requires extensive manual review of free-text notes, particularly when the goal is to ensure high specificity in cohort definition. Along with being time-consuming, this manual process is both costly and labor-intensive. A potential solution to the problem of classifying clinical texts in retrospective research is to use advanced natural language processing (NLP) models [ 4 – 6 ]. Yet, despite demonstrating strong performance, many of these models have limitations in sensitivity and specificity, and thus fall short of replacing human judgment in high-stakes clinical phenotyping tasks. This is particularly the case with rare conditions where research teams must rely on manual review as a confirmatory step for AI/NLP-ranked candidate records and where misclassification can cause a significant bias in findings [ 7 – 8 ]. Furthermore, dependency on manual validation creates a bottleneck in large-scale studies, particularly when it is unclear how long the review should continue. In the absence of principled stopping criteria, researchers typically adopt heuristic approaches, such as reviewing a fixed number of top-ranked cases until a perceived plateau in positive cases is reached [ 9 ] or stopping because of time or personnel constraints [ 10 ]. However, an important limitation of these ad hoc strategies is that they are not only non-reproducible, but also can lead to inefficiencies, either prematurely halting the review, and thus missing valuable cases, or unnecessarily extending the review beyond the point of diminishing returns. Another limitation of NLP models is that their performance is frequently evaluated under the implicit assumption that the data distribution remains stable over time, which rarely holds in clinical practice. Although previous studies validated models using random splits from a single dataset [ 11 ], such internal validation overlooks the reality that, due to changes in EHR systems, staff turnover, institutional policies, and population shifts, clinical documentation continuously evolves. Accordingly, when NLP models trained on earlier datasets are applied to newer records, these distributional shifts can lead to substantial performance degradation [ 12 – 13 ]. More specifically, when deployed on future datasets, standard classification metrics, such as sensitivity and specificity, calculated on historical data tend to fail to reflect a model’s actual performance. This concern is particularly relevant in clinical machine learning, where even small changes in documentation styles or coding practices can undermine model generalizability. For instance, in a recent systematic review, Guo et al. [ 14 ] found that temporal shifts frequently impact both calibration and discrimination of clinical prediction models, which highlights the need for regular re-evaluation and updating to maintain performance over time. In another study, the same research group demonstrated that a sepsis prediction model trained on the data from 2008–2010 experienced a marked decline in AUROC (Area Under the Receiver Operating Characteristic curve) when tested on the data from 2017–2019 [ 15 ], thus underscoring the risks associated with using static models in temporally dynamic environments. The aforementioned temporal fragility bears important implications for the use of NLP models in long-term cohort identification projects, where performance degradation over time can silently increase the number of missed cases, thereby reintroducing manual review burdens that these models intend to reduce. As a result, research teams face a fundamental planning challenge, namely, how to anticipate the actual utility of an NLP classifier trained on historical data when applied prospectively, and how to allocate manual review effort accordingly. This challenge is particularly salient when manual annotation budgets are limited and when the diminishing returns of reviewing lower-ranked records are unknown in advance. Indeed, when using EHR-based classifiers trained on historical data, it becomes increasingly challenging to estimate the amount of manual annotation required to achieve a desired level of case capture in newer datasets. Standard metrics such as precision, sensitivity, or AUROC calculated at the time of model development often fail to accurately reflect real-world utility at the time of deployment. As a result, long-term cohort identification projects must grapple with growing uncertainty around how much manual review is enough, and how much is too much. As a result, researchers may either overcommit effort to screening low-yield records ranked towards the bottom of the score distribution or prematurely terminate the review, which leads to a substantial fraction of true positive cases. What is missing to address this dilemma is a systematic method to estimate the residual yield, i.e., the number or proportion of positive cases that is likely to remain below a given classifier score threshold. Such a method would support more informed planning decisions, such as whether to continue using an older model despite its degradation, how much review effort is needed to reach a target capture rate (e.g., 80% of true positives), as well as when retraining a new model becomes necessary. To address this gap in the literature, this study develops and evaluates a predictive modeling framework that estimates the marginal annotation yield at each point along a relevance-ranked list of clinical notes. Specifically, we use historical manual review data from 2013 to train regression models that predict the proportion of remaining positive cases, namely, Established Status Epilepticus (ESE) or Refractory Status Epilepticus (RSE) beneath any given record in a 2020 test set, sorted by classifier score. The proposed approach provides a quantitative, model-agnostic tool to forecast yield during manual review and supports more efficient stopping strategies under real-world, temporally shifted conditions. Methods Data We used retrospective clinical data from Boston Children’s Hospital (BCH), extracted from the institutional EHR system (Cerner PowerChart) and stored in the BCH360 data warehouse. The dataset comprised over two million unstructured clinical notes spanning emergency, inpatient, and neurology-specific documentation recorded between 2013 and 2024. To evaluate model generalizability under real-world conditions of temporal dataset shift, we selected two non-consecutive years: 2013 for training and 2020 for testing. This approach is consistent with best practices in machine learning for simulating prospective deployment, as it exposes models to distributional drift and reduces the risk of overestimating performance due to temporal overlap or annotation homogeneity. All notes were processed using the Document Review Tool (DrT), a Clinician-in-the-Loop (CITL) NLP platform designed to support the identification of complex phenotypes within unstructured EHR text. Previously applied in a variety of clinical research settings [ 16 – 20 ], DrT has demonstrated adaptability across diverse phenotyping tasks. Specifically, prior applications included detecting invasive bacterial infections in pediatric emergency care [ 16 ], identifying cerebrospinal fluid pleocytosis in pediatric status epilepticus [ 17 ], analyzing cervical adenopathy to predict suppurative lymphadenitis [ 18 ], and monitoring safety events using NLP-based surveillance [ 19 ]. More recently, DrT has supported studies of febrile illness with focal musculoskeletal symptoms [ 20 ]. Together, these implementations reflect DrT’s generalizability and practical utility in real-world cohort discovery workflows [ 21 ]. In the present study, we used a validated version of DrT specifically trained to rank notes by likelihood of convulsive status epilepticus (SE) based on an SVM classifier with bag-of-n-grams features. In two previous peer-reviewed publications [ 22 , 23 ], this SE-focused model demonstrated high sensitivity in identifying pediatric patients with established and refractory SE and was found to outperform traditional manual review standards. These prior validations support the tool’s reliability in prioritizing high-yield records for manual chart review, thus forming the methodological foundation for the present study. Clinical phenotypes: ESE and RSE In this study, we focused on two major clinical phenotypes: (1) ESE, defined as seizure activity unresponsive to benzodiazepine administration, but terminated following a single non-benzodiazepine anti-seizure medication (ASM) and (2) RSE, characterized by seizures persisting after both a benzodiazepine and at least one non-benzodiazepine ASM or requiring ASM continuous infusion therapy. These definitions follow operational criteria established by the International League Against Epilepsy (ILAE) and are used in the pSERG (Pediatric Status Epilepticus Research Group) consortium’s prospective registry [ 24 ]. Scoring system: ML score and classification columns As previously described elsewhere [ 22 , 23 ], DrT assigns each note a machine-learning (ML) score based on a bag-of-n-grams model trained with an SVM. This model was previously trained on annotated data from 2017–2019, using Minimum Redundancy Maximum Relevance (MRMR) to select the top 500 most informative n-gram features [ 25 ]. Specifically, the ML score quantifies the likelihood that a note contains evidence of an SE event. The following annotations were used: Score Column . A continuous value derived from the SVM classifier, with higher values indicating a higher likelihood of SE. Score2 Column . Manual annotation categories included the following annotations: ESE – Established status epilepticus RSE – Refractory status epilepticus SE(NO) – Convulsive seizures not meeting ESE or RSE criteria NO – No SE event Excluded – Does not meet inclusion criteria Ineligible – Outside the defined age range (1 month–21 years) These annotations were applied to the highest-scoring note per Case Serial Number (CSN). The classification was then propagated automatically to other notes within the same CSN group to reduce the review burden. Category2 Column : This binary field was derived from the manual Score2 Column classification described above to facilitate predictive modeling. Specifically, the notes labeled ESE or RSE were mapped to the “positives” class, as they met inclusion criteria. All other manually reviewed notes (SE(NO), NO, Excluded, Ineligible) were mapped to the “negatives” class. Finally, the notes that inherited their classification automatically from another note via CSN (i.e., were not manually reviewed) were assigned the value “Automatic” in Category 2. To ensure that only manually verified labels contributed to model development and performance estimates, the aforementioned records were excluded from subsequent training and evaluation analyses. Document types To maximize clinical relevance and minimize noise, we filtered the notes based on document types that were most likely to contain SE-related content. Included note types included: Neurology and ICU medical documentation: “Emergency MD,” “Neurology Admission MD,” “Neurology Consultation,” “Neurology Inpatient MD,” “ICU Progress MD”; Nursing documentation: “Inpatient Nursing,” “ICU Process Nursing”; Critical care: “MSICU Admission MD,” “MSICU Progress MD,” and “MSICU Progress Attending MD.” Fig. 1 shows categorical heat maps of the distribution of classification outcomes (Category2 values) across the 2013 and 2020 datasets (A and B, respectively), sorted in descending order by the machine learning model’s assigned score. With a decrease in scores, a clear decline in green (positive) cases was observed, demonstrating the model’s prioritization effectiveness. This visualization confirms the utility of the model in frontloading high-yield records for efficient manual review. In 2013, the dataset included a total of 362,525 clinical notes, of which the 12,976 highest-scoring records were manually reviewed using the DrT interface. In 2020, a total of 397,956 notes were available, and 7,636 top-ranked notes underwent manual annotation. To simulate a realistic planning scenario in which a previously trained model was applied to new data, we trained the predictive yield model on the annotated 2013 dataset and prospectively evaluated it on the independently annotated 2020 data. In the data analysis for both years, reviewers employed a pragmatic yield-based stopping rule, namely, terminating annotation as soon as the discovery rate of new positive cases, defined as ESE or RSE, dropped to near zero. This approach is common in retrospective studies of rare conditions, where exhaustive review is infeasible and positive cases are sparse [ 31 ]. Under such conditions, traditional evaluation metrics such as sensitivity and specificity fail to reflect the diminishing marginal utility of continued annotation effort and thus become less actionable in terms of guiding review decisions [ 26 ]. An alternative approach increasingly adopted in both systematic reviews [ 27 , 28 ] and clinical NLP workflows [ 29 – 31 ] is to use saturation-based heuristics that provide a practical guide to stop annotation when further review is unlikely to add value Figure 1 shows that, on the same level of machine learning assigned scores, the year 2020 has a lower proportion of true positive annotations. To formally evaluate this observation, we employed non-parametric and rank-sensitive techniques. To illustrate the difference in the proportions of positive cases in the early-ranked observations, Fig. 2 shows cumulative gains curves [ 32 ] for both years. Each curve consists of the proportion of positive cases up to that rank. As can be seen in Fig. 2 , there is a systematically lower proportion of positive cases in the top-ranked instances, with a pronounced difference in the earliest instances. These cases are most important for triage and downstream medical review. To evaluate whether this difference is due to chance or represents a systematic phenomenon, we employed a permutation test [ 33 ] on the first 1000 cases. The p -value for this test was 0.0457, suggesting that there is indeed a significantly lower early sensitivity for the year 2020. Importantly, the 2013 and 2020 annotations were performed by different reviewers, a design choice made to enhance external validity of the evaluation. Real-world retrospective studies frequently span multiple years and teams, thus introducing unavoidable variation in annotation style and consistency (e.g., [ 34 ]). By exposing our regression-based stopping framework to temporally separated data and reviewer differences, we simulated the conditions where clinical NLP models are typically deployed. Previous studies highlighted how annotation yield declines steeply in ranked-review settings [ 28 , 31 ] and how stopping decisions are often informal or ad hoc [ 26 – 31 ]. Our method formalizes this intuition, thus offering a reproducible and principled model for determining when to stop manual review. Model development and evaluation To model the relationship between the machine learning-assigned score (Score) and the estimated remaining annotation yield (i.e., the likelihood of encountering additional positive cases beyond a given record), we trained the following four predictive regression models: (1) Linear Regression; (2) Polynomial Regression; (3) Support Vector Regression (SVR); and (4) a custom neural network implemented in PyTorch. All models were developed using Python (v3.10) with scikit-learn (v1.3) and PyTorch (v2.0). Data preparation All models were first trained on records from 2013, which was followed by their prospective evaluation on the records from 2020 to simulate a real-world environment, temporally shifting deployment scenarios. To ensure that the evaluation was limited to manually verified ground truth annotations, both datasets excluded notes classified as “automatic.” For each note, we computed the target variable as the positive ratio, defined as the proportion of remaining true positive records below a given score threshold. This was calculated using a custom function (see Notebook Cell 5). For each record, the following values were computed: Positive_ratio_above: Ratio of positives among all higher-ranked records; Positive_ratio_below: Ratio of positives among lower-ranked records, to be used as the regression target. The input feature used for modeling was the raw SVM score from the DrT tool (column Score). Model architectures Model architectures of the four predictive regression models were implemented as follows. First, we implemented Linear Regression using LinearRegression() from sklearn.linear_model. This model served as a baseline. Furthermore, Polynomial Regression was implemented as a pipeline combining PolynomialFeatures(degree = 2) and LinearRegression(). This enabled modeling non-linear relationships between the SVM score and yield. Next, using radial basis function (RBF) kernel, we implemented Support Vector Regression (SVR) via a pipeline with StandardScaler() and SVR(kernel='rbf'). Unless otherwise specified, hyperparameters were set to default (C = 1.0, epsilon = 0.1). Finally, regarding PyTorch Neural Network Regressor, a lightweight, fully connected feedforward neural network was implemented as a scikit-learn compatible wrapper (PyTorchRegressor). The corresponding model architecture consisted of (i) input layer (1 feature); (ii) hidden layer 1 (32 neurons, ReLU activation); (iii) hidden layer 2 (16 neurons, ReLU activation); and (iv) output layer (1 neuron, linear activation). Training was performed using the Adam optimizer (learning_rate = 0.001) and mean squared error loss over 50 epochs with a batch size of 32. All input features were standardized using StandardScaler. Training and evaluation protocol All models were trained using the 2013 data (n = 12976 records, mean SVM Score = -7920695.42, 95% confidence interval = [-7985784.17, -7855606.66]) and tested on the 2020 data (n = 7636, mean SVM Score = -7892003.17, 95% confidence interval = [-7987498.28, -7796508.07] records). To simulate a realistic review scenario, the 2020 dataset was sorted in descending order by the model-assigned score. Evaluation was performed by predicting the expected positive ratio below for each record in the 2020 test set. We computed the Mean Squared Error (MSE), which measures the average squared difference between predicted and actual positive ratios, and R² Score, which indicates the proportion of variance in the target variable explained by the model. These metrics were computed using sklearn.metrics.mean_squared_error and r2_score. Model performance is summarized in Table 1 (Section 3). Implementation details The analyses were conducted in Python 3.10 using libraries: scikit-learn 1.3, PyTorch 2.0, pandas, numpy, matplotlib, and seaborn. The models were trained and evaluated on a standard CPU-based environment (Intel i7, 16GB RAM). Finally, the set for reproducibility using np.random.seed(42) and torch.manual_seed(42) was employed. Results Model performance on training data (2013) We first evaluated each model’s ability to estimate annotation yield on the training data from 2013 using 5-fold cross-validation. To reflect precision and explained variance, the models were evaluated using MSE and the R² coefficient of determination. The results showed that, among all models, the PyTorch Neural Network model achieved the lowest MSE and the highest R² score, which clearly shows the model’s superior ability to capture the underlying patterns in the data with high accuracy and generalization. The Linear Regression model also performed exceptionally well, demonstrating a strong balance between predictive accuracy and consistency, as reflected in its low MSE and high R² score. By contrast, however, the Polynomial Regression model showed moderate predictive power, but was prone to higher variability, suggesting potential overfitting to the training data. Finally, the Support Vector Regression model delivered relatively poor performance, with both a high MSE and a significantly lower R² score, indicating this model’s limited capacity to capture the complexity of the data (see Table 1). Table 1 Cross-validated and test performance of models (Mean Squared Error (MSE) and R²) Model Cross-Validated MSE (2013) R² (2013) Linear Regression 0.000272 0.979970 Polynomial Regression (deg=2) 0.000540 0.961412 Support Vector Regression (RBF) 0.006234 0.538412 Deep Neural Network (PyTorch) 0.000040 0.997249 Model Test MSE (2020) R² (2020) Linear Regression 0.001287 0.935698 Polynomial Regression 0.000890 0.955544 Support Vector Regression (RBF) 0.009390 0.530947 Deep Neural Network (PyTorch) 0.002347 0.882763 Abbreviations: RBF = radial basis function Prospective generalization to temporal test data (2020) As a next step, to simulate real-world deployment under conditions reflecting potential shifts in population characteristics and workflow dynamics, all models were evaluated on held-out 2020 data. The results revealed that, despite an overall decline in performance due to temporal drift, the Polynomial Regression model achieved the lowest MSE and a high R² score, suggesting this model’s strong resilience to evolving data distributions. The Linear Regression model also performed competitively, maintaining a robust balance between simplicity and generalization under temporal shifts. However, the PyTorch Neural Network model experienced a notable decline in performance, with a higher MSE and a lower R² compared to its training performance. This outcome suggests that the model may be overly complex for the task at stake or could have overfitted during training, which should have limited its ability to adapt to unseen data. Finally, the Support Vector Regression model exhibited the weakest performance, with a significantly higher MSE and lower R², highlighting its challenges in handling real-world variability (see Table 1). Correlation of predictors with target yield We computed Pearson correlation coefficients between the input score and both outcome metrics (positive_ratio_above and positive_ratio_below) on the 2020 data. The score-to-yield correlations amounted to 0.94; 95%-confidence-interval: [0.918, 0.955] and 0.98; 95%-confidence,interval: [0.977, 0.982], respectively. This finding confirms a strong positive correlation: namely, with an increase in the score, the yield metrics also increase. This finding also supports our design rationale behind the score-ranked screening. Saturation-guided review efficiency To evaluate the practical utility of the four regression models, we simulated a stopping rule based on predicted positive_ratio_below values. The stopping point was defined as the first record in the score-ranked list where the predicted marginal yield (i.e., expected proportion of remaining positive cases) fell below a predefined threshold. In our analysis, this threshold was set at 20%. Subsequently, we measured (i) the total number of records reviewed before reaching this point; and (ii) the total number of positive cases captured up to that point, relative to all positives in the dataset. When the deep neural network regressor was used, out of 7636 total manually labeled records in the 2020 dataset, the review could be terminated after approximately 1268 records. This threshold-based early stopping captured over 79.6% of all positive cases, thus substantially front-loading the high-yield segment of the record list (see Table 2). Table 2 Stopping rule performance: Records reviewed and positives captured Stopping Rule N reviewed records % reviewed dataset % captured positives Threshold positive_ratio_below < 0.20 1268 16.6% 79.6% These results demonstrate that the screening saturation model can significantly reduce manual workload, saving approximately 6,368 annotations (83.4%) while still capturing most relevant cases (Fig. 3). This targeted stopping approach is particularly valuable in resource-constrained environments because it improves the efficiency of retrospective cohort studies without compromising cohort quality. The visualization in Fig. 3 enables researchers to define their yield-based stopping thresholds and adapt manual review strategies to time, budget, or statistical power constraints. Discussion Although retrospective cohort studies using EHRs are a cornerstone of real-world evidence generation, their manual review remains a major bottleneck, particularly when high specificity is required. To address this concern, the present study proposes a data-driven “screening saturation” framework that estimates the marginal annotation yield for AI-prioritized records. Regression models can accurately approximate the expected return, thereby enabling more efficient and transparent stopping decisions. Across different model types, we observed strong correlations between model-assigned relevance scores and annotation yield, which confirmed feasibility of learning a predictive mapping for stopping rule optimization. However, we also found that the models’ ability to generalize across temporal shifts in the data substantially varied. Specifically, while the neural network achieved the highest performance during training, it exhibited a notable decline when tested on the temporally shifted 2020 data, which could have been due to overfitting. By contrast, the Polynomial Regression model demonstrated the most robust performance, achieving the best balance of generalization and accuracy in the test set. These findings suggest that, in contrast to more complex architectures, their lightweight, interpretable counterparts may offer a more stable performance in real-world, non-stationary environments. Importantly, we also observed that yield-aware stopping rules based on these models can considerably reduce the manual annotation burden. Specifically, using a 20% yield threshold, we were able to successfully terminate the review process after examining only 16.6% of the dataset, while still capturing nearly 80% of all relevant cases. This substantial efficiency gain underscores the value of saturation modeling in resource-constrained settings. Therefore, compared to traditional stopping methods such as fixed quotas or yield plateaus, the proposed approach offers a principled, quantitative framework that aligns review effort with expected informational return. By enabling prospective planning and auditability in manual review processes, the proposed framework provides strong support to clinical research teams with limited time or staffing. Finally, via prospectively validating the models on unseen temporal data, our results also highlighted the challenges posed by distributional drift in clinical AI pipelines. Specifically, changes in documentation styles, case mix, and underlying EHR systems can significantly alter score distributions and case density, thereby reinforcing the need for adaptive, yield-based tools that can accommodate such shifts. Implications for practice and system design The proposed screening saturation framework has several important practical implications for the development and deployment of CITL systems in clinical research and quality improvement initiatives. In contrast to traditional manual review pipelines that frequently rely on arbitrary quotas, fixed-time efforts, or plateau-based heuristics to determine when to stop reviewing AI-ranked records, the proposed method introduces a quantifiable and transparent mechanism to stop the review process based on predicted marginal yield, thereby aligning human effort more precisely with expected benefit. Furthermore, embedding yield-aware stopping logic into clinical NLP systems, such as the DrT tool used in the present study, could allow researchers to visualize predicted annotation saturation in real time. For instance, users may receive system-generated alerts when a predefined low-yield threshold is reached or interactively adjust it based on evolving resource constraints. This would shift manual review from a fixed-scope task to a dynamically adaptive process, offering greater control over effort-quality trade-offs. By accelerating cohort construction and reducing annotation burden, such adaptive review workflows can reduce the time from data extraction to evidence generation, thereby enabling faster dissemination of research findings that will ultimately inform clinical practice and improve patient care. Similarly, from the workflow perspective, integrating regression-based saturation models into annotation platforms could support several practical use cases, including: Real-time monitoring dashboards to inform reviewers how many high-yield records remain based on current model projections. Adaptive review planning, allowing study coordinators to accurately forecast labor requirements and estimate marginal gains from reviewing additional cases. Auditable decision trails, where review stopping points can be justified using explicit model predictions, rather than subjective judgment. Model retraining pipelines, where the framework could help to select the most informative examples for manual labeling under active learning scenarios. Another important characteristic of the proposed regression-based framework that it is classifier-agnostic, meaning that, regardless of the specific architecture used, it can be deployed atop any base model that ranks records by relevance. While, in the present study, we used an SVM trained on n-gram features, the proposed saturation framework could easily integrate with more advanced transformer-based classifiers or hybrid rule-based systems. This compatibility promotes a broader applicability of the proposed framework across different clinical domains and annotation goals. Furthermore, by helping low-resource teams achieve acceptable cohort quality without extensive manual labor, the proposed saturation-guided review could contribute to more equitable research practices. In global health contexts or small-scale clinical studies with limited staffing, such models could maximize the return on scarce annotation resources, thus democratizing access to high-quality retrospective research. Finally, by allowing for user-defined trade-offs between effort and yield, the proposed approach supports tailored review strategies, for instance, stopping at a higher threshold (e.g., 30% remaining yield) in exploratory studies or at a lower threshold (e.g., 10%) in confirmatory analyses. Limitations The proposed screening saturation framework has several limitations. First, accuracy of the regression-based yield predictions inherently depends on quality, calibration, and stability of the underlying ranking model, in this case, the SVM-based classifier embedded in the DrT tool. If the base classifier assigns misleading scores due to miscalibration, distributional drift, or suboptimal feature representation, the saturation model’s predictions may become unreliable. Accordingly, it is essential to validate and update the base model regularly. Second, while the proposed framework assumes that manually reviewed records, particularly those at the top of the ranking, are exhaustively and correctly labeled in real-world scenarios with multiple annotators or variable reviewer experience, label noise or inconsistencies in annotation completeness may compromise fidelity of the training signal and adversely affect accuracy of the yield prediction model. Third, in the present study, we conducted a prospective evaluation using temporally shifted data (2013 and 2020) from the same institution. Yet, differences in documentation culture, clinical terminology, EHR platforms, and annotation protocols may introduce heterogeneity that could reduce generalizability. This warrants future research that would test the performance of the proposed framework in multi-institutional or multilingual settings, along with external validation across diverse sites and specialties. Fourth, in the current implementation, we used only the raw model-assigned score as input to the regression model. While this simplification aids interpretability and ease of integration, it may limit predictive precision. Accordingly, including additional contextual features (e.g., note type, department, or reviewer-specific factors) in future studies could enhance model accuracy and robustness. Fifth, our study focused solely on a binary positive-vs-negative yield prediction. Yet, in more nuanced annotation tasks, yield may not be adequately captured by simple binary outcomes. Accordingly, to better reflect clinical ambiguity and annotation effort, future implementations may need to incorporate richer label hierarchies or uncertainty measures. Sixth, considering that the saturation framework is currently retrospective and offline, although the results support its integration into real-time or interactive systems, additional software engineering and user testing would be needed to validate its usability, interpretability, and impact on end-user decision-making in live annotation settings. Finally, considering that the proposed framework focuses on yield prediction, rather than on classification performance, it does not produce standard metrics such as precision, recall, or specificity. This may limit its adoption in contexts where decision makers are more familiar with traditional model evaluation metrics. However, the present design choice reflects the framework’s major intent of guiding resource-efficient manual review, rather than automating case classification. Future research directions In further research, the proposed framework may be extended in several directions. First, regarding threshold optimization, future studies could use data-driven or Bayesian approaches to dynamically tune the stopping threshold based on cost-benefit trade-offs. Another meaningful future research path is multivariate modeling, where yield predictions could be further improved by incorporating metadata such as document type, reviewer experience, or visit context. Third, considering the novel opportunities for richer ranking functions afforded by large language models (LLMs), the proposed method could layer atop LLM outputs to guide a more efficient review. Fourth, to better reflect clinical ambiguity and support more nuanced annotation tasks, future implementations may extend beyond binary yield prediction to accommodate multi-class outcomes or incorporate uncertainty estimates. Finally, embedding saturation prediction into live annotation platforms could enable adaptive review workflows and immediate cost tracking. Conclusion Manual review of unstructured clinical notes remains a major barrier to scalable retrospective cohort identification, particularly in contexts requiring high specificity. While NLP models can rank candidate records for review, the actual cost and yield of manual validation remains difficult to forecast, particularly when models are applied to newer datasets. In the present study, we addressed this challenge by introducing a regression-based framework that predicts annotation yield at each point in a relevance-ranked list of EHR records. Our approach supports data-driven stopping rules that align review effort with expected benefit. Using manually annotated notes from 2013 to train predictive models and prospectively evaluating performance of these models on the 2020 data, we found that yield prediction is feasible and strongly correlates with model-assigned scores, even in the presence of temporal distribution shifts. Among the tested models, polynomial regression was found to offer the best balance of generalization and interpretability, while the neural network, despite its strong in-sample performance, proved to be more vulnerable to temporal overfitting. These findings highlight that the proposed screening saturation framework offers a practical way to manage the resource burden of manual review under conditions of model decay and uncertainty. By estimating the marginal return on continued review, the proposed framework enables tailored stopping decisions, for example, to identify 80% of positive cases while reviewing only a fraction of the dataset. The proposed approach moves beyond fixed or heuristic stopping rules, supporting a more transparent, reproducible, and resource-sensitive cohort construction. Finally, since the proposed method is classifier-agnostic, it can be integrated into various CITL NLP systems and applied meaningfully across multiple clinical domains. As retrospective studies increasingly rely on AI-assisted workflows, the proposed framework may prove essential for making the corresponding workflows more efficient, scalable, and responsive to real-world constraints. Abbreviations EHR Electronic Health Record NLP Natural Language Processing SE Status Epilepticus ESE Established Status Epilepticus RSE Refractory Status Epilepticus ASM Anti-Seizure Medication ML Machine Learning SVM Support Vector Machine MRMR Minimum Redundancy Maximum Relevance DrT Document Review Tool CSN Case Serial Number SVR Support Vector Regression MSE Mean Squared Error Declarations Acknowledgements This study was supported by the Epilepsy Research Fund. Author contributions T.L., A.L., F.M.A.C., M.A.P., L.A.: Study Design; A.L.: Conceptualization, Formal analysis, Software development, Data visualization; A.L., T. L.: Draft writing; F.M.A.C.: Data acquisition, Methodology development; F.M.A.C., T. L., L.A.: Project supervision; Personnel training; M.A.P., L.A.: Manual record review; T.L., J.V.G.: Clinical insights, Data interpretation; T.L.: Funding acquisition. All authors reviewed the manuscript. Funding This study was supported by the Epilepsy Research Fund. Data availability All code and de‑identified example data needed to reproduce the analysis are openly available at https://github.com/assaflandschaft/ehr-yield-prediction. The raw clinical notes contain protected health information and, under HIPAA regulations, cannot be shared. Competing interests The authors declare no competing interests. Ethics approval and consent to participate This retrospective study was conducted in accordance with the Declaration of Helsinki. The Boston Children’s Hospital Institutional Review Board (IRB‑ P00001207) approved the protocol and waived the requirement for written informed consent. Consent for publication Not applicable. References Kimia AA, Savova G, Landschaft A, Harper MB. An introduction to natural language processing: how you can get more from those electronic notes you are generating. Pediatr Emerg Care. 2015;31(7):536–41. Savova GK, Masanz JJ, Ogren, PV, Zheng J, Sohn S, Kipper-Schuler KC, et al. Mayo clinical text analysis and knowledge extraction system (cTAKES): architecture, component evaluation and applications. J Am Med Inform Assoc. 2010;17(5):507–13. Shickel B, Tighe PJ, Bihorac A, Rashidi P. Deep EHR: A survey of recent advances in deep learning techniques for electronic health record (EHR) analysis. IEEE J Biomed Health Inform. 2018; 22(5):1589–604. Veeranki SPK, Abdulnazar A, Kramer D, Kreuzthaler M, Benjaminet D. Multi-label text classification via secondary use of large clinical real-world data sets. Sci Rep. 2024;14, 26972. Xie Q, Chen Q, Chen A, Peng C, Hu Y, Lin F, et al. Medical foundation large language models for comprehensive text analysis and beyond. NPJ Digit Med. 2025;8:141. Spasic I, Nenadic G. Clinical text data in machine learning: systematic review. JMIR Med Inform. 2020;8(3): e17984. Walsh CG, Wilimitis D, Chen Q, Wright A, Kolli J, Robinson K, et al. Scalable incident detection via natural language processing and probabilistic language models. Sci Rep. 2024;14:23429. Yonamine S, Ma CJ, Alabi RO, Kaidonis G, Chan L, Borkar D, et al. Comparison of diagnosis codes to clinical notes in classifying patients with diabetic retinopathy. Ophthalmol Sci. 2024;4(6):100564. Berge GT, Granmo OC, Tveit TO, Ruthjersen AL, Sharmaet J. Combining unsupervised, supervised and rule-based learning: the case of detecting patient allergies in electronic health records. BMC Med Inform Decis Mak. 2023;23:188. Natukunda A, Muchene LK. Unsupervised title and abstract screening for systematic review: a retrospective case-study using topic modelling methodology. Syst Rev. 2023;12:1. Arbet J, Brokamp C, Meinzen-Derr J, Trinkley KE, Spratt HM. Lessons and tips for designing a machine learning study using EHR data. J Clin Transl Sci. 2020;5(1):e21. Zhou H, Chen Y, Lipton Z. Evaluating model performance in medical datasets over time. Proceedings of the Conference on Health, Inference, and Learning 2023;209:498–508. Guo LL, Steinberg E, Fleming SL, Posada J, Lemmon J, Pfohl SR, et al. EHR foundation models improve robustness in the presence of temporal distribution shift. Sci Rep. 2023;13:3767. Guo LL, Pfohl SR, Fries J, Posada J, Fleming SL, Aftandilian C, et al. Systematic review of approaches to preserve machine learning performance in the presence of temporal dataset shift in clinical medicine. Appl Clin Inform. 2021;12(4):808–15. Guo, LL, Pfohl SR, Fries J, Johnson AEW, Jose Posada J, Aftandilian C, et al . Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine. Sci Rep. 2022;12: 2726. Paydar-Darian N, Kimia AA, Monuteaux MC, Michelson KA, Landschaft A, Mauldenet AB, et al. C-reactive protein or erythrocyte sedimentation rate results reliably exclude invasive bacterial infections. Am J Emerg Med. 2019;37(8):1510–15. Johnson KB, Michelson KA, ∙ Lyons TW, Landschaft A, Loddenkemperb T, Kimiaet AA, et al. Pediatric status epilepticus: how common is cerebrospinal fluid pleocytosis in the absence of infection? Seizure. 2014;23(7):573–5. Kimia AA, Rudloe TF, Aprahamian N, McNamara J, Roberson D, Landschaft A, et al. Predictors of a drainable suppurative adenitis among children presenting with cervical adenopathy. Am J Emerg Med. 2019;37(1):109–13. Ozonoff A, Milliren CE, Fournier K, Welcher J, Landschaft A, Samnaliev M, et al. Electronic surveillance of patient safety events using natural language processing. Health Informatics J. 2022;28(4):146045822211324. El Helou R, Landschaft A, Harper MB, Kimia AA. Bacteremia in children with fever and acute lower extremity pain. Pediatrics. 2023;151(5):e2022059504. Kimia A, Landschaft A. Document review tools. www.documentreviewtools.com. Accessed June 23, 2025. Chafjiri FMA, Reece L, Voke L, Landschaft A, Clark J, Kimia AA, et al. Natural language processing for identification of refractory status epilepticus in children. Epilepsia. 2023;64(12):3227–37. Puckett MA, Chafjiri MA, Gettings JV, Landschaft A, Loddenkemper, T. Utilizing natural language processing to identify pediatric patients experiencing status epilepticus. Seizure. 2025;125:54–61. Trinka E, Cock H, Hesdorffer D, Rossetti AO, Scheffer IE, Shinnar S, et al. A definition and classification of status epilepticus - report of the ILAE Task Force on Classification of Status Epilepticus. Epilepsia. 2015;56(10):1515–23. Cavnar WB, Trenkle JM. N-gram-based text categorization. Proceedings of SDAIR-94, 3rd Annual Symposium on Document Analysis and Information Retrieval; 2021. p. 161–175. Yang E, Lewis DD, Frieder O. Heuristic stopping rules for technology-assisted review. Proceedings of the 21st ACM Symposium on Document Engineering (DocEng '21). Association for Computing Machinery, New York, NY, USA, Article 31; 2021. pp. 1–10. Stevenson M, Bin-Hezam R. Stopping methods for technology assisted reviews based on point processes. ACM Trans Inf Syst. 2024;42(3):1–37. Settles B. Active learning literature survey. Computer sciences technical report 1648, 2009. http://pages.cs.wisc.edu/~bsettles/pub/settles.activelearning.pdf. Accessed 23 June, 2025. Sun H, Xu C, Suominen H. Analyzing the granularity and cost of annotation in clinical sequence labeling. arXiv 2021. https://arxiv.org/abs/2108.09913. Accessed 23 June, 2025. Landschaft A, Antweiler D, Mackay S, Kugler S, Rüping S, Wrobel S, et al. Implementation and evaluation of an additional GPT-4-based reviewer in PRISMA-based medical systematic literature reviews. Int J Med Inform. 2024;189:105531. Cohen AM, Chamberlin S, Deloughery T, Nguyen M, Bedrick S, Meninger S, et al. Detecting rare diseases in electronic health records using machine learning and knowledge engineering: case study of acute hepatic porphyria. PloS One. 2020;15(7): e0235574. Manning CD, Raghavan P, Schütze H. Introduction to information retrieval. Cambridge: Cambridge University Press; 2008. Ernst MD. Permutation methods: a basis for exact inference. Statist Sci. 2004;19(4):676–85. Miñarro-Giménez JA, Cornet R, Jaulent MC, Dewenter H, Thun S, Gøeg KR, et al. Quantitative analysis of manual annotation of clinical text samples. Int J Med Inform. 2019;123:37–48. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 31 Oct, 2025 Read the published version in BMC Medical Informatics and Decision Making → Version 1 posted Editorial decision: Revision requested 13 Aug, 2025 Reviews received at journal 06 Aug, 2025 Reviews received at journal 04 Aug, 2025 Reviewers agreed at journal 27 Jul, 2025 Reviews received at journal 20 Jul, 2025 Reviewers agreed at journal 18 Jul, 2025 Reviewers agreed at journal 17 Jul, 2025 Reviewers invited by journal 17 Jul, 2025 Editor assigned by journal 17 Jul, 2025 Editor invited by journal 08 Jul, 2025 Submission checks completed at journal 07 Jul, 2025 First submitted to journal 07 Jul, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-6966149","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":488156511,"identity":"6a969a91-422d-4f4d-9336-93e3c3332a57","order_by":0,"name":"Assaf Landschaft","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABbUlEQVRIie2SP0vDQBTAXwhclytZL0Sar3AhkFZo6Ve5UqhLEEEoHUQLQroUXTPU75BVcHil0CyxXW+SFqFThYAgikW8WttSEdHNIb/hOO693713fwAyMv4rLSipUYAOUAADKOCEfATIZ4IAtbhDAmytuMxsK0X8SeGooj8pxmU8TRGYXcwdTB+PTvipG9/2UTTvCkUDyeTlZmAXO0KbPARrhUnfZUpxrrtz1wqHnHnJoUAxOnb3Q5FzurOB00smunO1UUBSsFJgWiR9olOyYJ6kHGuBqEUSCKPY0EImiJXfKPY4vn9WVaqRPJjp9I0zN1wpZ9EYibnARlUpudetwlF4y8bUnsLT8wFnnK0UoULEoliuLavoW8WRvldCzupRos6Sv+BmmPgqeSScSNYCaw/L9ZBOz83eaK0UVGMSW+VKFKsbo0/cMDqJm6ZNYfPxYGjOkVXCXL2fzpu7D8O/+xMAWvvrJCMjIyPjd7wDSoyMbctm3I4AAAAASUVORK5CYII=","orcid":"","institution":"Boston Children's Hospital","correspondingAuthor":true,"prefix":"","firstName":"Assaf","middleName":"","lastName":"Landschaft","suffix":""},{"id":488156515,"identity":"e23f1c9d-fd84-49fd-ae23-99215537eca8","order_by":1,"name":"Leena Abdelmoity","email":"","orcid":"","institution":"Boston Children's Hospital","correspondingAuthor":false,"prefix":"","firstName":"Leena","middleName":"","lastName":"Abdelmoity","suffix":""},{"id":488156517,"identity":"f8b1f9fb-7735-4e99-a6ba-c56d8712c063","order_by":2,"name":"Fatemeh Mohammad Alizadeh Chafjiri","email":"","orcid":"","institution":"Boston Children's Hospital","correspondingAuthor":false,"prefix":"","firstName":"Fatemeh","middleName":"Mohammad Alizadeh","lastName":"Chafjiri","suffix":""},{"id":488156519,"identity":"2293f2b3-b74b-488b-989d-70713b74e99a","order_by":3,"name":"Molly Ann Puckett","email":"","orcid":"","institution":"Boston Children's Hospital","correspondingAuthor":false,"prefix":"","firstName":"Molly","middleName":"Ann","lastName":"Puckett","suffix":""},{"id":488156522,"identity":"7f45d3a2-60b2-43ff-83af-004c763d99be","order_by":4,"name":"Jennifer Gettings","email":"","orcid":"","institution":"Boston Children's Hospital","correspondingAuthor":false,"prefix":"","firstName":"Jennifer","middleName":"","lastName":"Gettings","suffix":""},{"id":488156524,"identity":"fc66629c-f73b-4868-ab11-aabbd5b87e88","order_by":5,"name":"Tobias Loddenkemper","email":"","orcid":"","institution":"Boston Children's Hospital","correspondingAuthor":false,"prefix":"","firstName":"Tobias","middleName":"","lastName":"Loddenkemper","suffix":""}],"badges":[],"createdAt":"2025-06-24 13:23:11","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-6966149/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-6966149/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1186/s12911-025-03241-y","type":"published","date":"2025-10-31T15:58:14+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":87381080,"identity":"426c6955-9b3d-4a9c-a695-b41fb6f6c3c6","added_by":"auto","created_at":"2025-07-23 08:36:13","extension":"jpeg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":432114,"visible":true,"origin":"","legend":"\u003cp\u003eCategorical heat map of record classification in 2013 (A) and 2020 (B) datasets. Each rectangle represents a single reviewed or unreviewed record. Color coding: Green = Manually reviewed records classified as positive (ESE or RSE); Red = Manually reviewed records classified as negative (SE(NO), NO, Ineligible, Excluded, Question); Gray = Automatic label propagation from manually reviewed notes within the same admission (CSN-mapped), not used for analysis; White = Empty records (no data). Rows aggregate 100 sequential records (e.g., row 1 = records 1–100, row 2 = 101–200, etc.), with only the top 50 rows shown. The y-axis reflects the range of ML scores for the records within each row, with higher model-assigned scores positioned towards the top. Abbreviations: CSN = Case Serial Number, ESE = Established status epilepticus, RSE = Refractory status epilepticus, SE = status epilepticus.\u003c/p\u003e","description":"","filename":"floatimage1.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-6966149/v1/38eb6f38c8ddec3db108579c.jpeg"},{"id":87381072,"identity":"3808411b-75f6-4622-9346-e92fc1135ba4","added_by":"auto","created_at":"2025-07-23 08:36:13","extension":"jpeg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":132436,"visible":true,"origin":"","legend":"\u003cp\u003eCumulative gain curves (Precision@K) for 2013 and 2020 ranked predictions. Across all ranks, the 2020 instances produce lower yields of true positives.\u003c/p\u003e","description":"","filename":"floatimage2.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-6966149/v1/d10ce043ea197aca4c86685c.jpeg"},{"id":87381075,"identity":"b9899f53-9974-495f-806c-cd6324fa3cb4","added_by":"auto","created_at":"2025-07-23 08:36:13","extension":"jpeg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":123706,"visible":true,"origin":"","legend":"\u003cp\u003eSaturation curve visualization. A scatter plot displays positive_ratio_belowacross sequential records (ranked by model-assigned score). The x-axis shows record index, while the y-axis shows the predicted positive_ratio_below. A horizontal line at the 0.20 threshold intersects the curve, marking the recommended stopping point.\u003c/p\u003e","description":"","filename":"floatimage3.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-6966149/v1/74480e3b085dca9bb20a767b.jpeg"},{"id":95041185,"identity":"4834bf1f-61c2-423c-ac3f-e10b25a3f1ba","added_by":"auto","created_at":"2025-11-03 16:10:54","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1453693,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-6966149/v1/11ae5c67-626a-41eb-ad9e-69434a1e6834.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Predicting Annotation Yield in Artificial Intelligence-Ranked Electronic Health Record Cohorts: A Regression-Based Framework for Efficient Manual Review","fulltext":[{"header":"Background","content":"\u003cp\u003eRetrospective cohort studies that analyze patient charts from electronic health records (EHRs) become increasingly critical to generate real-world evidence in neurology and other medical fields [\u003cspan additionalcitationids=\"CR2\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e–\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. Since these patient charts often contain detailed clinical reasoning and symptom descriptions that are not captured in structured fields, they are crucial for the identification of complex or rare patient phenotypes. However, an accurate identification of the cases that meet strict inclusion criteria frequently requires extensive manual review of free-text notes, particularly when the goal is to ensure high specificity in cohort definition. Along with being time-consuming, this manual process is both costly and labor-intensive.\u003c/p\u003e\u003cp\u003eA potential solution to the problem of classifying clinical texts in retrospective research is to use advanced natural language processing (NLP) models [\u003cspan additionalcitationids=\"CR5\" citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e–\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]. Yet, despite demonstrating strong performance, many of these models have limitations in sensitivity and specificity, and thus fall short of replacing human judgment in high-stakes clinical phenotyping tasks. This is particularly the case with rare conditions where research teams must rely on manual review as a confirmatory step for AI/NLP-ranked candidate records and where misclassification can cause a significant bias in findings [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e–\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e].\u003c/p\u003e\u003cp\u003eFurthermore, dependency on manual validation creates a bottleneck in large-scale studies, particularly when it is unclear how long the review should continue. In the absence of principled stopping criteria, researchers typically adopt heuristic approaches, such as reviewing a fixed number of top-ranked cases until a perceived plateau in positive cases is reached [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e] or stopping because of time or personnel constraints [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. However, an important limitation of these ad hoc strategies is that they are not only non-reproducible, but also can lead to inefficiencies, either prematurely halting the review, and thus missing valuable cases, or unnecessarily extending the review beyond the point of diminishing returns.\u003c/p\u003e\u003cp\u003eAnother limitation of NLP models is that their performance is frequently evaluated under the implicit assumption that the data distribution remains stable over time, which rarely holds in clinical practice. Although previous studies validated models using random splits from a single dataset [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e], such internal validation overlooks the reality that, due to changes in EHR systems, staff turnover, institutional policies, and population shifts, clinical documentation continuously evolves. Accordingly, when NLP models trained on earlier datasets are applied to newer records, these distributional shifts can lead to substantial performance degradation [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e–\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e].\u003c/p\u003e\u003cp\u003eMore specifically, when deployed on future datasets, standard classification metrics, such as sensitivity and specificity, calculated on historical data tend to fail to reflect a model’s actual performance. This concern is particularly relevant in clinical machine learning, where even small changes in documentation styles or coding practices can undermine model generalizability. For instance, in a recent systematic review, Guo et al. [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e] found that temporal shifts frequently impact both calibration and discrimination of clinical prediction models, which highlights the need for regular re-evaluation and updating to maintain performance over time. In another study, the same research group demonstrated that a sepsis prediction model trained on the data from 2008–2010 experienced a marked decline in AUROC (Area Under the Receiver Operating Characteristic curve) when tested on the data from 2017–2019 [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e], thus underscoring the risks associated with using static models in temporally dynamic environments.\u003c/p\u003e\u003cp\u003eThe aforementioned temporal fragility bears important implications for the use of NLP models in long-term cohort identification projects, where performance degradation over time can silently increase the number of missed cases, thereby reintroducing manual review burdens that these models intend to reduce. As a result, research teams face a fundamental planning challenge, namely, how to anticipate the actual utility of an NLP classifier trained on historical data when applied prospectively, and how to allocate manual review effort accordingly. This challenge is particularly salient when manual annotation budgets are limited and when the diminishing returns of reviewing lower-ranked records are unknown in advance.\u003c/p\u003e\u003cp\u003eIndeed, when using EHR-based classifiers trained on historical data, it becomes increasingly challenging to estimate the amount of manual annotation required to achieve a desired level of case capture in newer datasets. Standard metrics such as precision, sensitivity, or AUROC calculated at the time of model development often fail to accurately reflect real-world utility at the time of deployment. As a result, long-term cohort identification projects must grapple with growing uncertainty around how much manual review is enough, and how much is too much. As a result, researchers may either overcommit effort to screening low-yield records ranked towards the bottom of the score distribution or prematurely terminate the review, which leads to a substantial fraction of true positive cases.\u003c/p\u003e\u003cp\u003eWhat is missing to address this dilemma is a systematic method to estimate the residual yield, i.e., the number or proportion of positive cases that is likely to remain below a given classifier score threshold. Such a method would support more informed planning decisions, such as whether to continue using an older model despite its degradation, how much review effort is needed to reach a target capture rate (e.g., 80% of true positives), as well as when retraining a new model becomes necessary.\u003c/p\u003e\u003cp\u003eTo address this gap in the literature, this study develops and evaluates a predictive modeling framework that estimates the marginal annotation yield at each point along a relevance-ranked list of clinical notes. Specifically, we use historical manual review data from 2013 to train regression models that predict the proportion of remaining positive cases, namely, Established Status Epilepticus (ESE) or Refractory Status Epilepticus (RSE) beneath any given record in a 2020 test set, sorted by classifier score. The proposed approach provides a quantitative, model-agnostic tool to forecast yield during manual review and supports more efficient stopping strategies under real-world, temporally shifted conditions.\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003e\u003cb\u003eData\u003c/b\u003e\u003c/p\u003e\u003cp\u003eWe used retrospective clinical data from Boston Children’s Hospital (BCH), extracted from the institutional EHR system (Cerner PowerChart) and stored in the BCH360 data warehouse. The dataset comprised over two million unstructured clinical notes spanning emergency, inpatient, and neurology-specific documentation recorded between 2013 and 2024. To evaluate model generalizability under real-world conditions of temporal dataset shift, we selected two non-consecutive years: 2013 for training and 2020 for testing. This approach is consistent with best practices in machine learning for simulating prospective deployment, as it exposes models to distributional drift and reduces the risk of overestimating performance due to temporal overlap or annotation homogeneity.\u003c/p\u003e\u003cp\u003eAll notes were processed using the Document Review Tool (DrT), a Clinician-in-the-Loop (CITL) NLP platform designed to support the identification of complex phenotypes within unstructured EHR text. Previously applied in a variety of clinical research settings [\u003cspan additionalcitationids=\"CR17 CR18 CR19\" citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e–\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e], DrT has demonstrated adaptability across diverse phenotyping tasks. Specifically, prior applications included detecting invasive bacterial infections in pediatric emergency care [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e], identifying cerebrospinal fluid pleocytosis in pediatric status epilepticus [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e], analyzing cervical adenopathy to predict suppurative lymphadenitis [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e], and monitoring safety events using NLP-based surveillance [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e]. More recently, DrT has supported studies of febrile illness with focal musculoskeletal symptoms [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]. Together, these implementations reflect DrT’s generalizability and practical utility in real-world cohort discovery workflows [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e].\u003c/p\u003e\u003cp\u003eIn the present study, we used a validated version of DrT specifically trained to rank notes by likelihood of convulsive status epilepticus (SE) based on an SVM classifier with bag-of-n-grams features. In two previous peer-reviewed publications [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e, \u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e], this SE-focused model demonstrated high sensitivity in identifying pediatric patients with established and refractory SE and was found to outperform traditional manual review standards. These prior validations support the tool’s reliability in prioritizing high-yield records for manual chart review, thus forming the methodological foundation for the present study.\u003c/p\u003e\u003cp\u003e\u003cb\u003eClinical phenotypes: ESE and RSE\u003c/b\u003e\u003c/p\u003e\u003cp\u003eIn this study, we focused on two major clinical phenotypes: (1) ESE, defined as seizure activity unresponsive to benzodiazepine administration, but terminated following a single non-benzodiazepine anti-seizure medication (ASM) and (2) RSE, characterized by seizures persisting after both a benzodiazepine and at least one non-benzodiazepine ASM or requiring ASM continuous infusion therapy. These definitions follow operational criteria established by the International League Against Epilepsy (ILAE) and are used in the pSERG (Pediatric Status Epilepticus Research Group) consortium’s prospective registry [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e].\u003c/p\u003e\u003cp\u003e\u003cb\u003eScoring system: ML score and classification columns\u003c/b\u003e\u003c/p\u003e\u003cp\u003eAs previously described elsewhere [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e, \u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e], DrT assigns each note a machine-learning (ML) score based on a bag-of-n-grams model trained with an SVM. This model was previously trained on annotated data from 2017–2019, using Minimum Redundancy Maximum Relevance (MRMR) to select the top 500 most informative n-gram features [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]. Specifically, the ML score quantifies the likelihood that a note contains evidence of an SE event. The following annotations were used:\u003c/p\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003e\u003cem\u003eScore Column\u003c/em\u003e. A continuous value derived from the SVM classifier, with higher values indicating a higher likelihood of SE.\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003e\u003cem\u003eScore2 Column\u003c/em\u003e. Manual annotation categories included the following annotations:\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eESE – Established status epilepticus\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eRSE – Refractory status epilepticus\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eSE(NO) – Convulsive seizures not meeting ESE or RSE criteria\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eNO – No SE event\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eExcluded – Does not meet inclusion criteria\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eIneligible – Outside the defined age range (1 month–21 years)\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003cp\u003e\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003cp\u003eThese annotations were applied to the highest-scoring note per Case Serial Number (CSN). The classification was then propagated automatically to other notes within the same CSN group to reduce the review burden.\u003c/p\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003e\u003cem\u003eCategory2 Column\u003c/em\u003e: This binary field was derived from the manual \u003cem\u003eScore2 Column\u003c/em\u003e classification described above to facilitate predictive modeling. Specifically, the notes labeled ESE or RSE were mapped to the “positives” class, as they met inclusion criteria. All other manually reviewed notes (SE(NO), NO, Excluded, Ineligible) were mapped to the “negatives” class. Finally, the notes that inherited their classification automatically from another note via CSN (i.e., were not manually reviewed) were assigned the value “Automatic” in Category 2. To ensure that only manually verified labels contributed to model development and performance estimates, the aforementioned records were excluded from subsequent training and evaluation analyses.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003cp\u003e\u003cb\u003eDocument types\u003c/b\u003e\u003c/p\u003e\u003cp\u003eTo maximize clinical relevance and minimize noise, we filtered the notes based on document types that were most likely to contain SE-related content. Included note types included:\u003c/p\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eNeurology and ICU medical documentation: “Emergency MD,” “Neurology Admission MD,” “Neurology Consultation,” “Neurology Inpatient MD,” “ICU Progress MD”;\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eNursing documentation: “Inpatient Nursing,” “ICU Process Nursing”;\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eCritical care: “MSICU Admission MD,” “MSICU Progress MD,” and “MSICU Progress Attending MD.”\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003cp\u003eFig. 1 shows categorical heat maps of the distribution of classification outcomes (Category2 values) across the 2013 and 2020 datasets (A and B, respectively), sorted in descending order by the machine learning model’s assigned score. With a decrease in scores, a clear decline in green (positive) cases was observed, demonstrating the model’s prioritization effectiveness. This visualization confirms the utility of the model in frontloading high-yield records for efficient manual review. \u003c/p\u003e\u003cp\u003eIn 2013, the dataset included a total of 362,525 clinical notes, of which the 12,976 highest-scoring records were manually reviewed using the DrT interface. In 2020, a total of 397,956 notes were available, and 7,636 top-ranked notes underwent manual annotation. To simulate a realistic planning scenario in which a previously trained model was applied to new data, we trained the predictive yield model on the annotated 2013 dataset and prospectively evaluated it on the independently annotated 2020 data. In the data analysis for both years, reviewers employed a pragmatic yield-based stopping rule, namely, terminating annotation as soon as the discovery rate of new positive cases, defined as ESE or RSE, dropped to near zero. This approach is common in retrospective studies of rare conditions, where exhaustive review is infeasible and positive cases are sparse [\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e]. Under such conditions, traditional evaluation metrics such as sensitivity and specificity fail to reflect the diminishing marginal utility of continued annotation effort and thus become less actionable in terms of guiding review decisions [\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e]. An alternative approach increasingly adopted in both systematic reviews [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e, \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e] and clinical NLP workflows [\u003cspan additionalcitationids=\"CR30\" citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e–\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e] is to use saturation-based heuristics that provide a practical guide to stop annotation when further review is unlikely to add value\u003c/p\u003e\u003cp\u003eFigure\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e shows that, on the same level of machine learning assigned scores, the year 2020 has a lower proportion of true positive annotations. To formally evaluate this observation, we employed non-parametric and rank-sensitive techniques. To illustrate the difference in the proportions of positive cases in the early-ranked observations, Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e shows cumulative gains curves [\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e] for both years. Each curve consists of the proportion of positive cases up to that rank. As can be seen in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e, there is a systematically lower proportion of positive cases in the top-ranked instances, with a pronounced difference in the earliest instances. These cases are most important for triage and downstream medical review. To evaluate whether this difference is due to chance or represents a systematic phenomenon, we employed a permutation test [\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e] on the first 1000 cases. The \u003cem\u003ep\u003c/em\u003e-value for this test was 0.0457, suggesting that there is indeed a significantly lower early sensitivity for the year 2020.\u003c/p\u003e\u003cp\u003eImportantly, the 2013 and 2020 annotations were performed by different reviewers, a design choice made to enhance external validity of the evaluation. Real-world retrospective studies frequently span multiple years and teams, thus introducing unavoidable variation in annotation style and consistency (e.g., [\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e]). By exposing our regression-based stopping framework to temporally separated data and reviewer differences, we simulated the conditions where clinical NLP models are typically deployed. Previous studies highlighted how annotation yield declines steeply in ranked-review settings [\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e, \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e] and how stopping decisions are often informal or ad hoc [\u003cspan additionalcitationids=\"CR27 CR28 CR29 CR30\" citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e–\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e]. Our method formalizes this intuition, thus offering a reproducible and principled model for determining when to stop manual review.\u003c/p\u003e\u003cp\u003e\u003cb\u003eModel development and evaluation\u003c/b\u003e\u003c/p\u003e\u003cp\u003eTo model the relationship between the machine learning-assigned score (Score) and the estimated remaining annotation yield (i.e., the likelihood of encountering additional positive cases beyond a given record), we trained the following four predictive regression models: (1) Linear Regression; (2) Polynomial Regression; (3) Support Vector Regression (SVR); and (4) a custom neural network implemented in PyTorch. All models were developed using Python (v3.10) with scikit-learn (v1.3) and PyTorch (v2.0).\u003c/p\u003e\u003cp\u003e\u003cem\u003eData preparation\u003c/em\u003e\u003c/p\u003e\u003cp\u003eAll models were first trained on records from 2013, which was followed by their prospective evaluation on the records from 2020 to simulate a real-world environment, temporally shifting deployment scenarios. To ensure that the evaluation was limited to manually verified ground truth annotations, both datasets excluded notes classified as “automatic.”\u003c/p\u003e\u003cp\u003eFor each note, we computed the target variable as the positive ratio, defined as the proportion of remaining true positive records below a given score threshold. This was calculated using a custom function (see Notebook Cell 5). For each record, the following values were computed:\u003c/p\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003ePositive_ratio_above: Ratio of positives among all higher-ranked records;\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003ePositive_ratio_below: Ratio of positives among lower-ranked records, to be used as the regression target.\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003cp\u003eThe input feature used for modeling was the raw SVM score from the DrT tool (column Score).\u003c/p\u003e\u003cp\u003e\u003cb\u003eModel architectures\u003c/b\u003e\u003c/p\u003e\u003cp\u003eModel architectures of the four predictive regression models were implemented as follows. First, we implemented Linear Regression using LinearRegression() from sklearn.linear_model. This model served as a baseline.\u003c/p\u003e\u003cp\u003eFurthermore, Polynomial Regression was implemented as a pipeline combining PolynomialFeatures(degree = 2) and LinearRegression(). This enabled modeling non-linear relationships between the SVM score and yield.\u003c/p\u003e\u003cp\u003eNext, using radial basis function (RBF) kernel, we implemented Support Vector Regression (SVR) via a pipeline with StandardScaler() and SVR(kernel='rbf'). Unless otherwise specified, hyperparameters were set to default (C = 1.0, epsilon = 0.1).\u003c/p\u003e\u003cp\u003eFinally, regarding PyTorch Neural Network Regressor, a lightweight, fully connected feedforward neural network was implemented as a scikit-learn compatible wrapper (PyTorchRegressor). The corresponding model architecture consisted of (i) input layer (1 feature); (ii) hidden layer 1 (32 neurons, ReLU activation); (iii) hidden layer 2 (16 neurons, ReLU activation); and (iv) output layer (1 neuron, linear activation). Training was performed using the Adam optimizer (learning_rate = 0.001) and mean squared error loss over 50 epochs with a batch size of 32. All input features were standardized using StandardScaler.\u003c/p\u003e\u003cp\u003e\u003cb\u003eTraining and evaluation protocol\u003c/b\u003e\u003c/p\u003e\u003cp\u003eAll models were trained using the 2013 data (n = 12976 records, mean SVM Score = -7920695.42, 95% confidence interval = [-7985784.17, -7855606.66]) and tested on the 2020 data (n = 7636, mean SVM Score = -7892003.17, 95% confidence interval = [-7987498.28, -7796508.07] records). To simulate a realistic review scenario, the 2020 dataset was sorted in descending order by the model-assigned score. Evaluation was performed by predicting the expected positive ratio below for each record in the 2020 test set.\u003c/p\u003e\u003cp\u003eWe computed the Mean Squared Error (MSE), which measures the average squared difference between predicted and actual positive ratios, and R² Score, which indicates the proportion of variance in the target variable explained by the model. These metrics were computed using sklearn.metrics.mean_squared_error and r2_score. Model performance is summarized in Table\u0026nbsp;1 (Section 3).\u003c/p\u003e\u003cp\u003e\u003cb\u003eImplementation details\u003c/b\u003e\u003c/p\u003e\u003cp\u003eThe analyses were conducted in Python 3.10 using libraries: scikit-learn 1.3, PyTorch 2.0, pandas, numpy, matplotlib, and seaborn. The models were trained and evaluated on a standard CPU-based environment (Intel i7, 16GB RAM). Finally, the set for reproducibility using np.random.seed(42) and torch.manual_seed(42) was employed.\u003c/p\u003e"},{"header":"Results","content":"\u003ch2\u003eModel performance on training data (2013)\u003c/h2\u003e\n\u003cp\u003eWe first evaluated each model\u0026rsquo;s ability to estimate annotation yield on the training data from 2013 using 5-fold cross-validation. To reflect precision and explained variance, the models were evaluated using MSE and the R\u0026sup2; coefficient of determination.\u003c/p\u003e\n\u003cp\u003eThe results showed that, among all models, the PyTorch Neural Network model achieved the lowest MSE and the highest R\u0026sup2; score, which clearly shows the model\u0026rsquo;s superior ability to capture the underlying patterns in the data with high accuracy and generalization. The Linear Regression model also performed exceptionally well, demonstrating a strong balance between predictive accuracy and consistency, as reflected in its low MSE and high R\u0026sup2; score. By contrast, however, the Polynomial Regression model showed moderate predictive power, but was prone to higher variability, suggesting potential overfitting to the training data. Finally, the Support Vector Regression model delivered relatively poor performance, with both a high MSE and a significantly lower R\u0026sup2; score, indicating this model\u0026rsquo;s limited capacity to capture the complexity of the data (see Table 1).\u003cbr\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable 1\u003c/strong\u003e Cross-validated and test performance of models (Mean Squared Error (MSE) and R\u0026sup2;)\u0026nbsp;\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"602\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 41.8605%;\"\u003e\n \u003cp\u003e\u003cstrong\u003eModel\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 42.5249%;\"\u003e\n \u003cp\u003e\u003cstrong\u003eCross-Validated MSE (2013)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 15.6146%;\"\u003e\n \u003cp\u003e\u003cstrong\u003eR\u0026sup2; (2013)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 41.8605%;\"\u003e\n \u003cp\u003eLinear Regression\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 42.5249%;\"\u003e\n \u003cp\u003e0.000272\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 15.6146%;\"\u003e\n \u003cp\u003e0.979970\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 41.8605%;\"\u003e\n \u003cp\u003ePolynomial Regression (deg=2)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 42.5249%;\"\u003e\n \u003cp\u003e0.000540\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 15.6146%;\"\u003e\n \u003cp\u003e0.961412\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 41.8605%;\"\u003e\n \u003cp\u003eSupport Vector Regression (RBF)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 42.5249%;\"\u003e\n \u003cp\u003e0.006234\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 15.6146%;\"\u003e\n \u003cp\u003e0.538412\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 41.8605%;\"\u003e\n \u003cp\u003eDeep Neural Network (PyTorch)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 42.5249%;\"\u003e\n \u003cp\u003e0.000040\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 15.6146%;\"\u003e\n \u003cp\u003e0.997249\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 41.8605%;\"\u003e\n \u003cp\u003e\u003cstrong\u003eModel\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 42.5249%;\"\u003e\n \u003cp\u003e\u003cstrong\u003eTest MSE (2020)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 15.6146%;\"\u003e\n \u003cp\u003e\u003cstrong\u003eR\u0026sup2; (2020)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 41.8605%;\"\u003e\n \u003cp\u003eLinear Regression\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 42.5249%;\"\u003e\n \u003cp\u003e0.001287\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 15.6146%;\"\u003e\n \u003cp\u003e0.935698\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 41.8605%;\"\u003e\n \u003cp\u003ePolynomial Regression\u0026nbsp;\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 42.5249%;\"\u003e\n \u003cp\u003e0.000890\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 15.6146%;\"\u003e\n \u003cp\u003e0.955544\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 41.8605%;\"\u003e\n \u003cp\u003eSupport Vector Regression (RBF)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 42.5249%;\"\u003e\n \u003cp\u003e0.009390\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 15.6146%;\"\u003e\n \u003cp\u003e0.530947\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 41.8605%;\"\u003e\n \u003cp\u003eDeep Neural Network (PyTorch)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 42.5249%;\"\u003e\n \u003cp\u003e0.002347\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 15.6146%;\"\u003e\n \u003cp\u003e0.882763\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eAbbreviations: RBF = radial basis function\u003c/p\u003e\n\u003ch2\u003eProspective generalization to temporal test data (2020)\u003c/h2\u003e\n\u003cp\u003eAs a next step, to simulate real-world deployment under conditions reflecting potential shifts in population characteristics and workflow dynamics, all models were evaluated on held-out 2020 data. The results revealed that, despite an overall decline in performance due to temporal drift, the Polynomial Regression model achieved the lowest MSE and a high R\u0026sup2; score, suggesting this model\u0026rsquo;s strong resilience to evolving data distributions. The Linear Regression model also performed competitively, maintaining a robust balance between simplicity and generalization under temporal shifts.\u003c/p\u003e\n\u003cp\u003eHowever, the PyTorch Neural Network model experienced a notable decline in performance, with a higher MSE and a lower R\u0026sup2; compared to its training performance. This outcome suggests that the model may be overly complex for the task at stake or could have overfitted during training, which should have limited its ability to adapt to unseen data. Finally, the Support Vector Regression model exhibited the weakest performance, with a significantly higher MSE and lower R\u0026sup2;, highlighting its challenges in handling real-world variability (see Table 1).\u003c/p\u003e\n\u003ch2\u003eCorrelation of predictors with target yield\u003c/h2\u003e\n\u003cp\u003eWe computed Pearson correlation coefficients between the input score and both outcome metrics (positive_ratio_above and positive_ratio_below) on the 2020 data. The score-to-yield correlations amounted to 0.94; 95%-confidence-interval: [0.918, 0.955] and 0.98; 95%-confidence,interval: [0.977, 0.982], respectively. This finding confirms a strong positive correlation: namely, with an increase in the score, the yield metrics also increase. This finding also supports our design rationale behind the score-ranked screening.\u003c/p\u003e\n\u003ch2\u003eSaturation-guided review efficiency\u003c/h2\u003e\n\u003cp\u003eTo evaluate the practical utility of the four regression models, we simulated a stopping rule based on predicted positive_ratio_below values. The stopping point was defined as the first record in the score-ranked list where the predicted marginal yield (i.e., expected proportion of remaining positive cases) fell below a predefined threshold. In our analysis, this threshold was set at 20%. Subsequently, we measured (i) the total number of records reviewed before reaching this point; and (ii) the total number of positive cases captured up to that point, relative to all positives in the dataset.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eWhen the deep neural network regressor was used, out of 7636 total manually labeled records in the 2020 dataset, the review could be terminated after approximately 1268 records. This threshold-based early stopping captured over 79.6% of all positive cases, thus substantially front-loading the high-yield segment of the record list (see Table 2).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable 2\u003c/strong\u003e Stopping rule performance: Records reviewed and positives captured\u0026nbsp;\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"602\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 32.5581%;\"\u003e\n \u003cp\u003e\u003cstrong\u003eStopping Rule\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 20.2658%;\"\u003e\n \u003cp\u003e\u003cstrong\u003eN reviewed records\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 22.5914%;\"\u003e\n \u003cp\u003e\u003cstrong\u003e% reviewed dataset\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 24.5847%;\"\u003e\n \u003cp\u003e\u003cstrong\u003e% captured positives\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd valign=\"top\" style=\"width: 32.5581%;\"\u003e\n \u003cp\u003eThreshold positive_ratio_below \u0026lt; 0.20\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 20.2658%;\"\u003e\n \u003cp\u003e1268\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 22.5914%;\"\u003e\n \u003cp\u003e16.6%\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd valign=\"top\" style=\"width: 24.5847%;\"\u003e\n \u003cp\u003e79.6%\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eThese results demonstrate that the screening saturation model can significantly reduce manual workload, saving approximately 6,368 annotations (83.4%) while still capturing most relevant cases (Fig. 3). This targeted stopping approach is particularly valuable in resource-constrained environments because it improves the efficiency of retrospective cohort studies without compromising cohort quality.\u003c/p\u003e\u003cp\u003eThe visualization in Fig. 3 enables researchers to define their yield-based stopping thresholds and adapt manual review strategies to time, budget, or statistical power constraints.\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eAlthough retrospective cohort studies using EHRs are a cornerstone of real-world evidence generation, their manual review remains a major bottleneck, particularly when high specificity is required. To address this concern, the present study proposes a data-driven \u0026ldquo;screening saturation\u0026rdquo; framework that estimates the marginal annotation yield for AI-prioritized records. Regression models can accurately approximate the expected return, thereby enabling more efficient and transparent stopping decisions.\u003c/p\u003e\n\u003cp\u003eAcross different model types, we observed strong correlations between model-assigned relevance scores and annotation yield, which confirmed feasibility of learning a predictive mapping for stopping rule optimization. However, we also found that the models\u0026rsquo; ability to generalize across temporal shifts in the data substantially varied. Specifically, while the neural network achieved the highest performance during training, it exhibited a notable decline when tested on the temporally shifted 2020 data, which could have been due to overfitting. By contrast, the Polynomial Regression model demonstrated the most robust performance, achieving the best balance of generalization and accuracy in the test set.\u003c/p\u003e\n\u003cp\u003eThese findings suggest that, in contrast to more complex architectures, their lightweight, interpretable counterparts may offer a more stable performance in real-world, non-stationary environments. Importantly, we also observed that yield-aware stopping rules based on these models can considerably reduce the manual annotation burden. Specifically, using a 20% yield threshold, we were able to successfully terminate the review process after examining only 16.6% of the dataset, while still capturing nearly 80% of all relevant cases. This substantial efficiency gain underscores the value of saturation modeling in resource-constrained settings.\u003c/p\u003e\n\u003cp\u003eTherefore, compared to traditional stopping methods such as fixed quotas or yield plateaus, the proposed approach offers a principled, quantitative framework that aligns review effort with expected informational return. By enabling prospective planning and auditability in manual review processes, the proposed framework provides strong support to clinical research teams with limited time or staffing.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eFinally, via prospectively validating the models on unseen temporal data, our results also highlighted the challenges posed by distributional drift in clinical AI pipelines. Specifically, changes in documentation styles, case mix, and underlying EHR systems can significantly alter score distributions and case density, thereby reinforcing the need for adaptive, yield-based tools that can accommodate such shifts.\u003c/p\u003e\n\u003ch2\u003eImplications for practice and system design\u003c/h2\u003e\n\u003cp\u003eThe proposed screening saturation framework has several important practical implications for the development and deployment of CITL systems in clinical research and quality improvement initiatives. In contrast to traditional manual review pipelines that frequently rely on arbitrary quotas, fixed-time efforts, or plateau-based heuristics to determine when to stop reviewing AI-ranked records, the proposed method introduces a quantifiable and transparent mechanism to stop the review process based on predicted marginal yield, thereby aligning human effort more precisely with expected benefit.\u003c/p\u003e\n\u003cp\u003eFurthermore, embedding yield-aware stopping logic into clinical NLP systems, such as the DrT tool used in the present study, could allow researchers to visualize predicted annotation saturation in real time. For instance, users may receive system-generated alerts when a predefined low-yield threshold is reached or interactively adjust it based on evolving resource constraints. This would shift manual review from a fixed-scope task to a dynamically adaptive process, offering greater control over effort-quality trade-offs. By accelerating cohort construction and reducing annotation burden, such adaptive review workflows can reduce the time from data extraction to evidence generation, thereby enabling faster dissemination of research findings that will ultimately inform clinical practice and improve patient care.\u003c/p\u003e\n\u003cp\u003eSimilarly, from the workflow perspective, integrating regression-based saturation models into annotation platforms could support several practical use cases, including:\u003c/p\u003e\n\u003cul\u003e\n \u003cli\u003eReal-time monitoring dashboards to inform reviewers how many high-yield records remain based on current model projections.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cul type=\"disc\"\u003e\n \u003cli\u003eAdaptive review planning, allowing study coordinators to accurately forecast labor requirements and estimate marginal gains from reviewing additional cases.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cul\u003e\n \u003cli\u003eAuditable decision trails, where review stopping points can be justified using explicit model predictions, rather than subjective judgment.\u003c/li\u003e\n \u003cli\u003eModel retraining pipelines, where the framework could help to select the most informative examples for manual labeling under active learning scenarios.\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAnother important characteristic of the proposed regression-based framework that it is classifier-agnostic, meaning that, regardless of the specific architecture used, it can be deployed atop any base model that ranks records by relevance. While, in the present study, we used an SVM trained on n-gram features, the proposed saturation framework could easily integrate with more advanced transformer-based classifiers or hybrid rule-based systems. This compatibility promotes a broader applicability of the proposed framework across different clinical domains and annotation goals.\u003c/p\u003e\n\u003cp\u003eFurthermore, by helping low-resource teams achieve acceptable cohort quality without extensive manual labor, the proposed saturation-guided review could contribute to more equitable research practices. In global health contexts or small-scale clinical studies with limited staffing, such models could maximize the return on scarce annotation resources, thus democratizing access to high-quality retrospective research.\u003c/p\u003e\n\u003cp\u003eFinally, by allowing for user-defined trade-offs between effort and yield, the proposed approach supports tailored review strategies, for instance, stopping at a higher threshold (e.g., 30% remaining yield) in exploratory studies or at a lower threshold (e.g., 10%) in confirmatory analyses.\u003c/p\u003e\n\u003ch2\u003eLimitations\u003c/h2\u003e\n\u003cp\u003eThe proposed screening saturation framework has several limitations. First, accuracy of the regression-based yield predictions inherently depends on quality, calibration, and stability of the underlying ranking model, in this case, the SVM-based classifier embedded in the DrT tool. If the base classifier assigns misleading scores due to miscalibration, distributional drift, or suboptimal feature representation, the saturation model\u0026rsquo;s predictions may become unreliable. Accordingly, it is essential to validate and update the base model regularly.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eSecond, while the proposed framework assumes that manually reviewed records, particularly those at the top of the ranking, are exhaustively and correctly labeled in real-world scenarios with multiple annotators or variable reviewer experience, label noise or inconsistencies in annotation completeness may compromise fidelity of the training signal and adversely affect accuracy of the yield prediction model.\u003c/p\u003e\n\u003cp\u003eThird, in the present study, we conducted a prospective evaluation using temporally shifted data (2013 and 2020) from the same institution. Yet, differences in documentation culture, clinical terminology, EHR platforms, and annotation protocols may introduce heterogeneity that could reduce generalizability. This warrants future research that would test the performance of the proposed framework in multi-institutional or multilingual settings, along with external validation across diverse sites and specialties.\u003c/p\u003e\n\u003cp\u003eFourth, in the current implementation, we used only the raw model-assigned score as input to the regression model. While this simplification aids interpretability and ease of integration, it may limit predictive precision. Accordingly, including additional contextual features (e.g., note type, department, or reviewer-specific factors) in future studies could enhance model accuracy and robustness.\u003c/p\u003e\n\u003cp\u003eFifth, our study focused solely on a binary positive-vs-negative yield prediction. Yet, in more nuanced annotation tasks, yield may not be adequately captured by simple binary outcomes. Accordingly, to better reflect clinical ambiguity and annotation effort, future implementations may need to incorporate richer label hierarchies or uncertainty measures.\u003c/p\u003e\n\u003cp\u003eSixth, considering that the saturation framework is currently retrospective and offline, although the results support its integration into real-time or interactive systems, additional software engineering and user testing would be needed to validate its usability, interpretability, and impact on end-user decision-making in live annotation settings.\u003c/p\u003e\n\u003cp\u003eFinally, considering that the proposed framework focuses on yield prediction, rather than on classification performance, it does not produce standard metrics such as precision, recall, or specificity. This may limit its adoption in contexts where decision makers are more familiar with traditional model evaluation metrics. However, the present design choice reflects the framework\u0026rsquo;s major intent of guiding resource-efficient manual review, rather than automating case classification.\u003c/p\u003e\n\u003ch2\u003eFuture research directions\u003c/h2\u003e\n\u003cp\u003eIn further research, the proposed framework may be extended in several directions. First, regarding threshold optimization, future studies could use data-driven or Bayesian approaches to dynamically tune the stopping threshold based on cost-benefit trade-offs. Another meaningful future research path is multivariate modeling, where yield predictions could be further improved by incorporating metadata such as document type, reviewer experience, or visit context. Third, considering the novel opportunities for richer ranking functions afforded by large language models (LLMs), the proposed method could layer atop LLM outputs to guide a more efficient review. Fourth, to better reflect clinical ambiguity and support more nuanced annotation tasks, future implementations may extend beyond binary yield prediction to accommodate multi-class outcomes or incorporate uncertainty estimates. Finally, embedding saturation prediction into live annotation platforms could enable adaptive review workflows and immediate cost tracking.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eManual review of unstructured clinical notes remains a major barrier to scalable retrospective cohort identification, particularly in contexts requiring high specificity. While NLP models can rank candidate records for review, the actual cost and yield of manual validation remains difficult to forecast, particularly when models are applied to newer datasets. In the present study, we addressed this challenge by introducing a regression-based framework that predicts annotation yield at each point in a relevance-ranked list of EHR records. Our approach supports data-driven stopping rules that align review effort with expected benefit.\u003c/p\u003e\n\u003cp\u003eUsing manually annotated notes from 2013 to train predictive models and prospectively evaluating performance of these models on the 2020 data, we found that yield prediction is feasible and strongly correlates with model-assigned scores, even in the presence of temporal distribution shifts. Among the tested models, polynomial regression was found to offer the best balance of generalization and interpretability, while the neural network, despite its strong in-sample performance, proved to be more vulnerable to temporal overfitting.\u003c/p\u003e\n\u003cp\u003eThese findings highlight that the proposed screening saturation framework offers a practical way to manage the resource burden of manual review under conditions of model decay and uncertainty. By estimating the marginal return on continued review, the proposed framework enables tailored stopping decisions, for example, to identify 80% of positive cases while reviewing only a fraction of the dataset. The proposed approach moves beyond fixed or heuristic stopping rules, supporting a more transparent, reproducible, and resource-sensitive cohort construction. Finally, since the proposed method is classifier-agnostic, it can be integrated into various CITL NLP systems and applied meaningfully across multiple clinical domains. As retrospective studies increasingly rely on AI-assisted workflows, the proposed framework may prove essential for making the corresponding workflows more efficient, scalable, and responsive to real-world constraints.\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cp\u003eEHR \u0026nbsp;Electronic Health Record\u003c/p\u003e\n\u003cp\u003eNLP \u0026nbsp;Natural Language Processing\u003c/p\u003e\n\u003cp\u003eSE \u0026nbsp;Status Epilepticus\u003c/p\u003e\n\u003cp\u003eESE \u0026nbsp;Established Status Epilepticus\u003c/p\u003e\n\u003cp\u003eRSE \u0026nbsp;Refractory Status Epilepticus\u003c/p\u003e\n\u003cp\u003eASM \u0026nbsp;Anti-Seizure Medication\u003c/p\u003e\n\u003cp\u003eML \u0026nbsp;Machine Learning\u003c/p\u003e\n\u003cp\u003eSVM \u0026nbsp;Support Vector Machine\u003c/p\u003e\n\u003cp\u003eMRMR \u0026nbsp;Minimum Redundancy Maximum Relevance\u003c/p\u003e\n\u003cp\u003eDrT \u0026nbsp;Document Review Tool\u003c/p\u003e\n\u003cp\u003eCSN \u0026nbsp;Case Serial Number\u003c/p\u003e\n\u003cp\u003eSVR \u0026nbsp;Support Vector Regression\u003c/p\u003e\n\u003cp\u003eMSE \u0026nbsp;Mean Squared Error\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003eAcknowledgements\u003c/p\u003e\n\u003cp\u003eThis study was supported by the Epilepsy Research Fund.\u003c/p\u003e\n\u003cp\u003eAuthor contributions\u003c/p\u003e\n\u003cp\u003eT.L., A.L., F.M.A.C., M.A.P., L.A.: Study Design; A.L.: Conceptualization, Formal analysis, Software development, Data visualization; A.L., T. L.: Draft writing; F.M.A.C.: Data acquisition, Methodology development; F.M.A.C., T. L., L.A.: Project supervision; Personnel training; M.A.P., \u0026nbsp;L.A.: Manual record review; T.L., J.V.G.: Clinical insights, Data interpretation; T.L.: Funding acquisition. All authors reviewed the manuscript.\u003c/p\u003e\n\u003cp\u003eFunding\u003c/p\u003e\n\u003cp\u003eThis study was supported by the Epilepsy Research Fund.\u003c/p\u003e\n\u003cp\u003eData availability\u003c/p\u003e\n\u003cp\u003eAll code and de‑identified example data needed to reproduce the analysis are openly available at https://github.com/assaflandschaft/ehr-yield-prediction. The raw clinical notes contain protected health information and, under HIPAA regulations, cannot be shared.\u003c/p\u003e\n\u003cp\u003eCompeting interests\u003c/p\u003e\n\u003cp\u003eThe authors declare no competing interests.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eEthics approval and consent to participate\u003c/p\u003e\n\u003cp\u003eThis retrospective study was conducted in accordance with the Declaration of Helsinki.\u0026nbsp;The Boston Children\u0026rsquo;s Hospital Institutional Review Board (IRB‑\u0026nbsp;P00001207) approved the protocol and waived the requirement for written informed consent. \u0026nbsp;\u003c/p\u003e\n\u003cp\u003eConsent for publication\u003c/p\u003e\n\u003cp\u003eNot applicable. \u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eKimia AA, Savova G, Landschaft A, Harper MB. An introduction to natural language processing: how you can get more from those electronic notes you are generating. Pediatr Emerg Care. 2015;31(7):536\u0026ndash;41. \u003c/li\u003e\n\u003cli\u003eSavova GK, Masanz JJ, Ogren, PV, Zheng J, Sohn S, Kipper-Schuler KC, et al. Mayo clinical text analysis and knowledge extraction system (cTAKES): architecture, component evaluation and applications. J Am Med Inform\u003cem\u003e \u003c/em\u003eAssoc. 2010;17(5):507\u0026ndash;13. \u003c/li\u003e\n\u003cli\u003eShickel B, Tighe PJ, Bihorac A, Rashidi P. Deep EHR: A survey of recent advances in deep learning techniques for electronic health record (EHR) analysis. IEEE J Biomed Health Inform. 2018; 22(5):1589\u0026ndash;604. \u003c/li\u003e\n\u003cli\u003eVeeranki SPK, Abdulnazar A, Kramer D, Kreuzthaler M, Benjaminet D. Multi-label text classification via secondary use of large clinical real-world data sets. Sci Rep. 2024;14, 26972. \u003c/li\u003e\n\u003cli\u003eXie Q, Chen Q, Chen A, Peng C, Hu Y, Lin F, et al. Medical foundation large language models for comprehensive text analysis and beyond. NPJ Digit Med. 2025;8:141. \u003c/li\u003e\n\u003cli\u003eSpasic I, Nenadic G. Clinical text data in machine learning: systematic review. JMIR Med Inform. 2020;8(3): e17984. \u003c/li\u003e\n\u003cli\u003eWalsh CG, Wilimitis D, Chen Q, Wright A, Kolli J, Robinson K, et al. Scalable incident detection via natural language processing and probabilistic language models. Sci Rep. 2024;14:23429.\u003c/li\u003e\n\u003cli\u003eYonamine S, Ma CJ, Alabi RO, Kaidonis G, Chan L, Borkar D, et al. Comparison of diagnosis codes to clinical notes in classifying patients with diabetic retinopathy. Ophthalmol Sci. 2024;4(6):100564. \u003c/li\u003e\n\u003cli\u003eBerge GT, Granmo OC, Tveit TO, Ruthjersen AL, Sharmaet J. Combining unsupervised, supervised and rule-based learning: the case of detecting patient allergies in electronic health records. BMC Med Inform Decis Mak. 2023;23:188. \u003c/li\u003e\n\u003cli\u003eNatukunda A, Muchene LK. Unsupervised title and abstract screening for systematic review: a retrospective case-study using topic modelling methodology. Syst Rev. 2023;12:1. \u003c/li\u003e\n\u003cli\u003eArbet J, Brokamp C, Meinzen-Derr J, Trinkley KE, Spratt HM. Lessons and tips for designing a machine learning study using EHR data. J Clin Transl Sci. 2020;5(1):e21. \u003c/li\u003e\n\u003cli\u003eZhou H, Chen Y, Lipton Z. Evaluating model performance in medical datasets over time. Proceedings of the Conference on Health, Inference, and Learning\u003cem\u003e \u003c/em\u003e2023;209:498\u0026ndash;508. \u003c/li\u003e\n\u003cli\u003eGuo LL, Steinberg E, Fleming SL, Posada J, Lemmon J, Pfohl SR, et al. EHR foundation models improve robustness in the presence of temporal distribution shift. Sci Rep. 2023;13:3767. \u003c/li\u003e\n\u003cli\u003eGuo LL, Pfohl SR, Fries J, Posada J, Fleming SL, Aftandilian C, et al. Systematic review of approaches to preserve machine learning performance in the presence of temporal dataset shift in clinical medicine. Appl Clin Inform. 2021;12(4):808\u0026ndash;15. \u003c/li\u003e\n\u003cli\u003eGuo, LL, Pfohl SR, Fries J, Johnson AEW, Jose Posada J, Aftandilian C, et al\u003cem\u003e.\u003c/em\u003e Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine. Sci Rep.\u003cem\u003e \u003c/em\u003e2022;12: 2726. \u003c/li\u003e\n\u003cli\u003ePaydar-Darian N, Kimia AA, Monuteaux MC, Michelson KA, Landschaft A, Mauldenet AB, et al. C-reactive protein or erythrocyte sedimentation rate results reliably exclude invasive bacterial infections. Am J Emerg Med. 2019;37(8):1510\u0026ndash;15. \u003c/li\u003e\n\u003cli\u003eJohnson KB, Michelson KA, ∙ Lyons TW, Landschaft A, Loddenkemperb T, Kimiaet AA, et al. Pediatric status epilepticus: how common is cerebrospinal fluid pleocytosis in the absence of infection? Seizure. 2014;23(7):573\u0026ndash;5. \u003c/li\u003e\n\u003cli\u003eKimia AA, Rudloe TF, Aprahamian N, McNamara J, Roberson D, Landschaft A, et al. Predictors of a drainable suppurative adenitis among children presenting with cervical adenopathy. Am J Emerg Med. 2019;37(1):109\u0026ndash;13. \u003c/li\u003e\n\u003cli\u003eOzonoff A, Milliren CE, Fournier K, Welcher J, Landschaft A, Samnaliev M, et al. Electronic surveillance of patient safety events using natural language processing. Health Informatics\u003cem\u003e \u003c/em\u003eJ. 2022;28(4):146045822211324. \u003c/li\u003e\n\u003cli\u003eEl Helou R, Landschaft A, Harper MB, Kimia AA. Bacteremia in children with fever and acute lower extremity pain. Pediatrics.\u003cem\u003e \u003c/em\u003e2023;151(5):e2022059504. \u003c/li\u003e\n\u003cli\u003eKimia A, Landschaft A. Document review tools. www.documentreviewtools.com. Accessed June 23, 2025. \u003c/li\u003e\n\u003cli\u003eChafjiri FMA, Reece L, Voke L, Landschaft A, Clark J, Kimia AA, et al. Natural language processing for identification of refractory status epilepticus in children. Epilepsia. 2023;64(12):3227\u0026ndash;37. \u003c/li\u003e\n\u003cli\u003ePuckett MA, Chafjiri MA, Gettings JV, Landschaft A, Loddenkemper, T. Utilizing natural language processing to identify pediatric patients experiencing status epilepticus. Seizure. 2025;125:54\u0026ndash;61. \u003c/li\u003e\n\u003cli\u003eTrinka E, Cock H, Hesdorffer D, Rossetti AO, Scheffer IE, Shinnar S, et al. A definition and classification of status epilepticus - report of the ILAE Task Force on Classification of Status Epilepticus. Epilepsia. 2015;56(10):1515\u0026ndash;23. \u003c/li\u003e\n\u003cli\u003eCavnar WB, Trenkle JM. N-gram-based text categorization.\u003cem\u003e \u003c/em\u003eProceedings of SDAIR-94, 3rd Annual Symposium on Document Analysis and Information Retrieval; 2021. p. 161\u0026ndash;175. \u003c/li\u003e\n\u003cli\u003eYang E, Lewis DD, Frieder O. Heuristic stopping rules for technology-assisted review. Proceedings of the 21st ACM Symposium on Document Engineering (DocEng \u0026apos;21). Association for Computing Machinery, New York, NY, USA, Article 31; 2021. pp. 1\u0026ndash;10. \u003c/li\u003e\n\u003cli\u003eStevenson M, Bin-Hezam R. Stopping methods for technology assisted reviews based on point processes. ACM Trans Inf Syst. 2024;42(3):1\u0026ndash;37. \u003c/li\u003e\n\u003cli\u003eSettles B. Active learning literature survey. Computer sciences technical report 1648, 2009. http://pages.cs.wisc.edu/~bsettles/pub/settles.activelearning.pdf. Accessed 23 June, 2025. \u003c/li\u003e\n\u003cli\u003eSun H, Xu C, Suominen H. Analyzing the granularity and cost of annotation in clinical sequence labeling. arXiv 2021. https://arxiv.org/abs/2108.09913. Accessed 23 June, 2025. \u003c/li\u003e\n\u003cli\u003eLandschaft A, Antweiler D, Mackay S, Kugler S, R\u0026uuml;ping S, Wrobel S, et al. Implementation and evaluation of an additional GPT-4-based reviewer in PRISMA-based medical systematic literature reviews. Int J Med Inform. 2024;189:105531. \u003c/li\u003e\n\u003cli\u003eCohen AM, Chamberlin S, Deloughery T, Nguyen M, Bedrick S, Meninger S, et al. Detecting rare diseases in electronic health records using machine learning and knowledge engineering: case study of acute hepatic porphyria. PloS One. 2020;15(7): e0235574. \u003c/li\u003e\n\u003cli\u003eManning CD, Raghavan P, Sch\u0026uuml;tze H. Introduction to information retrieval. Cambridge: Cambridge University Press; 2008.\u003c/li\u003e\n\u003cli\u003eErnst MD. Permutation methods: a basis for exact inference. Statist Sci. 2004;19(4):676\u0026ndash;85. \u003c/li\u003e\n\u003cli\u003eMi\u0026ntilde;arro-Gim\u0026eacute;nez JA, Cornet R, Jaulent MC, Dewenter H, Thun S, G\u0026oslash;eg KR, et al. Quantitative analysis of manual annotation of clinical text samples. Int J Med Inform. 2019;123:37\u0026ndash;48. \u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-informatics-and-decision-making","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"midm","sideBox":"Learn more about [BMC Medical Informatics and Decision Making](http://bmcmedinformdecismak.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/midm/default.aspx","title":"BMC Medical Informatics and Decision Making","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Electronic Health Records (EHRs), Natural Language Processing (NLP), Retrospective cohort studies, Status epilepticus, Machine learning, Annotation efficiency, Yield prediction, Artificial Intelligence (AI), Clinician in the Loop (CITL)","lastPublishedDoi":"10.21203/rs.3.rs-6966149/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-6966149/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground\u003c/h2\u003e\u003cp\u003ePainstaking manual chart review of EHRs is still the chief bottleneck in retrospective studies, especially when rare-disease cohorts demand high specificity. Automated NLP rankers help, yet when trained on dated data they leave teams guessing how long to keep reviewing charts. We therefore present a regression-based \u0026lsquo;screening-saturation\u0026rsquo; model that predicts residual yield at every point along the ranked list.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e\u003cp\u003eLeveraging a previously validated SVM that ranks notes for pediatric status epilepticus, we trained four predictive models: linear, polynomial, and support-vector regressions plus a lightweight neural net, on notes from 2013 and tested them on data from 2020. Our target was the proportion of true positives (ESE or RSE) expected below any score threshold.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e\u003cp\u003ePolynomial regression offered the best balance of generalizability and interpretability, which demonstrated a strong predictive performance even under temporal data shifts. Regression outputs were used to simulate stopping rules for manual review, which captured 80% of positives after reviewing just 16.6% of notes (an 83% workload cut).\u003c/p\u003e\u003ch2\u003eConclusion\u003c/h2\u003e\u003cp\u003eOur scalable, model-agnostic framework turns AI scores into actionable staffing decisions in clinical workflows. This screening-saturation model integrates with clinician-in-the-loop tools and adapts readily across medical domains that need lean chart review.\u003c/p\u003e","manuscriptTitle":"Predicting Annotation Yield in Artificial Intelligence-Ranked Electronic Health Record Cohorts: A Regression-Based Framework for Efficient Manual Review","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-07-23 08:36:09","doi":"10.21203/rs.3.rs-6966149/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-08-13T09:39:05+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-08-06T07:46:38+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-08-04T20:29:21+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"294205264415163087859389201997453183580","date":"2025-07-27T10:14:54+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-07-20T21:21:04+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"234382560172901941247328836521667912222","date":"2025-07-18T06:09:59+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"330612729208070185168488404876210604863","date":"2025-07-17T21:54:20+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-07-17T21:14:00+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2025-07-17T21:10:24+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2025-07-08T05:27:02+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-07-07T17:15:34+00:00","index":"","fulltext":""},{"type":"submitted","content":"BMC Medical Informatics and Decision Making","date":"2025-07-07T17:12:22+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-informatics-and-decision-making","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"midm","sideBox":"Learn more about [BMC Medical Informatics and Decision Making](http://bmcmedinformdecismak.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/midm/default.aspx","title":"BMC Medical Informatics and Decision Making","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"4dc6ef9b-daf7-4f69-918c-2795b7a037d3","owner":[],"postedDate":"July 23rd, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2025-11-03T16:09:11+00:00","versionOfRecord":{"articleIdentity":"rs-6966149","link":"https://doi.org/10.1186/s12911-025-03241-y","journal":{"identity":"bmc-medical-informatics-and-decision-making","isVorOnly":false,"title":"BMC Medical Informatics and Decision Making"},"publishedOn":"2025-10-31 15:58:14","publishedOnDateReadable":"October 31st, 2025"},"versionCreatedAt":"2025-07-23 08:36:09","video":"","vorDoi":"10.1186/s12911-025-03241-y","vorDoiUrl":"https://doi.org/10.1186/s12911-025-03241-y","workflowStages":[]},"version":"v1","identity":"rs-6966149","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-6966149","identity":"rs-6966149","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}