Evaluating Clinical Note Deidentification Tools and Transformer Transferability between Public and Private Data from the US Department of Veterans Affairs

Deidentification of clinical notes is critical for enabling secondary use of electronic health records (EHRs) while maintaining patient privacy. Existing deidentification tools vary in methodology, with rule-based, hybrid, and transformer-based approaches offering different trade-offs in performance. However, most evaluations of these tools are conducted on publicly available datasets, limiting their generalizability to private healthcare systems such as the U.S. Department of Veterans Affairs (VA).

We introduce an annotated corpus of 1,000 VA clinical notes and evaluated the performance of four open-source deidentification tools (Physionet Deid, Philter, CliniDeID, and Stanford Deid) using precision, recall, and F1-score. Additionally, we fine-tuned BioClinicalBERT using VA and I2B2 training data to assess the transferability of transformer models across public and private datasets.

Transformer-based models outperformed rule-based tools when identifying names and locations, while rule-based systems demonstrated higher recall for more structured entities such as dates. CliniDeID showed the highest performance among external tools, but no tool achieved optimal balance across all PHI categories. Transformer models trained on both public and VA datasets performed best, though challenges remained in handling VA-specific terminology and annotations.

These findings underscore the limitations of applying publicly trained models to private healthcare systems without adaptation. Differences in text structure, clinical terminology, and annotation standards impact deidentification effectiveness. While transformer models offer superior flexibility, computational cost and privacy concerns may hinder large-scale adoption.

The results emphasize the need for institution-specific evaluation and fine-tuning of deidentification models. Future work should explore large language model and hybrid approaches that integrate rule-based and transformer-based methods to enhance adaptability and accuracy in diverse clinical settings. Competing Interest Statement The authors have declared no competing interest. Funding Statement This work was supported using resources and facilities of the Department of Veterans Affairs (VA) Informatics and Computing Infrastructure (VINCI), funded under the research priority to Put VA Data to Work for Veterans (VA ORD 22-D4V). The views expressed are those of the authors and do not necessarily represent the views or policy of the Department of Veterans Affairs or the United States Government Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes The details of the IRB/oversight body that provided approval or exemption for the research described are given below: Ethics committee/IRB of the University of Utah waived ethical approval for this work I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Yes Footnotes minor proof/typo fixes and updates Data Availability The data for this study originates from the U.S. Department of Veterans Affairs and includes protected health information (PHI); therefore, it is not available for public distribution.