Optimizing Word Embeddings for Patient Portal Message Datasets with a Small Number of Samples

Optimizing Word Embeddings for Patient Portal Message Datasets with a Small Number of Samples | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Optimizing Word Embeddings for Patient Portal Message Datasets with a Small Number of Samples Qingyuan Song, Congning Ni, Jeremy L. Warner, Qingxia Chen, Lijun Song, and 3 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4350387/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 10 You are reading this latest preprint version Abstract Background Patient portal messages often relate to specific clinical phenomena (e.g., patients undergoing treatment for breast cancer) and, as a result, have received increasing attention in biomedical research. These messages require natural language processing and, while word embedding models, such as word2vec, have the potential to extract meaningful signals from text, they are not readily applicable to patient portal messages. This is because embedding models typically require millions of training samples to sufficiently represent semantics, while the volume of patient portal messages associated with a particular clinical phenomenon is often relatively small. Objective We introduce a novel adaptation of the word2vec model, PK-word2vec, for small-scale messages. Methods PK-word2vec incorporates the most similar terms for medical words (including problems, treatments, and tests) and non-medical words from two pre-trained embedding models as prior knowledge to improve the training process. We applied PK-word2vec on patient portal messages in the Vanderbilt University Medical Center electric health record system sent by patients diagnosed with breast cancer from December 2004 to November 2017. We evaluated the model through a set of 1000 tasks, each of which compared the relevance of a given word to a group of the five most similar words generated by PK-word2vec and a group of the five most similar words generated by the standard word2vec model. We recruited 200 Amazon Mechanical Turk (AMT) workers and 7 medical students to perform the tasks. Results The dataset was composed of 1,389 patient records and included 137,554 messages with 10,683 unique words. Prior knowledge was available for 7,981 non-medical and 1,116 medical words. In over 90% of the tasks, both reviewers indicated PK-word2vec generated more similar words than standard word2vec (p=0.01).The difference in the evaluation by AMT workers versus medical students was negligible for all comparisons of tasks’ choices between the two groups of reviewers (p =0.774 under a paired t-test). Conclusions . PK-word2vec can effectively learn word representations from a small message corpus, marking a significant advancement in processing patient portal messages. Biological sciences/Computational biology and bioinformatics/Machine learning Health sciences/Diseases/Cancer/Breast cancer breast cancer hormonal therapy natural language processing patient portal messages word embedding models word2vec Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Figure 7 1. Introduction Patient portals have become standard components of modern electronic health record (EHR) systems. 1 They enable patients to access health information, manage clinical appointments, and communicate securely with care providers in an asynchronous manner. 2,3 The messaging functionality of patient portals has gained in popularity over time 4,5 due to various benefits, such as the ability to increase patient involvement in medical decision-making 6 . At the same time, the information conveyed in patient portal messages can serve as a foundation for clinical research, such as studies of initiation and discontinuation of cancer therapies 7,8 , assessing readmission risk 9,10 , and creating taxonomies of patient engagement 11 . However, maximizing the utility of patient portal messages requires a means to extract meaningful signals from unstructured text. 12 Rule-based methods, which rely on manually pre-defined rules to perform a task, are ineffective as the volume of data or complexity of the task increases 12 . While specific computational methods, such as topic modeling, can be applied in this regard, these approaches typically rely on word frequency to determine the most representative words for each topic. By contrast, word embedding models, such as word2vec 13 , have been widely adopted by natural language processing (NLP) applications. These models can learn vector representations, typically from millions of documents, to capture each word’s semantic and linguistic meaning. However, it is challenging to procure a cohort that is sufficiently large to train a specific word embedding model. As an illustration, in this paper, we focus on breast cancer as it serves as an exemplar of a complex care coordination setting between various specialists who often communicate with patients via the patient portal 14 . The current approaches for generating word embeddings on relatively small corpora may learn semantic meaning inaccurately due to lack of sufficient occurrences of word usage. A natural solution to this problem is to fine-tune a word embedding model that is pre-trained on a larger data corpus. This is because the fitted parameters of a pre-trained model can serve as a better starting point for training than random initialization. Yet this strategy presents certain limitations as well. In particular, pre-trained models are incapable of handling words that are absent from the pre-trained model's vocabulary. Moreover, inconsistencies arise due to the incompatible linguistic contexts between the data used to fit the pre-trained model and the data for a specific domain of interest. 15 In this study, we introduce a novel approach, PK-word2vec, to train word2vec models on a patient portal message corpora of relatively small sample size. Our model leverages prior information, as defined by the most similar words sampled from pre-trained embedding models, to regularize the model training. We demonstrate the utility of this approach using a dataset of approximately 137,000 patient portal messages from 1,389 patients diagnosed with breast cancer at a large academic medical center. 1416 Our experiments show that the regularization process of PK-word2vec can guide and constrain the model training to learn word semantic meanings more effectively than standard word2vec models. 2. Methods Figure 1 depicts the workflow for this study: 1) data extraction and cleaning; 2) prior knowledge extraction from two pre-trained models, Snomed2vec 16 and the Google pre-trained word2vec (Google-word2vec) 13 for medical and non-medical words, respectively; 3) model training regularized by prior knowledge (PK-word2vec); and 4) similarity analysis and human evaluation of the model performance. This study was deemed exempt from human subjects research by the Vanderbilt University Institutional Review Board. 2.1. Data Collection and Preprocessing We collected the patient portal messages from the EHR system of Vanderbilt University Medical Center (VUMC), a large, private, non-profit, academic medical center in Nashville, Tennessee. VUMC launched its patient portal, My Health at Vanderbilt ( MHAV ), in 2004, which was migrated to Epic’s MyChart platform in 2017. MHAV supports secure messaging between healthcare providers and patients, appointment scheduling, billing management, and access to laboratory results or other EHR data. 5 In this study, we focused on the patient portal messages for patients with breast cancer who were prescribed the most common hormonal therapy medications: anastrozole , exemestane , letrozole , raloxifene , and/or tamoxifen . We masked URLs, email addresses, phone numbers, timestamps, and numerals by using the ekphrasis (v0.5.1) 17 . Afterwards, all personal names and usernames were manually masked, and misspelled words were corrected with the guidance of two domain experts (STR, JLW) by referring to the contexts from randomly selected messages that contain these misspelled words. We applied the CLAMP software package (v1.6.4) to detect medical words pertaining to diseases, laboratory tests, and medications from messages 18 . 2.2. Skip-Gram Model with Negative Sampling PK-word2vec is built on a skip-gram model with negative sampling (SGNS) 13 , a standard word2vec. In SGNS, each word has two vector representations: 1) a center vector when it is processed as a center word, and 2) a context vector when it serves as a neighbor of a center word. Figure 2 illustrates the center word and neighbor word within a context window size of 1 in the sentence " Please reschedule my appointment to Friday ”. In this example, reschedule is one of the neighbors of the center word appointment . Formally, SGNS structures the model training as a binary classification problem, where the objective is to predict whether a certain pair of words are neighbors. Formally, given a sequence of words \({w}_{1}, {w}_{2},\cdots , {w}_{T}\) and a context window of size \(m\) , the SGNS objective function is defined as: $${Loss}_{SGNS}=\sum _{t=1}^{T}\left[\sum _{-m\le j\le m}\left[-\text{log}\sigma \left({\mathbf{u}}_{t+j}^{T}{\mathbf{v}}_{t}\right)-\sum _{k=1, {w{\prime }}_{k}\sim P\left({w}_{t}\right)}^{K}\text{log}\sigma \left({-\mathbf{u}}_{k}^{T}{\mathbf{v}}_{t}\right)\right]\right]$$ 1 where \({\mathbf{u}}_{t}\) and \({\mathbf{v}}_{t}\) represent the context and center vector of \({w}_{t}\) , which are the parameters optimized in the loss function; \({w{\prime }}_{k}\) represents a random word that does not appear in the neighboring context of \({w}_{t}\) . \({w{\prime }}_{k}\) is sampled from \(P\left({w}_{t}\right)\) , a unigram distribution defined by the frequency of occurrence of each word 19 ; \(K\) is a hyperparameter that specifies the number of negative words sampled for \({w}_{t}\) ; and \(\sigma\) is the sigmoid function. The first term in Eq. ( 1 ) aims to maximize the probability of occurrence for words that are in the context window. By contrast, the second term in Eq. ( 1 ) aims to minimize the probability of occurrence for words that are not in the context window. 2.3. Prior Knowledge Word2vec Model The challenge of training a word2vec model on a small dataset arises from the insufficient context that such datasets provide. To address this issue, we define the prior knowledge of a word as its most similar words identified from other pre-trained models. Subsequently, the prior knowledge is applied to regularize the PK-word2vec model training. More formally, if a word in the patient portal messages \({w}_{t}\) also appears in a pre-trained embedding vocabulary, the prior knowledge of \({w}_{t}\) is defined by another word \({w}_{j}\) that is sampled with probability \({p}_{tj}\) from the patient portal message vocabulary. In this respect, the more similar \({w}_{j}\) is to \({w}_{t}\) in the pre-trained embedding model, the greater the likelihood that \({w}_{j}\) will be sampled. We introduce a prior knowledge loss function for the entire dataset, which is defined as follows: $${\text{L}\text{o}\text{s}\text{s}}_{\text{P}\text{K}}=\left\{\begin{array}{cc}{\sum }_{t=1}^{T}\left[\gamma \right({w}_{t})\bullet {\Psi }\left({\mathbf{u}}_{t},{\mathbf{u}}_{j},{\mathbf{v}}_{t},{\mathbf{v}}_{j}\right)]& {w}_{t} \text{w}\text{i}\text{t}\text{h} \text{p}\text{r}\text{i}\text{o}\text{r} \text{k}\text{n}\text{o}\text{w}\text{l}\text{e}\text{d}\text{g}\text{e}\\ 0& {w}_{t} \text{w}\text{i}\text{t}\text{h}\text{o}\text{u}\text{t} \text{p}\text{r}\text{i}\text{o}\text{r} \text{k}\text{n}\text{o}\text{w}\text{l}\text{e}\text{d}\text{g}\text{e}\end{array}\right.$$ 2 where \(\gamma \left({w}_{t}\right)=\tau /\text{f}\text{r}\text{e}\text{q}\left({w}_{t}\right)\) discounts the words proportionally to their frequency, \(\tau\) is a hyperparameter, and \({\Psi }\left(·\right)\) is a regularization factor to be defined in Eq. ( 4 ) below. Note that if a word does not exist in the pre-trained model, then it has no prior knowledge and its corresponding term in \({\text{L}\text{o}\text{s}\text{s}}_{\text{P}\text{K}}\) is set to zero. In this work, we calculate word similarity as the cosine similarity of two vectors \(\mathbf{u}\) and \(\mathbf{v}\) , \(\text{cos}(u,v)=u\bullet v/(\left|u\right|\bullet \left|v\right|)\) . We require that each sampled \({w}_{j}\) have a similarity score with \({w}_{t}\) above a pre-defined threshold \(\theta\) . As such, \({p}_{tj}\) is defined as: $${p}_{tj}=\left\{\begin{array}{cc}\text{s}\text{o}\text{f}\text{t}\text{m}\text{a}\text{x}\left[\text{cos}({\mathbf{u}}_{t},{\mathbf{u}}_{j})\right]& \text{cos}({\mathbf{u}}_{t},{\mathbf{u}}_{j})\ge \theta \\ 0& \text{cos}({\mathbf{u}}_{t},{\mathbf{u}}_{j})<\theta \end{array}\right.$$ 3 In our experiments, for each pre-trained model, we set \(\theta\) to the average of the top 10 similarity scores for words with prior knowledge in patient portal messages. In practice, each time \({w}_{t}\) is processed in model training using gradient descent optimization, a word \({w}_{j}\) is sampled and its cosine similarity with \({w}_{t}\) in both context and center vectors is calculated as its prior knowledge. Then, the corresponding item for \({w}_{t}\) that needs to be minimized in the PK-word2vec loss function in Eq. ( 2 ) is: $${\Psi }\left({\mathbf{u}}_{t},{\mathbf{u}}_{j},{\mathbf{v}}_{t},{\mathbf{v}}_{j}\right)=\left[1-\text{cos}\left({\mathbf{u}}_{t},{\mathbf{u}}_{j}\right)\right]+[1-\text{cos}({\mathbf{v}}_{t},{\mathbf{v}}_{j})]$$ 4 There are several reasons for incorporating context vector similarity into Eq. ( 4 ). First, a key characteristic of high-quality word embeddings is that the distance between the context vectors for similar words is relatively small 20 . Second, since the context vectors are applied to estimate the center vectors of neighboring words, the regularization of the context vectors can propagate to the center vectors of neighboring words without prior knowledge. Finally, the overall objective function for PK-word2vec can be defined as: $$\text{L}\text{o}\text{s}\text{s}={\text{L}\text{o}\text{s}\text{s}}_{\text{S}\text{G}\text{N}\text{S}}+\alpha \times {\text{L}\text{o}\text{s}\text{s}}_{\text{P}\text{K}}$$ 2 where \(\alpha\) is a hyperparameter that determines the degree to which the training of PK-word2vec relies on prior knowledge. A higher \(\alpha\) indicates a stronger dependency on prior knowledge. To evaluate the impact of α, we compared our approach to a baseline model where \(\alpha =0\) , a condition corresponding to SGNS. We generate prior knowledge for non-medical and medical words using two different models. Specifically, we applied Google-word2vec, which was trained on the Google News dataset (about 100 billion words), to build prior knowledge for non-medical words. We turned to a Snomed2vec model, which was trained on the SNOMED-CT knowledge graph 16 to obtain prior knowledge for medical words. 2.4. Hyperparameter Selection Hyperparameter tuning for word2vec models is challenging because it is difficult to define similarity. Since Google-word2vec was trained on a vast corpus, we selected hyperparameters, the dimension of word embedding d and the prior knowledge weight \(\alpha\) , such that PK-word2vec is similar to Google-word2vec in terms of the distribution of the similarity scores between a given word and its 10 most similar words. In this paper we name this distribution as similarity distribution for simplicity. We did not use Snomed2vec as a reference because it is inherently a node2vec 21 trained on a semantic network in the Unified Medical Language System 22 rather than text data. We set the embedding size \(d\) with a value that corresponds to the smallest Wasserstein distance 23 in the similarity distributions between the two models. To select \(\alpha\) , we first categorized words into high-, mid-, and low-frequency ranges based on their tertiles. Then, we set \(\alpha\) with a value that corresponds to the smallest Wasserstein distance in the similarity distribution between three distinct word frequency ranges. This resulted in a comparison in an unbiased manner, since high-frequency words tended to receive better representations. When training PK-word2vec, the \(\theta\) threshold was set to the average word similarity between all the words and their 10 most similar words in the pre-trained models. We set the threshold \(\tau\) in Eq. ( 2 ) to 0.00001, batch size to 10000, and trained the model for 10 epochs. 2.5. Human Evaluation To compare PK-word2vec with SGNS, we randomly sampled 1,000 words from the patient portal message vocabulary, with the same proportion of medical to non-medical words. For each term of interest, we created one group of the five most similar words generated from PK-word2vec and SGNS, respectively. To evaluate the system, we asked each reviewer to indicate which word group was more related to the word of interest. We randomized the order of the two groups in each task to avoid framing biases. We recruited two groups of reviewers for the model evaluation. The first group consisted of 330 Amazon Mechanical Turk (MTurk) workers 24 , recognized as “masters” due to their consistently submission of high-quality results in their history annotations. Each MTurk worker completed 100 out of the 1000 tasks and each task was answered by 33 different MTurk workers. The second group consisted of 7 medical students recruited through the Vanderbilt Institute for Clinical and Translational Research. Each medical student completed all 1,000 tasks. We set a 30-second time limit for each task; task not completed within this time limit were reassigned to a new worker. To analyze the human evaluation data, we first calculated the support rate, defined as the proportion of the 1,000 tasks for which MTurk workers or medical students preferred PK-word2vec by a majority vote. Next, we employed sample skewness 25 to examine the distribution of the proportion of reviewers who favored PK-word2vec in each task. This provided insights into the extent to which PK-word2vec was favored over SGNS. A larger value for left sample skew indicates a preference of PK-word2vec over SGNS. Finally, to account for the heterogeneity, we fitted a mixed-effects logistic regression model to assess which word embedding model was preferred by MTurk workers. The type of model (PK-word2vec or SGNS) and the category of the term of interest (medical or non-medical words) were variables with fixed effects. We assigned a value of 1 to the PK-word2vec model and medical words and a value of 0 to the SGNS and non-medical words in the corresponding variables. 3. Results 3.1. Data Summary The dataset was composed of 137,554 messages sent by 1,389 patients with breast cancer. After preprocessing, there was a total of 10,683 unique words in the vocabulary, of which 8,878 (80%) and 1,895 (20%) were non-medical and medical words, respectively. Prior knowledge was available for 7,981 (90%) non-medical and 1,116 (59%) medical words. 3.2. Hyperparameter Selection Embedding size ( \(d\) ) . The word embedding size was selected from a range of 10 to 300. Figure 3 depicts the Wasserstein distance between SGNS and the Google-word2vec as a function of the word embedding size. In the following experiments, we present results with an embedding size of 45 for SGNS and PK-word2vec because it corresponds to the smallest distance among all the candidate vector sizes. Prior knowledge weight ( \(\alpha\) ) . Figure 4 a shows the similarity distribution for PK-word2vec as a function of \(\alpha\) . When \(\alpha =0\) , this model corresponds to SGNS and most of the density concentrates around 0.6. As \(\alpha\) increases, the prior knowledge becomes dominant, and the density gradually shifts toward two directions, forming two peaks. Figure 4 b shows the Wasserstein distance in the similarity distribution between Google-Word2vec (three dashed lines at the bottom of the figure) and PK-word2vec (three solid lines at the top of the figure) in different word frequency ranges. The distributions for Google-word2vec were nearly flat and maintained relatively small values. Based on this observation, we set \(\alpha\) to \(5\times {10}^{-6}\) for the following experiments because it corresponds to the smallest Wasserstein distances as well as smallest variance across three pairs of comparisons. 3.3. Word Similarity Context vector regularization. Figure 5 depicts the similarity distribution for the models with (panels a-d) and without (panels e-g) context vector regularization. The blue and orange lines in each graph correspond to words with and without prior knowledge, respectively. The comparisons between two similarity distributions are illustrated in settings with different pre-selected \(\alpha\) values. As extreme examples, Figs. 5 d and 5 g show that when \(\alpha ={10}^{-4}\) , context vector regularization mitigates the skew of the similarity distribution for the words with prior knowledge. While a large \(\alpha\) value shifts the mode of the similarity distribution for words without prior knowledge in the model without context vector regularization to the right, the variance of similarity distribution is not significantly altered. Figures 5 c and 5 f depict the scenario where \(\alpha\) is set to its optimal value of \(5\times {10}^{-6}\) , where the difference between the two distributions is in an intermediate state compared with other \(\alpha\) values. These results show that context vector regularization mitigates the impact of prior knowledge on the similarity distribution between words with and without prior knowledge. 3.4. Human Evaluation Figure 6 displays the human evaluation results based on a majority vote from the MTurk workers and medical students. In addition to an overall comparison, the results were stratified by the word type, and the existence of prior knowledge. The difference was negligible for all comparisons between the two groups of reviewers ( \(\text{p}=0.774\) under a paired t-test). In over 90% of the tasks, both groups of reviewers indicated that PK-word2vec generates more similar words than SGNS. For medical words, the support rate for PK-word2vec was 78.8% for MTurk workers and 71.8% for medical students. For non-medical words, the support rate for PK-word2vec was 93.6% for MTurk workers and 93.4% for medical students. For words lacking prior knowledge, the support rate for PK-word2vec was 66.7% from MTurk workers and 61.4% from medical students. Figure 7 shows the distribution of the proportion of reviewers that favor PK-word2vec in each task. The three curves represent the proportion distribution for the low-, mid-, and high-frequency words, respectively. All distributions are strongly left-skewed and most of the density falls between 0.8 and 1.0, indicating reviewers consistently favored PK-word2vec. Notably, in the responses from MTurk workers, the sample skew was 1.19, 1.65, and 1.92 for the high-, mid-, and low-frequency words, respectively. In the responses from the medical students, the sample skew was 0.88, 1.46, and 1.84 for the high, mid-, and low-frequency words, respectively. In other words, the degree to which the distribution was left-skewed was inversely proportional to the word frequency. The results of the mixed-effect model showed that PK-word2vec was significantly more favored than SGNS ( \(\beta = 2.702\) , p-value = 0.01). By contrast, no difference was found between medical word and non-medical word ( \(\beta = 0\) , p-value = 1). 4. Discussion This study has several findings that are worth highlighting. First, relying on external word pairs to tune hyperparameters, as has been done in many prior studies 13,26,27 , is suboptimal for smaller datasets. This limitation arises from the fact that many word pairs in external datasets may not be included in the target datasets. To address this issue, we selected hyperparameters by minimizing the distance in the similarity distribution between Google-word2vec and PK-word2vec. Our experiments showed that the prior knowledge weight \(\alpha\) changed the word similarity distribution from a unimodal to a bimodal form, suggesting that introducing too much prior knowledge might degrade the quality of word embedding models. However, when comparing the similarity distribution between words across different frequency ranges, we observed that when the prior knowledge weight was very small or very large, it caused the similarity distribution to become dissimilar amongst these word frequency ranges. This finding highlights the necessity of carefully balancing the weight of prior knowledge within the model training. Second, including context vector regularization into the PK-word2vec training helped reduce the difference in the similarity distribution between words with and without prior knowledge. Specifically, in the model with context vector regularization, the distribution of words with prior knowledge exhibited a longer tail as the value of \(\alpha\) increased. By contrast, without context vector regularization, the similarity distribution for words without prior knowledge was right shifted without significant changes to its shape. To investigate the root cause, we inspected the most similar words for those without prior knowledge and located in the tail of the similarity distribution around cosine similarity of 0.9. For example, consider the word miscarriage . In the model with context vector regularization, its most similar words were trimester , gestational , gestation , fetal , and fetus , all of which are directly related to pregnancy. Conversely, without context vector regularization, the most similar words were tube , defects , defect , homocysteine , and suction . While these terms are associated with miscarriage, they revealed a different pattern of similarity compared to the model with context vector regularization. This effect suggests the importance of context vector regularization in enhancing the model's compatibility with a small sample size of data. Third, despite using a relatively small dataset, our human reviewers preferred the word groups produced by PK-word2vec over those produced by SGNS, and had a higher support rate for PK-word2vec for non-medical words than medical words. There are several possible explanations for this result. First, the medical words are inherently complex and more difficult for the reviewers to evaluate. Second, the number of medical words was relatively small, such that some medical words are associated with fewer than five other words with high similarity scores. In addition, the new PK-word2vec helped improve the vector representation, especially for low-frequency words (see Fig. 6 ). While medical students likely had more medical training than MTurk workers on average, our analysis suggested that the two types of reviewers were similar regarding their preferences for non-medical and medical words. This implies that for simple medical surveys, engaging a large number of participants pool can obtain accurate results, mitigating the need for, and potential higher costs associated with, specialized professionals. In the context of customizing word representation models, our PK-word2vec method offers a distinct approach compared to previous studies. Several earlier studies have adapted SGNS to enhance word representation, primary focusing on aligning the vector space of a general, pre-trained embedding model to a specific problem domain 28,29 . Other refined strategies included improving the embedding quality by retrofitting the pre-trained word vectors using lexical relational resources 30 , and augmenting SGNS with additional knowledge-based graph models and `anchor context models 31 . Different from these approaches, our PK-word2vec method does not require a large number of training samples to fine-tune the word embedding model. This significant advantage could substantially facilitate research within the field of biomedical informatics, mainly when working with small study cohorts. Despite the encouraging findings, this study has several limitations that can serve as a basis for future research. First, PK-word2vec was examined solely on patients with breast cancer. It has yet to be determined whether the method generalizes to clinical communications or notes for other types of patients. Second, we generated prior knowledge from two pre-trained models only. It is unknown if incorporating additional pre-trained models can enhance the generalizability of PK-word2vec. Third, complex medical concepts are often represented by long phrases rather than words, which begs the question of how to include more complex concepts in human evaluations. Finally, the model should be tested in downstream NLP tasks by jointly tuning the prior knowledge weight and other hyperparameters. 5. Conclusions This paper introduces PK-word2vec, an adapted word embedding model trained on a relatively small number of patient portal messages. This was achieved by incorporating prior knowledge from two pre-trained embedding models into the model training. Our evaluation with MTurk workers and medical students demonstrated that PK-word2vec outperformed the standard word2vec model in generating more similar words from small-size patient portal messages. While this study focused on extracting prior knowledge from Google-word2ec and Snomed2vec for patient portal messages, the proposed PK-word2vec can leverage prior knowledge from any pre-trained models, irrespective of the vector size or representation space, to build vector representations for other clinical text data. Declarations Author Contribution Q.S. and Z.Y. wrote the main manuscript text and Q.S. prepared figures. All authors reviewed the manuscript. Acknowledgments This work was supported by the National Cancer Institute under award number R37CA237452. Data Availability The data that support the findings of this study are available on the Vanderbilt University Medical Center Research Derivative, but restrictions apply to the availability of these data due to privacy issue, which were used under license for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of Vanderbilt University Office of Research Informatics. References Dendere R, Slade C, Burton-Jones A, Sullivan C, Staib A, Janda M. Patient Portals Facilitating Engagement With Inpatient Electronic Medical Records: A Systematic Review. J Med Internet Res. 2019;21(4):e12779. doi: 10.2196/12779 Goel MS, Brown TL, Williams A, Cooper AJ, Hasnain-Wynia R, Baker DW. Patient reported barriers to enrolling in a patient portal. Journal of the American Medical Informatics Association. 2011;18(Supplement_1):i8–i12. Kruse CS, Bolton K, Freriks G. The effect of patient portals on quality outcomes and its implications to meaningful use: a systematic review. J Med Internet Res. 2015;17(2):e44. Ralston JD, Carrell D, Reid R, Anderson M, Moran M, Hereford J. Patient web services integrated with a shared medical record: patient use and satisfaction. Journal of the American Medical Informatics Association. 2007;14(6):798–806. Osborn CY, Rosenbloom ST, Stenner SP, et al. MyHealthAtVanderbilt: policies and procedures governing patient portal functionality. Journal of the American Medical Informatics Association. 2011;18(Supplement_1):i18–i23. Griffin A, Skinner A, Thornhill J, Weinberger M. Patient portals. Appl Clin Inform. 2016;7(02):489–501. Yin Z, Warner JL, Chen Q, Malin BA. Patient Messaging Content Associated with Initiating Hormonal Therapy after a Breast Cancer Diagnosis. In: AMIA Annual Symposium Proceedings. Vol 2019.; 2019:962. Yin Z, Harrell M, Warner JL, Chen Q, Fabbri D, Malin BA. The therapy is making me sick: how online portal communications between breast cancer patients and physicians indicate medication discontinuation. Journal of the American Medical Informatics Association. 2018;25(11):1444–1451. doi: 10.1093/jamia/ocy118 Sulieman L, Yin Z, Malin BA. Why patient portal messages indicate risk of readmission for patients with ischemic heart disease. In: AMIA Annual Symposium Proceedings. Vol 2019.; 2019:828. Sulieman L, Gilmore D, French C, et al. Classifying patient portal messages using Convolutional Neural Networks. J Biomed Inform. 2017;74:59–70. doi: https://doi.org/10.1016/j.jbi.2017.08.014 Glöggler M, Ammenwerth E. Development and Validation of a Useful Taxonomy of Patient Portals Based on Characteristics of Patient Engagement. Methods Inf Med. 2021;60(S 01):e44–e55. Wang Z, Zhang W, Liu N, Wang J. Scalable rule-based representation learning for interpretable classification. Adv Neural Inf Process Syst. 2021;34:30479–30491. Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J. Distributed Representations of Words and Phrases and their Compositionality. In: Burges CJC, Bottou L, Welling M, Ghahramani Z, Weinberger KQ, eds. Advances in Neural Information Processing Systems. Vol 26. Curran Associates, Inc.; 2013. https://proceedings.neurips.cc/paper/2013/file/9aa42b 31882ec039965f3c4923ce901b-Paper.pdf Steitz BD, Levy MA. A Social Network Analysis of Cancer Provider Collaboration. AMIA Annu Symp Proc. 2016;2016:1987–1996. Galea D, Laponogov I, Veselkov K. Sub-word information in pre-trained biomedical word representations: evaluation and hyper-parameter optimization. In: Proceedings of the BioNLP 2018 Workshop.; 2018:56–66. Martinez Soriano I, Castro Peña JL, Fernandez Breis JT, San Román I, Alonso Barriuso A, Guevara Baraza D. Snomed2Vec: Representation of SNOMED CT Terms with Word2Vec. In: 2019 IEEE 32nd International Symposium on Computer-Based Medical Systems (CBMS).; 2019:678–683. doi: 10.1109/CBMS.2019.00138 Baziotis C, Pelekis N, Doulkeridis C. DataStories at SemEval-2017 Task 4: Deep LSTM with Attention for Message-level and Topic-based Sentiment Analysis. In: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Association for Computational Linguistics; 2017:747–754. doi: 10.18653/v1/S17-2126 Soysal E, Wang J, Jiang M, et al. CLAMP – a toolkit for efficiently building customized clinical natural language processing pipelines. Journal of the American Medical Informatics Association. 2017;25(3):331–336. doi: 10.1093/jamia/ocx132 Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J. Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems.; 2013:3111–3119. Mitra B, Nalisnick E, Craswell N, Caruana R. A dual embedding space model for document ranking. arXiv preprint arXiv:160201137. Published online 2016. Grover A, Leskovec J. node2vec: Scalable feature learning for networks. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.; 2016:855–864. Bodenreider O. The unified medical language system (UMLS): integrating biomedical terminology. Nucleic Acids Res. 2004;32(suppl_1):D267–D270. Villani C, others. Optimal Transport: Old and New. Vol 338. Springer; 2009. Buhrmester M, Kwang T, Gosling SD. Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality data? Published online 2016. Dean S, Illowsky B. Descriptive statistics: skewness and the mean, median, and mode. Connexions website. Published online 2018. Jansen S. Word and phrase translation with word2vec. arXiv preprint arXiv:170503127. Published online 2017. Chiu B, Crichton G, Korhonen A, Pyysalo S. How to train good word embeddings for biomedical NLP. In: Proceedings of the 15th Workshop on Biomedical Natural Language Processing.; 2016:166–174. Sarma PK, Liang Y, Sethares WA. Domain adapted word embeddings for improved sentiment classification. arXiv preprint arXiv:180504576. Published online 2018. Poerner N, Waltinger U, Schütze H. Inexpensive Domain Adaptation of Pretrained Language Models: Case Studies on Biomedical NER and Covid-19 QA. In: Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics; 2020:1482–1490. doi: 10.18653/v1/2020.findings-emnlp.134 Faruqui M, Dodge J, Jauhar SK, Dyer C, Hovy E, Smith NA. Retrofitting word vectors to semantic lexicons. arXiv preprint arXiv:14114166. Published online 2014. Yamada I, Shindo H, Takeda H, Takefuji Y. Joint learning of the embedding of words and entities for named entity disambiguation. arXiv preprint arXiv:160101343. Published online 2016. Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Editorial decision: Revision requested 05 Jun, 2024 Reviews received at journal 31 May, 2024 Reviewers agreed at journal 22 May, 2024 Reviews received at journal 16 May, 2024 Reviewers agreed at journal 16 May, 2024 Reviewers invited by journal 14 May, 2024 Editor assigned by journal 09 May, 2024 Editor invited by journal 07 May, 2024 Submission checks completed at journal 07 May, 2024 First submitted to journal 30 Apr, 2024 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4350387","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":301026744,"identity":"73a00d82-625c-486d-b580-ef7998234fd6","order_by":0,"name":"Qingyuan Song","email":"","orcid":"","institution":"Vanderbilt University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Qingyuan","middleName":"","lastName":"Song","suffix":""},{"id":301026745,"identity":"32177db1-357a-469e-90db-dc8018393047","order_by":1,"name":"Congning Ni","email":"","orcid":"","institution":"Vanderbilt University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Congning","middleName":"","lastName":"Ni","suffix":""},{"id":301026746,"identity":"c1cd90c3-68fe-4368-8983-0b06287cb844","order_by":2,"name":"Jeremy L. Warner","email":"","orcid":"","institution":"Brown University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Jeremy","middleName":"L.","lastName":"Warner","suffix":""},{"id":301026747,"identity":"8ebbcc36-1ee2-4c83-af7a-7e96922eaa33","order_by":3,"name":"Qingxia Chen","email":"","orcid":"","institution":"Vanderbilt University Medical Center","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Qingxia","middleName":"","lastName":"Chen","suffix":""},{"id":301026748,"identity":"ac55015c-6893-4d42-be23-eefa6fa54b20","order_by":4,"name":"Lijun Song","email":"","orcid":"","institution":"Vanderbilt University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Lijun","middleName":"","lastName":"Song","suffix":""},{"id":301026749,"identity":"dfdf956f-5730-40a8-903e-1d9293bc962c","order_by":5,"name":"S. Trent Rosenbloom","email":"","orcid":"","institution":"Vanderbilt University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"S.","middleName":"Trent","lastName":"Rosenbloom","suffix":""},{"id":301026750,"identity":"a656e64d-1cad-4bc1-a691-2e781a0cf48f","order_by":6,"name":"Bradley A. Malin","email":"","orcid":"","institution":"Vanderbilt University","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Bradley","middleName":"A.","lastName":"Malin","suffix":""},{"id":301026751,"identity":"76d6b586-1f66-4c43-be30-ab895a947ae1","order_by":7,"name":"Zhijun Yin","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA4UlEQVRIiWNgGAWjYDACCcYGAxDNBsQHPpCqhfHgDOK0IJjMh3mI0SE/u7mhmHcHQ2KfdAPDYdu2w9H8DczHPn7Bo4VxzsEGY94zDIltMgcYDue2Hc6dcYAtebYMHi3MEolALW1ALRIJDIdzzhzO3cDAY8wsgUcLG4oWC2K08KBoYaiAaGHEF9oSQC2Gc9skjNuAjIM9Fem5Mw6zJTPj0cEgPyP9mcHbNhvZ+TOSD3/4YWCd29/efJjxBz49QO8Ao1LCsYGBsQHCB1rBTCCCmB8ACXsUIUK2jIJRMApGwcgCAMUvSKxKlsYpAAAAAElFTkSuQmCC","orcid":"","institution":"Vanderbilt University","correspondingAuthor":true,"submittingAuthor":false,"prefix":"","firstName":"Zhijun","middleName":"","lastName":"Yin","suffix":""}],"badges":[],"createdAt":"2024-04-30 16:14:22","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4350387/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4350387/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":56550294,"identity":"8208dd63-7f5d-4430-9be3-ec6591928c73","added_by":"auto","created_at":"2024-05-15 15:50:16","extension":"jpeg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":104544,"visible":true,"origin":"","legend":"\u003cp\u003eWorkflow associated with the creation of the PK-word2vec model.\u003c/p\u003e","description":"","filename":"floatimage1.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-4350387/v1/a0d67a3dd93c7e079171d941.jpeg"},{"id":56550299,"identity":"0ca1b9f7-9420-459a-8cb4-bf7d670c77cc","added_by":"auto","created_at":"2024-05-15 15:50:18","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":76677,"visible":true,"origin":"","legend":"\u003cp\u003eAn illustration of a context window with a size of 1, a center word and its neighbors in the SGNS framework.\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-4350387/v1/c723c3f78464c6df5ddce980.png"},{"id":56550298,"identity":"41247b21-cd5a-4056-a26f-f43665af32ef","added_by":"auto","created_at":"2024-05-15 15:50:17","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":47837,"visible":true,"origin":"","legend":"\u003cp\u003eThe Wasserstein distance in the similarity distribution between Google-word2vec and SGNS as a function of embedding size. A\u003cstrong\u003e \u003c/strong\u003esensitivity analysis to optimize the word embedding size.\u003c/p\u003e","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-4350387/v1/3576db71d89f89c8086343cc.png"},{"id":56550296,"identity":"762bc0fe-530a-45ba-afc7-2427d63152de","added_by":"auto","created_at":"2024-05-15 15:50:17","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":111499,"visible":true,"origin":"","legend":"\u003cp\u003eSensitivity of similarity distribution (left) and Wasserstein distance between Google-word2vec and PK-word2vec on various word frequencies (right) across the range of \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;values.\u003c/p\u003e","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-4350387/v1/8153b9353f537ba4978772ea.png"},{"id":56550300,"identity":"d4228c6e-46de-46f4-bcab-e592743f203a","added_by":"auto","created_at":"2024-05-15 15:50:18","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":174435,"visible":true,"origin":"","legend":"\u003cp\u003eThe distribution of the words with the 10 highest similarity scores of words with and without context vector regularization during model training.\u003c/p\u003e","description":"","filename":"floatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-4350387/v1/eb4c016202da8f3ddadeda73.png"},{"id":56550295,"identity":"909005b3-62f6-4b60-b5e8-44873c39be0e","added_by":"auto","created_at":"2024-05-15 15:50:17","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":62957,"visible":true,"origin":"","legend":"\u003cp\u003eThe number of tasks where one model was preferred over the other based on a majority vote by MTurk workers (upper row) and medical students (lower row).\u003c/p\u003e","description":"","filename":"floatimage6.png","url":"https://assets-eu.researchsquare.com/files/rs-4350387/v1/bf422ff4b45becaff8f1b111.png"},{"id":56550293,"identity":"fbdf556b-66e5-41dd-98d7-a80683384b8c","added_by":"auto","created_at":"2024-05-15 15:50:16","extension":"png","order_by":7,"title":"Figure 7","display":"","copyAsset":false,"role":"figure","size":74870,"visible":true,"origin":"","legend":"\u003cp\u003eDistribution of the proportion of reviewers that choose PK-word2vec over SGNS per annotation task (n = 1000). Different colors indicate the word frequency ranges of the original word in each task. The vertical dashed line corresponds to 0.5, which indicate that there was no difference in the annotator preference.\u003c/p\u003e","description":"","filename":"floatimage7.png","url":"https://assets-eu.researchsquare.com/files/rs-4350387/v1/87f3d657895f7c3b838212ff.png"},{"id":56551022,"identity":"bb3af1e2-61cd-4fd3-9977-7b8bd2e4b45e","added_by":"auto","created_at":"2024-05-15 15:58:17","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1108403,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4350387/v1/3cdb1bc2-75e3-40a1-a5b0-7838fe4076fc.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Optimizing Word Embeddings for Patient Portal Message Datasets with a Small Number of Samples","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003ePatient portals have become standard components of modern electronic health record (EHR) systems.\u003csup\u003e1\u003c/sup\u003e They enable patients to access health information, manage clinical appointments, and communicate securely with care providers in an asynchronous manner.\u003csup\u003e2,3\u003c/sup\u003e The messaging functionality of patient portals has gained in popularity over time\u003csup\u003e4,5\u003c/sup\u003e due to various benefits, such as the ability to increase patient involvement in medical decision-making\u003csup\u003e6\u003c/sup\u003e. At the same time, the information conveyed in patient portal messages can serve as a foundation for clinical research, such as studies of initiation and discontinuation of cancer therapies\u003csup\u003e7,8\u003c/sup\u003e, assessing readmission risk\u003csup\u003e9,10\u003c/sup\u003e, and creating taxonomies of patient engagement\u003csup\u003e11\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eHowever, maximizing the utility of patient portal messages requires a means to extract meaningful signals from unstructured text.\u003csup\u003e12\u003c/sup\u003e Rule-based methods, which rely on manually pre-defined rules to perform a task, are ineffective as the volume of data or complexity of the task increases \u003csup\u003e12\u003c/sup\u003e. While specific computational methods, such as topic modeling, can be applied in this regard, these approaches typically rely on word frequency to determine the most representative words for each topic. By contrast, word embedding models, such as word2vec\u003csup\u003e13\u003c/sup\u003e, have been widely adopted by natural language processing (NLP) applications. These models can learn vector representations, typically from millions of documents, to capture each word\u0026rsquo;s semantic and linguistic meaning. However, it is challenging to procure a cohort that is sufficiently large to train a specific word embedding model. As an illustration, in this paper, we focus on breast cancer as it serves as an exemplar of a complex care coordination setting between various specialists who often communicate with patients via the patient portal\u003csup\u003e14\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003e The current approaches for generating word embeddings on relatively small corpora may learn semantic meaning inaccurately due to lack of sufficient occurrences of word usage. A natural solution to this problem is to fine-tune a word embedding model that is pre-trained on a larger data corpus. This is because the fitted parameters of a pre-trained model can serve as a better starting point for training than random initialization. Yet this strategy presents certain limitations as well. In particular, pre-trained models are incapable of handling words that are absent from the pre-trained model's vocabulary. Moreover, inconsistencies arise due to the incompatible linguistic contexts between the data used to fit the pre-trained model and the data for a specific domain of interest.\u003csup\u003e15\u003c/sup\u003e\u003c/p\u003e \u003cp\u003eIn this study, we introduce a novel approach, PK-word2vec, to train word2vec models on a patient portal message corpora of relatively small sample size. Our model leverages prior information, as defined by the most similar words sampled from pre-trained embedding models, to regularize the model training. We demonstrate the utility of this approach using a dataset of approximately 137,000 patient portal messages from 1,389 patients diagnosed with breast cancer at a large academic medical center. \u003csup\u003e1416\u003c/sup\u003eOur experiments show that the regularization process of PK-word2vec can guide and constrain the model training to learn word semantic meanings more effectively than standard word2vec models.\u003c/p\u003e"},{"header":"2. Methods","content":"\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eFigure 1 depicts the workflow for this study: 1) data extraction and cleaning; 2) prior knowledge extraction from two pre-trained models, Snomed2vec\u003csup\u003e16\u003c/sup\u003e and the Google pre-trained word2vec (Google-word2vec)\u003csup\u003e13\u003c/sup\u003e for medical and non-medical words, respectively; 3) model training regularized by prior knowledge (PK-word2vec); and 4) similarity analysis and human evaluation of the model performance. This study was deemed exempt from human subjects research by the Vanderbilt University Institutional Review Board.\u003c/p\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cdiv id=\"Sec3\" class=\"Section2\"\u003e\n\u003ch2\u003e2.1. Data Collection and Preprocessing\u003c/h2\u003e\n\u003cp\u003eWe collected the patient portal messages from the EHR system of Vanderbilt University Medical Center (VUMC), a large, private, non-profit, academic medical center in Nashville, Tennessee. VUMC launched its patient portal, \u003cem\u003eMy Health at Vanderbilt\u003c/em\u003e (\u003cem\u003eMHAV\u003c/em\u003e), in 2004, which was migrated to Epic\u0026rsquo;s MyChart platform in 2017. MHAV supports secure messaging between healthcare providers and patients, appointment scheduling, billing management, and access to laboratory results or other EHR data.\u003csup\u003e5\u003c/sup\u003e\u003c/p\u003e\n\u003cp\u003eIn this study, we focused on the patient portal messages for patients with breast cancer who were prescribed the most common hormonal therapy medications: \u003cem\u003eanastrozole\u003c/em\u003e, \u003cem\u003eexemestane\u003c/em\u003e, \u003cem\u003eletrozole\u003c/em\u003e, \u003cem\u003eraloxifene\u003c/em\u003e, and/or \u003cem\u003etamoxifen\u003c/em\u003e. We masked URLs, email addresses, phone numbers, timestamps, and numerals by using the \u003cem\u003eekphrasis\u003c/em\u003e (v0.5.1)\u003csup\u003e17\u003c/sup\u003e. Afterwards, all personal names and usernames were manually masked, and misspelled words were corrected with the guidance of two domain experts (STR, JLW) by referring to the contexts from randomly selected messages that contain these misspelled words. We applied the CLAMP software package (v1.6.4) to detect medical words pertaining to diseases, laboratory tests, and medications from messages\u003csup\u003e18\u003c/sup\u003e.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec4\" class=\"Section2\"\u003e\n\u003ch2\u003e2.2. Skip-Gram Model with Negative Sampling\u003c/h2\u003e\n\u003cp\u003e\u0026nbsp;\u003c/p\u003e\n\u003cp\u003ePK-word2vec is built on a skip-gram model with negative sampling (SGNS)\u003csup\u003e13\u003c/sup\u003e, a standard word2vec. In SGNS, each word has two vector representations: 1) a center vector when it is processed as a center word, and 2) a context vector when it serves as a neighbor of a center word. Figure\u0026nbsp;\u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e illustrates the center word and neighbor word within a context window size of 1 in the sentence \"\u003cem\u003ePlease reschedule my appointment to Friday\u003c/em\u003e\u0026rdquo;. In this example, \u003cem\u003ereschedule\u003c/em\u003e is one of the neighbors of the center word \u003cem\u003eappointment\u003c/em\u003e. Formally, SGNS structures the model training as a binary classification problem, where the objective is to predict whether a certain pair of words are neighbors. Formally, given a sequence of words \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({w}_{1}, {w}_{2},\\cdots , {w}_{T}\\)\u003c/span\u003e\u003c/span\u003eand a context window of size \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(m\\)\u003c/span\u003e\u003c/span\u003e, the SGNS objective function is defined as:\u003c/p\u003e\n\u003cdiv id=\"Equ1\" class=\"Equation\"\u003e\n\u003cdiv id=\"FileID_Equ1\" class=\"mathdisplay\"\u003e$${Loss}_{SGNS}=\\sum _{t=1}^{T}\\left[\\sum _{-m\\le j\\le m}\\left[-\\text{log}\\sigma \\left({\\mathbf{u}}_{t+j}^{T}{\\mathbf{v}}_{t}\\right)-\\sum _{k=1, {w{\\prime }}_{k}\\sim P\\left({w}_{t}\\right)}^{K}\\text{log}\\sigma \\left({-\\mathbf{u}}_{k}^{T}{\\mathbf{v}}_{t}\\right)\\right]\\right]$$\u003c/div\u003e\n\u003cdiv class=\"EquationNumber\"\u003e1\u003c/div\u003e\n\u003c/div\u003e\n\u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\mathbf{u}}_{t}\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\mathbf{v}}_{t}\\)\u003c/span\u003e\u003c/span\u003e represent the context and center vector of \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({w}_{t}\\)\u003c/span\u003e\u003c/span\u003e, which are the parameters optimized in the loss function; \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({w{\\prime }}_{k}\\)\u003c/span\u003e\u003c/span\u003e represents a random word that does not appear in the neighboring context of \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({w}_{t}\\)\u003c/span\u003e\u003c/span\u003e. \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({w{\\prime }}_{k}\\)\u003c/span\u003e\u003c/span\u003e is sampled from \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(P\\left({w}_{t}\\right)\\)\u003c/span\u003e\u003c/span\u003e, a unigram distribution defined by the frequency of occurrence of each word\u003csup\u003e19\u003c/sup\u003e; \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(K\\)\u003c/span\u003e\u003c/span\u003e is a hyperparameter that specifies the number of negative words sampled for \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({w}_{t}\\)\u003c/span\u003e\u003c/span\u003e; and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\sigma\\)\u003c/span\u003e\u003c/span\u003e is the sigmoid function. The first term in Eq.\u0026nbsp;(\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e) aims to maximize the probability of occurrence for words that are in the context window. By contrast, the second term in Eq.\u0026nbsp;(\u003cspan class=\"InternalRef\"\u003e1\u003c/span\u003e) aims to minimize the probability of occurrence for words that are not in the context window.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec5\" class=\"Section2\"\u003e\n\u003ch2\u003e2.3. Prior Knowledge Word2vec Model\u003c/h2\u003e\n\u003cp\u003eThe challenge of training a word2vec model on a small dataset arises from the insufficient context that such datasets provide. To address this issue, we define the prior knowledge of a word as its most similar words identified from other pre-trained models. Subsequently, the prior knowledge is applied to regularize the PK-word2vec model training. More formally, if a word in the patient portal messages \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({w}_{t}\\)\u003c/span\u003e\u003c/span\u003e also appears in a pre-trained embedding vocabulary, the prior knowledge of \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({w}_{t}\\)\u003c/span\u003e\u003c/span\u003e is defined by another word \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({w}_{j}\\)\u003c/span\u003e\u003c/span\u003e that is sampled with probability \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({p}_{tj}\\)\u003c/span\u003e\u003c/span\u003e from the patient portal message vocabulary. In this respect, the more similar \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({w}_{j}\\)\u003c/span\u003e\u003c/span\u003e is to \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({w}_{t}\\)\u003c/span\u003e\u003c/span\u003e in the pre-trained embedding model, the greater the likelihood that \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({w}_{j}\\)\u003c/span\u003e\u003c/span\u003e will be sampled.\u003c/p\u003e\n\u003cp\u003eWe introduce a prior knowledge loss function for the entire dataset, which is defined as follows:\u003c/p\u003e\n\u003cdiv id=\"Equ2\" class=\"Equation\"\u003e\n\u003cdiv id=\"FileID_Equ2\" class=\"mathdisplay\"\u003e$${\\text{L}\\text{o}\\text{s}\\text{s}}_{\\text{P}\\text{K}}=\\left\\{\\begin{array}{cc}{\\sum }_{t=1}^{T}\\left[\\gamma \\right({w}_{t})\\bullet {\\Psi }\\left({\\mathbf{u}}_{t},{\\mathbf{u}}_{j},{\\mathbf{v}}_{t},{\\mathbf{v}}_{j}\\right)]\u0026amp; {w}_{t} \\text{w}\\text{i}\\text{t}\\text{h} \\text{p}\\text{r}\\text{i}\\text{o}\\text{r} \\text{k}\\text{n}\\text{o}\\text{w}\\text{l}\\text{e}\\text{d}\\text{g}\\text{e}\\\\ 0\u0026amp; {w}_{t} \\text{w}\\text{i}\\text{t}\\text{h}\\text{o}\\text{u}\\text{t} \\text{p}\\text{r}\\text{i}\\text{o}\\text{r} \\text{k}\\text{n}\\text{o}\\text{w}\\text{l}\\text{e}\\text{d}\\text{g}\\text{e}\\end{array}\\right.$$\u003c/div\u003e\n\u003cdiv class=\"EquationNumber\"\u003e2\u003c/div\u003e\n\u003c/div\u003e\n\u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\gamma \\left({w}_{t}\\right)=\\tau /\\text{f}\\text{r}\\text{e}\\text{q}\\left({w}_{t}\\right)\\)\u003c/span\u003e\u003c/span\u003e discounts the words proportionally to their frequency, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\tau\\)\u003c/span\u003e\u003c/span\u003e is a hyperparameter, and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\Psi }\\left(\u0026middot;\\right)\\)\u003c/span\u003e\u003c/span\u003e is a regularization factor to be defined in Eq.\u0026nbsp;(\u003cspan class=\"InternalRef\"\u003e4\u003c/span\u003e) below. Note that if a word does not exist in the pre-trained model, then it has no prior knowledge and its corresponding term in \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({\\text{L}\\text{o}\\text{s}\\text{s}}_{\\text{P}\\text{K}}\\)\u003c/span\u003e\u003c/span\u003e is set to zero.\u003c/p\u003e\n\u003cp\u003eIn this work, we calculate word similarity as the cosine similarity of two vectors \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\mathbf{u}\\)\u003c/span\u003e\u003c/span\u003e and \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\mathbf{v}\\)\u003c/span\u003e\u003c/span\u003e, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\text{cos}(u,v)=u\\bullet v/(\\left|u\\right|\\bullet \\left|v\\right|)\\)\u003c/span\u003e\u003c/span\u003e. We require that each sampled \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({w}_{j}\\)\u003c/span\u003e\u003c/span\u003e have a similarity score with \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({w}_{t}\\)\u003c/span\u003e\u003c/span\u003e above a pre-defined threshold \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\theta\\)\u003c/span\u003e\u003c/span\u003e. As such, \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({p}_{tj}\\)\u003c/span\u003e\u003c/span\u003e is defined as:\u003c/p\u003e\n\u003cdiv id=\"Equ3\" class=\"Equation\"\u003e\n\u003cdiv id=\"FileID_Equ3\" class=\"mathdisplay\"\u003e$${p}_{tj}=\\left\\{\\begin{array}{cc}\\text{s}\\text{o}\\text{f}\\text{t}\\text{m}\\text{a}\\text{x}\\left[\\text{cos}({\\mathbf{u}}_{t},{\\mathbf{u}}_{j})\\right]\u0026amp; \\text{cos}({\\mathbf{u}}_{t},{\\mathbf{u}}_{j})\\ge \\theta \\\\ 0\u0026amp; \\text{cos}({\\mathbf{u}}_{t},{\\mathbf{u}}_{j})\u0026lt;\\theta \\end{array}\\right.$$\u003c/div\u003e\n\u003cdiv class=\"EquationNumber\"\u003e3\u003c/div\u003e\n\u003c/div\u003e\n\u003cp\u003eIn our experiments, for each pre-trained model, we set \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\theta\\)\u003c/span\u003e\u003c/span\u003e to the average of the top 10 similarity scores for words with prior knowledge in patient portal messages. In practice, each time \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({w}_{t}\\)\u003c/span\u003e\u003c/span\u003e is processed in model training using gradient descent optimization, a word \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({w}_{j}\\)\u003c/span\u003e\u003c/span\u003e is sampled and its cosine similarity with \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({w}_{t}\\)\u003c/span\u003e\u003c/span\u003e in both context and center vectors is calculated as its prior knowledge. Then, the corresponding item for \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\({w}_{t}\\)\u003c/span\u003e\u003c/span\u003e that needs to be minimized in the PK-word2vec loss function in Eq.\u0026nbsp;(\u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e) is:\u003c/p\u003e\n\u003cdiv id=\"Equ4\" class=\"Equation\"\u003e\n\u003cdiv id=\"FileID_Equ4\" class=\"mathdisplay\"\u003e$${\\Psi }\\left({\\mathbf{u}}_{t},{\\mathbf{u}}_{j},{\\mathbf{v}}_{t},{\\mathbf{v}}_{j}\\right)=\\left[1-\\text{cos}\\left({\\mathbf{u}}_{t},{\\mathbf{u}}_{j}\\right)\\right]+[1-\\text{cos}({\\mathbf{v}}_{t},{\\mathbf{v}}_{j})]$$\u003c/div\u003e\n\u003cdiv class=\"EquationNumber\"\u003e4\u003c/div\u003e\n\u003c/div\u003e\n\u003cp\u003eThere are several reasons for incorporating context vector similarity into Eq.\u0026nbsp;(\u003cspan class=\"InternalRef\"\u003e4\u003c/span\u003e). First, a key characteristic of high-quality word embeddings is that the distance between the context vectors for similar words is relatively small \u003csup\u003e20\u003c/sup\u003e. Second, since the context vectors are applied to estimate the center vectors of neighboring words, the regularization of the context vectors can propagate to the center vectors of neighboring words without prior knowledge. Finally, the overall objective function for PK-word2vec can be defined as:\u003c/p\u003e\n\u003cdiv id=\"Equ5\" class=\"Equation\"\u003e\n\u003cdiv id=\"FileID_Equ5\" class=\"mathdisplay\"\u003e$$\\text{L}\\text{o}\\text{s}\\text{s}={\\text{L}\\text{o}\\text{s}\\text{s}}_{\\text{S}\\text{G}\\text{N}\\text{S}}+\\alpha \\times {\\text{L}\\text{o}\\text{s}\\text{s}}_{\\text{P}\\text{K}}$$\u003c/div\u003e\n\u003cdiv class=\"EquationNumber\"\u003e2\u003c/div\u003e\n\u003c/div\u003e\n\u003cp\u003ewhere \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\alpha\\)\u003c/span\u003e\u003c/span\u003e is a hyperparameter that determines the degree to which the training of PK-word2vec relies on prior knowledge. A higher \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\alpha\\)\u003c/span\u003e\u003c/span\u003e indicates a stronger dependency on prior knowledge. To evaluate the impact of \u0026alpha;, we compared our approach to a baseline model where \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\alpha =0\\)\u003c/span\u003e\u003c/span\u003e, a condition corresponding to SGNS. We generate prior knowledge for non-medical and medical words using two different models. Specifically, we applied Google-word2vec, which was trained on the Google News dataset (about 100\u0026nbsp;billion words), to build prior knowledge for non-medical words. We turned to a \u003cem\u003eSnomed2vec\u003c/em\u003e model, which was trained on the SNOMED-CT knowledge graph\u003csup\u003e16\u003c/sup\u003e to obtain prior knowledge for medical words.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec6\" class=\"Section2\"\u003e\n\u003ch2\u003e2.4. Hyperparameter Selection\u003c/h2\u003e\n\u003cp\u003eHyperparameter tuning for word2vec models is challenging because it is difficult to define similarity. Since Google-word2vec was trained on a vast corpus, we selected hyperparameters, the dimension of word embedding \u003cem\u003ed\u003c/em\u003e and the prior knowledge weight \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\alpha\\)\u003c/span\u003e\u003c/span\u003e, such that PK-word2vec is similar to Google-word2vec in terms of the distribution of the similarity scores between a given word and its 10 most similar words. In this paper we name this distribution as similarity distribution for simplicity. We did not use Snomed2vec as a reference because it is inherently a node2vec\u003csup\u003e21\u003c/sup\u003e trained on a semantic network in the Unified Medical Language System\u003csup\u003e22\u003c/sup\u003e rather than text data.\u003c/p\u003e\n\u003cp\u003eWe set the embedding size \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(d\\)\u003c/span\u003e\u003c/span\u003e with a value that corresponds to the smallest Wasserstein distance\u003csup\u003e23\u003c/sup\u003e in the similarity distributions between the two models. To select \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\alpha\\)\u003c/span\u003e\u003c/span\u003e, we first categorized words into high-, mid-, and low-frequency ranges based on their tertiles. Then, we set \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\alpha\\)\u003c/span\u003e\u003c/span\u003e with a value that corresponds to the smallest Wasserstein distance in the similarity distribution between three distinct word frequency ranges. This resulted in a comparison in an unbiased manner, since high-frequency words tended to receive better representations. When training PK-word2vec, the \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\theta\\)\u003c/span\u003e\u003c/span\u003e threshold was set to the average word similarity between all the words and their 10 most similar words in the pre-trained models. We set the threshold \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\tau\\)\u003c/span\u003e\u003c/span\u003e in Eq.\u0026nbsp;(\u003cspan class=\"InternalRef\"\u003e2\u003c/span\u003e) to 0.00001, batch size to 10000, and trained the model for 10 epochs.\u003c/p\u003e\n\u003c/div\u003e\n\u003cdiv id=\"Sec7\" class=\"Section2\"\u003e\n\u003ch2\u003e2.5. Human Evaluation\u003c/h2\u003e\n\u003cp\u003eTo compare PK-word2vec with SGNS, we randomly sampled 1,000 words from the patient portal message vocabulary, with the same proportion of medical to non-medical words. For each term of interest, we created one group of the five most similar words generated from PK-word2vec and SGNS, respectively. To evaluate the system, we asked each reviewer to indicate which word group was more related to the word of interest. We randomized the order of the two groups in each task to avoid framing biases.\u003c/p\u003e\n\u003cp\u003eWe recruited two groups of reviewers for the model evaluation. The first group consisted of 330 Amazon Mechanical Turk (MTurk) workers\u003csup\u003e24\u003c/sup\u003e, recognized as \u0026ldquo;masters\u0026rdquo; due to their consistently submission of high-quality results in their history annotations. Each MTurk worker completed 100 out of the 1000 tasks and each task was answered by 33 different MTurk workers. The second group consisted of 7 medical students recruited through the Vanderbilt Institute for Clinical and Translational Research. Each medical student completed all 1,000 tasks. We set a 30-second time limit for each task; task not completed within this time limit were reassigned to a new worker.\u003c/p\u003e\n\u003cp\u003eTo analyze the human evaluation data, we first calculated the support rate, defined as the proportion of the 1,000 tasks for which MTurk workers or medical students preferred PK-word2vec by a majority vote. Next, we employed sample skewness\u003csup\u003e25\u003c/sup\u003e to examine the distribution of the proportion of reviewers who favored PK-word2vec in each task. This provided insights into the extent to which PK-word2vec was favored over SGNS. A larger value for left sample skew indicates a preference of PK-word2vec over SGNS. Finally, to account for the heterogeneity, we fitted a mixed-effects logistic regression model to assess which word embedding model was preferred by MTurk workers. The type of model (PK-word2vec or SGNS) and the category of the term of interest (medical or non-medical words) were variables with fixed effects. We assigned a value of 1 to the PK-word2vec model and medical words and a value of 0 to the SGNS and non-medical words in the corresponding variables.\u003c/p\u003e\n\u003c/div\u003e"},{"header":"3. Results","content":"\u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003e3.1. Data Summary\u003c/h2\u003e \u003cp\u003eThe dataset was composed of 137,554 messages sent by 1,389 patients with breast cancer. After preprocessing, there was a total of 10,683 unique words in the vocabulary, of which 8,878 (80%) and 1,895 (20%) were non-medical and medical words, respectively. Prior knowledge was available for 7,981 (90%) non-medical and 1,116 (59%) medical words.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003e3.2. Hyperparameter Selection\u003c/h2\u003e \u003cp\u003e \u003cb\u003eEmbedding size (\u003c/b\u003e \u003cspan class=\"InlineEquation\"\u003e \u003cspan class=\"mathinline\"\u003e\\(d\\)\u003c/span\u003e \u003c/span\u003e \u003cb\u003e)\u003c/b\u003e. The word embedding size was selected from a range of 10 to 300. Figure\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e3\u003c/span\u003e depicts the Wasserstein distance between SGNS and the Google-word2vec as a function of the word embedding size. In the following experiments, we present results with an embedding size of 45 for SGNS and PK-word2vec because it corresponds to the smallest distance among all the candidate vector sizes.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003ePrior knowledge weight (\u003c/b\u003e \u003cspan class=\"InlineEquation\"\u003e \u003cspan class=\"mathinline\"\u003e\\(\\alpha\\)\u003c/span\u003e \u003c/span\u003e \u003cb\u003e)\u003c/b\u003e. Figure\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e4\u003c/span\u003ea shows the similarity distribution for PK-word2vec as a function of \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\alpha\\)\u003c/span\u003e\u003c/span\u003e. When \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\alpha =0\\)\u003c/span\u003e\u003c/span\u003e, this model corresponds to SGNS and most of the density concentrates around 0.6. As \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\alpha\\)\u003c/span\u003e\u003c/span\u003e increases, the prior knowledge becomes dominant, and the density gradually shifts toward two directions, forming two peaks. Figure\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e4\u003c/span\u003eb shows the Wasserstein distance in the similarity distribution between Google-Word2vec (three dashed lines at the bottom of the figure) and PK-word2vec (three solid lines at the top of the figure) in different word frequency ranges. The distributions for Google-word2vec were nearly flat and maintained relatively small values. Based on this observation, we set \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\alpha\\)\u003c/span\u003e\u003c/span\u003e to \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(5\\times {10}^{-6}\\)\u003c/span\u003e\u003c/span\u003e for the following experiments because it corresponds to the smallest Wasserstein distances as well as smallest variance across three pairs of comparisons.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003e3.3. Word Similarity\u003c/h2\u003e \u003cp\u003e \u003cb\u003eContext vector regularization.\u003c/b\u003e Figure\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e5\u003c/span\u003e depicts the similarity distribution for the models with (panels a-d) and without (panels e-g) context vector regularization. The blue and orange lines in each graph correspond to words with and without prior knowledge, respectively. The comparisons between two similarity distributions are illustrated in settings with different pre-selected \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\alpha\\)\u003c/span\u003e\u003c/span\u003e values. As extreme examples, Figs.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e5\u003c/span\u003ed and \u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e5\u003c/span\u003eg show that when \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\alpha ={10}^{-4}\\)\u003c/span\u003e\u003c/span\u003e, context vector regularization mitigates the skew of the similarity distribution for the words with prior knowledge. While a large \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\alpha\\)\u003c/span\u003e\u003c/span\u003e value shifts the mode of the similarity distribution for words without prior knowledge in the model without context vector regularization to the right, the variance of similarity distribution is not significantly altered. Figures\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e5\u003c/span\u003ec and \u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e5\u003c/span\u003ef depict the scenario where \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\alpha\\)\u003c/span\u003e\u003c/span\u003e is set to its optimal value of \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(5\\times {10}^{-6}\\)\u003c/span\u003e\u003c/span\u003e, where the difference between the two distributions is in an intermediate state compared with other \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\alpha\\)\u003c/span\u003e\u003c/span\u003e values. These results show that context vector regularization mitigates the impact of prior knowledge on the similarity distribution between words with and without prior knowledge.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003e3.4. Human Evaluation\u003c/h2\u003e \u003cp\u003eFigure \u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e6\u003c/span\u003e displays the human evaluation results based on a majority vote from the MTurk workers and medical students. In addition to an overall comparison, the results were stratified by the word type, and the existence of prior knowledge. The difference was negligible for all comparisons between the two groups of reviewers (\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\text{p}=0.774\\)\u003c/span\u003e\u003c/span\u003e under a paired t-test). In over 90% of the tasks, both groups of reviewers indicated that PK-word2vec generates more similar words than SGNS. For medical words, the support rate for PK-word2vec was 78.8% for MTurk workers and 71.8% for medical students. For non-medical words, the support rate for PK-word2vec was 93.6% for MTurk workers and 93.4% for medical students. For words lacking prior knowledge, the support rate for PK-word2vec was 66.7% from MTurk workers and 61.4% from medical students.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eFigure \u003cspan refid=\"Fig6\" class=\"InternalRef\"\u003e7\u003c/span\u003e shows the distribution of the proportion of reviewers that favor PK-word2vec in each task. The three curves represent the proportion distribution for the low-, mid-, and high-frequency words, respectively. All distributions are strongly left-skewed and most of the density falls between 0.8 and 1.0, indicating reviewers consistently favored PK-word2vec. Notably, in the responses from MTurk workers, the sample skew was 1.19, 1.65, and 1.92 for the high-, mid-, and low-frequency words, respectively. In the responses from the medical students, the sample skew was 0.88, 1.46, and 1.84 for the high, mid-, and low-frequency words, respectively. In other words, the degree to which the distribution was left-skewed was inversely proportional to the word frequency.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eThe results of the mixed-effect model showed that PK-word2vec was significantly more favored than SGNS (\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\beta = 2.702\\)\u003c/span\u003e\u003c/span\u003e, p-value\u0026thinsp;=\u0026thinsp;0.01). By contrast, no difference was found between medical word and non-medical word (\u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\beta = 0\\)\u003c/span\u003e\u003c/span\u003e, p-value\u0026thinsp;=\u0026thinsp;1).\u003c/p\u003e \u003c/div\u003e"},{"header":"4. Discussion","content":"\u003cp\u003eThis study has several findings that are worth highlighting. First, relying on external word pairs to tune hyperparameters, as has been done in many prior studies\u003csup\u003e13,26,27\u003c/sup\u003e, is suboptimal for smaller datasets. This limitation arises from the fact that many word pairs in external datasets may not be included in the target datasets. To address this issue, we selected hyperparameters by minimizing the distance in the similarity distribution between Google-word2vec and PK-word2vec. Our experiments showed that the prior knowledge weight \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\alpha\\)\u003c/span\u003e\u003c/span\u003e changed the word similarity distribution from a unimodal to a bimodal form, suggesting that introducing too much prior knowledge might degrade the quality of word embedding models. However, when comparing the similarity distribution between words across different frequency ranges, we observed that when the prior knowledge weight was very small or very large, it caused the similarity distribution to become dissimilar amongst these word frequency ranges. This finding highlights the necessity of carefully balancing the weight of prior knowledge within the model training.\u003c/p\u003e \u003cp\u003eSecond, including context vector regularization into the PK-word2vec training helped reduce the difference in the similarity distribution between words with and without prior knowledge. Specifically, in the model with context vector regularization, the distribution of words with prior knowledge exhibited a longer tail as the value of \u003cspan class=\"InlineEquation\"\u003e\u003cspan class=\"mathinline\"\u003e\\(\\alpha\\)\u003c/span\u003e\u003c/span\u003e increased. By contrast, without context vector regularization, the similarity distribution for words without prior knowledge was right shifted without significant changes to its shape. To investigate the root cause, we inspected the most similar words for those without prior knowledge and located in the tail of the similarity distribution around cosine similarity of 0.9. For example, consider the word \u003cem\u003emiscarriage\u003c/em\u003e. In the model with context vector regularization, its most similar words were \u003cem\u003etrimester\u003c/em\u003e, \u003cem\u003egestational\u003c/em\u003e, \u003cem\u003egestation\u003c/em\u003e, \u003cem\u003efetal\u003c/em\u003e, and \u003cem\u003efetus\u003c/em\u003e, all of which are directly related to pregnancy. Conversely, without context vector regularization, the most similar words were \u003cem\u003etube\u003c/em\u003e, \u003cem\u003edefects\u003c/em\u003e, \u003cem\u003edefect\u003c/em\u003e, \u003cem\u003ehomocysteine\u003c/em\u003e, and \u003cem\u003esuction\u003c/em\u003e. While these terms are associated with miscarriage, they revealed a different pattern of similarity compared to the model with context vector regularization. This effect suggests the importance of context vector regularization in enhancing the model's compatibility with a small sample size of data.\u003c/p\u003e \u003cp\u003e Third, despite using a relatively small dataset, our human reviewers preferred the word groups produced by PK-word2vec over those produced by SGNS, and had a higher support rate for PK-word2vec for non-medical words than medical words. There are several possible explanations for this result. First, the medical words are inherently complex and more difficult for the reviewers to evaluate. Second, the number of medical words was relatively small, such that some medical words are associated with fewer than five other words with high similarity scores. In addition, the new PK-word2vec helped improve the vector representation, especially for low-frequency words (see Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e6\u003c/span\u003e). While medical students likely had more medical training than MTurk workers on average, our analysis suggested that the two types of reviewers were similar regarding their preferences for non-medical and medical words. This implies that for simple medical surveys, engaging a large number of participants pool can obtain accurate results, mitigating the need for, and potential higher costs associated with, specialized professionals.\u003c/p\u003e \u003cp\u003eIn the context of customizing word representation models, our PK-word2vec method offers a distinct approach compared to previous studies. Several earlier studies have adapted SGNS to enhance word representation, primary focusing on aligning the vector space of a general, pre-trained embedding model to a specific problem domain\u003csup\u003e28,29\u003c/sup\u003e. Other refined strategies included improving the embedding quality by retrofitting the pre-trained word vectors using lexical relational resources \u003csup\u003e30\u003c/sup\u003e, and augmenting SGNS with additional knowledge-based graph models and `anchor context models \u003csup\u003e31\u003c/sup\u003e. Different from these approaches, our PK-word2vec method does not require a large number of training samples to fine-tune the word embedding model. This significant advantage could substantially facilitate research within the field of biomedical informatics, mainly when working with small study cohorts.\u003c/p\u003e \u003cp\u003eDespite the encouraging findings, this study has several limitations that can serve as a basis for future research. First, PK-word2vec was examined solely on patients with breast cancer. It has yet to be determined whether the method generalizes to clinical communications or notes for other types of patients. Second, we generated prior knowledge from two pre-trained models only. It is unknown if incorporating additional pre-trained models can enhance the generalizability of PK-word2vec. Third, complex medical concepts are often represented by long phrases rather than words, which begs the question of how to include more complex concepts in human evaluations. Finally, the model should be tested in downstream NLP tasks by jointly tuning the prior knowledge weight and other hyperparameters.\u003c/p\u003e"},{"header":"5. Conclusions","content":"\u003cp\u003eThis paper introduces PK-word2vec, an adapted word embedding model trained on a relatively small number of patient portal messages. This was achieved by incorporating prior knowledge from two pre-trained embedding models into the model training. Our evaluation with MTurk workers and medical students demonstrated that PK-word2vec outperformed the standard word2vec model in generating more similar words from small-size patient portal messages. While this study focused on extracting prior knowledge from Google-word2ec and Snomed2vec for patient portal messages, the proposed PK-word2vec can leverage prior knowledge from any pre-trained models, irrespective of the vector size or representation space, to build vector representations for other clinical text data.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eQ.S. and Z.Y. wrote the main manuscript text and Q.S. prepared figures. All authors reviewed the manuscript.\u003c/p\u003e\u003ch2\u003eAcknowledgments\u003c/h2\u003e \u003cp\u003eThis work was supported by the National Cancer Institute under award number R37CA237452.\u003c/p\u003e\u003ch2\u003eData Availability\u003c/h2\u003e\u003cp\u003eThe data that support the findings of this study are available on the Vanderbilt University Medical Center Research Derivative, but restrictions apply to the availability of these data due to privacy issue, which were used under license for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of Vanderbilt University Office of Research Informatics.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eDendere R, Slade C, Burton-Jones A, Sullivan C, Staib A, Janda M. Patient Portals Facilitating Engagement With Inpatient Electronic Medical Records: A Systematic Review. J Med Internet Res. 2019;21(4):e12779. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.2196/12779\u003c/span\u003e\u003cspan address=\"10.2196/12779\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGoel MS, Brown TL, Williams A, Cooper AJ, Hasnain-Wynia R, Baker DW. Patient reported barriers to enrolling in a patient portal. Journal of the American Medical Informatics Association. 2011;18(Supplement_1):i8\u0026ndash;i12.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKruse CS, Bolton K, Freriks G. The effect of patient portals on quality outcomes and its implications to meaningful use: a systematic review. J Med Internet Res. 2015;17(2):e44.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRalston JD, Carrell D, Reid R, Anderson M, Moran M, Hereford J. Patient web services integrated with a shared medical record: patient use and satisfaction. Journal of the American Medical Informatics Association. 2007;14(6):798\u0026ndash;806.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eOsborn CY, Rosenbloom ST, Stenner SP, et al. MyHealthAtVanderbilt: policies and procedures governing patient portal functionality. Journal of the American Medical Informatics Association. 2011;18(Supplement_1):i18\u0026ndash;i23.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGriffin A, Skinner A, Thornhill J, Weinberger M. Patient portals. Appl Clin Inform. 2016;7(02):489\u0026ndash;501.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYin Z, Warner JL, Chen Q, Malin BA. Patient Messaging Content Associated with Initiating Hormonal Therapy after a Breast Cancer Diagnosis. In: AMIA Annual Symposium Proceedings. Vol 2019.; 2019:962.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYin Z, Harrell M, Warner JL, Chen Q, Fabbri D, Malin BA. The therapy is making me sick: how online portal communications between breast cancer patients and physicians indicate medication discontinuation. Journal of the American Medical Informatics Association. 2018;25(11):1444\u0026ndash;1451. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1093/jamia/ocy118\u003c/span\u003e\u003cspan address=\"10.1093/jamia/ocy118\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSulieman L, Yin Z, Malin BA. Why patient portal messages indicate risk of readmission for patients with ischemic heart disease. In: AMIA Annual Symposium Proceedings. Vol 2019.; 2019:828.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSulieman L, Gilmore D, French C, et al. Classifying patient portal messages using Convolutional Neural Networks. J Biomed Inform. 2017;74:59\u0026ndash;70. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doi.org/10.1016/j.jbi.2017.08.014\u003c/span\u003e\u003cspan address=\"10.1016/j.jbi.2017.08.014\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGl\u0026ouml;ggler M, Ammenwerth E. Development and Validation of a Useful Taxonomy of Patient Portals Based on Characteristics of Patient Engagement. Methods Inf Med. 2021;60(S 01):e44\u0026ndash;e55.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWang Z, Zhang W, Liu N, Wang J. Scalable rule-based representation learning for interpretable classification. Adv Neural Inf Process Syst. 2021;34:30479\u0026ndash;30491.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMikolov T, Sutskever I, Chen K, Corrado GS, Dean J. Distributed Representations of Words and Phrases and their Compositionality. In: Burges CJC, Bottou L, Welling M, Ghahramani Z, Weinberger KQ, eds. Advances in Neural Information Processing Systems. Vol 26. Curran Associates, Inc.; 2013. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://proceedings.neurips.cc/paper/2013/file/9aa42b\u003c/span\u003e\u003cspan address=\"https://proceedings.neurips.cc/paper/2013/file/9aa42b\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e31882ec039965f3c4923ce901b-Paper.pdf\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSteitz BD, Levy MA. A Social Network Analysis of Cancer Provider Collaboration. AMIA Annu Symp Proc. 2016;2016:1987\u0026ndash;1996.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGalea D, Laponogov I, Veselkov K. Sub-word information in pre-trained biomedical word representations: evaluation and hyper-parameter optimization. In: Proceedings of the BioNLP 2018 Workshop.; 2018:56\u0026ndash;66.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMartinez Soriano I, Castro Pe\u0026ntilde;a JL, Fernandez Breis JT, San Rom\u0026aacute;n I, Alonso Barriuso A, Guevara Baraza D. Snomed2Vec: Representation of SNOMED CT Terms with Word2Vec. In: 2019 IEEE 32nd International Symposium on Computer-Based Medical Systems (CBMS).; 2019:678\u0026ndash;683. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1109/CBMS.2019.00138\u003c/span\u003e\u003cspan address=\"10.1109/CBMS.2019.00138\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBaziotis C, Pelekis N, Doulkeridis C. DataStories at SemEval-2017 Task 4: Deep LSTM with Attention for Message-level and Topic-based Sentiment Analysis. In: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). Association for Computational Linguistics; 2017:747\u0026ndash;754. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.18653/v1/S17-2126\u003c/span\u003e\u003cspan address=\"10.18653/v1/S17-2126\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSoysal E, Wang J, Jiang M, et al. CLAMP \u0026ndash; a toolkit for efficiently building customized clinical natural language processing pipelines. Journal of the American Medical Informatics Association. 2017;25(3):331\u0026ndash;336. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1093/jamia/ocx132\u003c/span\u003e\u003cspan address=\"10.1093/jamia/ocx132\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMikolov T, Sutskever I, Chen K, Corrado GS, Dean J. Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems.; 2013:3111\u0026ndash;3119.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMitra B, Nalisnick E, Craswell N, Caruana R. A dual embedding space model for document ranking. arXiv preprint arXiv:160201137. Published online 2016.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGrover A, Leskovec J. node2vec: Scalable feature learning for networks. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.; 2016:855\u0026ndash;864.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBodenreider O. The unified medical language system (UMLS): integrating biomedical terminology. Nucleic Acids Res. 2004;32(suppl_1):D267\u0026ndash;D270.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eVillani C, others. Optimal Transport: Old and New. Vol 338. Springer; 2009.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBuhrmester M, Kwang T, Gosling SD. Amazon\u0026rsquo;s Mechanical Turk: A new source of inexpensive, yet high-quality data? Published online 2016.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDean S, Illowsky B. Descriptive statistics: skewness and the mean, median, and mode. Connexions website. Published online 2018.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJansen S. Word and phrase translation with word2vec. arXiv preprint arXiv:170503127. Published online 2017.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChiu B, Crichton G, Korhonen A, Pyysalo S. How to train good word embeddings for biomedical NLP. In: Proceedings of the 15th Workshop on Biomedical Natural Language Processing.; 2016:166\u0026ndash;174.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSarma PK, Liang Y, Sethares WA. Domain adapted word embeddings for improved sentiment classification. arXiv preprint arXiv:180504576. Published online 2018.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePoerner N, Waltinger U, Sch\u0026uuml;tze H. Inexpensive Domain Adaptation of Pretrained Language Models: Case Studies on Biomedical NER and Covid-19 QA. In: Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics; 2020:1482\u0026ndash;1490. doi:\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.18653/v1/2020.findings-emnlp.134\u003c/span\u003e\u003cspan address=\"10.18653/v1/2020.findings-emnlp.134\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFaruqui M, Dodge J, Jauhar SK, Dyer C, Hovy E, Smith NA. Retrofitting word vectors to semantic lexicons. arXiv preprint arXiv:14114166. Published online 2014.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYamada I, Shindo H, Takeda H, Takefuji Y. Joint learning of the embedding of words and entities for named entity disambiguation. arXiv preprint arXiv:160101343. Published online 2016.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"breast cancer, hormonal therapy, natural language processing, patient portal messages, word embedding models, word2vec","lastPublishedDoi":"10.21203/rs.3.rs-4350387/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4350387/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eBackground\u003c/strong\u003e Patient portal messages often relate to specific clinical phenomena (e.g., patients undergoing treatment for breast cancer) and, as a result, have received increasing attention in biomedical research. These messages require natural language processing and, while word embedding models, such as word2vec, have the potential to extract meaningful signals from text, they are not readily applicable to patient portal messages. This is because embedding models typically require millions of training samples to sufficiently represent semantics, while the volume of patient portal messages associated with a particular clinical phenomenon is often relatively small.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eObjective\u003c/strong\u003e We introduce a novel adaptation of the word2vec model, PK-word2vec, for small-scale messages.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMethods\u003c/strong\u003e PK-word2vec incorporates the most similar terms for medical words (including problems, treatments, and tests) and non-medical words from two pre-trained embedding models as prior knowledge to improve the training process. We applied PK-word2vec on patient portal messages in the Vanderbilt University Medical Center electric health record system sent by patients diagnosed with breast cancer from December 2004 to November 2017. We evaluated the model through a set of 1000 tasks, each of which compared the relevance of a given word to a group of the five most similar words generated by PK-word2vec and a group of the five most similar words generated by the standard word2vec model. We recruited 200 Amazon Mechanical Turk (AMT) workers and 7 medical students to perform the tasks.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults\u003c/strong\u003e The dataset was composed of 1,389 patient records and included 137,554 messages with 10,683 unique words. Prior knowledge was available for 7,981 non-medical and 1,116 medical words. In over 90% of the tasks, both reviewers indicated PK-word2vec generated more similar words than standard word2vec (p=0.01).The difference in the evaluation by AMT workers versus medical students was negligible for all comparisons of tasks’ choices between the two groups of reviewers (p =0.774 \u0026nbsp;under a paired t-test).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusions\u003c/strong\u003e. PK-word2vec can effectively learn word representations from a small message corpus, marking a significant advancement in processing patient portal messages.\u003c/p\u003e","manuscriptTitle":"Optimizing Word Embeddings for Patient Portal Message Datasets with a Small Number of Samples","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-05-15 15:50:11","doi":"10.21203/rs.3.rs-4350387/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2024-06-05T16:17:38+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-05-31T14:46:32+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"9539596560107394909515402175667460723","date":"2024-05-22T14:04:48+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2024-05-16T16:00:08+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"84058213127929099510472603588733082176","date":"2024-05-16T15:39:53+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2024-05-14T13:44:03+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2024-05-09T10:20:32+00:00","index":"","fulltext":""},{"type":"editorInvited","content":"","date":"2024-05-07T16:54:09+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2024-05-07T16:48:00+00:00","index":"","fulltext":""},{"type":"submitted","content":"Scientific Reports","date":"2024-04-30T16:12:38+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"scientific-reports","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"scirep","sideBox":"Learn more about [Scientific Reports](http://www.nature.com/srep/)","snPcode":"","submissionUrl":"","title":"Scientific Reports","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"Scientific Reports","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"4d1fec21-53a4-4529-85c7-e97db78058bb","owner":[],"postedDate":"May 15th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[{"id":31765529,"name":"Biological sciences/Computational biology and bioinformatics/Machine learning"},{"id":31765530,"name":"Health sciences/Diseases/Cancer/Breast cancer"}],"tags":[],"updatedAt":"2024-07-01T06:21:24+00:00","versionOfRecord":[],"versionCreatedAt":"2024-05-15 15:50:11","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4350387","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4350387","identity":"rs-4350387","version":["v1"]},"buildId":"7rjqhiLT3MXkJMwkYKINL","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}