Narrative Search Engine for Case Series Assessment Supported by Artificial Intelligence Query Suggestions.

Searching case narratives during signal assessment can be supported by suggesting additional search terms derived from artificial intelligence models pre-trained for understanding the similarity between words. This search approach identified additional relevant narratives as compared with standard search functionalities.

A narrative search engine with query suggestions was built for searching case narratives from case series in VigiBase, the World Health Organization global database of adverse event reports for medicines and vaccines. The search engine takes a series of case narratives and a query, which can consist of one or multiple search terms, and returns a ranked list of case narratives that are considered relevant to the search query. Additional query terms are suggested and presented for potential inclusion to the original search query. For an overview of this setup, see Fig. 1 . Fig. 1 Overview of the search engine supported by query suggestions Overview of the search engine supported by query suggestions The narrative search engine uses BM25 [ 15 ] as its search method and applies stemming to the search query terms (e.g. “exercise”, “exercising” and “exercises” are all stemmed to “exercis”). To enhance search results and to support the user in refining the search, the narrative search engine provides query suggestions, recommending other terms that are similar to the original query terms. The user may add any number of the suggested query terms to the initial search, forming queries such as “workout training exercise” to look for case narratives containing one or a multiple of these words. In practice, this can be an iterative process of adding and removing query terms depending on the search results. The suggested query terms are based on the meaning of the terms already included in the query. Similar terms are suggested per query term, and two different sets of suggestions are made using underlying word embedding models: a Global Vectors for Word Representation (GloVe) model [ 26 ] for general English text (with 300-dimensional embeddings) and a word2vec model [ 27 ] trained on biomedical literature (with 200-dimensional embeddings). Word2vec is a neural-network-based model that learns vector representations of words by predicting surrounding words, while GloVe is a model that generates word embeddings by leveraging word co-occurrence statistics from a large corpus. Both are relatively small models, making them computationally efficient. The reason for using two different word embedding models is that they may complement each other. For example, for “exercise”, the general English model could be more appropriate, while for medical terms such as “fever”, the biomedical one could be more helpful. We did not choose contextual embeddings such as those derived from BERT, since these rely on context, and we expected queries to often only contain single words without any context. These embeddings are also larger and the cost–benefit considerations were not favourable for their use. For general language, both word2vec and GloVe have shown good performance on word analogy datasets, word2vec in particular on word similarity datasets [ 28 ]. Another advantage of using GloVe was its availability within the spaCy python package. Not only are the suggested query terms semantically related to the original query terms, but they are also filtered to include only words that appear in the narratives of the case series. This helps the user build the best query for the given case series. In summary, throughout this process, each word embedding model recommends ten terms for each search term, not already present in the search query. All suggested terms are words present in the narratives. The suggested terms are ordered on the basis of their similarity to the original search term. There is no stipulated minimum similarity requirement, and thus, no threshold is applied. Figure 2 presents how the query suggestions may be presented in a user interface. Fig. 2 Query suggestions for the search query “exercise” on a case series used during development (built using the streamlit python package) Query suggestions for the search query “exercise” on a case series used during development (built using the streamlit python package) To systematically assess the performance of search engines, one can use evaluation datasets. These consist of a set of query topics that cover a range of themes that users might search for, a collection of texts and relevance judgements. The relevance judgements are the manual annotations for query topic–text pairs indicating whether the text is relevant for the topic. Evaluation metrics then compare the search engine output, i.e., a list of texts retrieved and ranked according to relevance by the search engine, to the evaluation dataset’s relevance judgements . For the evaluation of the narrative search engine, we created an evaluation dataset from COVID-19 vaccine case series in VigiBase. We chose to focus on COVID-19 vaccine reports since the narrative search engine was originally developed to address the challenge of an unprecedented increase in reports of adverse events following immunisation related to COVID-19 vaccination campaigns. We selected five COVID-19 vaccine case series on the basis of the European Medicines Agency’s public listing of safety signals discussed at the Pharmacovigilance Risk Assessment Committee. The COVID-19 vaccine case series selected were the adverse events heavy menstrual bleeding, myocarditis, erythema multiforme, deep vein thrombosis and myositis. We sampled 150 narratives for each of the case series, resulting in 750 annotated narratives in total. Each case narrative in the evaluation dataset was independently annotated by two experienced pharmacovigilance assessors. The experts evaluated each case narrative to determine its relevance to a predefined search topic. Each case narrative was assigned one label as defined in Table 1 . Disagreements between the annotators were discussed and joint decisions were made. Table 1 Label assigned during manual annotation of the dataset Label Description Relevant Case narrative matches the topic as it is defined or is related to the topic in some way Non-relevant Case narrative does not contain anything related to the topic or contains truncated entities (e.g. a narrative truncated for technical reasons to end with “traini” suggesting the mention of “training” but not including the complete word “training”) Label assigned during manual annotation of the dataset The following topics were chosen for annotation of the COVID-19 case series: For the adverse event heavy menstrual bleeding, the topic “ability to work”, which represents a qualifier of the event, indicating its impact on quality of life; For the adverse event myocarditis, the topic “autoimmune disease”, which represents a potential alternative cause for the event; For the adverse event erythema multiforme, the topic “oral lesions”, which represents a key diagnostic symptom of the event; For the adverse event deep vein thrombosis, the topic “cancer”, which represents a risk factor for the event; For the adverse event myositis, the topic “dysphagia”, which represents a qualifier of the event, indicating the severity of the event. For the adverse event heavy menstrual bleeding, the topic “ability to work”, which represents a qualifier of the event, indicating its impact on quality of life; For the adverse event myocarditis, the topic “autoimmune disease”, which represents a potential alternative cause for the event; For the adverse event erythema multiforme, the topic “oral lesions”, which represents a key diagnostic symptom of the event; For the adverse event deep vein thrombosis, the topic “cancer”, which represents a risk factor for the event; For the adverse event myositis, the topic “dysphagia”, which represents a qualifier of the event, indicating the severity of the event. These topics represent alternative causes, risk factors and other subsets of the case narratives, each with an evidence-based association to the respective adverse event. In the selection of the topics, we also made an informal estimate of their prevalence in the case narratives on the basis of clinical expertise and aimed to strike a prevalence balance. Specifically, we wanted to select topics that were likely to appear in the case narratives but not so prevalent that they would not require a search engine. The final annotated evaluation dataset included 55 relevant narratives: heavy menstrual bleeding and ability to work with 6% relevant case narratives, myocarditis and autoimmune diseases with 4%, erythema multiforme and oral lesions with 16%, deep vein thrombosis and cancer with 7% and myositis and dysphagia with 3%. Inter-annotator agreement was measured to evaluate their agreement during the creation of the evaluation dataset using Cohen’s kappa [ 29 ]. The overall Cohen’s kappa score for the evaluation dataset annotations was 0.64, which can be considered moderate agreement [ 29 ]. However, inter-annotator agreement varied between case series with a Cohen’s kappa score ranging from 0.39 (minimal agreement) for heavy menstrual bleeding to 0.81 (strong agreement) for erythema multiforme. To evaluate the search engine and test how well it retrieves relevant case narratives, we needed concrete search queries for the annotated topics. The search queries serve as the input made to the narrative search engine, simulating real-world user interactions. For the query selection, a medical assessor who had not seen the case narratives of the COVID-19 vaccine case series defined three single-word queries for each topic. This search query definition was based on experience from working with different case series, which would reflect of a pharmacovigilance assessor’s workflow. Single-word queries were chosen to facilitate comparison between different models. 1 The search queries, which should not be seen as complete definitions of the topics, were as follows: for the topic ability to work the queries “work”, “job” and “leave”; for the topic autoimmune disease the queries “autoimmune”, “lupus” and “rheumatoid”; for the topic oral lesions the queries “lip”, “oral” and “mouth”; for the topic cancer the queries “cancer”, “metastasis” and “malignant”; for the topic dysphagia the queries “dysphagia”, “swallowing” and “choking”. for the topic ability to work the queries “work”, “job” and “leave”; for the topic autoimmune disease the queries “autoimmune”, “lupus” and “rheumatoid”; for the topic oral lesions the queries “lip”, “oral” and “mouth”; for the topic cancer the queries “cancer”, “metastasis” and “malignant”; for the topic dysphagia the queries “dysphagia”, “swallowing” and “choking”. Relevance-based metrics, such as recall and precision, are most commonly used when evaluating the retrieval performance of information retrieval systems. These metrics quantify the number of relevant texts retrieved during a search. Recall is the proportion of relevant texts retrieved in the search and answers the question “what fraction of the relevant texts was retrieved?”: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text{Recall}= \frac{\text{number of relevant texts retrieved}}{\text{total number of relevant texts}}$$\end{document} Recall = number of relevant texts retrieved total number of relevant texts Precision is the proportion of texts retrieved in the search that were relevant. In other words, precision answers the question “what fraction of the retrieved texts were relevant?”: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text{Precision}= \frac{\text{number of relevant texts retrieved}}{\text{number of texts retrieved}}$$\end{document} Precision = number of relevant texts retrieved number of texts retrieved These metrics do not account for the ranking of the retrieved texts and do not capture the real-life search experience of the user, who will likely read the list of retrieved texts from top to bottom. A user reading the list in a sequential order might stop reading after a certain number of texts, particularly when many of the top-ranked results are not relevant. It is therefore desirable to have quality ranking with the most relevant texts on top. In this study, we used recall and precision for retrieval evaluation and evaluated the ranking quality by manually examining visualisations of the rankings in so called rank–recall curves. These curves present the recall at every position in the ranked list (see Fig. 3 in the Results section for an example). From these curves, it is possible to visually compare two rankings to see how recall differs at various positions in the ranked list. Depending on the use case, different rankings might be preferable. Fig. 3 Rank–recall curves for COVID-19 vaccine and ( a ) deep vein thrombosis with topic cancer and query cancer, ( b ) myocarditis with topic autoimmune disease and query rheumatoid, ( c ) erythema multiforme with topic oral lesions and query oral and ( d ) heavy menstrual bleeding with topic ability to work and query work. The rank–recall curves show the recall (proportion of all relevant narratives that were retrieved by the method) at every rank in the ranked list returned by the method. The “ x ” indicates the rank of the last document, i.e., the number of retrieved documents ( x -axis) and the overall recall of that method ( y -axis). Since none of the methods retrieved more than 50 cases, the x -axis is cut off at 50 instead of 150 Rank–recall curves for COVID-19 vaccine and ( a ) deep vein thrombosis with topic cancer and query cancer, ( b ) myocarditis with topic autoimmune disease and query rheumatoid, ( c ) erythema multiforme with topic oral lesions and query oral and ( d ) heavy menstrual bleeding with topic ability to work and query work. The rank–recall curves show the recall (proportion of all relevant narratives that were retrieved by the method) at every rank in the ranked list returned by the method. The “ x ” indicates the rank of the last document, i.e., the number of retrieved documents ( x -axis) and the overall recall of that method ( y -axis). Since none of the methods retrieved more than 50 cases, the x -axis is cut off at 50 instead of 150 We evaluated the narratives retrieved by the search engine after a human selection of additional query terms from the search engine’s suggestions. For each of the search queries defined above, the narrative search engine’s word embedding models produced 10 query suggestions, resulting in a total of 20 query suggestions. From those, a domain expert had the option to select and add any number of terms (0–20) to the query. For this experiment, the selection was done blindly without access to the narratives. We refer to this system as “BM25+ QS + Human”, where “QS” stands for “query suggestion”. For comparison, we used an exact-match search as a baseline method and BM25 with RM3, a commonly used query expansion method and available off-the-shelf, as a benchmark. In exact-match search, the system retrieves results that precisely match the provided search term, ignoring any word boundaries. RM3 performs automatic query expansions without the use of an external word embedding model for semantic meaning when choosing additional query terms. Instead, it uses a relevance-based language model [ 18 ], that expands the original query on the basis of term frequencies within the narratives retrieved by the initial query. We refer to this system as “BM25 + RM3”. We computed the recall and precision of the systems by running each query separately and averaging the true positive, false positive, true negative and false negative counts (classification outcomes) over the three queries per topic. This simulates a setting in which the user runs only one query. We then computed recall and precision as a micro average 2 over all case narratives from the five topics. Using micro averages makes it possible to perform significance testing. To assess whether observed differences were unlikely to be due to random chance, we performed a McNemar test [ 30 ] for the recall and a weighted generalised score statistic test for the precision [ 31 ], using a significance level of 0.05. For these calculations we created the required confusion matrices per query and averaged the cells across these matrices and rounded to whole numbers to obtain a single confusion matrix 3 used to perform the significance tests. As the aim of the evaluation was to provide initial results on the performance on a limited number of topics, we refrain from adjustments for multiple testing; i.e., the study should be interpreted as hypothesis-generating. We also computed the recall when combining retrieved narratives for all three queries per topic, i.e., using the union of the retrieved narratives. This simulates a setting in which a user runs three queries and combines the results. The quality of the ranking of case narratives was analysed by plotting the results per query as rank–recall curves for the 15 queries and manually examining the ranking visualisations. To analyse the query suggestions, we manually examined the suggestions from the word embedding models and the choices made by the domain expert and compared them to the automatically expanded terms according to RM3. We performed manual error analysis to get an in-depth understanding of the performance. We manually inspected the relevant narratives not retrieved by the methods (false negatives) and the narratives retrieved by the method but not manually annotated as relevant (false positives). To compute computational efficiency, we computed the time it took the search engine to index each case series with 150 narratives and how long it took to search for a query while at the same time computing the query suggestions. The narrative search engine was built using the Apache Lucene open-source search engine software library through the python package Pyserini [ 32 ] version 0.18.0. We used Pyserini to perform tokenisation and stemming using the Porter stemmer [ 33 ] with Lucene and to apply BM25 during search. For the query suggestions, we used the open-source natural language processing library spaCy to implement the word embeddings and scikit-learn’s NearestNeighbors class to find the ten nearest neighbours in the vector space using cosine similarity. We use spaCy’s GloVe model for general English text, en_core_web_lg version 3.4.1, and scispaCy’s word2vec model trained on biomedical literature, en_core_sci_lg version 0.5.1. As with BM25, we implemented RM3 using Pyserini. We expanded with up to three terms identified by RM3 from the top ten BM25-retrieved narratives and set the weight for the original query term to 0.5. For the evaluation, we used our own implementation of recall and precision. The recall values used for the rank–recall curves were calculated using the python package ir_measures version 0.3.1.

The retrieval performance of the search engine BM25 + QS + Human and the two comparator methods can be found in Table 2 . BM25 + QS + Human had a higher recall than the two comparators (comparison with exact-match search: p < 0.001; comparison with BM25 + RM3: p = 0.004). 4 Table 2 Retrieval performance results in terms of micro average recall and precision after averaging classification outcomes over the three queries per topic System Recall percentage (%) TP/(TP + FN) Precision percentage (%) TP/(TP + FP) Exact-match 21.8 12/55 54.5 12/22 BM25 + RM3 34.4 19/55 30.2 19/63 BM25 + QS + Human 56.4 31/55 43.1 31/72 TP true positives, FN false negatives, FP false positives Retrieval performance results in terms of micro average recall and precision after averaging classification outcomes over the three queries per topic TP true positives, FN false negatives, FP false positives BM25 + QS + Human had a higher precision than BM25 + RM3 ( p = 0.024). The precision was lower compared with the exact-match search; however, this difference was not statistically significant ( p = 0.13). For a more detailed overview of recall and precision per topic, see Electronic supplementary material Table S1 and Table S2. In total, when combining retrieved narratives for all queries, BM25 + QS + Human retrieved 49 of the 55 relevant narratives in the evaluation dataset, 15 more than the exact-match search (Electronic supplementary material Table S3). Comparing with BM25 + RM3, BM25 + QS + Human retrieved six additional relevant narratives. Qualitatively examining the 15 rank–recall curves shows that the ranking by BM25 + QS + Human is generally closer to the optimal than the exact-match search. The ranking provided by BM25 + QS + Human was inferior for three queries compared with exact-match search while at the same time retrieving the same number or more of relevant narratives. Figure 3 presents the rank–recall curves for selected queries from this experiment, exemplifying a variety of ranking results. For rank–recall curves for all queries, see Electronic supplementary material section Rank–Recall Curves. Figure 3 a presents an example where BM25 + QS + Human retrieved the optimal ranking. Figure 3 b presents an example where BM25 + QS + Human retrieved all relevant narratives, while the two comparator methods did not retrieve any narratives at all. Figure 3 c presents an example where the exact-match search led to more false positives than search with BM25 + QS + Human due to the occurrence of the letters “oral” within the word “temporal”, which is matched by exact-match search but not BM25 which only matches complete stemmed tokens. Figure 3 d presents an example where BM25 + QS + Human had the best recall but more false positives and inferior ranking as compared with exact-match search. As an illustration, Table 3 presents the query suggestions and expansions provided by the different models for the four queries presented in Fig 3 . A complete list can be found in Electronic supplementary material Table S4. Table 3 Selection of query suggestions from the different models Case series Topic Query Biomedical query suggestions General English query suggestions RM3 query expansion Heavy menstrual bleeding Ability to work Work Study Effort Experience a Lab Investigations Doing Way Professional Review Thinking Practice Experience a Professionals Writing Improvement Internist Functioning a Conversation Creating Especially Decemb So Amend Myocarditis Autoimmune disease Rheumatoid Ra a Autoimmune a Joint a Idiopathic a Crohn a Sclerosis a Systemic Inflammatory a Effusion Inflamed a Nephritis a Tonsillitis Rhinitis Idiopathic a Sclerosis a Colitis a Endocarditis Myocarditis Costochondritis Gastroenteritis - Erythema multiforme Oral lesions Oral Intravenous Administration Topical Pills Antihistamine Administered Tablet Amoxil Omeprazole Daily Intravenous Prophylaxis Topical Subcutaneous Intramuscularly Intramuscular Propranolol Antibiotic Dermatological Corticosteroid Requir He Cold Deep vein thrombosis Cancer Cancer Prostate Carcinoma a Metastatic a Colon Ovarian Advanced Malignancy a Tumour a Locally Cervical Disease Carcinoma a Tumour a Melanoma a Malignancy a Endometriosis Fibrosis Osteoporosis Metastatic a Diabetes Hi Her It is noteworthy that some suggestions overlap between the different models a Query suggestions selected by the human in the loop Selection of query suggestions from the different models Study Effort Experience a Lab Investigations Doing Way Professional Review Thinking Practice Experience a Professionals Writing Improvement Internist Functioning a Conversation Creating Especially Decemb So Amend Ra a Autoimmune a Joint a Idiopathic a Crohn a Sclerosis a Systemic Inflammatory a Effusion Inflamed a Nephritis a Tonsillitis Rhinitis Idiopathic a Sclerosis a Colitis a Endocarditis Myocarditis Costochondritis Gastroenteritis Intravenous Administration Topical Pills Antihistamine Administered Tablet Amoxil Omeprazole Daily Intravenous Prophylaxis Topical Subcutaneous Intramuscularly Intramuscular Propranolol Antibiotic Dermatological Corticosteroid Requir He Cold Prostate Carcinoma a Metastatic a Colon Ovarian Advanced Malignancy a Tumour a Locally Cervical Disease Carcinoma a Tumour a Melanoma a Malignancy a Endometriosis Fibrosis Osteoporosis Metastatic a Diabetes Hi Her It is noteworthy that some suggestions overlap between the different models a Query suggestions selected by the human in the loop For the non-medical search query “work”, the human in the loop selected two queries from the general English and one from the biomedical query suggestions. For the query “rheumatoid”, the top six terms from the biomedical query suggestions were all selected. None of the query suggestions for the query “oral” were selected. For the query “cancer”, four biomedical suggestions and five general English suggestions were selected. Overall, expansions made by RM3 had no obvious semantic similarity with the original query term, and in one case, RM3 did not expand the query at all. This happens when the BM25 search does not retrieve any narratives needed by RM3 to expand the query. It cannot therefore expand the query on the basis of words present in the search results. Inspecting the relevant narratives not identified by BM25 + QS + Human (false negatives), we found that, for three of the five topics, all relevant cases were retrieved by at least one of the queries built by the human in the loop. The following false negatives were never retrieved by BM25 + QS + Human: For ability to work the only false negative could have been identified by including “rest” in the query, a term provided in the query suggestions but not selected by the human in the loop (“Had to rest after every small activity for days.”). For oral lesions, all five false negatives included the words “mucosal” or “mucous” (“mucosal involvement”, “mucous membrane”, “mucosal lesions”) and could have been covered by adding these to the query; however, these were not included in the suggestions. For ability to work the only false negative could have been identified by including “rest” in the query, a term provided in the query suggestions but not selected by the human in the loop (“Had to rest after every small activity for days.”). For oral lesions, all five false negatives included the words “mucosal” or “mucous” (“mucosal involvement”, “mucous membrane”, “mucosal lesions”) and could have been covered by adding these to the query; however, these were not included in the suggestions. Inspecting the false positives of the BM25 + QS + Human, many false positives were caused by one included query term, either suggested or from the original query. For ability to work, three out of four contained the query term “experience” as in “I experience very heavy bleeding” or “Never had an experience like this before”. The word “experience” also appeared in four true-positive narratives, albeit alongside other words included in the selected or suggested query terms. For autoimmune, the query term “immune” caused 42 of the 58 false positives, mostly because “immune” is stemmed to “immun", which also matches “immunisation” and “immunization” (41 of the 42 “immun” matches). For oral lesions, 12 of 15 contained the original query term “oral”, which was commonly used in the narratives to describe administration routes or to describe locations of other reactions. For cancer, there was only one false positive, according to our evaluation dataset annotation, for “thymoma”: “in […] he was diagnsed [sic] with thymoma at this time which was surgically removed”. This appears to be an error in the evaluation dataset annotations. For dysphagia, three out of seven contained the query term “mouth” as in “dryness of mouth” or “bad taste in mouth”. The word “mouth” also appeared in one true-positive narrative, albeit alongside other words included in the selected query terms. For ability to work, three out of four contained the query term “experience” as in “I experience very heavy bleeding” or “Never had an experience like this before”. The word “experience” also appeared in four true-positive narratives, albeit alongside other words included in the selected or suggested query terms. For autoimmune, the query term “immune” caused 42 of the 58 false positives, mostly because “immune” is stemmed to “immun", which also matches “immunisation” and “immunization” (41 of the 42 “immun” matches). For oral lesions, 12 of 15 contained the original query term “oral”, which was commonly used in the narratives to describe administration routes or to describe locations of other reactions. For cancer, there was only one false positive, according to our evaluation dataset annotation, for “thymoma”: “in […] he was diagnsed [sic] with thymoma at this time which was surgically removed”. This appears to be an error in the evaluation dataset annotations. For dysphagia, three out of seven contained the query term “mouth” as in “dryness of mouth” or “bad taste in mouth”. The word “mouth” also appeared in one true-positive narrative, albeit alongside other words included in the selected query terms. Indexing 150 narratives per case series took less than 20 s respectively on Intel i7-10850H CPU @ 2.70 GHz with six cores, 12 logical processors and 32 GB RAM. While this can be done once per case series, each entered search term also requires time for searching the index. This took a maximum of 1 s per query including the retrieval of the new query suggestions based on the given query terms.

This study demonstrates that a narrative search engine supported by AI query suggestions with a human in the loop can facilitate the retrieval of additional relevant case narratives, compared with the commonly available exact-match search and a more traditional frequency-based automated query expansion. Enhancing the human capability by providing more powerful search functionalities has the potential to empower pharmacovigilance assessors in their interaction with case narratives during signal assessment.

To support pharmacovigilance assessors in identifying case narratives with specific characteristics of interest within a defined case series, we implemented and systematically evaluated a narrative search engine. Assuming the situation in which pharmacovigilance assessors apply exact-match search functionalities to work with large case series, we evaluated whether our narrative search engine could lead to better search performance. The evaluation showed the retrieval of more relevant case narratives, and better quality of ranking by the proposed method as compared with the baseline method, exact-match search. The retrieval of additional case narratives must be weighed against a potential loss in precision. Exact-match search could be the preferred choice for specific use cases that prioritise high precision over recall. However, in practice, terms returning too many false positives may be instantly removed from the search. In the case of the search term “immune” causing many false positives when aiming to retrieve mentions of “autoimmune disease” in our study, this term could have been removed from the search. Future work should evaluate the usability of the complete search engine system. The search engine uses word-embedding-based AI query suggestions to recommend additional search terms related to the user’s initial query and filtered on the basis of their presence in the searched case narratives. Additionally, it employs the BM25 algorithm to rank case narratives on the basis of their relevance to the search query, a helpful feature in prioritising within the search results. While this approach to searching is not novel within information retrieval, to our knowledge, its feasibility within the domain of pharmacovigilance is unknown. This study’s main contribution is the systematic evaluation of the performance of such a search engine in the context of signal assessments in pharmacovigilance. The human involvement in the process is fundamental for real-world use, giving the user control and building trust, mitigating some of the limitations of the AI model, such as the lack of explainability of the underlying word embeddings. The search engine with query suggestions and a human in the loop led to performance improvements over the search with automatic query expansion using RM3. A previous study in a different domain [ 19 ] found no improvement using word2vec query expansion; however, they did solely use automatic query expansion with no human in the loop. Our hypothesis and reason for using word embeddings was also that providing query suggestions on the basis of the meaning of the words would lead to suggestions that appear more relevant to the user than the ones from frequency-based RM3. While we did not test this specific hypothesis systematically, manual review of RM3 query expansions indicated that many suggestions lacked semantic relations to the original query. Another advantage of the search engine presented is the generation of query suggestions directly informed by the content of the case narratives in the case series of interest, allowing contextually relevant suggestions. By contrast, in an exact-match search, users would need to manually inspect the case narratives to identify relevant query terms for additional search steps. The query suggestions may also allow the users to explore different facets of their search topic, potentially generating new search ideas. Word2vec and GloVe are small embedding models. The use of small models makes this search engine computationally efficient even in a setting with limited computational resources. The short computational time is further important to meet users’ expectations on response time. Other relatively small embedding models could be explored in future work, such as FastText [ 34 ], which can handle misspellings and use subword information [ 28 ]. Another idea could be to prompt large language models such as a generative pretrained transformer (GPT) to suggest other query terms. However, hosting such models is currently still computationally more demanding than hosting word2vec and GloVe. Depending on the choice of model and way of implementation, they may further suffer from transparency and reproducibility issues. For the generation of query suggestions, the narrative search engine employs both a biomedical and a general English word embedding model. The rationale behind this choice is rooted in user flexibility since the availability of both models allows users to choose the suggestions that best align with their specific search topic. This was shown by the assessor choosing complementing query suggestions from both models. However, it is important to note that we conducted our evaluation using a small subset of queries. A larger-scope analysis would be needed to capture the extent of the capabilities of this dual-model approach. The overlap of the suggestions from the two models is likely caused by the potential overlap in their training data. This study focused on COVID-19 vaccine reports which, during the mass vaccination campaigns, led to large case series. The benefits of using a search engine may be limited in smaller case series where search might not be required to identify narratives of interest. Other limitations of this study are the small number of topics, as well as the restriction of the human-in-the-loop evaluation to one person. Further research is needed to provide confirmatory evidence and to understand how the findings might be applicable to other topics and case series. In this context it is good to bear in mind that annotating case narratives as relevant for a search topic can be challenging, and the interpretation of topics and narratives may vary between individuals. It may also be more difficult for some topics. These challenges can be seen in the wide range of inter-annotator agreement scores between the topics, as well as in the annotation error for the narrative including the word “thymoma” for the topic cancer, which was missed by both annotators. This study has not evaluated queries with more than one word, iterative query building using the suggestions and Boolean search using specific operators (“and”, “or” or “not”) to combine search terms. We expect that the possibility of building multi-word queries iteratively while interacting with the search results and the possibility of using Boolean operators will positively impact search results. We limited the study to focus on the search engine’s potential for use during signal assessment of large case series. However, additional possibilities for the usefulness of the search engine in other steps of the signal management process could be explored. For example, during signal detection or signal validation when generating hypotheses. An important feature that could be implemented for the search engine in the future is the handling of negations. Currently, any mention of a topic is considered relevant by the search engine even if it is negated, such as no malignancy . Automatically identifying which mentions are negated could allow these to be excluded from the search results. In some use cases, the assessor may want to know whether a risk factor was ruled out, and searching for negated topics might also be of interest to the user.

The increase of adverse event reporting in recent years, particularly during the coronavirus disease 2019 (COVID-19) vaccination campaigns, has strengthened the need for automated support during pharmacovigilance signal detection and assessments [ 1 – 4 ]. During signal assessment, manual evaluation of the case reports of interest (case series) is fundamental but may be challenging, especially when it involves large case series. Adverse event reports contain clinically relevant information in both coded structured sections, and in unstructured free-text sections, such as the case narrative. Case narratives describe the story of the case and contain information that is crucial for causality assessment and to understand the clinical course [ 5 ]. Within a large case series, identification of case narratives with specific characteristics relevant to the assessment can be supported by a text search. An assessor looking for all case narratives mentioning cardiovascular disease, for example, may use the search function of the document reader or search within the database to look for mentions of the word “cardiovascular”. A search allows the assessor to find the mentions of any search term of interest. In practice, a search, such as in PDF formats, is often performed as an exact-match search. The exact-match search process involves all potential search terms that need to be manually identified by the user and applied iteratively. The assessor needs to come up with all the alternative ways to describe “cardiovascular” such as “heart” or “arteries”. Using a search engine which supports the user by suggesting additional search terms can make the search process more effective, especially when also allowing searching for multiple terms at the same time. Search engines have been in everyday use for decades, helping people search the internet, scientific literature or other collections of documents. This idea has also been explored in the medical field [ 6 , 7 ]. In the context of electronic health records (EHR), search engines have been built to help find specific medical information and patient data, such as clinical records, diagnostic procedures, laboratory results, medications, treatment plans and other pertinent information [ 8 , 9 ]. In regard to EHR search, query suggestions with a human in the loop choosing the queries have played an important role in improving search quality and reducing user variation [ 10 ]. These suggestions, designed to enhance search performance, can be added to the original query. Typical sources of additional query terms are medical vocabularies, search logs or the retrieved information itself [ 10 – 12 ]. Search engines are information retrieval systems that retrieve relevant texts within a large collection of texts on the basis of users’ information needs presented as a search query. Both the texts, which in this study are case narratives, and the search queries themselves are unstructured free text. Instead of classifying text into binary relevance categories such as relevant to the search query or non-relevant to the search query, texts are commonly ranked according to their relevance to the query. In this relevance ranking process, the output of the search engine is a ranked list where the texts at the top of the list are considered most likely to be relevant to the user’s search query [ 13 , 14 ]. An approach of relevance ranking to retrieve texts is to assign each text a relevance score based on various factors, such as term frequency (TF), where the score is determined by how often the searched words appear in the text; term matching , in which the score is influenced by how closely the searched words match the words in the text; or semantic analysis , where the score evaluates how well the text's meaning and context align with the search query [ 13 ]. This study used the relevance ranking method Best Match 25 (BM25) to retrieve documents. BM25 follows term matching and term frequency principles, while also incorporating inverse document frequency (IDF), which measures how unique and informative a term is in a set of texts [ 15 ]. BM25 has been found to be effective for many diverse test collections [ 16 ]. It is good to note that many search engines add a reranking step after the initial retrieval step that improves the ranking of the retrieved texts, i.e., a search engine can contain first a relevance ranking model such as BM25 for retrieval of text, followed by another model to rerank the texts. However, this study does not use any reranking models. Search engines work on free-text natural language, which can be ambiguous and challenging for machines to understand. For example, the same concept can be represented by multiple words (synonyms). When looking for all patients who exercised, that is, did sports, we would like to find texts that mention different words that have the same semantic meaning: “training”, “workout”, “running”, “gym”, and so on. This difference between lexical representations of semantically similar concepts is called lexical gap [ 7 ]. The lexical gap can be approached by using traditional search methods such as BM25, combined with query expansion, which involves adding additional terms to the queries [ 17 ]. In query expansion, additional terms are added to the original search query to improve the match with relevant texts that use different words to represent the same concept the user is looking for. The identification of terms for query expansion can be performed using different information contexts. One approach to query expansion is to perform the initial search using the original query and to extract new additional search terms from the top-ranked search results. A common method for this is the so-called Relevance Model (RM3), which uses the frequency of words in the search results [ 18 ]. Another approach is to identify semantically similar terms from an external resource such as word embeddings [ 19 , 20 ]. Within natural language processing and artificial intelligence (AI), word embedding models capture semantic relationships between words in the form of vector representations and can be used to find related terms in the texts [ 7 ]. For instance, if the user’s query contains the word “cardiovascular”, these models can automatically add related terms such as “heart”, “circulation” or “arteries”. While automatic query expansion, meaning without any human intervention, can be useful to identify more relevant texts, the user might want more transparency and control over which words to add. A way to help users to build better queries themselves is to use query suggestions , where the terms identified by the query expansion method are suggested to the user (also known as interactive query refinement ) [ 17 ]. This approach where the user themselves can then manually choose which semantically similar terms they want to include in their search query is also applied in medical search [ 10 ]. While query suggestions based on word embeddings may not necessarily perform better than RM3 [ 19 ], suggested queries should potentially be more reasonable to the user as compared with purely frequency-based suggestions from RM3 due to them more explicitly capturing the semantic relationships. Manually created and maintained vocabularies such as Systematized Nomenclature of Medicine-Clinical Terms (SNOMED CT), Unified Medical Language System (UMLS) or WordNet could also provide semantically similar terms [ 20 , 21 ]. However, word embeddings are potentially advantageous over manually created knowledge bases because they are data-driven, can learn from large datasets to represent complex word relationships and are easily adaptable to new data. Another approach to addressing the lexical gap problem is to use deep neural networks to encode the semantic meaning of the query and the searched documents when matching the query to the words in the document. This approach, called dense retrieval, has gained popularity in recent years, while previously BM25 and other such sparse retrieval methods have been the state-of-the-art [ 22 , 23 ]. One example is ColBERT [ 24 ], which encodes the query and document using bidirectional encoder representations from transformers (BERT), an early large language model. These methods address the lexical gap but, as with automatic query expansion, they obscure part of the search from the user, reducing control over the results. For trust and clarity in the signal assessment process, a transparent approach using query suggestions is beneficial. While we suspect that searching narratives to identify cases with specific characteristics may be common within pharmacovigilance signal assessments, this process, and how it is supported by technology, is not well described in the literature. In their industry surveys in 2019–2021, TransCelerate found limited automation in the medical assessment part of pharmacovigilance work [ 25 ]. However, the extent of search functionality for medical assessment remains unclear. In the context of medical device surveillance, the US Food and Drug Administration has presented a semantic search tool for adverse event reports and other related documents used during signal detection and assessment [ 21 ]. The authors use TF–IDF with query expansions and query suggestions which are based on WordNet synonyms amongst other sources. The study [ 21 ] however lacks a systematic evaluation of the search engines performance, which is important for the trust of the user. The aim of this study was to explore the feasibility of identifying case narratives containing specific characteristics with a narrative search engine supported by AI query suggestions and systematically evaluate its performance as an alternative to exact-match search. We further compare how a more traditional frequency-based query expansion in form of RM3 compares to semantic query suggestions with a human in the loop.

Below is the link to the electronic supplementary material. Supplementary file1 (PDF 1110 KB) Supplementary file1 (PDF 1110 KB)