Implementation of inclusion and exclusion criteria of clinical studies in the software OHDSI ATLAS

Implementation of inclusion and exclusion criteria of clinical studies in the software OHDSI ATLAS | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Implementation of inclusion and exclusion criteria of clinical studies in the software OHDSI ATLAS Romina Blasini, Kornelia Marta Buchowicz, Henning Schneider, Birgit Samans, and 1 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-2985787/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background Clinical trials are an essential part of the medical research process, but trials are often canceled due to lack of enrollment. Clinical trial recruitment support systems can help increase the number of participants by finding more suitable subjects. The ATLAS software (developed by OHDSI) can support the initiation of a clinical trial by building cohorts of patients who meet certain eligibility criteria. The correct use of medical classification systems for clearly defined inclusion and exclusion criteria in studies is an important pillar of this software. Objectives The purpose of this study is to determine whether ATLAS can be used in a Clinical Trial Recruitment Support System to represent the eligibility criteria of clinical trials. Therefore, we have considered the set of criteria implemented in ATLAS and realistically identified the strengths and problems of the software. Methods We analyzed ATLAS using 223 eligibility criteria from 30 randomly selected studies from the past 10 years. Therefore, appropriate ICD, OPS, LOINC, or ATC codes were selected during the data collection process to feed the software. Each criterion and study was ranked for feasibility of implementation in the software. Results According to the observations, 49.33% of the analyzed inclusion criteria are fully implemented in ATLAS. In our selected sample set, 10% of the studies are classified as fully portrayable and 73.33% as portrayable. In addition, the software is evaluated with respect to its usability and its interaction with medical classification systems. Conclusions To improve and extend the coverage of criteria in cohort definition in a real-world setting, we recommend working closely with the individuals involved in the study to precisely define the criteria and carefully select terminology systems. The selected criteria will be combined according to the individual setting. Further work is needed to specify the relevance and quantity of the extracted criteria. Clinical Trial Recruitment Support System Patient Selection Logical Observation Identifiers Names and Codes Interoperability OMOP Figures Figure 1 Figure 2 1 Background In the field of medicine, the goal of clinical trials is always to learn more about the human body in an effective and positive way. The ultimate aim of a medical study is always to improve patient care and therapeutic standards through the application of clinical trials. In order to observe the influence of the new interventions and preventive measures in terms of safety and efficiency, a certain number of participants are recruited according to certain criteria. Such recruitment is always a challenging and time-consuming stage in the medical trial pipeline, resulting in a large number of canceled or incomplete trials ( 1 – 3 ). In each clinical trial, eligibility criteria (eligibility criteria) are defined at the beginning of study development describing the relevant characteristics shared by the participants. Patient cohort identification and recruitment usually is carried out by research staff or primary care personnel through querying the clinical systems manually for patients matching the eligibility criteria. The latter process is a time-consuming and cost-intensive part of a clinical research study pipeline. Therefore, utilizing digital tools can significantly improve the recruitment of subjects and reduce costs and required labor resources ( 4 ). Clinical Trial Recruitment Support Systems (CTRSS) or patient recruitment systems (PRS) can booster patient inclusion of clinical trials by automatically analyzing eligibility criteria based on electronic health records ( 4 – 6 ). Although these systems nowadays are integrated increasingly in many information and communication technology information and communication technology medical research projects, most of the study centers do not tend to use such digital tools for patient recruitment ( 7 ). The main reason is the requirement to domain technical staff as well as equipment to perform implementation or supportive tasks for the CTRSS. To address this issue, an individual system can be launched for multiple platforms or to integrate the CTRSS with the existing research systems. In addition to the existing CTRSS or commercial solutions developed as part of different projects, several open-source tools like the Observational Medical Outcomes Partnership Common Data Model (OMOP CDM) developed by Observational Health Data Sciences and Informatics (OHDSI) or Informatics for Integrating Biology and the Beside (i2b2) established by National Institutes of Health , are free to use for patient recruitment. Both systems are based on the idea of using standard terminologies within data repositories however, they are not bound to a specific nomenclature. In addition, data can be extracted from different sources and in various models, so they may be utilized in different medical contexts and at multiple study centers with different prerequisites ( 8 , 9 ). Therefore an architecture for a CTRSS utilizing OMOP as its data basis can be used for a patient recruitment system for example.( 10 , 11 ) 1.1 Observational Health Data Sciences and Informatics (OHDSI) The OMOP CDM is part of a whole tool suite created and maintained by OHDSI, as a collaboration among 150 organizations around the world that collect and process healthcare data. OHDSI offers a wide range of open-source tools to support various data-analytics use cases on observational patient-level data, all are interacting with databases using the OMOP CDM. ( 8 , 12 ) Inspired by OHDSI, the ATLAS software was designed and developed as a web-based open source application. The primary goal of ATLAS is to be used for the observational analysis and generating real-world evidence through patient data extracted from clinical practice. ATLAS operates as a user interface on the OMOP CDM. According to ( 10 ) ATLAS allows defining cohorts, selecting analysis configurations, and tagging diseases with appropriate codes. Also, it allows sharing the project easily with other researchers ( 8 ). ATLAS can also be used for subject recruitment by formalizing the eligibility criteria of a trial for the corresponding cohort. The definition of trials in ATLAS works with two main designs: cohorts and concepts. Concepts are individual codes that belong to a terminology system integrated in the OMOP CDM. They can be grouped into concept sets. A concept set contains several concepts from the standardized vocabulary in combination with logical indicators. It allows the user to specify whether related concepts should be included or excluded from the vocabulary hierarchy. Cohorts are the collection of individuals searched for by a particular query based on specific criteria, so all of a study's eligibility criteria are part of the cohort definition ( 8 ). 1.2 Terminology systems The database of ATLAS enforces the feeding data to follow standardized classification systems like Logical Observation Identifiers Names and Codes (LOINC) or Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT). These nomenclatures work as an interface for the study centers and built the basis for generating patient cohorts. ( 13 , 14 ) Several analyses have been conducted to investigate the use of digital systems based on international terminologies.( 15 , 16 ) For example, ( 17 ) investigated the use of a Clinical Data Interchange Standard Consortium (CDISC) based model in conjunction with a patient-centered model to structure eligibility criteria of clinical studies. International classification systems, such as International Statistical Classification of Diseases and Related Health Problems (ICD), formed the basis for this project. The authors used 200 randomly selected inclusion criteria from other studies, and the majority of the criteria could be represented in the system. ( 17 ) Automatic tools like criteria2query use natural language processing (NLP) to automate formalization of eligibility criteria into the ATLAS cohorts. The latter system works just on specific terminology systems and local specialties are not considered. As the definition of an ATLAS cohort depends highly on the engaged terminology systems, such tools cannot be applied in the presence of other supported coding systems. ( 18 – 21 ) Although there are more studies available addressing the latter issue, there still is a gap left between the theory and practically introducing the eligibility criteria with different terminology systems into the software ( 15 , 22 ). Therefore, further analysis should be conducted regarding to this matter in order to explore the hidden potentials behind the different software and coding systems. This study considers the possibility to apply inclusion and exclusion criteria of different clinical trials in ATLAS as a proof of concept. 2 Methods 2.1 Objectives The first step in applying the eligibility criteria into ATLAS is to discover the feasibility of their implementation (in terms of quantity) with respect to the utilized standardized open-source terminology system. Also, the technical difficulties in the implementation process will be analyzed as the qualitative drawbacks. The selected terminology systems in our study are the standard classification systems ICD and Operation and Procedure Code (OPS) , which is a German modification of the International Classification of Procedures in Medicine (ICPM), utilized for billing in all German hospitals. Also, the Anatomical Therapeutic Chemical classification (ATC) system is selected for the medications and LOINC coding system for laboratory results as well. In a next step, all eligibility criteria which could not be introduced into ATLAS are considered. They are categorized based on the shortlisted drawbacks above. This paper works on establishing classification systems and standards, as well as dealing with the mapping of inclusion and exclusion criteria generally and in ATLAS especially. 2.2 Study design To initiate the research, we randomly selected 30 studies without any preference with respect to a medical discipline. For this sake, we searched on the platform "ClinicalTrials.gov", one of the largest registries of clinical trials including about 425,817 studies from 222 different countries ( 23 ). The selected studies had all to be started within the last 10 years resulted in 114 studies wherein 30 studies have been selected randomly. Also, to investigate the extent to which the eligibility criteria can be implemented in ATLAS, we documented all 331 eligibility criteria of the 30 studies in a table. 2.3 Procedure in ATLAS To begin the process of introducing criteria into ATLAS, we selected an appropriate terminology system with appropriate codes for each eligibility criteria, as shown in Fig. 1 . The appropriate codes were identified using official online catalogs ( 13 , 24 – 26 ). If no appropriate codes were found in these catalogs, SNOMED CT would have added to the analysis to check for an alternative way to introduce or embed an approval criterion. ( 14 ) In ATLAS all fitting codes describing a single eligibility criterion are summarized in one concept set, which is a collection of codes e.g. concepts. This is necessary, because a single criterion can be constituted of one up to hundreds of codes. For example, type I diabetes mellitus can have different characteristics, which can be marked with different ICD codes. Codes for these disease individualities create one concept set. In comparison, Hemophilia A could be created as a concept set using a single code, as there is only one ICD code for this specific disease (Fig. 2 ). If there was any doubt about an eligibility criterion or related codes, medical staff was consulted. Afterwards, a separate concept set had to be created for each eligibility criterion, although no concept sets were created for demographic data such as age or gender. The aforementioned criteria were entered directly during the later process within a cohort by means of the corresponding functions available in ATLAS. Once all the concept sets required for a study were established, cohorts, that included all eligibility criteria for a study, had to be defined. To outline a cohort, at first it is necessary to define an initial event, which is a main criterion and used for a preselection of subjects. All individuals, which fulfill the requirements of the initial event, build the main cohort of the patients. To set further constraints, it is possible to set other criteria to get more specified results. We always selected the examined disease of the trial as an initial event for the cohort. For example, if a study needed test group who meet selection criteria such as type I diabetes mellitus, age under 18 years and elevated Body-Mass-Index (BMI), we would define diabetes mellitus as initial event. All patients with type I diabetes built the main cohort. Criteria “age” and “BMI” would have added as further eligibility criteria. 3 Evaluation of the studies and eligibility criteria 3.1 Categorization of eligibility criteria and studies Of all 331 eligibility criteria, 108 criteria were excluded because of missing relevance. These criteria were the ones, which do not exist prior to a face-to-face interview with a study physician so, they cannot be analyzed in the process of the electronic patient screening. To categorize the eligibility criteria, we analyzed the remaining 223 suitability criteria and classified them in terms of their implementation in ATLAS into the categories implementable, partially implementable, and not implementable (s. Table 1 ). If a criterion was only partially or not implementable, we would also add short notes expressing our decision. Table 1 Criteria categories Criteria Category Explanation implementable Criterion can be implemented in ATLAS with given terminology systems. partially implementable Criterion can be implemented, but not with the given terminologies (only SNOMED CT) or only a part of criterion is mapped (for example a disease can be implemented, but a timely constraint is not possible). not implementable Criterion cannot be implemented with the help of the given terminologies or SNOMED CT and the functions available in ATLAS. After weighting the studies with respect to their feasibility, four categories have been established: fully portrayable, portrayable, partially portrayable, and not portrayable. (see Table 2 ). Table 2 Trial Categories Trial Category Explanation fully portrayable All criteria of a study can be implemented in ATLAS. portrayable The initial event is classified as implementable and can be mapped completely in ATLAS. Further inclusion criteria can be classified as partially implemented or not implementable according to Table 1 . partially portrayable The initial event is classified as partially implementable in ATLAS according to Table 1 . not portrayable The initial event cannot be implemented in ATLAS. Explanation of trial categories in the overall evaluation about feasibility of measures taken in ATLAS 3.2 Classification of partially/not implementable criteria Finally, we made a secondary analysis to discover the main difficulties in the introducing into or implementation of eligibility criteria in the software ATLAS. Therefore, all reasons and argumentations have been considered to justify the categorization and the implementation issues of the eligible criteria. 3.3 Ethical considerations No ethical vote was needed. 4 Results The described categorization resulted in an implementation of 30 cohorts with one initial event per cohort and an additional number of 223 eligibility criteria. For implementing all eligibility criteria and initial events, we built 129 different concept sets, including 11.2 concepts in average. Each cohort included in average 5.33 concept sets with 59.73 concepts in total. The whole table can be found in Appendix 1. 4.1 Evaluation of eligibility criteria According to our observation, 49.33% (n = 110) of all eligibility criteria identified in the studies could be mapped in ATLAS so classified as implementable (see. Table 1 ). This set is corresponding with the eligibility criteria referring to clearly defined diseases, procedures, or drugs respectively with the corresponding ICD, OPS, ATC or LOINC codes. Also, 20.63% (n = 46) of the criteria were defined as partially implementable and 30.04% (n = 67) could not be implemented in ATLAS at all. 4.2 Overall evaluation of analyzed studies Based on our evaluation, in 10% of the studies (n = 3) all criteria could be fully mapped, so these studies were classified as fully portrayable, (see. Table 2 ). The latter set includes a few simple and clearly defined criteria, such as a well-defined diseases, forming a group of subjects which could be filtered more by one or two additional criteria, like age or gender. In these cases, the disease has been introduced into ATLAS using the appropriate codes and as an initial event. Also, 73.33% of the studies (n = 22) have been classified as portrayable, where at least one of the eligibility criteria was only partially or not implementable according to the rules in Table 1 . However, the initial event could be fully implemented as a concept set using ICD, OPS, LOINC or ATC. In 16.67% (n = 5) it was not possible to fully implement the initial event and these were assigned to the category partially portrayable. Finally, none of the studies was classified as not portrayable. 4.3 Partially implementable criteria After analyzing the justifications corresponding to the criteria which could not be introduced into ATLAS, we have categorized them in four groups described in Table 3 . Table 3 Categories of partially implementable criteria Criteria Category Explanation Missing information Description is not precise Highly specialized criterion Highly specialized diagnosis, procedure or laboratory result Technical reasons Partial aspect is not feasible due to technical reasons Terminology system Only implementable with SNOMED CT According to the results, 39.13% (n = 18) of the partially implemented criteria are belonging to the associated class due to technical reasons, with different causes. For example, in some studies, the ages are quoted in years but in others like pediatric studies dealing with the infants quoted in a monthly basis. In ATLAS it is only possible to query the age in years. Some criteria also need a cause-and-effect relationship. In such criteria, the information whether the patient is involved in cross studies is required which cannot be extracted from an OMOP data model. Among the rest of the criteria, 26.09% (n = 12) could not be fully implemented due to missing information. Those criteria mainly refer to “norm“ or “stable” values even if there are no defined norm quantities valid in all countries or locations but different standards from site to site. The latter set includes criteria that refer to the groups of medicaments, diagnosis, procedures, or laboratory results, which are not globally defined. Another set of criteria includes 17.39% (n = 8) criteria which were too specific and no terminology system has been found to have a specific corresponding code, except for the codes that also include a group of diseases/procedures/laboratory/results/medications. Finally, 17.39% (n = 8) eligible criteria, have been classified with SNOMED CT. 4.4 Not implementable criteria To categorize the criteria which are not implementable, we have selected the same categories, without the category corresponding to the wrong terminology system. In 47.76% (n = 32) eligible criteria, missing information leads to the categorizing as “not implementable”. 28.36% (n = 19) of the criteria were too specific and 23.88% (n = 16) were not implementable due to technical reasons. 4.5 Technical usability aspects of ATLAS According to the practices performed with ATLAS in this work, it is observed that the intuitive operations in the software have a poor performance which necessitates a training phase prior launching the main study work. Individual adaptations are needed to enter complex or nested study criteria. For example, the scores cannot be noted by selecting an available function in ATLAS or by introducing one specific code as a concept set. The analyzed scores usually consist of several sub-items, such as diseases or laboratory values, which are assigned a value depending on their severity. In these cases, the content of each score first has to be reviewed more in detail. This means that all aspects of a score are treated in the first step like individual criteria that have to be introduced into the system as separate concept sets, which is a major effort for complex scale studies. ATLAS has the possibility to input basic demographic data in an easy way and without assigning a concept, but these data are limited to age and gender. Other basic information like height and weight has to be input like other observations. For this purpose, the appropriate LOINC code has to be identified, created as a concept set and time constraints might have to be set, when only the latest data on height and weight are inquired. At this point, this information can be integrated into the cohort. Also, temporal information, like the starting point of an event or a treatment can only be introduced as an absolute date, not as relative time information like “six month ago”. Therefore, the selection criteria for a cohort of a time period must be updated on a daily basis. 5 Discussion 5.1 Digital subject recruitment OMOP and ATLAS as the products of OHDSI have been developed to support multi-institutional research projects. In this study, we intended to test the possibility of using OMOP and ATLAS not only to run analysis on the existing data sets, but also to portray the eligibility criteria of clinical trials. This makes it suitable as a digital tool for recruiting of candidates for clinical studies. We have observed that the implementation of trial eligibility criteria in ATLAS cohorts is more complex than a 1:1 translation: about half of the eligibility criteria could not be fully implemented in ATLAS. One main reason (n = 44) is that the eligibility criteria from study descriptions are defined for the use of study physicians or study personnel and often leave scope for individual professional assessment. These are easy to interpret for medical domain experts while reading the criterion, such as “stable laboratory results”, in an intuitive way. However, these criteria are not globally defined and each physician can have a slightly different interpretation of “stable” for example. A deep understanding of the criteria is essential for the process and the lack of such an understanding, results in criteria not being introduced into the system or being incorrectly implemented. The precise formulation of subject characteristics is required for a clinical research initially, so well-chosen criteria are crucial for the feasibility and internal validity of studies ( 27 ). In terms of ATLAS, carefully chosen and precisely formulated eligibility criteria form the basis for being able to use the software correctly and to exploit its potential functionalities. To get better results in the process of formalizing eligibility criteria for an automated search, it is helpful to collaborate tightly with the study personnel, so that imprecise criteria can be refined by professionals. Subsequently, all these criteria and therefore more trials, can be implemented in ATLAS. 5.2 The utilized terminology systems Since the ATLAS software is based on the involvement of terminology systems, the analysis highlights the importance of the choice of terminology systems and thus the availability of codes contained in terminology system. ICD, ATC and LOINC are international classification systems widely used in German clinics in hospital information systems and therefore they are easy to access and use if hospital data are queried. Also, OPS is a terminology system only used in Germany applied for the code procedures for billing ( 28 ). In addition to the 49.33% of the terms that could be mapped by ICD, ATC, LOINC and OPS, further 4% of the criteria could be mapped by using SNOMED CT, which is the standard terminology system used in OMOP ( 8 ). Terminology systems have a huge impact on the opportunities in building concept sets and cohorts in ATLAS. The medical terminology in SNOMED CT is significantly in more details than in ICD. SNOMED CT includes not only diagnoses itself, but also medical terms, which are placed in relation to each other. The use of ICD is recommended to record the diagnoses for billing and has a hierarchically arranged structure of diagnoses, so it is more widely used in the patient care systems and therefore provides the ease of access. Consequently, it is essential to inspect the utility and target group before making a statement about nomenclatures being better suited for a specific purpose. A combination of the coding methods, for example combination of ICD and SNOMED CT, is also recommended by ( 29 ). However, there were also many exclusion and inclusion criteria, which could not be mapped by means of chosen classification systems (neither by ICD, OPS, ATC, or LOINC, nor by SNOMED CT) due to high individuality of the subjects (n = 27). Clinical trials are often designed to gain knowledge about novel or under-researched diseases. The codes for these conditions do not yet exist in the medical classification systems, which makes it difficult or impossible to introduce some selection criteria into ATLAS. In some cases, eligibility criteria refer to non-medical settings like the proof of a vaccination. Those criteria are not commonly included in the medical terminologies, as the corresponding domain is a medical and not an administrative scenery. According to our observations, we can conclude that the formulated eligibility criteria have an impact on using correct terms, but on the other hand, for the specialized criteria no fitting codes are available. Also, it has been shown that the choice and correct use of medical classification systems, like the definition of inclusion and exclusion criteria, is an important basis for working with ATLAS. Therefore, there is a need for the correct use of nomenclatures and standardization, as well as continual adaptations to medical progress, which researchers, physicians, and others involved in health care need to strive for. In this study, the effectiveness and suitability of the individual classification systems has not been analyzed. So, we cannot state a verdict on the usability of some nomenclatures in ATLAS. 5.3 Limitations of ATLAS As expressed before, the requirement to a training phase before working with the software is a drawback. Also, in some cases, individual adaptations are required to introduce complex or nested data sets such as scores. This indicates problems with usability of ATLAS that have been also confirmed by other studies. In 2021 ( 30 ) focused on the timeline availability of clinical data and bio specimens that can be identified using digital applications. Three systems, i2b2, OHDSI's ATLAS, and Sample Locator by the German Biobank Alliance, were tested in terms of strengths, weaknesses, and associated usability via qualitative exploratory study followed by a web-based usability test. The results of this study show that on one hand ATLAS has significantly more functionalities than other tools and a very good interface visualization. However, multiple usability issues exist as well. These problems can be associated to the large number of functions and different approaches when introducing selection criteria. However, the authors take into account that ATLAS has been primarily developed for the researchers and experts who would often define complex cohort queries. Therefore it is important to provide them with the variety of selection and input options, however, it increases the operational costs ( 30 ). This study also suggests that ATLAS is more suitable for people with the appropriate technical background, due to the existing features and the need for a certain level of development for the sake of customization. However, according to the developers, the application aims to provide the greatest scientific impact for researchers. Therefore, promoting the usability and adaptability of the software for this target group is of great importance. This should be supported by members of the target group as described in ( 31 ) wherein the medical students and licensed physicians were asked to test electronic health record used in Finland in terms of usability. It has been concluded that the physicians were able to identify potential problems corresponding to the usability aspect and provide the corresponding feedback. So, it is recommended that target groups of a specific application to be involved in the usability development process in the test phases to detect problems in a timely manner and avoid stress in the future. It is worth mentioning that an inaccurate implementation of eligibility criteria in ATLAS can lead to less precise subject recommendations. If the entry of studies into ATLAS is intended to lead to suitable subject recruitment, the involvement of specialized study physicians or study personnel in the process of cohort definition will be helpful. 6 Conclusions In this study, we have considered the selection criteria of 30 studies for the possibility to portray them in the software ATLAS. We have found out that about 50% of the criteria could be implemented into the software, about 20% could only be partially implemented, and the rest of 30% could not be implemented at all. The four main justifications are the existence of missing information about the criteria, limitations of the software, limitations of the chosen terminology systems and finally highly specialized criteria. To use ATLAS as a CTRSS, we recommend the following points to improve the coverage of eligibility criteria in a cohort definition. As ATLAS offers a variety of holistic functionalities, the usability degree is not trivial. It can be useful to plan a training phase before starting with the main work. Even for the planning phase of a project, it is helpful to know all possibilities and limitations of ATLAS right from the start. Another difficulty is the limitation of the selected terminology systems. It can be helpful to examine the options and possibilities of different terminology systems to find the best and suitable system for the project. The combination of different terminology systems can also be useful to improve the coverage of eligibility criteria in cohort definitions. The main problem in portraying clinical trials in ATLAS is the missing information in eligibility criteria. A tight cooperation with study personnel, which is usually responsible for the subject recruitment and works with the criteria daily, can help to specify these uncertainties. If all missing information can be cleared with the study personnel, this could raise the percentage of implementable criteria up to about 70%. This could help with criteria that are highly specialized to be covered by terminology systems, and therefore it is only possible to implement parts of the criteria or use collective codes, which are not precise. Usually, the study personnel has similar issues when searching for eligible persons, so they can help in identifying other ways to portray these criteria. Even if the criteria are not fully implementable, it is sometimes possible to use similar criteria, which are valid for a preselection. In addition, the study personnel can prioritize the relevance of different criteria, as some of the criteria are more important for a selection of candidates than others. In this work, we picked one of the criteria as the main criterion for the cohort, and leave the rest with no prioritizing. Although some criteria are more relevant for the selection process of the subjects than others. The importance level of a criterion, can only be estimated by the study personnel working in the recruitment process. Further work is planned to examine which and how many of the criteria are relevant for the selection process of clinical trials and to survey how many of the relevant criteria can be implemented. Abbreviations ATC Anatomical Therapeutic Chemical classification BMI Body-Mass-Index CDISC Clinical Data Interchange Standard Consortium CDM Common Data Model CTRSS Clinical Trial Recruitment Support System ICD International Statistical Classification of Diseases and Related Health Problems ICPM International Classification of Procedures in Medicine i2b2 Informatics for Integrating Biology and the Beside LOINC Logical Observation Identifiers Names and Codes OHDSI Observational Health Data Sciences and Informatics OMOP Observational Medical Outcomes Partnership OPS Operation and Procedure Code PRS Patient Recruitment System SNOMED CT Systematized Nomenclature of Medicine Clinical Terms Declarations Ethics approval and consent to participate Not applicable Consent for publications Not applicable Availability of data and materials All data generated or analysed during this study are included in this published article and its supplementary information files. Competing interests The authors declare no conflict of interest. Funding This work is part of the project MIRACUM, funded by the German Ministry of Education and Research (FKZ 01ZZ1801D). Authors’ contributions RB and KMB wrote the original texts of this work. RB and KS did the conceptual work of this study. KMB analyzed all trials and criteria, as well as implemented and categorized the criteria. HS and BS were involved in the conceptual work of this article. All authors read and approved the final manuscript. Acknowledgements We are thankful to Dr. rer. nat. Seyed Amir Hossein Tabatabaei for reviewing this work and offering useful advice. References Carlisle B, Kimmelman J, Ramsay T, MacKinnon N. Unsuccessful trial accrual and human subjects protections: an empirical analysis of recently closed trials. Clin Trials Lond Engl. 2015 Feb;12(1):77–83. Desai M. Recruitment and retention of participants in clinical studies: Critical issues and challenges. Perspect Clin Res. 2020;11(2):51–3. Sully BGO, Julious SA, Nicholl J. A reinvestigation of recruitment to randomised, controlled, multicenter trials: a review of trials funded by two UK funding agencies. Trials. 2013 Jun 9;14(1):166. Thadani SR, Weng C, Bigger JT, Ennever JF, Wajngurt D. Electronic Screening Improves Efficiency in Clinical Trial Recruitment. J Am Med Inform Assoc. 2009 Nov 1;16(6):869–73. Trinczek B, Köpcke F, Leusch T, Majeed RW, Schreiweis B, Wenk J, et al. Design and multicentric Implementation of a generic Software Architecture for Patient Recruitment Systems re-using existing HIS tools and Routine Patient Data. Appl Clin Inform. 2014;05(01):264–83. Penberthy L, Brown R, Puma F, Dahman B. Automated matching software for clinical trials eligibility: Measuring efficiency and flexibility. Contemp Clin Trials. 2010 May 1;31(3):207–17. Fitzer K, Haeuslschmid R, Blasini R, Altun FB, Hampf C, Freiesleben S, et al. Patient Recruitment System for Clinical Trials: Mixed Methods Study About Requirements at Ten University Hospitals. JMIR Med Inform. 2022 Apr 20;10(4):e28696. Reich C, Ostropolets A. Chapter 5 Standardized Vocabularies | The Book of OHDSI [Internet]. San Bernardino, CA: OHDSI; 2019 [cited 2022 Nov 7]. Available from: https://ohdsi.github.io/TheBookOfOhdsi/ Prokosch HU, Sedlmayr M. Datenaustausch in der Forschung via OMOP/OHDSI [Data exchange in research via OMOP/OHDSI]. EHealth.com. 2018;2018(4):34–7. Reinecke I, Gulden C, Kümmel M, Nassirian A, Blasini R, Sedlmayr M. Design for a Modular Clinical Trial Recruitment Support System Based on FHIR and OMOP. Stud Health Technol Inform. 2020 Jun 16;270:158–62. Meystre SM, Heider PM, Kim Y, Aruch DB, Britten CD. Automatic Trial Eligibility Surveillance Based on Unstructured Clinical Data. Int J Med Inf. 2019 Sep;129:13–9. Hripcsak G, Duke JD, Shah NH, Reich CG, Huser V, Schuemie MJ, et al. Observational Health Data Sciences and Informatics (OHDSI): Opportunities for Observational Researchers. Stud Health Technol Inform. 2015;216:574–8. Regenstrief Institute, Inc. LOINC [Internet]. LOINC. [cited 2022 Aug 24]. Available from: https://loinc.org/ SNOMED International. SNOMED International Browser [Internet]. [cited 2022 Aug 24]. Available from: https://browser.ihtsdotools.org/ Garza M, Del Fiol G, Tenenbaum J, Walden A, Zozus MN. Evaluating common data models for use with a longitudinal community registry. J Biomed Inform. 2016 Dec;64:333–41. Peterson KJ, Jiang G, Brue SM, Liu H. Leveraging Terminology Services for Extract-Transform-Load Processes: A User-Centered Approach. AMIA Annu Symp Proc. 2017 Feb 10;2016:1010–9. Chondrogiannis E, Andronikou V, Tagaris A, Karanastasis E, Varvarigou T, Tsuji M. A novel semantic representation for eligibility criteria in clinical trials. J Biomed Inform. 2017 May;69:10–23. Yuan C, Ryan PB, Ta C, Guo Y, Li Z, Hardin J, et al. Criteria2Query: a natural language interface to clinical databases for cohort definition. J Am Med Inform Assoc. 2019 Apr 1;26(4):294–305. Liu X, Hersch GL, Khalil I, Devarakonda M. Clinical Trial Information Extraction with BERT. In: 2021 IEEE 9th International Conference on Healthcare Informatics (ICHI) [Internet]. Victoria, BC, Canada: IEEE; 2021 [cited 2022 Sep 26]. p. 505–6. Available from: https://ieeexplore.ieee.org/document/9565728/ Tseo Y, Salkola MI, Mohamed A, Kumar A, Abnousi F. Information Extraction of Clinical Trial Eligibility Criteria. 2020 [cited 2022 Sep 26]; Available from: https://arxiv.org/abs/2006.07296 Kang B, Yoon J, Kim HY, Jo SJ, Lee Y, Kam HJ. Deep-learning-based automated terminology mapping in OMOP-CDM. J Am Med Inform Assoc JAMIA. 2021 May 13;28(7):1489–96. Kang T, Zhang S, Tang Y, Hruby GW, Rusanov A, Elhadad N, et al. EliIE: An open-source information extraction system for clinical trial eligibility criteria. J Am Med Inform Assoc. 2017 Nov 1;24(6):1062–71. National Library of Medicine (NLM). Home - ClinicalTrials.gov [Internet]. [cited 2022 Aug 24]. Available from: https://clinicaltrials.gov/ Bundesinstitut für Arzneimittel und Medizinprodukte (BfArM). BfArM - ICD-10-GM Version 2022 [Internet]. [cited 2022 Aug 24]. Available from: https://www.dimdi.de/static/de/klassifikationen/icd/icd-10-gm/kode-suche/htmlgm2022/ Bundesinstitut für Arzneimittel und Medizinprodukte (BfArM). BfArM - OPS Version 2022 [Internet]. [cited 2022 Aug 24]. Available from: https://www.dimdi.de/static/de/klassifikationen/ops/kode-suche/opshtml2022/ Vidal MMI Germany GmbH. ATC Codes von A bis Z | Gelbe Liste [Internet]. [cited 2022 Aug 24]. Available from: https://www.gelbe-liste.de/atc Van Spall HGC, Toren A, Kiss A, Fowler RA. Eligibility criteria of randomized controlled trials published in high-impact general medical journals: a systematic sampling review. JAMA. 2007 Mar 21;297(11):1233–40. Maier C, Lang L, Storf H, Vormstein P, Bieber R, Bernarding J, et al. Towards Implementation of OMOP in a German University Hospital Consortium. Appl Clin Inform. 2018 Jan;09(1):54–61. Thun S, Dewenter H. ICD-11, ICHI und SNOMED CT – was bedeuten die Systematiken für E‑Health-Anwendungen? Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz. 2018 Jul;61(7):812–20. Schüttler C, Prokosch HU, Sedlmayr M, Sedlmayr B. Evaluation of Three Feasibility Tools for Identifying Patient Data and Biospecimen Availability: Comparative Usability Study. JMIR Med Inform. 2021 Jul 21;9(7):e25531. Vehmas N, Kaipio J. Physicians as usability evaluators – first aid for poor EHR usability? Finn J EHealth EWelfare [Internet]. 2018 May 21 [cited 2021 Sep 15];10(2–3). Available from: https://journal.fi/finjehew/article/view/69162 Additional Declarations No competing interests reported. Supplementary Files Appendix1AllTrialsandCriteriawithcategories.xlsx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-2985787","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":205184716,"identity":"1cc68f7c-52c7-4a30-8723-fa38e26ff48a","order_by":0,"name":"Romina Blasini","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABhUlEQVRIie3SPWvCQBjA8QcOLkOf6BpR9CucCBWp1K+SINQlSsClg1hBSIdWZ0uFfgVFsGvkgbj0hW6FDrU4dHGQClJokJ7xBYdSHEvJHy6ES35cLglAUNAfzZFDD/knIovAHH8WNf8oB1/fs0zskNW8OEHgun/tVwIbIiPYENiS3SRJn18YZFWgxJXhG1nWYyzNmfvxXPFikdqBMZoAlUKgPDlTrxpPA+sCxO7uO9RyoczRFNQSL5ixef7adAVGQe0l20BlDmgNrmxKZWq8DKBpxQ4hB8MGEwglEa+XKWZygXFQ+1EEMuzERCe15hgdBw83ZCFJ+H0kyQMKFyVZ+OT2yyeAOnle9UyS9JqotiSaLiRxVqRo+w/WZyuiOASc6WK1Csq9qE2tzLXJkuQl4XlWbKYwUld7kbYoLFeBQcOmZIf8vSiN3gzn2VI4XBjP0DvOCZcRM+fxnDZsdKeT0yPjpqaMp59eNSGG9e72dWs//RVs++1Q7MzslzLa/96goKCg/9c3d9OPwAUDTssAAAAASUVORK5CYII=","orcid":"","institution":"Institute of Medical Informatics, University of Giessen","correspondingAuthor":true,"submittingAuthor":false,"prefix":"","firstName":"Romina","middleName":"","lastName":"Blasini","suffix":""},{"id":205184718,"identity":"34354b5b-528f-4b3d-b170-4c58f3645ef0","order_by":1,"name":"Kornelia Marta Buchowicz","email":"","orcid":"","institution":"Faculty of Health Sciences, University of Applied Sciences","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Kornelia","middleName":"Marta","lastName":"Buchowicz","suffix":""},{"id":205184719,"identity":"8415dd27-b9ec-4327-90b3-d1427e539a4e","order_by":2,"name":"Henning Schneider","email":"","orcid":"","institution":"Institute of Medical Informatics, University of Giessen","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Henning","middleName":"","lastName":"Schneider","suffix":""},{"id":205184720,"identity":"773fe47f-e6e4-47d3-84ee-a99cc22ab05e","order_by":3,"name":"Birgit Samans","email":"","orcid":"","institution":"Faculty of Health Sciences, University of Applied Sciences","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Birgit","middleName":"","lastName":"Samans","suffix":""},{"id":205184721,"identity":"a7140408-8f6c-49eb-8757-11e376934712","order_by":4,"name":"Keywan Sohrabi","email":"","orcid":"","institution":"Faculty of Health Sciences, University of Applied Sciences","correspondingAuthor":false,"submittingAuthor":false,"prefix":"","firstName":"Keywan","middleName":"","lastName":"Sohrabi","suffix":""}],"badges":[],"createdAt":"2023-05-26 13:14:13","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-2985787/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-2985787/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":37848406,"identity":"1e3e815c-0096-46ab-831c-98acb33013fa","added_by":"auto","created_at":"2023-06-01 14:40:52","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":104524,"visible":true,"origin":"","legend":"\u003cp\u003eProcedure in ATLAS\u003c/p\u003e","description":"","filename":"Fig1ProcedureATLAS.png","url":"https://assets-eu.researchsquare.com/files/rs-2985787/v1/e10da3998128b68b4771befe.png"},{"id":37850606,"identity":"9d2cb619-0826-45f8-932e-c7d236f4c94e","added_by":"auto","created_at":"2023-06-01 14:48:52","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":65276,"visible":true,"origin":"","legend":"\u003cp\u003eComparison of concept sets between type I diabetes mellitus type (left) and hemophilia A (right)\u003c/p\u003e","description":"","filename":"Fig2ExampleConceptSets.png","url":"https://assets-eu.researchsquare.com/files/rs-2985787/v1/4caf914e7b74aeda203f00ef.png"},{"id":38202222,"identity":"d97f0e2e-dcdd-40ef-8006-72fc36dfec56","added_by":"auto","created_at":"2023-06-08 05:44:41","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":505305,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-2985787/v1/b19b6a21-d735-40b1-9832-4559b808a5b4.pdf"},{"id":37848405,"identity":"d4e2ac23-ce26-4381-907f-a009341cd42b","added_by":"auto","created_at":"2023-06-01 14:40:52","extension":"xlsx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":48762,"visible":true,"origin":"","legend":"","description":"","filename":"Appendix1AllTrialsandCriteriawithcategories.xlsx","url":"https://assets-eu.researchsquare.com/files/rs-2985787/v1/d68e573da9e989b352ddf399.xlsx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Implementation of inclusion and exclusion criteria of clinical studies in the software OHDSI ATLAS","fulltext":[{"header":"1 Background","content":"\u003cp\u003eIn the field of medicine, the goal of clinical trials is always to learn more about the human body in an effective and positive way. The ultimate aim of a medical study is always to improve patient care and therapeutic standards through the application of clinical trials. In order to observe the influence of the new interventions and preventive measures in terms of safety and efficiency, a certain number of participants are recruited according to certain criteria. Such recruitment is always a challenging and time-consuming stage in the medical trial pipeline, resulting in a large number of canceled or incomplete trials (\u003cspan additionalcitationids=\"CR2\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eIn each clinical trial, eligibility criteria (eligibility criteria) are defined at the beginning of study development describing the relevant characteristics shared by the participants. Patient cohort identification and recruitment usually is carried out by research staff or primary care personnel through querying the clinical systems manually for patients matching the eligibility criteria. The latter process is a time-consuming and cost-intensive part of a clinical research study pipeline. Therefore, utilizing digital tools can significantly improve the recruitment of subjects and reduce costs and required labor resources (\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eClinical Trial Recruitment Support Systems (CTRSS) or patient recruitment systems (PRS) can booster patient inclusion of clinical trials by automatically analyzing eligibility criteria based on electronic health records (\u003cspan additionalcitationids=\"CR5\" citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e). Although these systems nowadays are integrated increasingly in many information and communication technology information and communication technology medical research projects, most of the study centers do not tend to use such digital tools for patient recruitment (\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e). The main reason is the requirement to domain technical staff as well as equipment to perform implementation or supportive tasks for the CTRSS. To address this issue, an individual system can be launched for multiple platforms or to integrate the CTRSS with the existing research systems.\u003c/p\u003e \u003cp\u003eIn addition to the existing CTRSS or commercial solutions developed as part of different projects, several open-source tools like the \u003cem\u003eObservational Medical Outcomes Partnership Common Data Model\u003c/em\u003e (OMOP CDM) developed by \u003cem\u003eObservational Health Data Sciences and Informatics\u003c/em\u003e (OHDSI) or \u003cem\u003eInformatics for Integrating Biology and the Beside\u003c/em\u003e (i2b2) established by \u003cem\u003eNational Institutes of Health\u003c/em\u003e, are free to use for patient recruitment. Both systems are based on the idea of using standard terminologies within data repositories however, they are not bound to a specific nomenclature. In addition, data can be extracted from different sources and in various models, so they may be utilized in different medical contexts and at multiple study centers with different prerequisites (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e, \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e). Therefore an architecture for a CTRSS utilizing OMOP as its data basis can be used for a patient recruitment system for example.(\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e)\u003c/p\u003e \u003cdiv id=\"Sec2\" class=\"Section2\"\u003e \u003ch2\u003e1.1 Observational Health Data Sciences and Informatics (OHDSI)\u003c/h2\u003e \u003cp\u003eThe OMOP CDM is part of a whole tool suite created and maintained by OHDSI, as a collaboration among 150 organizations around the world that collect and process healthcare data. OHDSI offers a wide range of open-source tools to support various data-analytics use cases on observational patient-level data, all are interacting with databases using the OMOP CDM. (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e)\u003c/p\u003e \u003cp\u003eInspired by OHDSI, the ATLAS software was designed and developed as a web-based open source application. The primary goal of ATLAS is to be used for the observational analysis and generating real-world evidence through patient data extracted from clinical practice. ATLAS operates as a user interface on the OMOP CDM. According to (\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e) ATLAS allows defining cohorts, selecting analysis configurations, and tagging diseases with appropriate codes. Also, it allows sharing the project easily with other researchers (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eATLAS can also be used for subject recruitment by formalizing the eligibility criteria of a trial for the corresponding cohort. The definition of trials in ATLAS works with two main designs: cohorts and concepts. Concepts are individual codes that belong to a terminology system integrated in the OMOP CDM. They can be grouped into concept sets. A concept set contains several concepts from the standardized vocabulary in combination with logical indicators. It allows the user to specify whether related concepts should be included or excluded from the vocabulary hierarchy. Cohorts are the collection of individuals searched for by a particular query based on specific criteria, so all of a study's eligibility criteria are part of the cohort definition (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e).\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003e1.2 Terminology systems\u003c/h2\u003e \u003cp\u003eThe database of ATLAS enforces the feeding data to follow standardized classification systems like Logical Observation Identifiers Names and Codes (LOINC) or Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT). These nomenclatures work as an interface for the study centers and built the basis for generating patient cohorts. (\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e, \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e)\u003c/p\u003e \u003cp\u003eSeveral analyses have been conducted to investigate the use of digital systems based on international terminologies.(\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e, \u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e) For example, (\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e) investigated the use of a \u003cem\u003eClinical Data Interchange Standard Consortium\u003c/em\u003e (CDISC) based model in conjunction with a patient-centered model to structure eligibility criteria of clinical studies. International classification systems, such as \u003cem\u003eInternational Statistical Classification of Diseases and Related Health Problems\u003c/em\u003e (ICD), formed the basis for this project. The authors used 200 randomly selected inclusion criteria from other studies, and the majority of the criteria could be represented in the system. (\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e)\u003c/p\u003e \u003cp\u003eAutomatic tools like criteria2query use natural language processing (NLP) to automate formalization of eligibility criteria into the ATLAS cohorts. The latter system works just on specific terminology systems and local specialties are not considered. As the definition of an ATLAS cohort depends highly on the engaged terminology systems, such tools cannot be applied in the presence of other supported coding systems. (\u003cspan additionalcitationids=\"CR19 CR20\" citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e)\u003c/p\u003e \u003cp\u003eAlthough there are more studies available addressing the latter issue, there still is a gap left between the theory and practically introducing the eligibility criteria with different terminology systems into the software (\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e, \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e). Therefore, further analysis should be conducted regarding to this matter in order to explore the hidden potentials behind the different software and coding systems. This study considers the possibility to apply inclusion and exclusion criteria of different clinical trials in ATLAS as a proof of concept.\u003c/p\u003e \u003c/div\u003e"},{"header":"2 Methods","content":"\u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e2.1 Objectives\u003c/h2\u003e \u003cp\u003eThe first step in applying the eligibility criteria into ATLAS is to discover the feasibility of their implementation (in terms of quantity) with respect to the utilized standardized open-source terminology system. Also, the technical difficulties in the implementation process will be analyzed as the qualitative drawbacks.\u003c/p\u003e \u003cp\u003eThe selected terminology systems in our study are the standard classification systems ICD and \u003cem\u003eOperation and Procedure Code (OPS)\u003c/em\u003e, which is a German modification of the \u003cem\u003eInternational Classification of Procedures in Medicine\u003c/em\u003e (ICPM), utilized for billing in all German hospitals. Also, the \u003cem\u003eAnatomical Therapeutic Chemical classification\u003c/em\u003e (ATC) system is selected for the medications and LOINC coding system for laboratory results as well.\u003c/p\u003e \u003cp\u003eIn a next step, all eligibility criteria which could not be introduced into ATLAS are considered. They are categorized based on the shortlisted drawbacks above.\u003c/p\u003e \u003cp\u003eThis paper works on establishing classification systems and standards, as well as dealing with the mapping of inclusion and exclusion criteria generally and in ATLAS especially.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e2.2 Study design\u003c/h2\u003e \u003cp\u003eTo initiate the research, we randomly selected 30 studies without any preference with respect to a medical discipline. For this sake, we searched on the platform \"ClinicalTrials.gov\", one of the largest registries of clinical trials including about 425,817 studies from 222 different countries (\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e). The selected studies had all to be started within the last 10 years resulted in 114 studies wherein 30 studies have been selected randomly. Also, to investigate the extent to which the eligibility criteria can be implemented in ATLAS, we documented all 331 eligibility criteria of the 30 studies in a table.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003e2.3 Procedure in ATLAS\u003c/h2\u003e \u003cp\u003eTo begin the process of introducing criteria into ATLAS, we selected an appropriate terminology system with appropriate codes for each eligibility criteria, as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. The appropriate codes were identified using official online catalogs (\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e, \u003cspan additionalcitationids=\"CR25\" citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eIf no appropriate codes were found in these catalogs, SNOMED CT would have added to the analysis to check for an alternative way to introduce or embed an approval criterion. (\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e)\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eIn ATLAS all fitting codes describing a single eligibility criterion are summarized in one concept set, which is a collection of codes e.g. concepts. This is necessary, because a single criterion can be constituted of one up to hundreds of codes. For example, type I diabetes mellitus can have different characteristics, which can be marked with different ICD codes. Codes for these disease individualities create one concept set. In comparison, Hemophilia A could be created as a concept set using a single code, as there is only one ICD code for this specific disease (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). If there was any doubt about an eligibility criterion or related codes, medical staff was consulted. Afterwards, a separate concept set had to be created for each eligibility criterion, although no concept sets were created for demographic data such as age or gender. The aforementioned criteria were entered directly during the later process within a cohort by means of the corresponding functions available in ATLAS.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eOnce all the concept sets required for a study were established, cohorts, that included all eligibility criteria for a study, had to be defined. To outline a cohort, at first it is necessary to define an initial event, which is a main criterion and used for a preselection of subjects. All individuals, which fulfill the requirements of the initial event, build the main cohort of the patients. To set further constraints, it is possible to set other criteria to get more specified results. We always selected the examined disease of the trial as an initial event for the cohort. For example, if a study needed test group who meet selection criteria such as type I diabetes mellitus, age under 18 years and elevated Body-Mass-Index (BMI), we would define diabetes mellitus as initial event. All patients with type I diabetes built the main cohort. Criteria \u0026ldquo;age\u0026rdquo; and \u0026ldquo;BMI\u0026rdquo; would have added as further eligibility criteria.\u003c/p\u003e \u003c/div\u003e"},{"header":"3 Evaluation of the studies and eligibility criteria","content":"\u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Categorization of eligibility criteria and studies\u003c/h2\u003e \u003cp\u003eOf all 331 eligibility criteria, 108 criteria were excluded because of missing relevance. These criteria were the ones, which do not exist prior to a face-to-face interview with a study physician so, they cannot be analyzed in the process of the electronic patient screening.\u003c/p\u003e \u003cp\u003eTo categorize the eligibility criteria, we analyzed the remaining 223 suitability criteria and classified them in terms of their implementation in ATLAS into the categories implementable, partially implementable, and not implementable (s. Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). If a criterion was only partially or not implementable, we would also add short notes expressing our decision.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eCriteria categories\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCriteria Category\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eExplanation\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eimplementable\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCriterion can be implemented in ATLAS with given terminology systems.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003epartially \u003c/p\u003e \u003cp\u003eimplementable\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCriterion can be implemented, but not with the given terminologies (only SNOMED CT) or only a part of criterion is mapped (for example a disease can be implemented, but a timely constraint is not possible).\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003enot implementable\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCriterion cannot be implemented with the help of the given terminologies or SNOMED CT and the functions available in ATLAS.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eAfter weighting the studies with respect to their feasibility, four categories have been established: fully portrayable, portrayable, partially portrayable, and not portrayable. (see Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eTrial Categories\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTrial Category\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eExplanation\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003efully portrayable\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAll criteria of a study can be implemented in ATLAS.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eportrayable\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThe initial event is classified as implementable and can be mapped completely in ATLAS. Further inclusion criteria can be classified as partially implemented or not implementable according to Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003epartially portrayable\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThe initial event is classified as partially implementable in ATLAS according to Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003enot portrayable\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThe initial event cannot be implemented in ATLAS.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eExplanation of trial categories in the overall evaluation about feasibility of measures taken in ATLAS\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003e3.2 Classification of partially/not implementable criteria\u003c/h2\u003e \u003cp\u003eFinally, we made a secondary analysis to discover the main difficulties in the introducing into or implementation of eligibility criteria in the software ATLAS. Therefore, all reasons and argumentations have been considered to justify the categorization and the implementation issues of the eligible criteria.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003e3.3 Ethical considerations\u003c/h2\u003e \u003cp\u003eNo ethical vote was needed.\u003c/p\u003e \u003c/div\u003e"},{"header":"4 Results","content":"\u003cp\u003eThe described categorization resulted in an implementation of 30 cohorts with one initial event per cohort and an additional number of 223 eligibility criteria. For implementing all eligibility criteria and initial events, we built 129 different concept sets, including 11.2 concepts in average. Each cohort included in average 5.33 concept sets with 59.73 concepts in total. The whole table can be found in Appendix 1.\u003c/p\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003e4.1 Evaluation of eligibility criteria\u003c/h2\u003e \u003cp\u003eAccording to our observation, 49.33% (n\u0026thinsp;=\u0026thinsp;110) of all eligibility criteria identified in the studies could be mapped in ATLAS so classified as implementable (see. Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). This set is corresponding with the eligibility criteria referring to clearly defined diseases, procedures, or drugs respectively with the corresponding ICD, OPS, ATC or LOINC codes. Also, 20.63% (n\u0026thinsp;=\u0026thinsp;46) of the criteria were defined as partially implementable and 30.04% (n\u0026thinsp;=\u0026thinsp;67) could not be implemented in ATLAS at all.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003e4.2 Overall evaluation of analyzed studies\u003c/h2\u003e \u003cp\u003eBased on our evaluation, in 10% of the studies (n\u0026thinsp;=\u0026thinsp;3) all criteria could be fully mapped, so these studies were classified as fully portrayable, (see. Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). The latter set includes a few simple and clearly defined criteria, such as a well-defined diseases, forming a group of subjects which could be filtered more by one or two additional criteria, like age or gender. In these cases, the disease has been introduced into ATLAS using the appropriate codes and as an initial event.\u003c/p\u003e \u003cp\u003eAlso, 73.33% of the studies (n\u0026thinsp;=\u0026thinsp;22) have been classified as portrayable, where at least one of the eligibility criteria was only partially or not implementable according to the rules in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. However, the initial event could be fully implemented as a concept set using ICD, OPS, LOINC or ATC. In 16.67% (n\u0026thinsp;=\u0026thinsp;5) it was not possible to fully implement the initial event and these were assigned to the category partially portrayable. Finally, none of the studies was classified as not portrayable.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003e4.3 Partially implementable criteria\u003c/h2\u003e \u003cp\u003eAfter analyzing the justifications corresponding to the criteria which could not be introduced into ATLAS, we have categorized them in four groups described in Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eCategories of partially implementable criteria\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCriteria Category\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eExplanation\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMissing information\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eDescription is not precise\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHighly specialized criterion\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHighly specialized diagnosis, procedure or laboratory result\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTechnical reasons\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003ePartial aspect is not feasible due to technical reasons\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTerminology system\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eOnly implementable with SNOMED CT\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eAccording to the results, 39.13% (n\u0026thinsp;=\u0026thinsp;18) of the partially implemented criteria are belonging to the associated class due to technical reasons, with different causes. For example, in some studies, the ages are quoted in years but in others like pediatric studies dealing with the infants quoted in a monthly basis. In ATLAS it is only possible to query the age in years. Some criteria also need a cause-and-effect relationship. In such criteria, the information whether the patient is involved in cross studies is required which cannot be extracted from an OMOP data model.\u003c/p\u003e \u003cp\u003eAmong the rest of the criteria, 26.09% (n\u0026thinsp;=\u0026thinsp;12) could not be fully implemented due to missing information. Those criteria mainly refer to \u0026ldquo;norm\u0026ldquo; or \u0026ldquo;stable\u0026rdquo; values even if there are no defined norm quantities valid in all countries or locations but different standards from site to site. The latter set includes criteria that refer to the groups of medicaments, diagnosis, procedures, or laboratory results, which are not globally defined.\u003c/p\u003e \u003cp\u003eAnother set of criteria includes 17.39% (n\u0026thinsp;=\u0026thinsp;8) criteria which were too specific and no terminology system has been found to have a specific corresponding code, except for the codes that also include a group of diseases/procedures/laboratory/results/medications. Finally, 17.39% (n\u0026thinsp;=\u0026thinsp;8) eligible criteria, have been classified with SNOMED CT.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003e4.4 Not implementable criteria\u003c/h2\u003e \u003cp\u003eTo categorize the criteria which are not implementable, we have selected the same categories, without the category corresponding to the wrong terminology system. In 47.76% (n\u0026thinsp;=\u0026thinsp;32) eligible criteria, missing information leads to the categorizing as \u0026ldquo;not implementable\u0026rdquo;. 28.36% (n\u0026thinsp;=\u0026thinsp;19) of the criteria were too specific and 23.88% (n\u0026thinsp;=\u0026thinsp;16) were not implementable due to technical reasons.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec17\" class=\"Section2\"\u003e \u003ch2\u003e4.5 Technical usability aspects of ATLAS\u003c/h2\u003e \u003cp\u003eAccording to the practices performed with ATLAS in this work, it is observed that the intuitive operations in the software have a poor performance which necessitates a training phase prior launching the main study work. Individual adaptations are needed to enter complex or nested study criteria. For example, the scores cannot be noted by selecting an available function in ATLAS or by introducing one specific code as a concept set. The analyzed scores usually consist of several sub-items, such as diseases or laboratory values, which are assigned a value depending on their severity. In these cases, the content of each score first has to be reviewed more in detail. This means that all aspects of a score are treated in the first step like individual criteria that have to be introduced into the system as separate concept sets, which is a major effort for complex scale studies.\u003c/p\u003e \u003cp\u003eATLAS has the possibility to input basic demographic data in an easy way and without assigning a concept, but these data are limited to age and gender. Other basic information like height and weight has to be input like other observations. For this purpose, the appropriate LOINC code has to be identified, created as a concept set and time constraints might have to be set, when only the latest data on height and weight are inquired. At this point, this information can be integrated into the cohort.\u003c/p\u003e \u003cp\u003eAlso, temporal information, like the starting point of an event or a treatment can only be introduced as an absolute date, not as relative time information like \u0026ldquo;six month ago\u0026rdquo;. Therefore, the selection criteria for a cohort of a time period must be updated on a daily basis.\u003c/p\u003e \u003c/div\u003e"},{"header":"5 Discussion","content":"\u003cdiv id=\"Sec19\" class=\"Section2\"\u003e \u003ch2\u003e5.1 Digital subject recruitment\u003c/h2\u003e \u003cp\u003eOMOP and ATLAS as the products of OHDSI have been developed to support multi-institutional research projects. In this study, we intended to test the possibility of using OMOP and ATLAS not only to run analysis on the existing data sets, but also to portray the eligibility criteria of clinical trials. This makes it suitable as a digital tool for recruiting of candidates for clinical studies. We have observed that the implementation of trial eligibility criteria in ATLAS cohorts is more complex than a 1:1 translation: about half of the eligibility criteria could not be fully implemented in ATLAS.\u003c/p\u003e \u003cp\u003eOne main reason (n\u0026thinsp;=\u0026thinsp;44) is that the eligibility criteria from study descriptions are defined for the use of study physicians or study personnel and often leave scope for individual professional assessment. These are easy to interpret for medical domain experts while reading the criterion, such as \u0026ldquo;stable laboratory results\u0026rdquo;, in an intuitive way. However, these criteria are not globally defined and each physician can have a slightly different interpretation of \u0026ldquo;stable\u0026rdquo; for example. A deep understanding of the criteria is essential for the process and the lack of such an understanding, results in criteria not being introduced into the system or being incorrectly implemented. The precise formulation of subject characteristics is required for a clinical research initially, so well-chosen criteria are crucial for the feasibility and internal validity of studies (\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e). In terms of ATLAS, carefully chosen and precisely formulated eligibility criteria form the basis for being able to use the software correctly and to exploit its potential functionalities. To get better results in the process of formalizing eligibility criteria for an automated search, it is helpful to collaborate tightly with the study personnel, so that imprecise criteria can be refined by professionals. Subsequently, all these criteria and therefore more trials, can be implemented in ATLAS.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec20\" class=\"Section2\"\u003e \u003ch2\u003e5.2 The utilized terminology systems\u003c/h2\u003e \u003cp\u003eSince the ATLAS software is based on the involvement of terminology systems, the analysis highlights the importance of the choice of terminology systems and thus the availability of codes contained in terminology system. ICD, ATC and LOINC are international classification systems widely used in German clinics in hospital information systems and therefore they are easy to access and use if hospital data are queried. Also, OPS is a terminology system only used in Germany applied for the code procedures for billing (\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e). In addition to the 49.33% of the terms that could be mapped by ICD, ATC, LOINC and OPS, further 4% of the criteria could be mapped by using SNOMED CT, which is the standard terminology system used in OMOP (\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eTerminology systems have a huge impact on the opportunities in building concept sets and cohorts in ATLAS. The medical terminology in SNOMED CT is significantly in more details than in ICD. SNOMED CT includes not only diagnoses itself, but also medical terms, which are placed in relation to each other. The use of ICD is recommended to record the diagnoses for billing and has a hierarchically arranged structure of diagnoses, so it is more widely used in the patient care systems and therefore provides the ease of access. Consequently, it is essential to inspect the utility and target group before making a statement about nomenclatures being better suited for a specific purpose. A combination of the coding methods, for example combination of ICD and SNOMED CT, is also recommended by (\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eHowever, there were also many exclusion and inclusion criteria, which could not be mapped by means of chosen classification systems (neither by ICD, OPS, ATC, or LOINC, nor by SNOMED CT) due to high individuality of the subjects (n\u0026thinsp;=\u0026thinsp;27). Clinical trials are often designed to gain knowledge about novel or under-researched diseases. The codes for these conditions do not yet exist in the medical classification systems, which makes it difficult or impossible to introduce some selection criteria into ATLAS. In some cases, eligibility criteria refer to non-medical settings like the proof of a vaccination. Those criteria are not commonly included in the medical terminologies, as the corresponding domain is a medical and not an administrative scenery.\u003c/p\u003e \u003cp\u003eAccording to our observations, we can conclude that the formulated eligibility criteria have an impact on using correct terms, but on the other hand, for the specialized criteria no fitting codes are available. Also, it has been shown that the choice and correct use of medical classification systems, like the definition of inclusion and exclusion criteria, is an important basis for working with ATLAS. Therefore, there is a need for the correct use of nomenclatures and standardization, as well as continual adaptations to medical progress, which researchers, physicians, and others involved in health care need to strive for.\u003c/p\u003e \u003cp\u003eIn this study, the effectiveness and suitability of the individual classification systems has not been analyzed. So, we cannot state a verdict on the usability of some nomenclatures in ATLAS.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec21\" class=\"Section2\"\u003e \u003ch2\u003e5.3 Limitations of ATLAS\u003c/h2\u003e \u003cp\u003eAs expressed before, the requirement to a training phase before working with the software is a drawback. Also, in some cases, individual adaptations are required to introduce complex or nested data sets such as scores. This indicates problems with usability of ATLAS that have been also confirmed by other studies. In 2021 (\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e) focused on the timeline availability of clinical data and bio specimens that can be identified using digital applications. Three systems, i2b2, OHDSI's ATLAS, and Sample Locator by the German Biobank Alliance, were tested in terms of strengths, weaknesses, and associated usability via qualitative exploratory study followed by a web-based usability test. The results of this study show that on one hand ATLAS has significantly more functionalities than other tools and a very good interface visualization. However, multiple usability issues exist as well. These problems can be associated to the large number of functions and different approaches when introducing selection criteria. However, the authors take into account that ATLAS has been primarily developed for the researchers and experts who would often define complex cohort queries. Therefore it is important to provide them with the variety of selection and input options, however, it increases the operational costs (\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eThis study also suggests that ATLAS is more suitable for people with the appropriate technical background, due to the existing features and the need for a certain level of development for the sake of customization. However, according to the developers, the application aims to provide the greatest scientific impact for researchers. Therefore, promoting the usability and adaptability of the software for this target group is of great importance. This should be supported by members of the target group as described in (\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e) wherein the medical students and licensed physicians were asked to test electronic health record used in Finland in terms of usability. It has been concluded that the physicians were able to identify potential problems corresponding to the usability aspect and provide the corresponding feedback. So, it is recommended that target groups of a specific application to be involved in the usability development process in the test phases to detect problems in a timely manner and avoid stress in the future.\u003c/p\u003e \u003cp\u003eIt is worth mentioning that an inaccurate implementation of eligibility criteria in ATLAS can lead to less precise subject recommendations. If the entry of studies into ATLAS is intended to lead to suitable subject recruitment, the involvement of specialized study physicians or study personnel in the process of cohort definition will be helpful.\u003c/p\u003e \u003c/div\u003e"},{"header":"6 Conclusions","content":"\u003cp\u003eIn this study, we have considered the selection criteria of 30 studies for the possibility to portray them in the software ATLAS. We have found out that about 50% of the criteria could be implemented into the software, about 20% could only be partially implemented, and the rest of 30% could not be implemented at all.\u003c/p\u003e \u003cp\u003eThe four main justifications are the existence of missing information about the criteria, limitations of the software, limitations of the chosen terminology systems and finally highly specialized criteria. To use ATLAS as a CTRSS, we recommend the following points to improve the coverage of eligibility criteria in a cohort definition.\u003c/p\u003e \u003cp\u003eAs ATLAS offers a variety of holistic functionalities, the usability degree is not trivial. It can be useful to plan a training phase before starting with the main work. Even for the planning phase of a project, it is helpful to know all possibilities and limitations of ATLAS right from the start. Another difficulty is the limitation of the selected terminology systems. It can be helpful to examine the options and possibilities of different terminology systems to find the best and suitable system for the project. The combination of different terminology systems can also be useful to improve the coverage of eligibility criteria in cohort definitions.\u003c/p\u003e \u003cp\u003eThe main problem in portraying clinical trials in ATLAS is the missing information in eligibility criteria. A tight cooperation with study personnel, which is usually responsible for the subject recruitment and works with the criteria daily, can help to specify these uncertainties. If all missing information can be cleared with the study personnel, this could raise the percentage of implementable criteria up to about 70%. This could help with criteria that are highly specialized to be covered by terminology systems, and therefore it is only possible to implement parts of the criteria or use collective codes, which are not precise. Usually, the study personnel has similar issues when searching for eligible persons, so they can help in identifying other ways to portray these criteria. Even if the criteria are not fully implementable, it is sometimes possible to use similar criteria, which are valid for a preselection. In addition, the study personnel can prioritize the relevance of different criteria, as some of the criteria are more important for a selection of candidates than others.\u003c/p\u003e \u003cp\u003eIn this work, we picked one of the criteria as the main criterion for the cohort, and leave the rest with no prioritizing. Although some criteria are more relevant for the selection process of the subjects than others. The importance level of a criterion, can only be estimated by the study personnel working in the recruitment process. Further work is planned to examine which and how many of the criteria are relevant for the selection process of clinical trials and to survey how many of the relevant criteria can be implemented.\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.70860927152318%\" valign=\"top\"\u003e\n \u003cp\u003eATC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"81.29139072847683%\" valign=\"top\"\u003e\n \u003cp\u003eAnatomical Therapeutic Chemical classification\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.70860927152318%\" valign=\"top\"\u003e\n \u003cp\u003eBMI\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"81.29139072847683%\" valign=\"top\"\u003e\n \u003cp\u003eBody-Mass-Index\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.70860927152318%\" valign=\"top\"\u003e\n \u003cp\u003eCDISC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"81.29139072847683%\" valign=\"top\"\u003e\n \u003cp\u003eClinical Data Interchange Standard Consortium\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.70860927152318%\" valign=\"top\"\u003e\n \u003cp\u003eCDM\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"81.29139072847683%\" valign=\"top\"\u003e\n \u003cp\u003eCommon Data Model\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.70860927152318%\" valign=\"top\"\u003e\n \u003cp\u003eCTRSS\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"81.29139072847683%\" valign=\"top\"\u003e\n \u003cp\u003eClinical Trial Recruitment Support System\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.70860927152318%\" valign=\"top\"\u003e\n \u003cp\u003eICD\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"81.29139072847683%\" valign=\"top\"\u003e\n \u003cp\u003eInternational Statistical Classification of Diseases and Related Health Problems\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.70860927152318%\" valign=\"top\"\u003e\n \u003cp\u003eICPM\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"81.29139072847683%\" valign=\"top\"\u003e\n \u003cp\u003eInternational Classification of Procedures in Medicine\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.70860927152318%\" valign=\"top\"\u003e\n \u003cp\u003ei2b2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"81.29139072847683%\" valign=\"top\"\u003e\n \u003cp\u003eInformatics for Integrating Biology and the Beside\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.70860927152318%\" valign=\"top\"\u003e\n \u003cp\u003eLOINC\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"81.29139072847683%\" valign=\"top\"\u003e\n \u003cp\u003eLogical Observation Identifiers Names and Codes\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.70860927152318%\" valign=\"top\"\u003e\n \u003cp\u003eOHDSI\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"81.29139072847683%\" valign=\"top\"\u003e\n \u003cp\u003eObservational Health Data Sciences and Informatics\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.70860927152318%\" valign=\"top\"\u003e\n \u003cp\u003eOMOP\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"81.29139072847683%\" valign=\"top\"\u003e\n \u003cp\u003eObservational Medical Outcomes Partnership\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.70860927152318%\" valign=\"top\"\u003e\n \u003cp\u003eOPS\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"81.29139072847683%\" valign=\"top\"\u003e\n \u003cp\u003eOperation and Procedure Code\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.70860927152318%\" valign=\"top\"\u003e\n \u003cp\u003ePRS\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"81.29139072847683%\" valign=\"top\"\u003e\n \u003cp\u003ePatient Recruitment System\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd width=\"18.70860927152318%\" valign=\"top\"\u003e\n \u003cp\u003eSNOMED CT\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd width=\"81.29139072847683%\" valign=\"top\"\u003e\n \u003cp\u003eSystematized Nomenclature of Medicine Clinical Terms\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e"},{"header":"Declarations","content":"\u003cp\u003eEthics approval and consent to participate\u003c/p\u003e\n\u003cp\u003eNot applicable\u003c/p\u003e\n\u003cp\u003eConsent for publications\u003c/p\u003e\n\u003cp\u003eNot applicable\u003c/p\u003e\n\u003cp\u003eAvailability of data and materials\u003c/p\u003e\n\u003cp\u003eAll data generated or analysed during this study are included in this published article and its supplementary information files.\u003c/p\u003e\n\u003cp\u003eCompeting interests\u003c/p\u003e\n\u003cp\u003eThe authors declare no conflict of interest.\u003c/p\u003e\n\u003cp\u003eFunding\u003c/p\u003e\n\u003cp\u003eThis work is part of the project MIRACUM, funded by the German Ministry of Education and Research (FKZ 01ZZ1801D).\u003c/p\u003e\n\u003cp\u003eAuthors\u0026rsquo; contributions\u003c/p\u003e\n\u003cp\u003eRB and KMB wrote the original texts of this work. RB and KS did the conceptual work of this study. KMB analyzed all trials and criteria, as well as implemented and categorized the criteria. HS and BS were involved in the conceptual work of this article. All authors read and approved the final manuscript.\u003c/p\u003e\n\u003cp\u003eAcknowledgements\u003c/p\u003e\n\u003cp\u003eWe are thankful to Dr. rer. nat. Seyed Amir Hossein Tabatabaei for reviewing this work and offering useful advice. \u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eCarlisle B, Kimmelman J, Ramsay T, MacKinnon N. Unsuccessful trial accrual and human subjects protections: an empirical analysis of recently closed trials. Clin Trials Lond Engl. 2015 Feb;12(1):77\u0026ndash;83. \u003c/li\u003e\n\u003cli\u003eDesai M. Recruitment and retention of participants in clinical studies: Critical issues and challenges. Perspect Clin Res. 2020;11(2):51\u0026ndash;3. \u003c/li\u003e\n\u003cli\u003eSully BGO, Julious SA, Nicholl J. A reinvestigation of recruitment to randomised, controlled, multicenter trials: a review of trials funded by two UK funding agencies. Trials. 2013 Jun 9;14(1):166. \u003c/li\u003e\n\u003cli\u003eThadani SR, Weng C, Bigger JT, Ennever JF, Wajngurt D. Electronic Screening Improves Efficiency in Clinical Trial Recruitment. J Am Med Inform Assoc. 2009 Nov 1;16(6):869\u0026ndash;73. \u003c/li\u003e\n\u003cli\u003eTrinczek B, K\u0026ouml;pcke F, Leusch T, Majeed RW, Schreiweis B, Wenk J, et al. Design and multicentric Implementation of a generic Software Architecture for Patient Recruitment Systems re-using existing HIS tools and Routine Patient Data. Appl Clin Inform. 2014;05(01):264\u0026ndash;83. \u003c/li\u003e\n\u003cli\u003ePenberthy L, Brown R, Puma F, Dahman B. Automated matching software for clinical trials eligibility: Measuring efficiency and flexibility. Contemp Clin Trials. 2010 May 1;31(3):207\u0026ndash;17. \u003c/li\u003e\n\u003cli\u003eFitzer K, Haeuslschmid R, Blasini R, Altun FB, Hampf C, Freiesleben S, et al. Patient Recruitment System for Clinical Trials: Mixed Methods Study About Requirements at Ten University Hospitals. JMIR Med Inform. 2022 Apr 20;10(4):e28696. \u003c/li\u003e\n\u003cli\u003eReich C, Ostropolets A. Chapter 5 Standardized Vocabularies | The Book of OHDSI [Internet]. San Bernardino, CA: OHDSI; 2019 [cited 2022 Nov 7]. Available from: https://ohdsi.github.io/TheBookOfOhdsi/\u003c/li\u003e\n\u003cli\u003eProkosch HU, Sedlmayr M. Datenaustausch in der Forschung via OMOP/OHDSI [Data exchange in research via OMOP/OHDSI]. EHealth.com. 2018;2018(4):34\u0026ndash;7. \u003c/li\u003e\n\u003cli\u003eReinecke I, Gulden C, K\u0026uuml;mmel M, Nassirian A, Blasini R, Sedlmayr M. Design for a Modular Clinical Trial Recruitment Support System Based on FHIR and OMOP. Stud Health Technol Inform. 2020 Jun 16;270:158\u0026ndash;62. \u003c/li\u003e\n\u003cli\u003eMeystre SM, Heider PM, Kim Y, Aruch DB, Britten CD. Automatic Trial Eligibility Surveillance Based on Unstructured Clinical Data. Int J Med Inf. 2019 Sep;129:13\u0026ndash;9. \u003c/li\u003e\n\u003cli\u003eHripcsak G, Duke JD, Shah NH, Reich CG, Huser V, Schuemie MJ, et al. Observational Health Data Sciences and Informatics (OHDSI): Opportunities for Observational Researchers. Stud Health Technol Inform. 2015;216:574\u0026ndash;8. \u003c/li\u003e\n\u003cli\u003eRegenstrief Institute, Inc. LOINC [Internet]. LOINC. [cited 2022 Aug 24]. Available from: https://loinc.org/\u003c/li\u003e\n\u003cli\u003eSNOMED International. SNOMED International Browser [Internet]. [cited 2022 Aug 24]. Available from: https://browser.ihtsdotools.org/\u003c/li\u003e\n\u003cli\u003eGarza M, Del Fiol G, Tenenbaum J, Walden A, Zozus MN. Evaluating common data models for use with a longitudinal community registry. J Biomed Inform. 2016 Dec;64:333\u0026ndash;41. \u003c/li\u003e\n\u003cli\u003ePeterson KJ, Jiang G, Brue SM, Liu H. Leveraging Terminology Services for Extract-Transform-Load Processes: A User-Centered Approach. AMIA Annu Symp Proc. 2017 Feb 10;2016:1010\u0026ndash;9. \u003c/li\u003e\n\u003cli\u003eChondrogiannis E, Andronikou V, Tagaris A, Karanastasis E, Varvarigou T, Tsuji M. A novel semantic representation for eligibility criteria in clinical trials. J Biomed Inform. 2017 May;69:10\u0026ndash;23. \u003c/li\u003e\n\u003cli\u003eYuan C, Ryan PB, Ta C, Guo Y, Li Z, Hardin J, et al. Criteria2Query: a natural language interface to clinical databases for cohort definition. J Am Med Inform Assoc. 2019 Apr 1;26(4):294\u0026ndash;305. \u003c/li\u003e\n\u003cli\u003eLiu X, Hersch GL, Khalil I, Devarakonda M. Clinical Trial Information Extraction with BERT. In: 2021 IEEE 9th International Conference on Healthcare Informatics (ICHI) [Internet]. Victoria, BC, Canada: IEEE; 2021 [cited 2022 Sep 26]. p. 505\u0026ndash;6. Available from: https://ieeexplore.ieee.org/document/9565728/\u003c/li\u003e\n\u003cli\u003eTseo Y, Salkola MI, Mohamed A, Kumar A, Abnousi F. Information Extraction of Clinical Trial Eligibility Criteria. 2020 [cited 2022 Sep 26]; Available from: https://arxiv.org/abs/2006.07296\u003c/li\u003e\n\u003cli\u003eKang B, Yoon J, Kim HY, Jo SJ, Lee Y, Kam HJ. Deep-learning-based automated terminology mapping in OMOP-CDM. J Am Med Inform Assoc JAMIA. 2021 May 13;28(7):1489\u0026ndash;96. \u003c/li\u003e\n\u003cli\u003eKang T, Zhang S, Tang Y, Hruby GW, Rusanov A, Elhadad N, et al. EliIE: An open-source information extraction system for clinical trial eligibility criteria. J Am Med Inform Assoc. 2017 Nov 1;24(6):1062\u0026ndash;71. \u003c/li\u003e\n\u003cli\u003eNational Library of Medicine (NLM). Home - ClinicalTrials.gov [Internet]. [cited 2022 Aug 24]. Available from: https://clinicaltrials.gov/\u003c/li\u003e\n\u003cli\u003eBundesinstitut f\u0026uuml;r Arzneimittel und Medizinprodukte (BfArM). BfArM - ICD-10-GM Version 2022 [Internet]. [cited 2022 Aug 24]. Available from: https://www.dimdi.de/static/de/klassifikationen/icd/icd-10-gm/kode-suche/htmlgm2022/\u003c/li\u003e\n\u003cli\u003eBundesinstitut f\u0026uuml;r Arzneimittel und Medizinprodukte (BfArM). BfArM - OPS Version 2022 [Internet]. [cited 2022 Aug 24]. Available from: https://www.dimdi.de/static/de/klassifikationen/ops/kode-suche/opshtml2022/\u003c/li\u003e\n\u003cli\u003eVidal MMI Germany GmbH. ATC Codes von A bis Z | Gelbe Liste [Internet]. [cited 2022 Aug 24]. Available from: https://www.gelbe-liste.de/atc\u003c/li\u003e\n\u003cli\u003eVan Spall HGC, Toren A, Kiss A, Fowler RA. Eligibility criteria of randomized controlled trials published in high-impact general medical journals: a systematic sampling review. JAMA. 2007 Mar 21;297(11):1233\u0026ndash;40. \u003c/li\u003e\n\u003cli\u003eMaier C, Lang L, Storf H, Vormstein P, Bieber R, Bernarding J, et al. Towards Implementation of OMOP in a German University Hospital Consortium. Appl Clin Inform. 2018 Jan;09(1):54\u0026ndash;61. \u003c/li\u003e\n\u003cli\u003eThun S, Dewenter H. ICD-11, ICHI und SNOMED CT \u0026ndash; was bedeuten die Systematiken f\u0026uuml;r E‑Health-Anwendungen? Bundesgesundheitsblatt - Gesundheitsforschung - Gesundheitsschutz. 2018 Jul;61(7):812\u0026ndash;20. \u003c/li\u003e\n\u003cli\u003eSch\u0026uuml;ttler C, Prokosch HU, Sedlmayr M, Sedlmayr B. Evaluation of Three Feasibility Tools for Identifying Patient Data and Biospecimen Availability: Comparative Usability Study. JMIR Med Inform. 2021 Jul 21;9(7):e25531. \u003c/li\u003e\n\u003cli\u003eVehmas N, Kaipio J. Physicians as usability evaluators \u0026ndash; first aid for poor EHR usability? Finn J EHealth EWelfare [Internet]. 2018 May 21 [cited 2021 Sep 15];10(2\u0026ndash;3). Available from: https://journal.fi/finjehew/article/view/69162 \u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Clinical Trial Recruitment Support System, Patient Selection , Logical Observation Identifiers Names and Codes, Interoperability, OMOP","lastPublishedDoi":"10.21203/rs.3.rs-2985787/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-2985787/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cb\u003eBackground\u003c/b\u003e\u003c/p\u003e \u003cp\u003eClinical trials are an essential part of the medical research process, but trials are often canceled due to lack of enrollment. Clinical trial recruitment support systems can help increase the number of participants by finding more suitable subjects. The ATLAS software (developed by OHDSI) can support the initiation of a clinical trial by building cohorts of patients who meet certain eligibility criteria. The correct use of medical classification systems for clearly defined inclusion and exclusion criteria in studies is an important pillar of this software.\u003c/p\u003e\u003cp\u003e\u003cb\u003eObjectives\u003c/b\u003e\u003c/p\u003e \u003cp\u003eThe purpose of this study is to determine whether ATLAS can be used in a Clinical Trial Recruitment Support System to represent the eligibility criteria of clinical trials. Therefore, we have considered the set of criteria implemented in ATLAS and realistically identified the strengths and problems of the software.\u003c/p\u003e\u003cp\u003e\u003cb\u003eMethods\u003c/b\u003e\u003c/p\u003e \u003cp\u003eWe analyzed ATLAS using 223 eligibility criteria from 30 randomly selected studies from the past 10 years. Therefore, appropriate ICD, OPS, LOINC, or ATC codes were selected during the data collection process to feed the software. Each criterion and study was ranked for feasibility of implementation in the software.\u003c/p\u003e\u003cp\u003e\u003cb\u003eResults\u003c/b\u003e\u003c/p\u003e \u003cp\u003eAccording to the observations, 49.33% of the analyzed inclusion criteria are fully implemented in ATLAS. In our selected sample set, 10% of the studies are classified as fully portrayable and 73.33% as portrayable. In addition, the software is evaluated with respect to its usability and its interaction with medical classification systems.\u003c/p\u003e\u003cp\u003e\u003cb\u003eConclusions\u003c/b\u003e\u003c/p\u003e \u003cp\u003eTo improve and extend the coverage of criteria in cohort definition in a real-world setting, we recommend working closely with the individuals involved in the study to precisely define the criteria and carefully select terminology systems. The selected criteria will be combined according to the individual setting. Further work is needed to specify the relevance and quantity of the extracted criteria.\u003c/p\u003e","manuscriptTitle":"Implementation of inclusion and exclusion criteria of clinical studies in the software OHDSI ATLAS","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2023-06-01 14:40:47","doi":"10.21203/rs.3.rs-2985787/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"db627f17-7ec8-4b15-9a7b-d39a2c628202","owner":[],"postedDate":"June 1st, 2023","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2023-06-08T05:44:29+00:00","versionOfRecord":[],"versionCreatedAt":"2023-06-01 14:40:47","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-2985787","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-2985787","identity":"rs-2985787","version":["v1"]},"buildId":"7rjqhiLT3MXkJMwkYKINL","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}