GPT-4 based AI agents – the new expert system for detection of antimicrobial resistance mechanisms?

(299): 26

EUCAST recommends a two -step process for beta -lactamases in Gram -27 negative bacteria. Screening with minimal inhibitory concentrations (MICs) or inhibition zone 28 diameters for potential extended spectrum beta-lactamase (ESBL), plasmid-mediated AmpC 29 beta-lactamase, or carbapenemase production is followed by confirmatory tests. GPT -4 and 30 its newly released customized GPT -agent may support the initial EUCAST-screening 31 process. We aimed to validate a customized GPT -agent to identify potential resistance 32 mechanisms. 33 34

We used 225 Gram -negative isolates. Based on phenotypic resistances against 35 beta-lactam antibiotics, we formed four categories: “none”, “ESBL”, “AmpC”, or 36 “carbapenemase”. We included 862 phenotypic categories . Next, w e customized a GPT-37 agent with EUCAST -guidelines, expert rule s, and EUCAST-breakpoint table (v13.1). We 38 compared routine diagnostic outputs (reference) to (i) EUCAST -GPT-expert, (ii) medical 39 microbiologists, and (iii) GPT -4 without customization . We determined performance as 40 sensitivities and specificities to flag suspect resistance mechanisms. 41 42

Three human readers showed concordance in 814/862 (94.4%) phenotypic 43 categories and used in median eight words (IQR 4 -11) for reasoning. Median sensitivity and 44 specificity for ESBL, AmpC, and carbapenemase were 98%/99.1%, 96.8%/97.1%, and 45 95.5%/98.5%, respectively. Three independent prompting rounds of the GPT -agent showed 46 concordance in 706/862 (81.9%) categories but used in median 158 words (IQR 140 -174) 47 for reasoning,. Median sensitivity and specificity for ESBL, AmpC, and carbapenemase 48 prediction were 95.4%/69.23%, 96.9%/86.3%, and 100%/98.8%, respectively. In the non-49 customized GPT -4, 169/862 (19.6 %) categories could be interpreted. Of these 137/169 50 (81.1%) categories agreed with routine diagnostic. The non -customized GPT -4 used in 51 median 85 words (IQR 72-105) for reasoning. 52 53

Human experts showed higher concordance and shorter argumentations 54 compared to GPT -agents. Human experts showed comparable median sensitivities and 55 higher specificities compared to GPT -agents. GPT-agents showed more unspecific flagging 56 of ESBL and AmpC, p otentially, resulting in additional testing, diagnostic delays, and higher 57 costs. GPT-4 and GPT-agents are not IVDR/FDA-approved, but validation of LLMs is critical 58 and datasets for benchmarking are needed. 59 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted May 6, 2024. ; https://doi.org/10.1101/2024.05.06.592800doi: bioRxiv preprint

60 As the healthcare sector grapples with the escalating challenge of antimicrobial resistance 61 (AMR), the need for advanced diagnostic methods increas es (1) . For beta-lactamases in 62 Gram-negative bacteria this is usually a two-step process (2): first , based on screening 63 breakpoints for minimal inhibitory concentrations (MICs) or inhibition zone diameters 64 suspected isolates with potential extended spectrum beta-lactamase (ESBL), plasmid-65 mediated AmpC, and carbapenemase production are flagged; second, suspected resistance 66 is then confirmed with additional tests e.g., molecular assay for specific resistance genes (3). 67 Kirby-Bauer disk diffusion is a commonly used method for determining bacterial susceptibility 68 and exemplifies the complexities of microbiological diagnostics (4, 5). This is particularly true 69 for detection of extended-spectrum b eta-lactamases (ESBLs), AmpC beta-lactamases, and 70 carbapenemases. Precise interpretation within this framework is vital, yet challenged by 71 technical and human variability, and the necessity for continual expert knowledge updating. 72 Reproducibility in reading and interpretation of disk diffusion has been reported to be 73 variable (6-8). 74 In this context, the potential integration of AI technologies, such as generative models like 75 GPT-4, customized GPT-agents (openAI), or other large language models (LLMs) into 76 laboratory medicine is an area of growing interest (9). However, it is crucial to note that AI-77 tools are not routinely established due to the current absence of compliance with In Vitro 78 Diagnostic Regulation (IVDR) and Food and Drug Administration (FDA) regulations (10) . 79 This regulatory gap underscores the importance of validation to ensure their reliability, 80 accuracy, and safety in clinical diagnostics. 81 Our study aimed to contribute to this validation process. By utilizing GPT-4 and a customized 82 GPT-agent to interpret test results. W e aimed to understand how such AI-tools can be 83 calibrated and utilized within the stringent frameworks of laboratory medicine. The study 84 provides an opportunity to gather valuable insights into the integration of AI in diagnostics, 85 for future validation and regulatory approval. This is a critical step in ensuring that AI-tools 86 can be safely and effectively used to enhance patient care in clinical laboratories (9, 11, 12). 87

88 Study Design and Sample Collection. We conducted a retrospective study involving 225 89 Gram-negative isolates from routine diagnostics. Laboratory processes are ISO/IEC 90 accredited. We randomly included four Acinetobacter baumannii, three Citrobacter freundii, 91 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted May 6, 2024. ; https://doi.org/10.1101/2024.05.06.592800doi: bioRxiv preprint two C. koseri , 13 Enterobacter cloacae complex, 132 Escherichia coli , five Klebsiella 92 aerogenes, five K. oxytoca , 40 K. pneumoniae , three Morganella morganii , ten Proteus 93 mirabilis, one P. vulgaris, one Pseudomonas aeruginosa, and six Serratia marcescens. 94 The isolates measured with disk diffusion according to EUCAST -guidelines (13) using 95 antibiotic discs purchased from i2 a (Perols Cedex, France) and Mueller -Hinton agar plates 96 (BD, Franklin Lakes, NJ). The SIRweb/SIRscan system (i2a) was used to measure the 97 inhibition zone diameters (8). The automated SIRweb expert system based on EUCAST -98 guidelines for the detectio n of resistance mechanisms enabled the categorization of four 99 potential phenotypic resistance categories: suspected «none» (n=75) , «ESBL» (n=111) , 100 «AmpC» (n=32), and «carbapenemase» (n=23). A total of 13.8% of isolates were excluded. 101 Main reasons were poor image quality, non-resulting interpretation by GPT -4, or no valid 102 interpretation guideline e.g., in A. baumannii and P. aeruginosa for AmpC. We finally 103 included a total of 862 valid category phenotypes for subsequent analysis. All data can be 104 downloaded at (weblink upon publication). 105 Ethics and data protection. The study focused only on quality assessment of a new 106 diagnostics tool and was not including any patient characteristics. No personal or patient-107 related data was shared with GPT-4 or the customized GPT-agent. The study was approved 108 by the local ethical committee (Req-2023-00752). 109 Development of the Customized GPT-Agent. We generated a customized GPT-agent named 110 “EUCAST-GPT-expert” using the commercial version of ChatGPT (openAI). The specific 111 GPT-4 version used was dated 06/11/2023 to assist in the interpretation of EUCAST 112 antimicrobial susceptibility testing. The GPT-agent was customized through the following 113 steps (Figure 1): 114 1. Knowledge Acquisition. The GPT-agent was equipped with the following documents: 115 latest breakpoint tables (v13.1 , (14)), EUCAST-guidelines for detection of resistance 116 mechanisms and specific resistances of clinical and/or epidemiological importance (v2, 117 July 2017, (3)), and EUCAST expert rules (15, 16). This knowledge base was intended to 118 enable the GPT-agent to understand and apply EUCAST-guidelines accurately. 119 2. Model Refinement. Preliminary testing was performed with some examples to fine-tune its 120 interpretative capabilities. Corrections were made to ensure the GPT -agent did not repeat 121 identifiable errors, such as miss -listing species known to have chromosomal AmpC with 122 common de-repression, e.g., Citrobacter freundii , Enterobacter cloacae , Klebsiella 123 aerogenes, etc. or providing awareness of intrinsic resistances such as ampicillin in 124 Klebsiella spp.. 125 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted May 6, 2024. ; https://doi.org/10.1101/2024.05.06.592800doi: bioRxiv preprint 3. Input Preparation. For each isolate, the input comprised images of disk diffusion plates 126 from SIRscan. Accompanying these images was a table detailing the measured inhibition 127 zones for each antibiotic (Supplementary Figure 1 and 2, Supplementary Table 1). 128 4. Standardized Prompting . The GPT-agent was prompted in a structured manner to 129 interpret the provided data (see below for exact prompt). The same prompt was also used 130 for the non-customized GPT-4 and provided to human experts. Each group was tasked 131 with categorizing the resistance level into one of four categories and to elaborate on its 132 reasoning in a brief argumentative text. 133 5. Output Analysis. The analysis included a detailed output table containing the resistance 134 categories and a section where the GPT-agent provided its argumentation for each 135 categorization. For each bacterial isolate, the agent was prompted to analyse images and 136 generate an output table identifying the resistance mechanisms. The agent was 137 instructed to force a binary (yes/no) decision on the presence of resistance mechanisms 138 and to assess the likelihood of each identified mechanism. 139 Standardized Prompting Procedure. For each sample, the identical queries were used. 140 Example “Sample 6.70.1. Escherichia coli - Make an output table. In that output table identify 141 the resistance mechanisms you have detected from the analysis of the provided images and 142 information: (i) None, (ii) ESBL-production, (iii) AmpC-production, or (iv) Carbapenemase 143 production. Make a specific call with yes/no answers - force yourself to provide an answer. 144 Add into this table a likelihood analysis for each resistance mechanism: (a) very likely, (b) 145 likely, (c) unlikely, (d) very unlikely. For samples with likely or very likely results make a 146 recommendation on the potential confirmation tests which should be used according to 147 EUCAST. In the title of each table mention the sample ID and the bacterial species. Provide 148 a short argumentation for each resistance phenotypes (none, ESBL, AmpC, and 149 Carbapenemase) based on the measurement and image analysis.” 150 Output Analysis and Argument ation. The output table from the EUCAST-GPT-expert 151 included sample identification and resistance mechanism detection, with a likelihood 152 analysis (S upplementary Table 2 ). The GPT-agent was also required to provide a short 153 argumentation for each decision, drawing on the measured inhibition zones and image 154 analysis. This approach aimed to mimic the reasoning process of human experts. 155 Benchmarking and Validation. The outputs of all groups, GPT-4, GPT-agent, and three 156 medical microbiologists were compared against the previous ly reported results in routine 157 diagnostics (reference standard). As there was some variability in the interpretation of the 158 results, we showed the median outputs of the three microbiological experts and performed 159 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted May 6, 2024. ; https://doi.org/10.1101/2024.05.06.592800doi: bioRxiv preprint three independent prompting rounds for the EUCAST -GPT-expert, where also the median 160 outputs were used. The output of the non-customized GPT-4 was poor, and therefore we did 161 not repeat the prompting. We calculated concordance rates for categories identified by 162 human experts and the EUCAST-GPT-expert. We recorded the median number of words 163 used for reasoning. 164 Statistical Analysis. Descriptive statistics were used to summarize the performance of the 165 human experts, the EUCAST-GPT-expert and non-customized GPT-4 . Sensitivity, 166 specificity, and negative and positive predictive value were compar ed. The median 167 sensitivities and specificities for human readers was calculated with interquartile ranges 168 (https://www.medcalc.org) and compared to evaluate the diagnostic accuracy of the AI-169 models. 170 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted May 6, 2024. ; https://doi.org/10.1101/2024.05.06.592800doi: bioRxiv preprint

171 The customized GPT-agent (“EUCAST-GPT-expert”), informed by EUCAST-guidelines and 172 trained on expert rules, analyzed 862 phenotypes from 225 Gram-negative bacterial isolates. 173 When compared to the reference standards, the GPT- agent demonstrated a median 174 sensitivity of 95.4% for suspected ESBL detection, 96.9% for suspected AmpC, and 100% 175 for suspected carbapenemase detection. Specificity, however, varied, with 69.2% for ESBL, 176 86.3% for AmpC, and 98.8% for carbapenemases, indicating a propensity for over-flagging 177 potential resistances, particularly for ESBL and AmpC (Table 1). The GPT-agent's tendency 178 to over-flag, particularly ESBL-producing isolates with amoxicillin resistance and 179 amoxicillin/clavulanic acid susceptibility due to a narrow spectrum beta-lactamase, might 180 lead to unnecessary confirmatory testing and higher costs. Similarly, the over-flagging of 181 AmpC with cefoxitin susceptibility suggests areas for model refinement. Of note, when the 182 GPT-agent was asked to re-analyze and carefully consider the knowledge and rules, then 183 the output was often adapted and corrected. For this analysis, we have however used only 184 the first reply of the GPT-agent. 185 Interestingly, when we focused on individual bacterial species, we observed a potential 186 species-specific effect. As an example, in E. coli (n=132) we noted lower ESBL detection 187 rates compared to all samples (median sensitivity 86.4% vs. 95.4%) , and a lower median 188 sensitivity compared to K. pneumoniae and K. oxytoca (n=45, median sensitivity 100% , 189 Table 2). However, the specificity in ESBL -producing E. coli was higher compared to all 190 samples (median specificity 76.9% vs. 69.2%), and higher compared to K. pneumoniae and 191 K. oxytoca (median specificity 61.9%). This could potential ly be explained by the previously 192 mentioned misinterpretation of ESBL in the case of amoxicillin resistance and 193 amoxicillin/clavulanic acid susceptibility. 194 195 Comparison of AI and Human Performance. Human experts generally exhibited higher 196 specificity, particularly in the detection of ESBL and AmpC phenotypes, compared to the 197 EUCAST-GPT-expert. However, the sensitivity was comparable between the human experts 198 and the AI -agent (Table 1 ). Positive and negative predictive values were in general also 199 higher in human experts compared to EUCAST -GPT-expert (Table 1 ). We also wanted to 200 explore the performance of a customized GPT -agent compared to a non -customized GPT-4 201 prompt. Therefore, we also generated outputs using the same images and prompting 202 strategy but with a non -customized GPT -4. In the non -customized GPT -4, only 169/862 203 (19.6%) categories could be interpreted. Of these 137/169 (81.1%) categories agreed with 204 routine diagnostic. For this subgroup, the available phenotypic categories were too low to 205 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted May 6, 2024. ; https://doi.org/10.1101/2024.05.06.592800doi: bioRxiv preprint provide robust enough calculations for sensitivities and specificities for individual resistance 206 mechanisms. 207 Analysis of Argumentation. The EUCAST-GPT-Expert's argumentation was more detailed 208 than replies from human experts, which could be beneficial for educational purposes or in 209 complex cases where thorough explanations are warranted. However, this verbosity may not 210 be practical in a routine diagnostic setting where brevity is preferred. The human experts 211 used in m edian eight words (IQR 4-11), indicating concise rationales for their decisions. In 212 contrast, the EUCAST-GPT-expert provided more extensive explanations, with a median of 213 158 words (IQR 140-174) and suggested in addition also confirmation steps with a median of 214 five words (IQR 4-9). Although we did not perform an in-depth analysis of the quality , we 215 noted that the EUCAST-GPT-expert provided in some cases correct interpretations of e.g. 216 the phenotypic resistance category, but the argumentation for the interpretation was not 217 correct. 218 EUCAST-GPT-expert provided an additional text to specifically recommend a next 219 confirmational step. In most situations a correct follow-up assay was described, e.g., with a 220 specific PCR, but occasionally it was simply noted that a confirmational assay is necessary. 221 The non -customized GPT -4 used in median 85 words (IQR 72 -105) for reasoning and 0 222 words (IQR 0-0) to suggest confirmation steps. In very few situations the next confirmational 223 steps were properly described. 224 Human Expert and EUCAST-GPT-expert concordance. The three human experts showed in 225 814/862 (94.4%) concordance across all phenotypic categories. Concordance for ESBL, 226 AmpC, and carbapenemases were 94.0%, 96.6%, and 98.6%, respectively. In contrast, 227 three separated runs with the EUCAST-GPT-expert showed in 706/862 (81.9%) phenotypic 228 concordance across the categories. Concordance for ESBL, AmpC, and carbapenemase 229 was 74.4%, 72.3%, and 97.7%, respectively. 230 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted May 6, 2024. ; https://doi.org/10.1101/2024.05.06.592800doi: bioRxiv preprint

231 We highlight the potential and limitations of integrating a customized AI-agent, specifically 232 GPT-4 and GPT-agents , in interpreting antimicrobial susceptibility tests. The GPT-agent’s 233 comparable sensitivity to human experts in detecting ESBL, AmpC beta-lactamases, and 234 carbapenemases is promising and underscores its ability to identify resistance mechanisms 235 effectively, a crucial aspect in combating AMR. However, the lower specificity and over-flag 236 of certain resistances call for a careful approach during integration in clinical practice as it 237 may result in critical delays as well as increased workloads and costs during confirmation. 238 The detailed argumentation provided by the GPT-agent , while beneficial for educational 239 purposes, may need to be streamlined for practical application in busy diagnostic labs. The 240 customized-GPT agent showed a clear benefit over a non-customized version. 241 The variability in interpretation among human experts illustrates the subjective nature of 242 manual readings and the potential for AI to provide more standardized interpretations (8) . On 243 this occasion also different interpretation of AmpC, considering only plasmid-mediated or 244 both plasmid- and chromosomal-mediated resistance. Variability in plate reading and 245 interpretation, and as a matter of fact also human error needs to be considered in any 246 diagnostic test validation. However, the expertise and clinical judgment of human 247 microbiologists remain invaluable, especially in complex or ambiguous cases. Also different 248 LLM models may show variable results as indicated in a comparison of clinical microbiology 249 scenarios between GPT-3.5 and GPT-4 (17). Importantly, GPT-4 and the customized-GPT 250 agent do not provide detailed insights on how the data is analysed and interpreted. These 251 systems remain a black box. Open-source LLMs, such as LLAMA-2, will become very 252 important to explore the technology and understand how AI algorithms work with real-world 253 data (9). 254 Our results demonstrate the importance of ongoing AI refinement, considering both the rapid 255 evolution of AMR patterns and advancements in technology. The study also underscores the 256 necessity of regulatory compliance and validation for AI-tools in healthcare, as highlighted by 257 the lack of IVDR/FDA approval for GPT-4 in clinical diagnostics (9). Thereby, our study may 258 also be seen as a blueprint for a dataset which allows tracking of LLM progress and as a 259 benchmark in silico dataset. Repetitive testing of the same dataset a llows track ing 260 performance evolution of LLMs. 261 Future studies should focus on expanding the dataset to include a wider range of bacterial 262 species and resistance mechanisms. Additionally, exploring AI's role in interpreting other 263 diagnostic tests could provide a more comprehensive understanding of its capabilities and 264 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted May 6, 2024. ; https://doi.org/10.1101/2024.05.06.592800doi: bioRxiv preprint

(11). Collaborations between AI developers, medical microbiologists, and 265 regulatory bodies are essential to ensure that AI-tools are safely and effectively integrated 266 into clinical workflows. 267 Our study has important limitations. It is limited by the specific version of GPT-4 used, as this 268 field rapidly evolves and releases are published on a regular basis, by the time this article is 269 published, the performance has likely improved. Also, we have focused on Gram-negative 270 bacteria and beta-lactamases. Future studies should explore different bacterial species and 271 resistance mechanisms e.g. with methicillin-resistant Staphylococcus aureus or vancomycin- 272 resistant Enterococcus faecium . Moreover, the AI's performance in a real-world clinical 273 setting may differ from this controlled study environment. Prospective trials are needed in the 274 field of AI and laboratory investigations; however, a first step must be a retrospective 275 assessment to ensure its safety and baseline performance. Next, we have only few 276 Acinetobacter baumannii and Pseudomonas aeruginosa isolates included, future work needs 277 a more balanced dataset. Finally, our dataset has only been used in a GPT-4 and GPT-278 agent related context and not been used to explore other LLMs. 279 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted May 6, 2024. ; https://doi.org/10.1101/2024.05.06.592800doi: bioRxiv preprint Acknowledgments: 280 We want to thank the laboratory technical person nel at the Institute of Medical Microbiology 281 at the University of Zurich for processing and measurements of antimicrobial resistance 282 profiles. 283 284 285 Funding: 286 The study has been financed by a grant of the Swiss National Science Foundation (Ref. 287 310030_213019) to AE and an unrestricted grant to AE from the University of Zurich to 288 conduct research in the field of medical microbiology. 289 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted May 6, 2024. ; https://doi.org/10.1101/2024.05.06.592800doi: bioRxiv preprint Figures: 290 Figure 1. Workflow for validation of GPT-4 based generative AI-agent. 291 292 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted May 6, 2024. ; https://doi.org/10.1101/2024.05.06.592800doi: bioRxiv preprint Tables: 293 294 295 Table 1. Sensitivity and specificity of human experts and the customized EUCAST -GPT-296 expert. a, three human experts (median) . b, three independent prompting outputs from the 297 customized GPT -4 agent “EUCAST -GPT-expert”. As reference standard, we used the 298

reported according to our ISO -accredited laboratory information system. ESBL, 299 extended spectrum beta-lactamase; None, no specific molecular resistance mechanism. 300 301 302 303 Table 2. Comparison of common bacterial species and the performance of the EUCAST -304 GPT-expert. Only ESBL was analyzed, as for AmpC and carbapenemase producing bacteria 305 the numbers where too low and not balanced. Outputs of K. pneumoniae and K. oxytoca 306 were pool to generate more robust numbers. 307 308 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted May 6, 2024. ; https://doi.org/10.1101/2024.05.06.592800doi: bioRxiv preprint Supplementary material. 309 310 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted May 6, 2024. ; https://doi.org/10.1101/2024.05.06.592800doi: bioRxiv preprint Supplementary Figure 1. Representative photography of a disk diffusion assay 311 generated by the SIRscan device. The image originates from routine diagnostics and 312 shows measured circles to indicate the inhibition zone diameter in mm and a categorical 313 interpretation according to EUCAST. As representative example image 2.3.1 was used. 314 315 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted May 6, 2024. ; https://doi.org/10.1101/2024.05.06.592800doi: bioRxiv preprint Supplementary Figure 2. Representative photography of a disk diffusion assay 316 generated by the SIRscan device. The image originates from routine diagnostics and 317 shows no circles to indicate the measured inhibition zone diameter. As representative 318 example image 2.3.1 was used. 319 320 321 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted May 6, 2024. ; https://doi.org/10.1101/2024.05.06.592800doi: bioRxiv preprint Supplementary Table 1. A representative isolate (isolate 2.3.1) with measured 322 inhibition zone diameters . The measured inhibition zones were interpreted according to 323 EUCAST. As representative example image 2.3.1 was used. R, resistant; I, susceptible, 324 increased exposure; S, susceptible; N. read, not read. 325 326 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted May 6, 2024. ; https://doi.org/10.1101/2024.05.06.592800doi: bioRxiv preprint Supplementary Table 2 . Representative output table from the EUCAST-GPT-expert. In 327 this example sample 6.70.1: Escherichia coli is shown. 328 Argumentation: 329  None: Based on the disk diffusion readings and adherence to EUCAST guidelines, 330 there is no indication of reduced susceptibility that would suggest the presence of 331 ESBL, AmpC, or carbapenemase resistance mechanisms. The inhibition zones are 332 consistent with a susceptible Escherichia coli profile. 333  ESBL-production: The inhibition zones for cefotaxime and ceftazidime are within 334 the susceptible range without any signs of blunting, making ESBL production 335 unlikely. 336  AmpC-production: No diminished zone diameters for cefoxitin observed, suggesting 337 AmpC production is unlikely. Furthermore, the susceptibility to amoxicillin-clavulanate 338 supports the absence of AmpC. 339  Carbapenemase production: The preserved zone sizes for meropenem and 340 ertapenem indicate that carbapenemase production is very unlikely. There is no 341 phenotypic evidence suggesting carbapenem resistance. 342 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted May 6, 2024. ; https://doi.org/10.1101/2024.05.06.592800doi: bioRxiv preprint

343 1. Shanmugakani RK, Srinivasan B, Glesby MJ, Westblade LF, Cardenas WB, Raj T, et al. 344 Current state of the art in rapid diagnostics for antimicrobial resistance. Lab Chip. 345 2020;20(15):2607-25. 346 2. Pereckaite L, Tatarunas V, Giedraitiene A. Current antimicrobial susceptibility testing 347 for beta-lactamase-producing Enterobacteriaceae in clinical settings. J Microbiol Methods. 348 2018;152:154-64. 349 3. EUCAST. EUCAST guidelines for detection of resistance mechanisms and specific 350 resistances of clinical and/or epidemiological importance, v2.01. 2017 July 2017. 351 4. Pascucci M, Royer G, Adamek J, Asmar MA, Aristizabal D, Blanche L, et al. AI -based 352 mobile application to fight antibiotic resistance. Nat Commun. 2021;12(1):1173. 353 5. Rajasekar SJS, Thiyagarajan S, Mohamed Ali S. Ab.ai - A Novel Automated AI Tool for 354 Reporting Antibiograms. Stud Health Technol Inform. 2022;299:202-7. 355 6. Ballestero-Tellez M, Jimenez -Morgades E, Arjona -Camacho P , Blanco -Suarez A, 356 Padilla-Esteba E, Perez-Jove J. Inter-technique variability between antimicrobial susceptibility 357 testing methods affects clinical classification of cefuroxime in strains close to breakpoint. Clin 358 Microbiol Infect. 2020;26(5):648 e1- e3. 359 7. Hombach M, Ochoa C, Maurer FP , Pfiffner T, Bottger EC, Furrer R. Relative 360 contribution of biological variation and technical variables to zone diameter variations of 361 disc diffusion susceptibility testing. J Antimicrob Chemother. 2016;71(1):141-51. 362 8. Hombach M, Zbinden R, Bottger EC. Standardisation of disk diffusion results for 363 antibiotic susceptibility testing using the sirscan automated zone reader. BMC Microbiol. 364 2013;13:225. 365 9. Egli A. ChatGPT, GPT -4, and Other Large Language Models: The Next Revolution for 366 Clinical Microbiology? Clin Infect Dis. 2023;77(9):1322-8. 367 10. Saenz AD, Harned Z, Banerjee O, Abramoff MD, Rajpurkar P . Autonomous AI systems 368 in the face of liability, regulations and costs. NPJ Digit Med. 2023;6(1):185. 369 11. Burns BL, Rhoads DD, Misra A. The Use of Machine Learning for Image Analysis 370 Artificial Intelligence in Clinical Microbiology. J Clin Microbiol. 2023;61(9):e0233621. 371 12. Egli A, Schrenzel J, Greub G. Digital microbiology. Clin Microbiol Infect. 372 2020;26(10):1324-31. 373 13. EUCAST. Antimicrobial susceptibility testing - EUCAST disk diffusion method. 2023 Jan 374 2023. 375 14. EUCAST. The European Committee on Antimicrobial Susceptibility Testing. Breakpoint 376 tables for interpretation of MICs and zone diameters. In: v_13.1_breatpoint_tables.pdf, 377 editor. 13.1 ed. http://www.eucast.org: European Society for Clinical Microbiology and 378 Infectious Diseases; 2023. 379 15. EUCAST. Expert Rules: Intrinsic Resistance and Unusual Phenotypes Tables v3.2. In: 380 Intrinsic_Resistance_and_Unusual_Phenotypes_Tables_v3.2_20200225.pdf, editor. v3.2. ed: 381 European Society of Clinical Microbiology and Infectious Diseases; 2020. 382 16. EUCAST. Expert Rules: Enterobacterales v3.2.; 2023 23.01.2023. 383 17. Sallam M, Al -Salahat K, Al -Ajlouni E. ChatGPT Performance in Diagnostic Clinical 384 Microbiology Laboratory-Oriented Case Scenarios. Cureus. 2023;15(12):e50629. 385 386 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted May 6, 2024. ; https://doi.org/10.1101/2024.05.06.592800doi: bioRxiv preprint Step 1: Generation of a GPT powered generative AI agent. Step 2: Acquisition of knowledge. Using document from EUCAST.org - EUCAST breakpoint table v13.1 - Expert rules Step 3: Within model testing. Checked with few examples. Improving rules for obvious mistakes e.g. list of species with chromosomal AmpC. + same image without measurement circles. + Table with measured inhibition zones Step 4: Input for prompt. Step 5: Standardized prompting. Ask for: (i) Interpretation of image and table; (ii) Provide output table with 4 categories: ÒNoneÓ, ÒESBLÓ, ÒAmpCÓ, and ÒCarbapenemaseÓ; (iii) recommended confirmation; (iv) and short argumentation text. Step 5: Output analysis Output table Argumentation Step 6: Calculation of output performance e.g., sensitivity and specificity. .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted May 6, 2024. ; https://doi.org/10.1101/2024.05.06.592800doi: bioRxiv preprint Human expertsa EUCAST-GPT-exp ertb ESBL Sensitivity 98.0% (91.8 – 100) 95.4% (94.5 – 96.3) Specificity 99.1% (97.1 – 100) 69.2% (63.8 – 85.7) PPV 99.0% (97.4 – 100) 76.3% (73.4 – 87.3) NPV 98.1% (92.0 – 100) 93.8% (93.5 – 94.4) AmpC Sensitivity 96.8% (93.3 – 100) 96.9% (87.5 – 96.9) Specificity 97.1% (95.9 – 97.7) 86.3% (84.1 – 91.8) PPV 84.9% (81.6 – 88.2) 54.9% (53.5 – 68.9 NPV 99.4% (98.8 – 100) 99.3% (97.3 – 99.4) Carbapenemases Sensitivity 95.5% (90.9 – 100) 100% (90 .9 – 100) Specificity 98.5% (98.5 – 98.5) 98.8% (98.8 – 98.8) PPV 88.0% (87.0 – 91.3) 91.7% (90.9 – 91.7) NPV 99.5% (99.0 – 100) 98.8% (98.8 – 100) T able 1. Sensitivity and specificity of human experts and the customized EUCAST -GPT- expert. a, three human experts (median). b, three independent prompting outputs from the customized GPT -4 agent <EUCAST -GPT-expert=. As reference standard, we used the

reported according to our ISO -accredited laboratory information system. ESBL, extended spectrum beta-lactamase; None, no specific molecular resistance mechanism. .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted May 6, 2024. ; https://doi.org/10.1101/2024.05.06.592800doi: bioRxiv preprint Escherichia coli Klebsiella pneumoniae and K. o xytoca ESBL Sensitivity 86.4% (83.7 - 92.7) 100% (100 - 100) Specificity 76.9% (71.2 - 95.8) 61.9% (52.4 - 76.2) PPV 86.4% (83.7 - 97.4) 71.4% (65.5 - 80.0) NPV 76.9% (71.2 - 88.5) 100% (100 - 100) T able 2. Comparison of common bacterial species and the performance of the EUCAST - GPT-expert. Only ESBL was analyzed, as for AmpC and carbapenemase producing bacteria the numbers where too low and not balanced. Outputs of K. pneumoniae and K. oxytoca were pool to generate more robust numbers. .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted May 6, 2024. ; https://doi.org/10.1101/2024.05.06.592800doi: bioRxiv preprint .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted May 6, 2024. ; https://doi.org/10.1101/2024.05.06.592800doi: bioRxiv preprint .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted May 6, 2024. ; https://doi.org/10.1101/2024.05.06.592800doi: bioRxiv preprint Abbrevia琀椀on An琀椀bio琀椀c Inhibi琀椀on z one (mm) Interpreta琀椀on (S/I/R) Resistant (mm) Suscep琀椀ble (mm) CPD Cefpodoxime 6 R <21 ≥21 AMC Amoxicillin/Clavulanic acid 16 R <16 ≥16 CRO Ce昀琀riaxone 6 R <22 ≥25 CIP Cipro昀氀oxacin 23 I <22 ≥25 FOX Cefoxi琀椀n 22 S <19 ≥19 CAZ Ce昀琀azidime 10 R <19 ≥22 TPZ Piperacillin/Tazobactam 19 R <20 ≥20 SXT Sufamethoxazol-T rimethoprim 6 R <11 ≥14 MEM Meropenem 30 S <16 ≥22 CN Gentamicin (10ug) 23 S <17 ≥17 F100 Nitrofurantoin (100ug) 15 S <11 ≥11 ETP Ertapenem 29 S <25 ≥25 FF Fosfomycin 17 N. read <24 ≥24 PEF Pe昀氀oxacin 13 R <24 ≥24 FEP Cefepime 18 R <24 ≥27 AM10 Ampicillin (10ug) 6 R <14 ≥14 Suppl ementary Table 1. A representative isolate (isolate 2.3.1) with measured inhibition zone diameters. The measured inhibition zones were interpreted according to EUCAST. As representative example image 2.3.1 was used. R, resistant; I, susceptible, increased exposure; S, susceptible; N. read, not read. .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted May 6, 2024. ; https://doi.org/10.1101/2024.05.06.592800doi: bioRxiv preprint