⚙
AI-generated deep summary
by claude@2026-07, 2026-07-05
· read from full text
ⓘ
This study evaluated whether generative AI models can help users interact via natural language with the WHO 2023 Mutation Catalogue for Mycobacterium tuberculosis, including mutation search/retrieval across full and antibiotic-specific tables and grading rules for novel mutations. Four models (Google Gemini 2.5 Pro, OpenAI ChatGPT 4.1, Perplexity AI, and DeepSeek R1) were tested, with performance assessed for accuracy, completeness, clarity, source citation, and hallucinations. Gemini 2.5 Pro performed best overall in accuracy, completeness, and hallucination avoidance, while DeepSeek R1 was strongest for applying grading rules but still showed some hallucinations; ChatGPT 4.1 lacked proper citations and Perplexity AI had variable performance and more hallucinations. This paper does not explicitly discuss endometriosis or adenomyosis; it was included in the corpus via a keyword match in the upstream search index.
Abstract
ABSTRACT The World Health Organization (WHO) 2023 Mutation Catalogue for Mycobacterium tuberculosis is a crucial knowledgebase and tool for clinical interpretation of mutations associated with drug-resistant TB. However, the document’s complexity and size pose challenges for many users. This study evaluated the potential of generative artificial intelligence (AI) models to facilitate natural language user interaction with the catalogue. Four prominent AI models—Google Gemini 2.5 Pro, OpenAI ChatGPT 4.1, Perplexity AI, and DeepSeek R1—were assessed through general test questions, mutation search and retrieval tasks using both full catalogue queries and antibiotic-specific tables, and the application of additional grading rules to score novel mutations. Performance was measured based on accuracy, completeness, clarity, source citation, and the presence of hallucinations. Google Gemini 2.5 Pro consistently demonstrated superior performance in accuracy, completeness, and avoidance of hallucinations across most evaluations, especially in general queries and large dataset searches. DeepSeek R1 excelled in applying grading rules to novel mutations and showed high accuracy in focused datasets, but exhibited some hallucinations. ChatGPT 4.1 was strong in clarity but lacked proper citations, and Perplexity AI showed variable performance with a higher frequency of hallucinations. The findings highlight the potential of AI tools to enhance accessibility and utility of complex knowledgebases like the WHO Mutation Catalogue, while emphasizing the critical need for careful model selection and rigorous benchmarking to ensure accuracy and reliability. The results suggest that Google Gemini 2.5 Pro and DeepSeek R1 are strong candidates for developing a custom clinical AI agent to assist all levels of healthcare users in navigating and interpreting the complex information within the catalogue, ultimately contributing to improved TB control efforts.
Full text
2,041 characters
· extracted from
oa-doi-fallback
· click to expand
ABSTRACT
The World Health Organization (WHO) 2023 Mutation Catalogue for Mycobacterium tuberculosis is a crucial knowledgebase and tool for clinical interpretation of mutations associated with drug-resistant TB. However, the document’s complexity and size pose challenges for many users. This study evaluated the potential of generative artificial intelligence (AI) models to facilitate natural language user interaction with the catalogue. Four prominent AI models—Google Gemini 2.5 Pro, OpenAI ChatGPT 4.1, Perplexity AI, and DeepSeek R1—were assessed through general test questions, mutation search and retrieval tasks using both full catalogue queries and antibiotic-specific tables, and the application of additional grading rules to score novel mutations. Performance was measured based on accuracy, completeness, clarity, source citation, and the presence of hallucinations. Google Gemini 2.5 Pro consistently demonstrated superior performance in accuracy, completeness, and avoidance of hallucinations across most evaluations, especially in general queries and large dataset searches. DeepSeek R1 excelled in applying grading rules to novel mutations and showed high accuracy in focused datasets, but exhibited some hallucinations. ChatGPT 4.1 was strong in clarity but lacked proper citations, and Perplexity AI showed variable performance with a higher frequency of hallucinations. The findings highlight the potential of AI tools to enhance accessibility and utility of complex knowledgebases like the WHO Mutation Catalogue, while emphasizing the critical need for careful model selection and rigorous benchmarking to ensure accuracy and reliability. The results suggest that Google Gemini 2.5 Pro and DeepSeek R1 are strong candidates for developing a custom clinical AI agent to assist all levels of healthcare users in navigating and interpreting the complex information within the catalogue, ultimately contributing to improved TB control efforts.
Competing Interest Statement
The authors have declared no competing interest.
Text is read by the "Ask this paper" AI Q&A widget below.
Extraction quality varies by source — PMC NXML preserves structure
cleanly, OA-HTML may include some navigation residue, and OA-PDF can
have broken hyphenation. The publisher copy
(via DOI)
is the canonical version.