A Domain-specific Large Language Model for Diabetes Care and Management

A Domain-specific Large Language Model for Diabetes Care and Management | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article A Domain-specific Large Language Model for Diabetes Care and Management Gift Zandile Silinda, Patrick Ngassa Piotie, Olawande Daramola This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9563605/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract The prevalence of diabetes is rapidly increasing in low- and middle-income countries (LMIC), making it one of the fastest-growing global health emergencies of the modern era. Despite efforts by healthcare practitioners, the government, and communities to minimise the associated complications and mortality, there are significant challenges that can be potentially alleviated through innovative digital health technologies. This study developed a domain-specific large language model (DS-LLM) aimed at improving diabetes care and management using a case study of South Africa. To achieve this, local data was collected and supplemented with benchmark and medical Hugging Face datasets. Medical pre-trained large language models (LLMs): BioMedLM (2.7B) and BioMistral-7B were selected as base models, along with Qwen3-8B (a non-specialised LLM). Two Parameter-Efficient Fine-Tuning (PEFT) techniques: prompt tuning and Quantised Low-Rank Adaptation (QLoRA) were applied, with Retrieval-Augmented Generation (RAG) applied on the best-performing LLM. The fine-tuned LLMs were evaluated by comparing their respective performance with Diabetica-7B , a specialised diabetes LLM. The final dataset comprised 18,079 processed question-answer pairs (14% artificially generated for the South African context) and 1,596 documents, covering medication, management, diagnosis, screening, and general diabetes topics that pertain to South Africa. For fill-in-the-blank and multiple-choice questions formats, Qwen QLoRA outperformed all LLMs (ROUGE-1 = 0.793, ROUGE-L= 0.792, and BERTScore F1 = 0.940). Diabetica had the highest BLEU score (0.465), while Qwen3-8B had 0.365. For multiple-choice questions only, Qwen3-8B QLoRA achieved a top accuracy of 80.7%. For short and long answers, BioMistral-7B QLoRA performed slightly better, with all models scoring above 0.800. These findings highlight the promising use of LLMs for diabetes care. Diabetes care and management Domain-Specific LLM Fine-tuning South Africa Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 1. Introduction The prevalence of diabetes mellitus (diabetes) is rapidly increasing, especially in low- and middle-income countries (LMICs), making it one of the fastest-growing global health emergencies of the modern era. From 2024 to 2050, the global prevalence of diabetes is expected to increase by 45%, from 590 million to 853 million cases [1]. In 2024, a majority (81%) of people living with diabetes were from LMICs, which also have high proportions of undiagnosed cases. The increase in diabetes cases, and high percentage of underdiagnosis of diabetes contribute to increased risks of complications such as cardiovascular disease, dementia, eye disorders, mortality, and escalating healthcare costs. Over the past 17 years, global diabetes-related spending has increased by 338% [1]. In Africa, the increase is expected to be even higher, with a 142% increase from 24 million to 60 million cases [1]. This increase in diabetes cases is attributed to economic, socio-cultural, demographic (urbanisation), environmental, and genetic factors, especially for Type 2 diabetes mellitus (T2D), which accounts for 95% of the total diabetes cases. In 2024, African people made up 19% of the world’s population, and 4% of Africa’s population was living with diabetes. Despite the sizeable number of African cases, only 1% of the total global diabetes-related expenditure was for African people, leaving 73% of adults with diabetes undiagnosed. Africa, however, faces challenges in addressing diabetes prevalence, including diagnostics, education, low financial investment, healthcare practitioners’ training and access to medication. South Africa is among the top 5 countries in Africa with the highest number of people living with diabetes [1, 2, 3]. In 2024, 6.1% of adult South Africans, aged between 20 and 79, were living with diabetes [1]. The number of diabetes cases is expected to increase by 74%, from 2.3 million to 3.9 million cases, by 2050 [1]. The total diabetes-related expenditure was 4.6 billion USD and is expected to increase by 37% in 2050 [1, 4]. Despite having the highest diabetes-related expenditure in Africa in 2024, 50% of people living with diabetes in South Africa remained unaware or undiagnosed, putting them at risk for developing complications and increased mortality [1]. Moreover, in 2021, the ratio of doctors to patients was 0.8 per 1000 individuals, lower than similar middle-income countries at 1.4, and significantly lower than the estimate for adequate coverage set by the World Health Organisation (WHO) of at least 2.5 medical practitioners per 1000 individuals [5, 6]. This is an indication of low coverage and care limitation, impacting the time an individual can spend with a medical practitioner or can access a doctor. Currently, a majority (84%) of the South African population relies on public healthcare for medical needs, as they cannot afford costly private healthcare [7]. However, 70% of the country’s doctors are employed by private healthcare [8]. One of the risk factors for diabetes is urbanisation and urban poverty [2, 9]. This is a relevant challenge for South Africa, as a majority (66%) of South Africans reside in urban areas [10] and rely on public health for medical needs due to affordability. South Africa’s Department of Health has set goals to minimise the impact of diabetes and assist in managing and preventing diabetes at the primary care level. The goals are [9]: educating patients, families, and healthcare practitioners (including nonconventional ones); ensuring early detection through effective screening; providing appropriate treatment and management plans; promoting self-monitoring to improve patient outcomes; assessing the effectiveness of services; and maintaining good medical records and history for patients. However, there are challenges in achieving these goals, such as resistance to lifestyle changes, attitudes and beliefs, social and economic issues, limited consultation duration with healthcare providers, and complex treatment regimens [3, 9]. Diabetes in South Africa continues to constrain the public healthcare system and the country’s primary healthcare goals. This presents an opportunity to explore innovative digital health solutions that can improve diabetes care and management. Large Language Models (LLMs) are deep learning models trained on a large amount of unlabelled data, harbours a large amount of information and knowledge. They possess natural language processing capabilities such as text generation, language understanding, question-answering, and text summarization [11]. Recently, LLMs are being increasingly used in the healthcare domain. Examples include using LLM for continuous glucose monitoring [12], as an educational tool for ordinary people [13, 14], personalised care and self-management, medical question-answering, electronic health record summarisation, and clinical decision support [13, 15, 16, 17]. Thus, the application of LLMs to diabetes care and management may enhance education, symptom screening, diagnostic support, treatment adherence, and self-management (diet and exercise) [16, 18]. However, so far there is limited research on the use of context-specific datasets for fine-tuning LLMs for diabetes, particularly in addressing the unique challenges faced by patients, caregivers, and healthcare practitioners. Also, instances of a solution approach that combines patient care and management to aid both medical practitioners and patients are limited. Thus, the unique contributions of this study are the following: i. Curation of South African-specific diabetes dataset for training and fine-tuning of LLMs; ii. Comparative evaluation of medical LLMs and a general LLM; iii. Comparative evaluation of two Parameter-Efficient Fine-Tuning (PEFT) techniques on medical LLMs; iv. Application of Retrieval-Augmented Generation (RAG) on the best-performing LLM to improve factual accuracy and reduce the tendency for hallucination; and v. Development of a domain-specific LLM for diabetes care and management in South Africa with a focus on providing assistance to both medical practitioners and patients, which is new to the best of our knowledge. The remaining part of this paper is structured as follows. Section 2 provides the theoretical background and related work. The adopted research methodology is detailed in Section 3, while Section 4 presents a comprehensive overview of the results from our experiments. Section 5 discusses the findings, while the paper is concluded in Section 6 with a summary and an outlook of future work. 2. Background and Related Work This section presents theoretical background on key topics, and a review of related work. 2.1 Theoretical Background Diabetes Care and Management Diabetes mellitus is a chronic disease characterised by prolonged periods of high blood sugar levels. The most common types are Type 1 Diabetes Mellitus and Type 2 Diabetes Mellitus, with gestational diabetes, ketosis-prone Type 2 diabetes, and monogenic diabetes being less common [2, 3]. Type 1 diabetes is an autoimmune disease that results from limited or no production of insulin by the pancreas. People living with type 1 diabetes depend on lifelong insulin medication to supplement their deficiency. This type is often referred to as juvenile- or childhood-onset diabetes because it is usually diagnosed in the early years. On the other hand, type 2 diabetes occurs when the body uses insulin inefficiently or ineffectively. While adopting a healthy lifestyle can delay or prevent the onset of type 2 diabetes can be delayed with a healthy lifestyle, it can develop at any age [2, 3, 9] Proper diabetes care and management are essential for living a healthier life and reducing complications and risk factors such as premature death, blindness, kidney failure, cardiovascular diseases and amputation, particularly for type 2 diabetes [2, 3]. Diabetes care is multidisciplinary, involving a care team that works collaboratively with the person living with diabetes to enable effective self-management. Part of the care team, healthcare professionals provide medical examinations, diagnoses, prevention/treatment plans, advice, and education. Caregivers and family members together proactively provide non-medical support and advice. Governments and states design policies, regulations and guidelines to promote and ensure proper care. For the person living with diabetes, self-management includes day-to-day activities such as continuous glucose level monitoring, treatment adherence, and adhering to a diet and exercise plan [19]. Large Language Models Large Language Models (LLMs) are deep learning models that are trained to perform cognitive generative tasks through self-supervised or semi-supervised learning techniques. These models are trained on a large amount of unlabelled data, mostly from open-source data sources with limited focus on specialisation. The models have a large amount of information and knowledge that can be queried using natural language [11, 20]. LLMs have shown state-of-the-art performance in understanding context, grammatical structures, and semantic relationships to generate new content, such as text, images, audio and videos [19]. LLMs are currently available as either commercial (closed-source) products or open-source for public, business and academic usage. Depending on performance (including computing requirements, purpose, cost, and availability) [19], a suitable LLM can be selected for use. In general, closed-source models tend to outperform open-source models on the massive multitask language understanding benchmark [20]. However, for complex medical diagnostics, when comparing closed-source (GPT-4 – Generative Pre-trained Transformer 4, by OpenAI) and open-source (LLaMA 3.1 – Large Language Model Meta AI 3.1, by Meta), LLMs showed relatively similar performance on limited diagnostics cases based on a scoring evaluation method [21]. Fine-tuning Large Language Models Generic pre-trained LLMs may not perform optimally for domain-specific tasks, such as diabetes care and management. The pre-trained LLMs are trained on a large amount of unlabelled, diverse and raw data, which incurs an exorbitant amount of cost to understand general language structures. As a result, the generic LLMs are well-suited for handling general queries but may provide generic responses for specialised domains for tasks such as summarisation, text classification, sentiment analysis, and question-answering [22, 23]. Fine-tuning can then enhance the performance of pre-trained LLMs by using customised data to adapt them for specific downstream tasks, often at low computational cost [20, 22]. To fine-tune or adapt a pre-trained LLM, Retrieval-Augmented Generation (RAG) [22], conventional fine-tuning, and Parameter-efficient Fine-Tuning (PEFT) techniques can be used [13]. The most common PEFT methods are Prefix Tuning, P-Tuning, Soft and Hard Prompt Tuning, Adapter Methods, Low-Rank Adaptation (LoRA), and Quantised Low-Rank Adaptation (QLoRA) [19, 20, 22]. 2.2 Related Work Scholars have identified the potential of using LLMs, either alone or in combination with other AI techniques, such as machine learning and deep learning, in the prevention, care, and management of diabetes. These techniques have shown potential in improving patient outcomes and reducing complications and the burden on healthcare systems. Data such as renal images, electronic medical records, clinical metadata, wearable device data, MRI scans, and metabolic markers can be used to fine-tune general-purpose LLMs to perform medical tasks [24]. These tasks include providing support, coaching, and educating healthcare practitioners, communities, and patients. However, these potentials are not without limitations or risks. Some of the limitations include the quality and reliability of training data, lack of external validation beyond the research teams, and various ethical challenges, such as privacy, accountability, bias, transparency, trustworthiness, and consent [24, 25, 26]. Additionally, there are financial costs associated with potential medical errors [26]. Addressing these limitations will require multidisciplinary approaches and regulations to mitigate the associated risks LLMs have been practically employed as recommender systems [27] for continuous glucose monitoring [12], as potential educators [13] and to educate ordinary people [14], personalised care and self-management, medical question-answering, electronic health record summarisation, and clinical decision support [13, 15, 16, 17]. LLMs have also been evaluated in answering medical queries [28] using open- and closed-source models, showing promising application results. Previous studies [13, 15, 17, 29], have primarily concentrated on clinical activities and the fine-tuning of generic LLMs, rather than medical-specific models that possess foundational medical knowledge. Most of these studies have focused on clinical support provided by care teams, with less attention given to patient self-management. Other studies, such as [12, 16, 18], have explored the daily prevention and management of diabetes, with an emphasis on self-management and preventative care, and have shown promising outcomes. However, many of these studies have been conducted outside South Africa, raising questions about the relevance of certain types of support for the South African context. Across these studies, a consistent finding has been that the quality of fine-tuning data, which should provide relevant and context-specific information, is critical to achieving improved model performance. The majority of work to date has focused on either clinical support or patient self-management using generic LLMs that have only limited awareness of medical specialisation. In response to this gap, this study aimed to fine-tune a medically specialised LLM using curated South African data. The aim is to create a model capable of educating, providing diagnostic support, offering guidance on diet and exercise, and screening symptoms for referral, thereby integrating both care and management aspects of diabetes within the South African context. Table 2.1 : Summary of Related work Study Objective Model(s) Technique(s) Setting Data Source(s) Key Observation [13] Generate clinical summaries from time-series glucose monitoring data for patients and clinicians GPT-4 (Data Analyst plugin) Prompt-based approach; qualitative and quantitative evaluation Not specified Synthetic glucose monitoring data Use of synthetic data limits real-world applicability [14] Deliver diabetes and foot care education tailored to individuals with lower health literacy GPT-4 Retrieval-Augmented Generation (RAG) with vector database; qualitative evaluation Not specified Scientific articles and NIH diabetes self-management guidelines Limited evaluation of clinical accuracy and real-world use [15] Support diabetes care tasks including personalised consultation, education, and medical record summarisation Qwen2 LoRA; qualitative and quantitative evaluation China Public MCQs, clinical guidelines, textbooks, drug labels, patient records Context-specific to China; limited generalisability to other settings [16] Support diabetes prevention through health tracking, resource access, and communication tools GPT-3.5 RAG with FAISS Not specified Web-based sources, WHO and national health information Reliance on curated web data; limited clinical validation [17] Recommend diabetes care solutions including screening, referral, and personalised recommendations LLaMA + deep learning (image-based) Multimodal approach; qualitative evaluation China Retinal images, electronic medical records Focus on clinical decision support; limited patient-facing applications [18] Support dietary monitoring and diabetes self-management GPT-3.5 (OpenAI-based chatbot) Prompt-based interaction; qualitative evaluation USA Dietary guidelines and nutrition databases Focus limited to diet; lacks broader care integration [28] Evaluate performance of multiple LLMs on diabetes-related primary care queries GPT-4, GPT-3.5, Bard, LLaMA variants Benchmark evaluation of accuracy China Examination datasets (Chinese and UK diabetes exams) Focus on exam-style queries; may not reflect real-world use [29] Provide personalised treatment strategies and laboratory recommendations for outpatients ChatGLM P-tuning, LoRA; qualitative and quantitative evaluation China Hospital big data platform (de-identified medical records, diagnoses, complaints) Single-institution dataset; limited external validation This study Develop an integrated LLM to support both clinical decision-making and patient self-management in diabetes care BioMedLM (2.7B), BioMistral-7B, Qwen3-8B Fine-tuning (PEFT), RAG; quantitative evaluation South Africa South African guidelines, policy documents, drug labels, FAQs, and curated local dataset Addresses contextual gap by incorporating locally relevant data and dual user focus 3. Methodology This study adopted an experimental research design. The experimental research process that consists of five phases is shown in Fig. 1 . Phase 1 focused on defining the tasks the LLM is expected to perform after successful training. Phase 2 involved data preparation, comprising sub-phases for data collection, curation, augmentation, preprocessing, and splitting. Phase 3 entailed identifying and selecting suitable pre-trained base LLMs. Phase 4 was fine-tuning. The final phase, Phase 5, entailed comparative model evaluation. Phase 1 focused on defining the tasks the LLM is expected to perform after successful training. Phase 2 was data preparation, which included sub-phases concerned with preparing the data for modelling, including data collection, curation, augmentation, pre-processing and splitting. Phase 3 involves identifying suitable and selecting pre-trained base LLMs. Phase 4 was where fine-tuning was done. Phase 5 focussed on comparative evaluation of the trained models. 3.1 Define Requirements Based on the studies reported in [ 1 , 2 , 3 ] and the goals and challenges highlighted by the South Africa’s Department of Health [ 9 ], the fine-tuned LLM is expected to generate text or answer questions to educate, screen symptoms for referral, support diagnosis, summarise or explain treatment for adherence, and self-management (diet and exercise), with a focus on South Africa. Thereby supporting both healthcare practitioners and patients in the care and management of diabetes. 3.2 Data Preparation Data preparation included the collection of relevant data from local sources and South Africa-relevant international sources, curation, augmentation, pre-processing, and splitting for training and evaluation. To ensure the collection of relevant and accurate data for diabetes care and management, keywords and phrases that were identified from literature and reviewed by a medical professional to provide a holistic view, covering synonyms/related terms, diagnosis, treatment, management and terms that could be mistaken for diabetes, referred to as negative matches. These keywords were used to collect and screen data, forming a filtering layer specific to diabetes care and management data. Datasets specific to the South African context were collected from reliable government and non-profit organisation websites, such as the Department of Health, the South African Health Products Regulatory Authority (SAHPRA), and the World Health Organisation (WHO), using web scraping and by downloading available PDFs. The focus was on guidelines, factsheets, drug labels, frequently asked questions (FAQs), and other related formats to create the South African contextual custom data. The collected custom data, consisting of 1,596 documents. The data was then used to extract knowledge and generate question-answer pairs by prompting OpenAI GPT-5-nano to generate short/long answer questions and fill-in-the-blank pairs. Relevant data to supplement the South African contextual custom data from Hugging Face [ 20 ] and referenced in LLMXplorer [ 11 ] was sourced. The focus was on data in question-answer pairs, medical data, and applying the keywords and phrases to filter through the data. The final processed dataset (see Table 1 ), consist of 18,079 question-answer pairs in multiple formats (MCQs, fill-in-the-blanks, short and long answers), covering all topics related to diabetes, medication, diagnosis, and management, with questions representing the expected topics, keywords, or phrases associated with diabetes care and management. Table 1 Data source relevant question-answer pairs count Data Source Relevant QA pairs PubMedQA [ 35 ] 9,130 South African curated artificially generated question-answer pairs 2,507 MedMCQA [ 36 ] 2,204 MedQA (USMLE) [ 34 ] 1,390 medical-o1-reasoning-SFT 1,300 Health Care Facts 1,072 Massive Multitask Language Understanding (MMLU) [ 37 ] 455 Stanford Question Answering Data 2.0 (SQuAD) [38] 12 AfriMed-QA [ 33 ] 9 Total 18,079 3.3 Selecting base pre-trained LLMs Medical-specialised pre-trained LLMs were identified from the literature to select two base models. The LLMs were then evaluated by considering size, purpose, availability, and cost [ 11 ]. The encoder-only architecture LLMs, which focus on understanding context but offer limited capacity for text generation [ 20 ], a critical requirement for this study were excluded. BioMistral [ 30 ], Meditron [ 31 ], BioGPT [ 32 ], BioMedLM[ 33 ] and BiomedGPT [ 34 ], including decoder-only and encoder-decoder architectures, were evaluated for the study. These LLMs’ architecture makes them suitable for the task. Encoder-decoder and decoder-only LLMs are flexible at text generation, performing well in tasks such as completing sentences [ 20 ], open-ended discussions [ 35 ], summarisation [ 35 ], and translation [ 20 ]. The majority of these decoder-only LLMs were trained on PubMed abstracts and papers, with Meditron also leveraging clinical guidelines. Notably, the encoder-decoder model BiomedGPT, despite being smaller in size, offers the unique advantage of training on both rich imagery data and diverse text sources such as publications, electronic hospital records, literature, and clinical notes. This makes BiomedGPT particularly advantageous for visual question answering tasks. When performance was evaluated on benchmark data (MMLU, MedQA, PubMedQA and MedMCQA) on downstream activities, by the authors in [ 30 – 34 ], it was noted that BioMedLM and BioMistral outperformed all the other LLMs on average. Based on benchmark performance, task relevance, open-source accessibility, and computational feasibility, BioMistral-7B and BioMedLM (PubMedGPT) were selected as the base LLMs. BioMistral-7B demonstrates strong performance in three-shot learning across multiple medical QA benchmark datasets, while BioMedLM, despite its smaller size (2.7B parameters), has shown competitive or superior results when fine-tuned, making it highly viable under constrained computational resources. Qwen3-8B was also used in this study, as it is an open-source LLM that has shown promising results in the application of diabetes care and management, according to the authors in [ 15 ] 3.4 Fine-tuning LLM The fine-tuning pipeline shown in Fig. 2 was applied to fine-tune two medical LLMs (BioMistral-7B and BioMedLM) as well as a general LLM (Qwen3-8B). The environment was configured on Google Colab using a single Graphics Processing Unit (GPU), specifically the NVIDIA A100-SXM4-40GB, with Python version 3.12.12. The necessary libraries were installed and imported (including Python Libraries such as accelerate:1.11.0 ; peft:0.17.1 ; trl:0.24.0 ; Bitsandbytes:0.48.2 ; pandas:2.2.2 ; transformers:4.57.1 ; evaluate:0.4.6 ; datasets:4.0.0 ; sacrebleu:2.5.1 ; rouge-score:0.1.2 ; bert-score:0.3.1 ). The selected pre-trained LLMs were downloaded from Hugging Face, saved to Google Drive for reuse, and initialised with their respective tokenizers upon use. The training data was formatted by adding end-of-sequence, beginning-of-sequence, and specialised control tokens for each LLM, respectively. The training data was then split into training and validation sets to facilitate performance evaluation during the training process. PEFT fine-tuning algorithms (Prompt Tuning and QLoRA) were configured with consistent training arguments, batch sizes, and hardware to enable a fair comparison of training outputs. During experimentation, training and validation loss, mean token accuracy, and epochs were tracked and logged. Checkpoints were enabled throughout the training process to allow for continuation and optimisation of computational resources and time. Baseline and post-training inference were conducted. RAG was implemented on the collected data that provided a South African context and was used to query a sample of questions. To implement the experimental design and execute the fine-tuning pipelines, assumptions were made. It was assumed that the data sourced from Hugging Face and benchmark datasets, together with the artificially generated question-answer pairs, were accurate, sufficient and representative of the South African context after pre-processing and expert review. It was also assumed that the PEFT QLoRA parameter configurations, including the scaling factor, dropout rate, and LoRA rank, were consistent across LLMs. Moreover, it is assumed that standardising training arguments across all experimental conditions provides a fair basis for comparing the performance of QLoRA and prompt tuning. Furthermore, all experiments were conducted on the same Google Colab GPU and data, with accuracy and loss tracked throughout training and checkpoints saved to ensure continuity, assuming that the experimental environment allowed for consistent training. 3.5 Evaluation Quantitative measures were used to evaluate the LLMs, comparing the performance of the base LLMs, fine-tuned LLMs and Diabetica-7B, a diabetes specialised LLM. This was done to evaluate how the fine-tuned LLMs performed compared to the previous diabetes specialised LLM. The metrics used to evaluate LLM performance in text generation and question answering are Accuracy, Bilingual Evaluation Understudy (BLEU), Recall-Oriented Understudy for Gisting Evaluation (ROUGE), and Bidirectional Encoder Representations from Transformers (BERT) Score. 4. Results This section provides a detailed overview of the results from the base and fine-tuned PEFT models (Prompt Tuning and QLoRA, alongside Diabetica-7B). 4.1 Model training performance Figure 3 shows the training and validation accuracy and loss of BioMistral-7B QLoRA, BioMedLM QLoRA and Qwen3-8B QLoRA. The training loss for BioMistral-7B QLoRA started at approximately 2.2 and decreased rapidly to below 1.0 within the first 100 steps, after which it converged more slowly. Accuracy increased from 0.54 to above 0.75 during the first 100 steps and showed minimal fluctuation thereafter. The training curves remained close to the validation curves at all steps after 100, suggesting effective training. Qwen3-8B QLoRA, similar to BioMistral-7B QLoRA, converges and stabilises within the first 100 steps, although at a slightly slower rate than BioMistral-7B. However, BioMedLM QLoRA started with a training loss of approximately 4.25 and decreased slowly to around 2.5 within 500 steps, indicating a slower convergence compared to BioMistral-7B QLoRA. The accuracy increased to above 0.50 only after 400 steps, and the gap between the training and validation loss and accuracy curves remains wider throughout the training steps, suggesting less efficient optimisation and potential generalisation challenges compared to the other LLMs. Figure 4 shows that the validation and training loss and accuracy for the Prompt technique, which shows minimal improvement over the same number of training steps for BioMistral-7B and Qwen3-8B. The fluctuations observed indicate instability and limited learning. Due to the poor performance of QLoRA on BioMedLM, prompt tuning was not applied to BioMedLM. 4.2 Quantitative performance evaluation The responses generated by BioMistral-7B Base, BioMistral-7B QLoRA, Qwen3-8B Base, Qwen3-8B QLoRA, and Diabetica-7B were evaluated using BLEU, ROUGE, BERTScore and accuracy. BioMedLM was not evaluated further quantitatively due to the observation that the generated answers were inaccurate and unrelated to the questions. For fill-in-the-blank and multiple-choice questions, the evaluation focused on ROUGE-1, ROUGE-L, and BERTScore. These metrics were used to measure the degree of exact matches, contextual similarity, and overlap between the reference answers and the generated responses. According to Table 2 , QLoRA fine-tuned Qwen3-8B demonstrated slightly better performance compared to both its base version and Diabetica-7B. Notably, Diabetica-7B outperformed all other models in terms of the BLEU metric. In contrast, BioMistral-7B QLoRA showed lower performance than the base BioMistral-7B model. When focusing solely on the accuracy of answers for multiple-choice questions, both Qwen3-8B LLMs displayed consistently higher results. Specifically, Qwen3-8B QLoRA achieved the highest accuracy at 80.6%, slightly higher than the base Qwen3-8B LLM by 0.1%. Diabetica-7B also remained competitive with an accuracy of 77.7%. For long answer question pairs, excluding multiple-choice and fill-in-the-blank types, BERTScore was used to evaluate contextual similarities between generated and reference texts. BioMistral-7B QLoRA demonstrated slightly superior performance in this category, with all LLMs achieving average BERTScores above 0.800, as presented in Table 2 . Table 2 LLMs quantitative performance for MCQ and fill-in-the-blanks questions Model MCQ MCQ and fill-in-the-blanks questions Long answer questions Accuracy BLEU Rouge-1 Rouge-L BERTScore BERTScore Base BioMistral-7B 0.886 0.062 0.378 0.376 0886 0.840 Qwen3-8B 0.939 0.359 0.789 0.789 0.939 0.826 Domain Specific (fine-tuned) BioMistral-7B QLoRA 0.841 0.032 0.180 0.180 0.841 0.844 Qwen3-8B QLoRA 0.940 0.363 0.793 0.792 0.940 0.826 Diabetica-7B 0.929 0.465 0.778 0.776 0.929 0.841 4.3 Observation from the Models’ Generated Answers The author observed that the answers generated by BioMedLM, both base and fine-tuned, were often irrelevant to the questions and difficult to comprehend. The observed answers generated by BioMistral-7B and Qwen3-8B QLoRA fine-tuned LLM provided an ethical declaration that advises on medical consultation. For the evaluation that combined multiple-choice questions and fill-in-the-blank question-answer pairs, although BioMistral-7B QLoRA responds to questions correctly, it provides answers that include variations of 'Diabetes Virtual Doctor' before the actual answer, which impacts the BLEU, ROUGE-1, ROUGE-L, and BERTScore performance. For a sample of 10 questions, a RAG approach was implemented and queried against the collected South African contextual documents, which were categorised in the vector database as collections. The top five related chunks were used to provide additional context to the questions. Figure 5 shows an answer generated by Qwen3-8B QLORA (the best performing LLM on the majority of the Quantitative metrics) using RAG, with the relevant sources referenced for additional information. The top-referenced sources are those related to Metplitin drug leaflets and guidelines, showing that the relevant documents were identified from the collection to construct the answer. 5. Discussion BioMedLM exhibited limited learning, with minimal improvement across training steps when using QLoRA, indicating challenges in its suitability for this study. Prompt tuning showed minimal learning and instability during the first 500 training steps for the LLMs, supporting authors in [ 20 ] that, although prompt tuning is efficient, it converges more slowly compared to QLoRA. On the other hand, QLoRA demonstrated effective and stable learning for BioMistral-7B and Qwen3-8B, as evidenced by fast convergence and lower validation loss and increased accuracy. Notably, Qwen3-8B, a general LLM, achieved the lowest validation loss of 0.84 and the highest mean token accuracy of 0.79, outperforming the medically specialised LLMs. This may be because Qwen3-8B is slightly larger, with 8 billion parameters, while BioMistral-7B has 7 billion and BioMedLM has 2.7 billion. Additionally, the Qwen3-8B model used in the study was trained in 2025 with more recent data, whereas BioMistral-7B and BioMedLM were trained in 2024 [ 30 , 36 , 37 ]. For further quantitative evaluation, prompt tuning was excluded, as training and validation loss indicated minimal learning and instability. The question-answer pairs consisted of three types of formats: multiple-choice questions, fill-in-the-blank questions, and long/short answer questions. The initial evaluation combined multiple-choice questions and fill-in-the-blank question-answer pairs. From Table 2 , BioMistral-7B performed poorly on BLEU, ROUGE-1, ROUGE-L, and BERTScore when compared with the other LLMs. The performance of the BioMistral-7B QLoRA is lower than that of the BioMistral-7B base. Upon analysis of the generated answers, although BioMistral-7B QLoRA responds to questions correctly, it provides answers that include variations of 'Diabetes Virtual Doctor' before the actual answer. Qwen QLoRA, an improvement of Qwen base, outperforms all models, excluding Diabetica, on BLEU. The second evaluation focused on multiple-choice question-answer pairs only. BioMistral’s poor performance, with an accuracy of less than 50% in generating correct answers, while Qwen3-8B QLoRA and base achieved competitively high results. Qwen3-8B QLoRA reached an accuracy of 80.7%, compared to 80.6% for Qwen3-8B base. However, an added benefit of the long answers for multiple-choice questions generated by QLoRA, it includes an ethical declaration as a footnote, unlike the base models and Diabetica. And finally, for long- and short-answer questions, evaluation based on contextual similarity using BERTScore showed that responses generated by BioMistral-7B are, on average, superior to those generated by Qwen3-8B, although competitive with Diabetica-7B. The generated outputs from Qwen3-8B QLoRA through RAG implementation ensured that when guidelines or drug labels are updated, cancelled, or discontinued, the collections in the vector database can be updated to allow for the use of recent, factual, and relevant data sources. Training both medical and non-medical specialised LLMs demonstrated that non-medical models can be highly competitive when fine-tuned on custom datasets, likely due to their exposure to large and diverse data sources, including medical information. General LLMs were applied in [ 12 – 17 ], and [ 29 , 31 , 32 ], showed promising results, as in this study. Fine-tuning LLMs with QLoRA improved the results to add ethical and safety declarations and notes, while RAG integration improved the answers by referencing relevant and recent documentation. From a practical perspective, the results suggest that LLMs could play a supportive role in diabetes care and management, particularly in areas such as patient education, symptom screening, and decision support for healthcare providers. In resource-constrained settings such as South Africa, where healthcare systems face workforce shortages and high patient loads, such tools could help improve access to reliable health information and support self-management among individuals living with diabetes. The integration of retrieval-augmented generation (RAG) further enhances the potential utility of the system by enabling access to up-to-date and contextually relevant information, such as national guidelines and drug formularies. This is particularly important in dynamic healthcare environments where recommendations and treatment protocols evolve over time. However, the application of LLMs in healthcare must be approached with caution. While the models demonstrated promising performance, there remains a risk of generating incorrect or misleading information, which could have serious implications for patient safety. Ensuring appropriate safeguards, including human oversight, validation mechanisms, and clear communication of limitations, is essential before deployment in real-world settings. Furthermore, while this study focused on developing a context-specific model for South Africa, additional considerations such as language diversity, digital access, and health literacy must be addressed to ensure equitable implementation. The potential benefits of such technologies will depend not only on technical performance but also on their integration into existing healthcare systems and workflows. 5.1 Limitations of the study This study has several limitations. First, the dataset used for fine-tuning had limited representation of certain subgroups, including individuals living with type 1 diabetes, which may affect the generalisability of the model across different patient populations. Additionally, the presence of abbreviations and technical terminology in the dataset may limit accessibility for individuals with lower health literacy. Second, the evaluation of the models was primarily quantitative and did not include human expert assessment. Incorporating clinical and patient perspectives would provide a more comprehensive evaluation of the relevance, safety, and usability of the generated responses. Third, the study did not assess the performance of the model in real-world clinical or community settings. As such, the practical feasibility, acceptability, and impact of the system remain to be established. Finally, ethical and regulatory considerations, including issues related to data privacy, accountability, and potential bias in model outputs, were not fully explored. These factors are critical for the safe and responsible deployment of LLM-based solutions in healthcare and should be addressed in future work. 6. Conclusion This study developed and evaluated a context-specific LLM for diabetes care and management using locally sourced South African data. By combining curated datasets with parameter-efficient fine-tuning techniques and retrieval-augmented generation, the study demonstrates that both medical and general-purpose LLMs can be adapted to perform diabetes-related tasks with promising levels of accuracy. The findings suggest that general-purpose models, when fine-tuned on contextually relevant data, can achieve performance comparable to or exceeding that of medically specialised models. This highlights the potential for leveraging widely available LLMs and adapting them to local healthcare contexts, particularly in resource-constrained settings. From a public health perspective, this work contributes to the growing body of evidence on the potential role of artificial intelligence in supporting diabetes care. Such tools may enhance patient education, support self-management, and assist healthcare providers in decision-making, particularly in settings where healthcare systems are under pressure. However, these potential benefits must be interpreted with caution. Significant challenges remain, including the risk of inaccurate or misleading outputs, issues related to bias and accountability, and the need for robust validation. At present, the system has not been evaluated in real-world clinical or community settings, and its practical applicability remains to be established. Ensuring patient safety, building trust among users, and aligning such tools with existing healthcare systems will be critical for future implementation. Future work should focus on expanding and improving the quality of the dataset, including broader representation of diabetes subtypes and user groups, as well as conducting comprehensive human evaluations involving both healthcare professionals and individuals living with diabetes. In addition, efforts to enhance inclusivity, such as incorporating South African languages and addressing varying levels of health literacy, will be essential to ensure equitable access. Ultimately, further research is needed to assess the feasibility, acceptability, and impact of such systems within real-world healthcare settings. Declarations Supplementary Information Supplementary information has been provided in the Appendix. Author Contributions O.D. conceptualised the study, G.S. collected the data, undertook the investigation, performed the experiments and drafted the manuscript. O.D and N.P.P provided guidance on the methodological design, validation of results ethical considerations, supervision of the study, and critically reviewed and revised the manuscript. Funding This work was supported in part through internal funding by the University of Pretoria. Data Availability Data is provided within the manuscript or supplementary information files. Ethical Approval. We used publicly available dataset, with no human subject involvement. Thus, this study was not subject to IRB review, in accordance with institutional guidelines. Competing interests The authors declare no competing interests. Clinical Trial Number Not applicable. References Federation ID IDF Diabetes Atlas 11th Edition – 2025, diabetesatlas.org. Accessed: Apr. 30, 2025. [Online]. Available: https://diabetesatlas.org/resources/idf-diabetes-atlas-2025/ Thant WP et al Analytical Fact Sheet: Diabetes, a Silent Killer in Africa, Integrated Africa Health Observatory (iAHO) - World Health Organization (WHO) African Region. Accessed: Apr. 01, 2025. [Online]. Available: https://web.archive.org/web/20250110032137/https://files.aho.afro.who.int/afahobckpcontainer/production/files/iAHO_Diabetes_Regional_Factsheet.pdf National Institute of Diabetes and Digestive and, Diseases K Diabetes | NIDDK, National Institute of Diabetes and Digestive and Kidney Diseases. Accessed: Apr. 30, 2025. [Online]. Available: https://www.niddk.nih.gov/health-information/diabetes Federation ID, South Africa Diabetes Country Report 2000 — 2050, diabetesatlas.org. Accessed: Apr. 30, 2025. [Online]. Available: https://diabetesatlas.org/data-by-location/country/south-africa/ World Bank Group Physicians (per 1,000 people) - South Africa, World Bank Group. Accessed: Apr. 01, 2025. [Online]. Available: https://data.worldbank.org/indicator/SH.MED.PHYS.ZS?locations=ZA Rispel L (2016) Analysing the progress and fault lines of health sector transformation in South Africa, S Afr Health Rev, vol. no. 1, Jan. 2016, Accessed: Jun. 08, 2025. [Online]. Available: https://journals.co.za/doi/abs/ 10.10520/EJC189322 Ramaphosa C One step closer to affordable quality health care for all, The Presidency Republic of South Africa. Accessed: Apr. 30, 2025. [Online]. Available: https://www.thepresidency.gov.za/node/8149 Mayosi BM, Benatar SR (2014) Health and Health Care in South Africa — 20 Years after Mandela, New England Journal of Medicine, vol. 371, no. 14, pp. 1344–1353, Oct. 10.1056/NEJMsr1405012 DEPARTMENT OF HEALTH DAG, DIRECTORATE: CHRONIC DISEASES, DEPARTMENT OF HEALTH, Type 2 Diabetes Management in Adults at Primary Care Level - January 2014, Accessed: Apr. 30, 2025. [Online]. Available: https://www.health.gov.za/wp-content/uploads/2021/09/Management-of-type-2-Diabetes-Electronic-copy-2014_compresseda.pdf Worldometer South Africa Population (LIVE), Worldometers. Accessed: Apr. 30, 2025. [Online]. Available: https://www.worldometers.info/world-population/south-africa-population/ Gijs E, Zuercher E, Henry V, Morin D, Bize R, Peytremann-Bridevaux I (Aug. 2017) Diabetes care: Comparison of patients’ and healthcare professionals’ assessment using the PACIC instrument. J Eval Clin Pract 23(4):803–811. 10.1111/jep.12720 Jeong C (2024) Fine-tuning and Utilization Methods of Domain-specific LLMs. ArXiv Jan. 10.13088/jiis.2024.30.1.093 H. and O. O. Langa Kelly and Wang, Parameter-Efficient Fine-Tuning of Pre-trained Large Language Models for Financial Text Analysis, in Artificial Intelligence Research, J. and P. A. W. Gerber Aurona and Maritz, Ed., Cham: Springer Nature Switzerland, (2025) pp. 3–20 Buckley TA, Crowe B, Abdulnour R-EE, Rodman A, Manrai AK (Mar. 2025) Comparison of Frontier Open-Source and Proprietary Large Language Models for Complex Diagnoses. JAMA Health Forum 6(3):e250040. 10.1001/jamahealthforum.2025.0040 Parthasarathy VB, Zafar A, Khan A, Shahid A (2024) The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities, Aug Nexla LLM Fine-Tuning—Overview with Code Example, Nexla. Accessed: Aug. 19, 2025. [Online]. Available: https://nexla.com/enterprise-ai/llm-fine-tuning/ Sheng B et al (Aug. 2024) Artificial intelligence for diabetes care: current and future prospects. Lancet Diabetes Endocrinol 12(8):569–595. 10.1016/S2213-8587(24)00154-2 Sheng B et al (2024) Large language models for diabetes care: Potentials and prospects, Sci Bull (Beijing), vol. 69, no. 5, pp. 583–588, Mar. 10.1016/j.scib.2024.01.004 Moodley K (Dec. 2023) Artificial intelligence (AI) or augmented intelligence? How big data and AI are transforming healthcare: Challenges and opportunities. South Afr Med J 114(1):22–26. 10.7196/SAMJ.2024.v114i1.1631 Pavon JM, Schlientz D, Maciejewski ML, Economou-Zavlanos N, Lee RH (2025) Large Language Models in Diabetes Management: The Need for Human and Artificial Intelligence Collaboration, Diabetes Care, vol. 48, no. 2, pp. 182–184, Feb. 10.2337/dci24-0079 Healey E, Tan ALM, Flint KL, Ruiz JL, Kohane I (Jan. 2025) A case study on using a large language model to analyze continuous glucose monitoring data. Sci Rep 15(1):1143. 10.1038/s41598-024-84003-0 Sharma S et al (2023) A Critical Review of ChatGPT as a Potential Substitute for Diabetes Educators. Cureus May. 10.7759/cureus.38380 Mashatian S et al (2024) Building Trustworthy Generative Artificial Intelligence for Diabetes Care and Limb Preservation: A Medical Knowledge Extraction Case. J Diabetes Sci Technol May. 10.1177/19322968241253568 Wei L et al (Sep. 2024) Diabetica: Adapting Large Language Model to Enhance Multiple Medical Tasks in Diabetes Care and Management. ArXiv Dao D, Teo JYC, Wang W, Nguyen HD (2024) LLM-Powered Multimodal AI Conversations for Diabetes Prevention, in Proceedings of the 1st ACM Workshop on AI-Powered Q&A Systems for Multimedia, New York, NY, USA: ACM, Jun. pp. 1–6. 10.1145/3643479.3662049 Li J et al (Oct. 2024) Integrated image-based deep learning and language models for primary diabetes care. Nat Med 30(10):2886–2896. 10.1038/s41591-024-03139-8 Li H et al (2025) Large language models for diabetes training: a prospective study, Sci Bull (Beijing), vol. 70, no. 6, pp. 934–942, Mar. 10.1016/j.scib.2025.01.034 Yang H et al (2023) Exploring the Potential of Large Language Models in Personalized Diabetes Treatment Strategies. medRxiv Jul. 10.1101/2023.06.30.23292034 Abbasian M et al (2024) Knowledge-Infused LLM-Powered Conversational Health Agent: A Case Study for Diabetes Patients. Feb Labrak Y, Bazoge A, Morin E, Gourraud P-A, Rouvier M, Dufour R (2024) BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains, Jul Chen Z et al (2023) MEDITRON-70B: Scaling Medical Pretraining for Large Language Models. Nov Luo R et al (Nov. 2022) BioGPT: generative pre-trained transformer for biomedical text generation and mining. Brief Bioinform 23(6). 10.1093/bib/bbac409 Bolton E et al (2024) BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text. Mar Zhang K et al (2024) A generalist vision–language foundation model for diverse biomedical tasks, Nat Med, vol. 30, no. 11, pp. 3129–3141, Nov. 10.1038/s41591-024-03185-2 Zhang D et al (2024) A survey of datasets in medicine for large language models, Intelligence & Robotics, vol. 4, no. 4, pp. 457–78, Dec. 10.20517/ir.2024.27 Yang A et al (2025) Qwen3 Technical Report, May Bolton E et al (2024) BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text. Mar Additional Declarations No competing interests reported. Supplementary Files SupplementaryInformation.docx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9563605","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":631679970,"identity":"1a6f595f-823b-4130-947d-a0fd02f2d4b3","order_by":0,"name":"Gift Zandile Silinda","email":"","orcid":"","institution":"University of Pretoria","correspondingAuthor":false,"prefix":"","firstName":"Gift","middleName":"Zandile","lastName":"Silinda","suffix":""},{"id":631679971,"identity":"21f1296b-da58-44df-b53d-5db55af1bd29","order_by":1,"name":"Patrick Ngassa Piotie","email":"","orcid":"","institution":"University of Pretoria","correspondingAuthor":false,"prefix":"","firstName":"Patrick","middleName":"Ngassa","lastName":"Piotie","suffix":""},{"id":631679972,"identity":"3b38e617-5206-42f5-92f9-bd2f95244b48","order_by":2,"name":"Olawande Daramola","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA+0lEQVRIiWNgGAWjYJACCQYGZgjrAYMNqVoSGNJI13KYsHKD42cP3vjAYC1v3n468UFCzfnE7QzMDz8w/LHDreVMXrLlDIZ0wzlncjcbJBy7nbizgc1YgoEnGacWswM5ZtI8DIcZZzDkbpNIYLuduOEAgxnQscy4tZx/Yyb9h+Gw/Qz+t0At/84BtbB/A9pej1vLDaAtDAyHE2dIAG1JbDsA1MIDtCUBdzjY33hjbNljkJ48Q+LtZoPEvmTjnc08xRIJB47j1CLZn2N440eFte0M/tyNDz58s5Pdzt6+8cOHP9U4tUCAATIb5PEEAhpwax8Fo2AUjIJRAAIA/PJTfCjvs4YAAAAASUVORK5CYII=","orcid":"","institution":"University of Pretoria","correspondingAuthor":true,"prefix":"","firstName":"Olawande","middleName":"","lastName":"Daramola","suffix":""}],"badges":[],"createdAt":"2026-04-29 09:38:18","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9563605/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9563605/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":108409800,"identity":"4ffabc15-7690-48c0-8466-3cefb0b05537","added_by":"auto","created_at":"2026-05-04 10:02:48","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":385050,"visible":true,"origin":"","legend":"\u003cp\u003eOverview of the Research Design\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-9563605/v1/d9dc9c480a65dedb6ac6a2d1.png"},{"id":108493279,"identity":"1889f22b-2401-4452-ad4d-37e27112c8a5","added_by":"auto","created_at":"2026-05-05 09:59:50","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":239009,"visible":true,"origin":"","legend":"\u003cp\u003eFine-tuning pipeline\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-9563605/v1/e3bf964a4baea0b588517183.png"},{"id":108493296,"identity":"405be362-9476-4e5b-90c3-42174ca6e4af","added_by":"auto","created_at":"2026-05-05 09:59:51","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":318281,"visible":true,"origin":"","legend":"\u003cp\u003eTraining and validation accuracy and loss of BioMistral-7B QLoRA, BioMedLM QLoRA and Qwen3-8B QLoRA\u003cstrong\u003e.\u003c/strong\u003e\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-9563605/v1/e527680d5121b7d3efdd0116.png"},{"id":108493464,"identity":"832e448b-b37a-462a-a85a-c1fd0e578cf9","added_by":"auto","created_at":"2026-05-05 10:00:34","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":390639,"visible":true,"origin":"","legend":"\u003cp\u003ePrompt tuning training and evaluation loss\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-9563605/v1/66435cec611dac0489cd0721.png"},{"id":108409805,"identity":"5e66cccd-f1bc-4a0b-8ea0-3009ac2a88e4","added_by":"auto","created_at":"2026-05-04 10:02:48","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":446051,"visible":true,"origin":"","legend":"\u003cp\u003eAnswer generated by Qwen3-8B QLoRA using RAG\u003c/p\u003e","description":"","filename":"5.png","url":"https://assets-eu.researchsquare.com/files/rs-9563605/v1/828573154a0bc13b9c8157d9.png"},{"id":108804053,"identity":"ec235cc8-cf71-42bc-a2e7-899648d47776","added_by":"auto","created_at":"2026-05-08 15:15:07","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1855708,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9563605/v1/1ce16d40-7e49-4372-8599-5a3ad7fa37b0.pdf"},{"id":108409802,"identity":"ecc83406-e97d-4e40-ace4-58128ff8808b","added_by":"auto","created_at":"2026-05-04 10:02:48","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":60306,"visible":true,"origin":"","legend":"","description":"","filename":"SupplementaryInformation.docx","url":"https://assets-eu.researchsquare.com/files/rs-9563605/v1/c5dbc6b366356519caee7328.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"A Domain-specific Large Language Model for Diabetes Care and Management","fulltext":[{"header":"1. Introduction","content":"\u003cp\u003eThe prevalence of diabetes mellitus (diabetes) is rapidly increasing, especially in low- and middle-income countries (LMICs), making it one of the fastest-growing global health emergencies of the modern era. From 2024 to 2050, the global prevalence of diabetes is expected to increase by 45%, from 590 million to 853 million cases [1]. In 2024, a majority (81%) of people living with diabetes were from LMICs, which also have high proportions of undiagnosed cases. The increase in diabetes cases, and high percentage of underdiagnosis of diabetes contribute to increased risks of complications such as cardiovascular disease, dementia, eye disorders, mortality, and escalating healthcare costs. Over the past 17 years, global diabetes-related spending has increased by 338% [1].\u003c/p\u003e\n\u003cp\u003eIn Africa, the increase is expected to be even higher, with a 142% increase from 24 million to 60 million cases [1]. This increase in diabetes cases is attributed to economic, socio-cultural, demographic (urbanisation), environmental, and genetic factors, especially for Type 2 diabetes mellitus (T2D), which accounts for 95% of the total diabetes cases. In 2024, African people made up 19% of the world\u0026rsquo;s population, and 4% of Africa\u0026rsquo;s population was living with diabetes. Despite the sizeable number of African cases, only 1% of the total global diabetes-related expenditure was for African people, leaving 73% of adults with diabetes undiagnosed. Africa, however, faces challenges in addressing diabetes prevalence, including diagnostics, education, low financial investment, healthcare practitioners\u0026rsquo; training and access to medication. South Africa is among the top 5 countries in Africa with the highest number of people living with diabetes [1, 2, 3].\u003c/p\u003e\n\u003cp\u003eIn 2024, 6.1% of adult South Africans, aged between 20 and 79, were living with diabetes [1]. The number of diabetes cases is expected to increase by 74%, from 2.3 million to 3.9 million cases, by 2050 [1]. The total diabetes-related expenditure was 4.6 billion USD and is expected to increase by 37% in 2050 [1, 4]. Despite having the highest diabetes-related expenditure in Africa in 2024, 50% of people living with diabetes in South Africa remained unaware or undiagnosed, putting them at risk for developing complications and increased mortality [1]. Moreover, in 2021, the ratio of doctors to patients was 0.8 per 1000 individuals, lower than similar middle-income countries at 1.4, and significantly lower than the estimate for adequate coverage set by the World Health Organisation (WHO) of at least 2.5 medical practitioners per 1000 individuals [5, 6]. This is an indication of low coverage and care limitation, impacting the time an individual can spend with a medical practitioner or can access a doctor. Currently, a majority (84%) of the South African population relies on public healthcare for medical needs, as they cannot afford costly private healthcare [7]. However, 70% of the country\u0026rsquo;s doctors are employed by private healthcare [8]. One of the risk factors for diabetes is urbanisation and urban poverty [2, 9]. This is a relevant challenge for South Africa, as a majority (66%) of South Africans reside in urban areas [10] and rely on public health for medical needs due to affordability.\u003c/p\u003e\n\u003cp\u003eSouth Africa\u0026rsquo;s Department of Health has set goals to minimise the impact of diabetes and assist in managing and preventing diabetes at the primary care level. The goals are [9]: educating patients, families, and healthcare practitioners (including nonconventional ones); ensuring early detection through effective screening; providing appropriate treatment and management plans; promoting self-monitoring to improve patient outcomes; assessing the effectiveness of services; and maintaining good medical records and history for patients. However, there are challenges in achieving these goals, such as resistance to lifestyle changes, attitudes and beliefs, social and economic issues, limited consultation duration with healthcare providers, and complex treatment regimens [3, 9].\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eDiabetes in South Africa continues to constrain the public healthcare system and the country\u0026rsquo;s primary healthcare goals. This presents an opportunity to explore innovative digital health solutions that can improve diabetes care and management.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eLarge Language Models (LLMs) are deep learning models trained on a large amount of unlabelled data, harbours a large amount of information and knowledge. They possess natural language processing capabilities such as text generation, language understanding, question-answering, and text summarization [11].\u0026nbsp;Recently, LLMs are being increasingly used in the healthcare domain. Examples include using LLM for continuous glucose monitoring\u0026nbsp;[12], as an educational tool for ordinary people\u0026nbsp;[13, 14], personalised care and self-management, medical question-answering, electronic health record summarisation, and clinical decision support\u0026nbsp;[13, 15, 16, 17]. Thus, \u0026nbsp;the application of LLMs to diabetes care and management may enhance education, symptom screening, diagnostic support, treatment adherence, and self-management (diet and exercise) [16, 18]. However, so far there is limited research on the use of context-specific datasets for fine-tuning LLMs for diabetes, particularly in addressing the unique challenges faced by patients, caregivers, and healthcare practitioners. Also, instances of a solution approach that combines patient care and management to aid both medical practitioners and patients are limited.\u003c/p\u003e\n\u003cp\u003eThus, the unique contributions of this study are the following:\u0026nbsp;\u003c/p\u003e\n\u003cp\u003ei.\u0026nbsp; \u0026nbsp; \u0026nbsp;Curation of South African-specific diabetes dataset for training and fine-tuning of LLMs;\u003c/p\u003e\n\u003cp\u003eii.\u0026nbsp; \u0026nbsp;\u0026nbsp;Comparative evaluation of medical LLMs and a general LLM;\u003c/p\u003e\n\u003cp\u003eiii.\u0026nbsp; \u0026nbsp;Comparative evaluation of two Parameter-Efficient Fine-Tuning (PEFT) techniques on medical LLMs;\u003c/p\u003e\n\u003cp\u003eiv.\u0026nbsp; \u0026nbsp;Application of Retrieval-Augmented Generation (RAG) on the best-performing LLM to improve factual accuracy and reduce the tendency for hallucination; and\u003c/p\u003e\n\u003cp\u003ev.\u0026nbsp; \u0026nbsp;\u0026nbsp;Development of a domain-specific LLM for diabetes care and management in South Africa with a focus on providing assistance to both medical practitioners and patients, which is new to the best of our knowledge.\u003c/p\u003e\n\u003cp\u003eThe remaining part of this paper is structured as follows. Section 2 provides the theoretical background and related work. The adopted research methodology is detailed in Section 3, while Section 4 presents a comprehensive overview of the results from our experiments. Section 5 discusses the findings, while the paper is concluded in Section 6 with a summary and an outlook of future work.\u0026nbsp;\u003c/p\u003e"},{"header":"2. Background and Related Work","content":"\u003cp\u003eThis section presents theoretical background on key topics, and a review of related work.\u003c/p\u003e\n\u003cp\u003e2.1 \u0026nbsp; \u0026nbsp; \u0026nbsp;Theoretical Background\u003c/p\u003e\n\u003ch3 id=\"_Toc213259373\"\u003eDiabetes Care and Management\u0026nbsp;\u003c/h3\u003e\n\u003cp\u003eDiabetes mellitus is a chronic disease characterised by prolonged periods of high blood sugar levels. The most common types are Type 1 Diabetes Mellitus and Type 2 Diabetes Mellitus, with gestational diabetes, ketosis-prone Type 2 diabetes, and monogenic diabetes being less common [2, 3].\u003c/p\u003e\n\u003cp\u003eType 1 diabetes is an autoimmune disease that results from limited or no production of insulin by the pancreas. People living with type 1 diabetes depend on lifelong insulin medication to supplement their deficiency. This type is often referred to as juvenile- or childhood-onset diabetes because it is usually diagnosed in the early years. On the other hand, type 2 diabetes occurs when the body uses insulin inefficiently or ineffectively. While adopting a healthy lifestyle can delay or prevent the onset of type 2 diabetes can be delayed with a healthy lifestyle, it can develop at any age [2, 3, 9]\u003c/p\u003e\n\u003cp\u003eProper diabetes care and management are essential for living a healthier life and reducing complications and risk factors such as premature death, blindness, kidney failure, cardiovascular diseases and amputation, particularly for type 2 diabetes [2, 3]. Diabetes care is multidisciplinary, involving a care team that works collaboratively with the person living with diabetes to enable effective self-management. Part of the care team, healthcare professionals provide medical examinations, diagnoses, prevention/treatment plans, advice, and education. Caregivers and family members together proactively provide non-medical support and advice. Governments and states design policies, regulations and guidelines to promote and ensure proper care. For the person living with diabetes, self-management includes day-to-day activities such as continuous glucose level monitoring, treatment adherence, and adhering to a diet and exercise plan [19].\u003c/p\u003e\n\u003ch3 id=\"_Toc213259374\"\u003eLarge Language Models\u0026nbsp;\u003c/h3\u003e\n\u003cp\u003eLarge Language Models (LLMs) are deep learning models that are trained to perform cognitive generative tasks through self-supervised or semi-supervised learning techniques. These models are trained on a large amount of unlabelled data, mostly from open-source data sources with limited focus on specialisation. The models have a large amount of information and knowledge that can be queried using natural language [11, 20]. LLMs have shown state-of-the-art performance in understanding context, grammatical structures, and semantic relationships to generate new content, such as text, images, audio and videos [19].\u003c/p\u003e\n\u003cp\u003eLLMs are currently available as either commercial (closed-source) products or open-source for public, business and academic usage. Depending on performance (including computing requirements, purpose, cost, and availability) [19], a suitable LLM can be selected for use. In general, closed-source models tend to outperform open-source models on the massive multitask language understanding benchmark [20]. However, for complex medical diagnostics, when comparing closed-source (GPT-4 \u0026ndash; Generative Pre-trained Transformer 4, by OpenAI) and open-source (LLaMA 3.1 \u0026ndash; Large Language Model Meta AI 3.1, by Meta), LLMs showed relatively similar performance on limited diagnostics cases based on a scoring evaluation method [21].\u003c/p\u003e\n\u003ch3 id=\"_Toc213259376\"\u003eFine-tuning Large Language Models\u0026nbsp;\u003c/h3\u003e\n\u003cp\u003eGeneric pre-trained LLMs may not perform optimally for domain-specific tasks, such as diabetes care and management. The pre-trained LLMs are trained on a large amount of unlabelled, diverse and raw data, which incurs an exorbitant amount of cost to understand general language structures. As a result, the generic LLMs are well-suited for handling general queries but may provide generic responses for specialised domains for tasks such as summarisation, text classification, sentiment analysis, and question-answering [22, 23].\u003c/p\u003e\n\u003cp\u003eFine-tuning can then enhance the performance of pre-trained LLMs by using customised data to adapt them for specific downstream tasks, often at low computational cost [20, 22]. To fine-tune or adapt a pre-trained LLM, Retrieval-Augmented Generation (RAG) [22], conventional fine-tuning, and Parameter-efficient Fine-Tuning (PEFT) techniques can be used [13].\u0026nbsp;The most common PEFT methods are Prefix Tuning, P-Tuning, Soft and Hard Prompt Tuning, Adapter Methods, Low-Rank Adaptation (LoRA), and Quantised Low-Rank Adaptation (QLoRA) [19, 20, 22].\u003c/p\u003e\n\u003cp id=\"_Toc213259383\"\u003e2.2\u0026nbsp; \u0026nbsp; \u0026nbsp;\u0026nbsp;Related Work\u003c/p\u003e\n\u003cp\u003eScholars have identified the potential of using LLMs, either alone or in combination with other AI techniques, such as machine learning and deep learning, in the prevention, care, and management of diabetes. These techniques have shown potential in improving patient outcomes and reducing complications and the burden on healthcare systems. Data such as renal images, electronic medical records, clinical metadata, wearable device data, MRI scans, and metabolic markers can be used to fine-tune general-purpose LLMs to perform medical tasks\u0026nbsp;[24]. These tasks include providing support, coaching, and educating healthcare practitioners, communities, and patients. However, these potentials are not without limitations or risks. Some of the limitations include the quality and reliability of training data, lack of external validation beyond the research teams, and various ethical challenges, such as privacy, accountability, bias, transparency, trustworthiness, and consent\u0026nbsp;[24, 25, 26]. Additionally, there are financial costs associated with potential medical errors [26]. Addressing these limitations will require multidisciplinary approaches and regulations to mitigate the associated risks\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eLLMs have been practically employed as recommender systems\u0026nbsp;[27]\u0026nbsp;for continuous glucose monitoring\u0026nbsp;[12], as potential educators\u0026nbsp;[13]\u0026nbsp;and to educate ordinary people\u0026nbsp;[14], personalised care and self-management, medical question-answering, electronic health record summarisation, and clinical decision support\u0026nbsp;[13, 15, 16, 17]. LLMs have also been evaluated in answering medical queries [28] using open- and closed-source models, showing promising application results.\u003c/p\u003e\n\u003cp\u003ePrevious studies [13, 15, 17, 29], have primarily concentrated on clinical activities and the fine-tuning of generic LLMs, rather than medical-specific models that possess foundational medical knowledge. Most of these studies have focused on clinical support provided by care teams, with less attention given to patient self-management. Other studies, such as\u0026nbsp;[12, 16, 18], have explored the daily prevention and management of diabetes, with an emphasis on self-management and preventative care, and have shown promising outcomes. However, many of these studies have been conducted outside South Africa, raising questions about the relevance of certain types of support for the South African context.\u003c/p\u003e\n\u003cp\u003eAcross these studies, a consistent finding has been that the quality of fine-tuning data, which should provide relevant and context-specific information, is critical to achieving improved model performance. The majority of work to date has focused on either clinical support or patient self-management using generic LLMs that have only limited awareness of medical specialisation. In response to this gap, this study aimed to fine-tune a medically specialised LLM using curated South African data. The aim is to create a model capable of educating, providing diagnostic support, offering guidance on diet and exercise, and screening symptoms for referral, thereby integrating both care and management aspects of diabetes within the South African context.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTable 2.1\u003c/strong\u003e: Summary of Related work\u003c/p\u003e\n\u003ctable border=\"1\" cellspacing=\"0\" cellpadding=\"0\" width=\"669\"\u003e\n \u003ctbody\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eStudy\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 143px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eObjective\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 99px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eModel(s)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 110px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eTechnique(s)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 58px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eSetting\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 114px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eData Source(s)\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 98px;\"\u003e\n \u003cp\u003e\u003cstrong\u003eKey Observation\u003c/strong\u003e\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e[13]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 143px;\"\u003e\n \u003cp\u003eGenerate clinical summaries from time-series glucose monitoring data for patients and clinicians\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 99px;\"\u003e\n \u003cp\u003eGPT-4 (Data Analyst plugin)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 110px;\"\u003e\n \u003cp\u003ePrompt-based approach; qualitative and quantitative evaluation\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 58px;\"\u003e\n \u003cp\u003eNot specified\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 114px;\"\u003e\n \u003cp\u003eSynthetic glucose monitoring data\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 98px;\"\u003e\n \u003cp\u003eUse of synthetic data limits real-world applicability\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e[14]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 143px;\"\u003e\n \u003cp\u003eDeliver diabetes and foot care education tailored to individuals with lower health literacy\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 99px;\"\u003e\n \u003cp\u003eGPT-4\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 110px;\"\u003e\n \u003cp\u003eRetrieval-Augmented Generation (RAG) with vector database; qualitative evaluation\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 58px;\"\u003e\n \u003cp\u003eNot specified\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 114px;\"\u003e\n \u003cp\u003eScientific articles and NIH diabetes self-management guidelines\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 98px;\"\u003e\n \u003cp\u003eLimited evaluation of clinical accuracy and real-world use\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e[15]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 143px;\"\u003e\n \u003cp\u003eSupport diabetes care tasks including personalised consultation, education, and medical record summarisation\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 99px;\"\u003e\n \u003cp\u003eQwen2\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 110px;\"\u003e\n \u003cp\u003eLoRA; qualitative and quantitative evaluation\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 58px;\"\u003e\n \u003cp\u003eChina\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 114px;\"\u003e\n \u003cp\u003ePublic MCQs, clinical guidelines, textbooks, drug labels, patient records\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 98px;\"\u003e\n \u003cp\u003eContext-specific to China; limited generalisability to other settings\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e[16]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 143px;\"\u003e\n \u003cp\u003eSupport diabetes prevention through health tracking, resource access, and communication tools\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 99px;\"\u003e\n \u003cp\u003eGPT-3.5\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 110px;\"\u003e\n \u003cp\u003eRAG with FAISS\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 58px;\"\u003e\n \u003cp\u003eNot specified\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 114px;\"\u003e\n \u003cp\u003eWeb-based sources, WHO and national health information\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 98px;\"\u003e\n \u003cp\u003eReliance on curated web data; limited clinical validation\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e[17]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 143px;\"\u003e\n \u003cp\u003eRecommend diabetes care solutions including screening, referral, and personalised recommendations\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 99px;\"\u003e\n \u003cp\u003eLLaMA + deep learning (image-based)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 110px;\"\u003e\n \u003cp\u003eMultimodal approach; qualitative evaluation\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 58px;\"\u003e\n \u003cp\u003eChina\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 114px;\"\u003e\n \u003cp\u003eRetinal images, electronic medical records\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 98px;\"\u003e\n \u003cp\u003eFocus on clinical decision support; limited patient-facing applications\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e[18]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 143px;\"\u003e\n \u003cp\u003eSupport dietary monitoring and diabetes self-management\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 99px;\"\u003e\n \u003cp\u003eGPT-3.5 (OpenAI-based chatbot)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 110px;\"\u003e\n \u003cp\u003ePrompt-based interaction; qualitative evaluation\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 58px;\"\u003e\n \u003cp\u003eUSA\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 114px;\"\u003e\n \u003cp\u003eDietary guidelines and nutrition databases\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 98px;\"\u003e\n \u003cp\u003eFocus limited to diet; lacks broader care integration\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e[28]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 143px;\"\u003e\n \u003cp\u003eEvaluate performance of multiple LLMs on diabetes-related primary care queries\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 99px;\"\u003e\n \u003cp\u003eGPT-4, GPT-3.5, Bard, LLaMA variants\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 110px;\"\u003e\n \u003cp\u003eBenchmark evaluation of accuracy\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 58px;\"\u003e\n \u003cp\u003eChina\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 114px;\"\u003e\n \u003cp\u003eExamination datasets (Chinese and UK diabetes exams)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 98px;\"\u003e\n \u003cp\u003eFocus on exam-style queries; may not reflect real-world use\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003e[29]\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 143px;\"\u003e\n \u003cp\u003eProvide personalised treatment strategies and laboratory recommendations for outpatients\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 99px;\"\u003e\n \u003cp\u003eChatGLM\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 110px;\"\u003e\n \u003cp\u003eP-tuning, LoRA; qualitative and quantitative evaluation\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 58px;\"\u003e\n \u003cp\u003eChina\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 114px;\"\u003e\n \u003cp\u003eHospital big data platform (de-identified medical records, diagnoses, complaints)\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 98px;\"\u003e\n \u003cp\u003eSingle-institution dataset; limited external validation\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003ctr\u003e\n \u003ctd style=\"width: 47px;\"\u003e\n \u003cp\u003eThis study\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 143px;\"\u003e\n \u003cp\u003eDevelop an integrated LLM to support both clinical decision-making and patient self-management in diabetes care\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 99px;\"\u003e\n \u003cp\u003eBioMedLM (2.7B), BioMistral-7B, Qwen3-8B\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 110px;\"\u003e\n \u003cp\u003eFine-tuning (PEFT), RAG; quantitative evaluation\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 58px;\"\u003e\n \u003cp\u003eSouth Africa\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 114px;\"\u003e\n \u003cp\u003eSouth African guidelines, policy documents, drug labels, FAQs, and curated local dataset\u003c/p\u003e\n \u003c/td\u003e\n \u003ctd style=\"width: 98px;\"\u003e\n \u003cp\u003eAddresses contextual gap by incorporating locally relevant data and dual user focus\u003c/p\u003e\n \u003c/td\u003e\n \u003c/tr\u003e\n \u003c/tbody\u003e\n\u003c/table\u003e"},{"header":"3. Methodology","content":"\u003cp\u003eThis study adopted an experimental research design. The experimental research process that consists of five phases is shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. Phase 1 focused on defining the tasks the LLM is expected to perform after successful training. Phase 2 involved data preparation, comprising sub-phases for data collection, curation, augmentation, preprocessing, and splitting. Phase 3 entailed identifying and selecting suitable pre-trained base LLMs. Phase 4 was fine-tuning. The final phase, Phase 5, entailed comparative model evaluation.\u003c/p\u003e \u003cp\u003ePhase 1 focused on defining the tasks the LLM is expected to perform after successful training. Phase 2 was data preparation, which included sub-phases concerned with preparing the data for modelling, including data collection, curation, augmentation, pre-processing and splitting. Phase 3 involves identifying suitable and selecting pre-trained base LLMs. Phase 4 was where fine-tuning was done. Phase 5 focussed on comparative evaluation of the trained models.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003e3.1 Define Requirements\u003c/h2\u003e \u003cp\u003eBased on the studies reported in [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e] and the goals and challenges highlighted by the South Africa\u0026rsquo;s Department of Health [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e], the fine-tuned LLM is expected to generate text or answer questions to educate, screen symptoms for referral, support diagnosis, summarise or explain treatment for adherence, and self-management (diet and exercise), with a focus on South Africa. Thereby supporting both healthcare practitioners and patients in the care and management of diabetes.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003e3.2 Data Preparation\u003c/h2\u003e \u003cp\u003eData preparation included the collection of relevant data from local sources and South Africa-relevant international sources, curation, augmentation, pre-processing, and splitting for training and evaluation. To ensure the collection of relevant and accurate data for diabetes care and management, keywords and phrases that were identified from literature and reviewed by a medical professional to provide a holistic view, covering synonyms/related terms, diagnosis, treatment, management and terms that could be mistaken for diabetes, referred to as negative matches. These keywords were used to collect and screen data, forming a filtering layer specific to diabetes care and management data.\u003c/p\u003e \u003cp\u003eDatasets specific to the South African context were collected from reliable government and non-profit organisation websites, such as the Department of Health, the South African Health Products Regulatory Authority (SAHPRA), and the World Health Organisation (WHO), using web scraping and by downloading available PDFs. The focus was on guidelines, factsheets, drug labels, frequently asked questions (FAQs), and other related formats to create the South African contextual custom data. The collected custom data, consisting of 1,596 documents. The data was then used to extract knowledge and generate question-answer pairs by prompting OpenAI GPT-5-nano to generate short/long answer questions and fill-in-the-blank pairs.\u003c/p\u003e \u003cp\u003eRelevant data to supplement the South African contextual custom data from Hugging Face [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e] and referenced in LLMXplorer [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e] was sourced. The focus was on data in question-answer pairs, medical data, and applying the keywords and phrases to filter through the data. The final processed dataset (see Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e1\u003c/span\u003e), consist of 18,079 question-answer pairs in multiple formats (MCQs, fill-in-the-blanks, short and long answers), covering all topics related to diabetes, medication, diagnosis, and management, with questions representing the expected topics, keywords, or phrases associated with diabetes care and management.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eData source relevant question-answer pairs count\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eData Source\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eRelevant QA pairs\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePubMedQA [\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e9,130\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSouth African curated artificially generated question-answer pairs\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2,507\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMedMCQA [\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2,204\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMedQA (USMLE) [\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1,390\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003emedical-o1-reasoning-SFT\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1,300\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHealth Care Facts\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1,072\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMassive Multitask Language Understanding (MMLU) [\u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e455\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eStanford Question Answering Data 2.0 (SQuAD) [38]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e12\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAfriMed-QA [\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e9\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTotal\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e18,079\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003e3.3 Selecting base pre-trained LLMs\u003c/h2\u003e \u003cp\u003eMedical-specialised pre-trained LLMs were identified from the literature to select two base models. The LLMs were then evaluated by considering size, purpose, availability, and cost [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. The encoder-only architecture LLMs, which focus on understanding context but offer limited capacity for text generation [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e], a critical requirement for this study were excluded.\u003c/p\u003e \u003cp\u003eBioMistral [\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e], Meditron [\u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e], BioGPT [\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e], BioMedLM[\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e] and BiomedGPT [\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e], including decoder-only and encoder-decoder architectures, were evaluated for the study. These LLMs\u0026rsquo; architecture makes them suitable for the task. Encoder-decoder and decoder-only LLMs are flexible at text generation, performing well in tasks such as completing sentences [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e], open-ended discussions [\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e], summarisation [\u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e], and translation [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]. The majority of these decoder-only LLMs were trained on PubMed abstracts and papers, with Meditron also leveraging clinical guidelines. Notably, the encoder-decoder model BiomedGPT, despite being smaller in size, offers the unique advantage of training on both rich imagery data and diverse text sources such as publications, electronic hospital records, literature, and clinical notes. This makes BiomedGPT particularly advantageous for visual question answering tasks.\u003c/p\u003e \u003cp\u003eWhen performance was evaluated on benchmark data (MMLU, MedQA, PubMedQA and MedMCQA) on downstream activities, by the authors in [\u003cspan additionalcitationids=\"CR31 CR32 CR33\" citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e], it was noted that BioMedLM and BioMistral outperformed all the other LLMs on average.\u003c/p\u003e \u003cp\u003eBased on benchmark performance, task relevance, open-source accessibility, and computational feasibility, BioMistral-7B and BioMedLM (PubMedGPT) were selected as the base LLMs. BioMistral-7B demonstrates strong performance in three-shot learning across multiple medical QA benchmark datasets, while BioMedLM, despite its smaller size (2.7B parameters), has shown competitive or superior results when fine-tuned, making it highly viable under constrained computational resources.\u003c/p\u003e \u003cp\u003eQwen3-8B was also used in this study, as it is an open-source LLM that has shown promising results in the application of diabetes care and management, according to the authors in [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec8\" class=\"Section2\"\u003e \u003ch2\u003e3.4 Fine-tuning LLM\u003c/h2\u003e \u003cp\u003eThe fine-tuning pipeline shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e was applied to fine-tune two medical LLMs (BioMistral-7B and BioMedLM) as well as a general LLM (Qwen3-8B). The environment was configured on Google Colab using a single Graphics Processing Unit (GPU), specifically the NVIDIA A100-SXM4-40GB, with Python version 3.12.12. The necessary libraries were installed and imported (including Python Libraries such as \u003cem\u003eaccelerate:1.11.0\u003c/em\u003e; \u003cem\u003epeft:0.17.1\u003c/em\u003e; \u003cem\u003etrl:0.24.0\u003c/em\u003e; \u003cem\u003eBitsandbytes:0.48.2\u003c/em\u003e; \u003cem\u003epandas:2.2.2\u003c/em\u003e; \u003cem\u003etransformers:4.57.1\u003c/em\u003e; \u003cem\u003eevaluate:0.4.6\u003c/em\u003e; \u003cem\u003edatasets:4.0.0\u003c/em\u003e; \u003cem\u003esacrebleu:2.5.1\u003c/em\u003e; \u003cem\u003erouge-score:0.1.2\u003c/em\u003e; \u003cem\u003ebert-score:0.3.1\u003c/em\u003e). The selected pre-trained LLMs were downloaded from Hugging Face, saved to Google Drive for reuse, and initialised with their respective tokenizers upon use.\u003c/p\u003e \u003cp\u003eThe training data was formatted by adding end-of-sequence, beginning-of-sequence, and specialised control tokens for each LLM, respectively. The training data was then split into training and validation sets to facilitate performance evaluation during the training process. PEFT fine-tuning algorithms (Prompt Tuning and QLoRA) were configured with consistent training arguments, batch sizes, and hardware to enable a fair comparison of training outputs.\u003c/p\u003e \u003cp\u003eDuring experimentation, training and validation loss, mean token accuracy, and epochs were tracked and logged. Checkpoints were enabled throughout the training process to allow for continuation and optimisation of computational resources and time. Baseline and post-training inference were conducted. RAG was implemented on the collected data that provided a South African context and was used to query a sample of questions.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eTo implement the experimental design and execute the fine-tuning pipelines, assumptions were made. It was assumed that the data sourced from Hugging Face and benchmark datasets, together with the artificially generated question-answer pairs, were accurate, sufficient and representative of the South African context after pre-processing and expert review. It was also assumed that the PEFT QLoRA parameter configurations, including the scaling factor, dropout rate, and LoRA rank, were consistent across LLMs. Moreover, it is assumed that standardising training arguments across all experimental conditions provides a fair basis for comparing the performance of QLoRA and prompt tuning. Furthermore, all experiments were conducted on the same Google Colab GPU and data, with accuracy and loss tracked throughout training and checkpoints saved to ensure continuity, assuming that the experimental environment allowed for consistent training.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003e3.5 Evaluation\u003c/h2\u003e \u003cp\u003eQuantitative measures were used to evaluate the LLMs, comparing the performance of the base LLMs, fine-tuned LLMs and Diabetica-7B, a diabetes specialised LLM. This was done to evaluate how the fine-tuned LLMs performed compared to the previous diabetes specialised LLM.\u003c/p\u003e \u003cp\u003eThe metrics used to evaluate LLM performance in text generation and question answering are Accuracy, Bilingual Evaluation Understudy (BLEU), Recall-Oriented Understudy for Gisting Evaluation (ROUGE), and Bidirectional Encoder Representations from Transformers (BERT) Score.\u003c/p\u003e \u003c/div\u003e"},{"header":"4. Results","content":"\u003cp\u003eThis section provides a detailed overview of the results from the base and fine-tuned PEFT models (Prompt Tuning and QLoRA, alongside Diabetica-7B).\u003c/p\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003e4.1 Model training performance\u003c/h2\u003e \u003cp\u003eFigure \u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e shows the training and validation accuracy and loss of BioMistral-7B QLoRA, BioMedLM QLoRA and Qwen3-8B QLoRA. The training loss for BioMistral-7B QLoRA started at approximately 2.2 and decreased rapidly to below 1.0 within the first 100 steps, after which it converged more slowly. Accuracy increased from 0.54 to above 0.75 during the first 100 steps and showed minimal fluctuation thereafter. The training curves remained close to the validation curves at all steps after 100, suggesting effective training. Qwen3-8B QLoRA, similar to BioMistral-7B QLoRA, converges and stabilises within the first 100 steps, although at a slightly slower rate than BioMistral-7B. However, BioMedLM QLoRA started with a training loss of approximately 4.25 and decreased slowly to around 2.5 within 500 steps, indicating a slower convergence compared to BioMistral-7B QLoRA. The accuracy increased to above 0.50 only after 400 steps, and the gap between the training and validation loss and accuracy curves remains wider throughout the training steps, suggesting less efficient optimisation and potential generalisation challenges compared to the other LLMs.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eFigure \u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e shows that the validation and training loss and accuracy for the Prompt technique, which shows minimal improvement over the same number of training steps for BioMistral-7B and Qwen3-8B. The fluctuations observed indicate instability and limited learning. Due to the poor performance of QLoRA on BioMedLM, prompt tuning was not applied to BioMedLM.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003e4.2 Quantitative performance evaluation\u003c/h2\u003e \u003cp\u003eThe responses generated by BioMistral-7B Base, BioMistral-7B QLoRA, Qwen3-8B Base, Qwen3-8B QLoRA, and Diabetica-7B were evaluated using BLEU, ROUGE, BERTScore and accuracy. BioMedLM was not evaluated further quantitatively due to the observation that the generated answers were inaccurate and unrelated to the questions.\u003c/p\u003e \u003cp\u003eFor fill-in-the-blank and multiple-choice questions, the evaluation focused on ROUGE-1, ROUGE-L, and BERTScore. These metrics were used to measure the degree of exact matches, contextual similarity, and overlap between the reference answers and the generated responses. According to Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e2\u003c/span\u003e, QLoRA fine-tuned Qwen3-8B demonstrated slightly better performance compared to both its base version and Diabetica-7B. Notably, Diabetica-7B outperformed all other models in terms of the BLEU metric. In contrast, BioMistral-7B QLoRA showed lower performance than the base BioMistral-7B model. When focusing solely on the accuracy of answers for multiple-choice questions, both Qwen3-8B LLMs displayed consistently higher results. Specifically, Qwen3-8B QLoRA achieved the highest accuracy at 80.6%, slightly higher than the base Qwen3-8B LLM by 0.1%. Diabetica-7B also remained competitive with an accuracy of 77.7%. For long answer question pairs, excluding multiple-choice and fill-in-the-blank types, BERTScore was used to evaluate contextual similarities between generated and reference texts. BioMistral-7B QLoRA demonstrated slightly superior performance in this category, with all LLMs achieving average BERTScores above 0.800, as presented in Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e2\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eLLMs quantitative performance for MCQ and fill-in-the-blanks questions\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"7\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c6\" colnum=\"6\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c7\" colnum=\"7\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003eModel\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMCQ\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"4\" nameend=\"c6\" namest=\"c3\"\u003e \u003cp\u003eMCQ and fill-in-the-blanks questions\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eLong answer questions\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eAccuracy\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eBLEU\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eRouge-1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eRouge-L\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c6\"\u003e \u003cp\u003eBERTScore\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c7\"\u003e \u003cp\u003eBERTScore\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colspan=\"7\" nameend=\"c7\" namest=\"c1\"\u003e \u003cp\u003eBase\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBioMistral-7B\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.886\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.062\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.378\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.376\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0886\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.840\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eQwen3-8B\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.939\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.359\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.789\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.789\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.939\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.826\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colspan=\"7\" nameend=\"c7\" namest=\"c1\"\u003e \u003cp\u003e\u003cb\u003eDomain Specific (fine-tuned)\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBioMistral-7B QLoRA\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.841\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.032\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.180\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.180\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.841\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.844\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eQwen3-8B QLoRA\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.940\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.363\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.793\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.792\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.940\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.826\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eDiabetica-7B\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e0.929\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e0.465\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e0.778\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e0.776\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c6\"\u003e \u003cp\u003e0.929\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c7\"\u003e \u003cp\u003e0.841\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec13\" class=\"Section2\"\u003e \u003ch2\u003e4.3 Observation from the Models\u0026rsquo; Generated Answers\u003c/h2\u003e \u003cp\u003eThe author observed that the answers generated by BioMedLM, both base and fine-tuned, were often irrelevant to the questions and difficult to comprehend. The observed answers generated by BioMistral-7B and Qwen3-8B QLoRA fine-tuned LLM provided an ethical declaration that advises on medical consultation. For the evaluation that combined multiple-choice questions and fill-in-the-blank question-answer pairs, although BioMistral-7B QLoRA responds to questions correctly, it provides answers that include variations of 'Diabetes Virtual Doctor' before the actual answer, which impacts the BLEU, ROUGE-1, ROUGE-L, and BERTScore performance.\u003c/p\u003e \u003cp\u003eFor a sample of 10 questions, a RAG approach was implemented and queried against the collected South African contextual documents, which were categorised in the vector database as collections. The top five related chunks were used to provide additional context to the questions. Figure\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e shows an answer generated by Qwen3-8B QLORA (the best performing LLM on the majority of the Quantitative metrics) using RAG, with the relevant sources referenced for additional information. The top-referenced sources are those related to Metplitin drug leaflets and guidelines, showing that the relevant documents were identified from the collection to construct the answer.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"5. Discussion","content":"\u003cp\u003eBioMedLM exhibited limited learning, with minimal improvement across training steps when using QLoRA, indicating challenges in its suitability for this study. Prompt tuning showed minimal learning and instability during the first 500 training steps for the LLMs, supporting authors in [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e] that, although prompt tuning is efficient, it converges more slowly compared to QLoRA. On the other hand, QLoRA demonstrated effective and stable learning for BioMistral-7B and Qwen3-8B, as evidenced by fast convergence and lower validation loss and increased accuracy. Notably, Qwen3-8B, a general LLM, achieved the lowest validation loss of 0.84 and the highest mean token accuracy of 0.79, outperforming the medically specialised LLMs. This may be because Qwen3-8B is slightly larger, with 8\u0026nbsp;billion parameters, while BioMistral-7B has 7\u0026nbsp;billion and BioMedLM has 2.7\u0026nbsp;billion. Additionally, the Qwen3-8B model used in the study was trained in 2025 with more recent data, whereas BioMistral-7B and BioMedLM were trained in 2024 [\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e, \u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e, \u003cspan citationid=\"CR37\" class=\"CitationRef\"\u003e37\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eFor further quantitative evaluation, prompt tuning was excluded, as training and validation loss indicated minimal learning and instability.\u003c/p\u003e \u003cp\u003eThe question-answer pairs consisted of three types of formats: multiple-choice questions, fill-in-the-blank questions, and long/short answer questions. The initial evaluation combined multiple-choice questions and fill-in-the-blank question-answer pairs. From Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e2\u003c/span\u003e, BioMistral-7B performed poorly on BLEU, ROUGE-1, ROUGE-L, and BERTScore when compared with the other LLMs. The performance of the BioMistral-7B QLoRA is lower than that of the BioMistral-7B base. Upon analysis of the generated answers, although BioMistral-7B QLoRA responds to questions correctly, it provides answers that include variations of 'Diabetes Virtual Doctor' before the actual answer. Qwen QLoRA, an improvement of Qwen base, outperforms all models, excluding Diabetica, on BLEU. The second evaluation focused on multiple-choice question-answer pairs only. BioMistral\u0026rsquo;s poor performance, with an accuracy of less than 50% in generating correct answers, while Qwen3-8B QLoRA and base achieved competitively high results. Qwen3-8B QLoRA reached an accuracy of 80.7%, compared to 80.6% for Qwen3-8B base. However, an added benefit of the long answers for multiple-choice questions generated by QLoRA, it includes an ethical declaration as a footnote, unlike the base models and Diabetica. And finally, for long- and short-answer questions, evaluation based on contextual similarity using BERTScore showed that responses generated by BioMistral-7B are, on average, superior to those generated by Qwen3-8B, although competitive with Diabetica-7B.\u003c/p\u003e \u003cp\u003e The generated outputs from Qwen3-8B QLoRA through RAG implementation ensured that when guidelines or drug labels are updated, cancelled, or discontinued, the collections in the vector database can be updated to allow for the use of recent, factual, and relevant data sources.\u003c/p\u003e \u003cp\u003eTraining both medical and non-medical specialised LLMs demonstrated that non-medical models can be highly competitive when fine-tuned on custom datasets, likely due to their exposure to large and diverse data sources, including medical information. General LLMs were applied in [\u003cspan additionalcitationids=\"CR13 CR14 CR15 CR16\" citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e], and [\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e, \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e, \u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e], showed promising results, as in this study. Fine-tuning LLMs with QLoRA improved the results to add ethical and safety declarations and notes, while RAG integration improved the answers by referencing relevant and recent documentation.\u003c/p\u003e \u003cp\u003eFrom a practical perspective, the results suggest that LLMs could play a supportive role in diabetes care and management, particularly in areas such as patient education, symptom screening, and decision support for healthcare providers. In resource-constrained settings such as South Africa, where healthcare systems face workforce shortages and high patient loads, such tools could help improve access to reliable health information and support self-management among individuals living with diabetes.\u003c/p\u003e \u003cp\u003e The integration of retrieval-augmented generation (RAG) further enhances the potential utility of the system by enabling access to up-to-date and contextually relevant information, such as national guidelines and drug formularies. This is particularly important in dynamic healthcare environments where recommendations and treatment protocols evolve over time.\u003c/p\u003e \u003cp\u003eHowever, the application of LLMs in healthcare must be approached with caution. While the models demonstrated promising performance, there remains a risk of generating incorrect or misleading information, which could have serious implications for patient safety. Ensuring appropriate safeguards, including human oversight, validation mechanisms, and clear communication of limitations, is essential before deployment in real-world settings.\u003c/p\u003e \u003cp\u003eFurthermore, while this study focused on developing a context-specific model for South Africa, additional considerations such as language diversity, digital access, and health literacy must be addressed to ensure equitable implementation. The potential benefits of such technologies will depend not only on technical performance but also on their integration into existing healthcare systems and workflows.\u003c/p\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003e5.1 Limitations of the study\u003c/h2\u003e \u003cp\u003eThis study has several limitations. First, the dataset used for fine-tuning had limited representation of certain subgroups, including individuals living with type 1 diabetes, which may affect the generalisability of the model across different patient populations. Additionally, the presence of abbreviations and technical terminology in the dataset may limit accessibility for individuals with lower health literacy.\u003c/p\u003e \u003cp\u003eSecond, the evaluation of the models was primarily quantitative and did not include human expert assessment. Incorporating clinical and patient perspectives would provide a more comprehensive evaluation of the relevance, safety, and usability of the generated responses.\u003c/p\u003e \u003cp\u003eThird, the study did not assess the performance of the model in real-world clinical or community settings. As such, the practical feasibility, acceptability, and impact of the system remain to be established.\u003c/p\u003e \u003cp\u003eFinally, ethical and regulatory considerations, including issues related to data privacy, accountability, and potential bias in model outputs, were not fully explored. These factors are critical for the safe and responsible deployment of LLM-based solutions in healthcare and should be addressed in future work.\u003c/p\u003e \u003c/div\u003e"},{"header":"6. Conclusion","content":"\u003cp\u003eThis study developed and evaluated a context-specific LLM for diabetes care and management using locally sourced South African data. By combining curated datasets with parameter-efficient fine-tuning techniques and retrieval-augmented generation, the study demonstrates that both medical and general-purpose LLMs can be adapted to perform diabetes-related tasks with promising levels of accuracy.\u003c/p\u003e \u003cp\u003eThe findings suggest that general-purpose models, when fine-tuned on contextually relevant data, can achieve performance comparable to or exceeding that of medically specialised models. This highlights the potential for leveraging widely available LLMs and adapting them to local healthcare contexts, particularly in resource-constrained settings.\u003c/p\u003e \u003cp\u003eFrom a public health perspective, this work contributes to the growing body of evidence on the potential role of artificial intelligence in supporting diabetes care. Such tools may enhance patient education, support self-management, and assist healthcare providers in decision-making, particularly in settings where healthcare systems are under pressure. However, these potential benefits must be interpreted with caution.\u003c/p\u003e \u003cp\u003eSignificant challenges remain, including the risk of inaccurate or misleading outputs, issues related to bias and accountability, and the need for robust validation. At present, the system has not been evaluated in real-world clinical or community settings, and its practical applicability remains to be established. Ensuring patient safety, building trust among users, and aligning such tools with existing healthcare systems will be critical for future implementation.\u003c/p\u003e \u003cp\u003eFuture work should focus on expanding and improving the quality of the dataset, including broader representation of diabetes subtypes and user groups, as well as conducting comprehensive human evaluations involving both healthcare professionals and individuals living with diabetes. In addition, efforts to enhance inclusivity, such as incorporating South African languages and addressing varying levels of health literacy, will be essential to ensure equitable access. Ultimately, further research is needed to assess the feasibility, acceptability, and impact of such systems within real-world healthcare settings.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003eSupplementary Information \u0026nbsp;Supplementary information has been provided in the Appendix.\u003c/p\u003e\n\u003cp\u003eAuthor Contributions\u0026nbsp;O.D. conceptualised the study, G.S. collected the data, undertook the investigation, performed the experiments and drafted the manuscript. O.D and N.P.P provided guidance on the methodological design, validation of results ethical considerations, supervision of the study, and critically reviewed and revised the manuscript.\u003c/p\u003e\n\u003cp\u003eFunding\u0026nbsp;This work was supported in part through internal funding by the University of Pretoria.\u003c/p\u003e\n\u003cp\u003eData Availability Data is provided within the manuscript or supplementary information files.\u003c/p\u003e\n\u003cp\u003eEthical Approval.\u0026nbsp;We used publicly available dataset, with no human subject involvement. Thus, this study was not subject to IRB review, in accordance with institutional guidelines.\u003c/p\u003e\n\u003cp\u003eCompeting interests The authors declare no competing interests.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eClinical Trial Number Not applicable.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eFederation ID IDF Diabetes Atlas 11th Edition\u0026thinsp;\u0026ndash;\u0026thinsp;2025, diabetesatlas.org. Accessed: Apr. 30, 2025. [Online]. Available: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://diabetesatlas.org/resources/idf-diabetes-atlas-2025/\u003c/span\u003e\u003cspan address=\"https://diabetesatlas.org/resources/idf-diabetes-atlas-2025/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eThant WP et al Analytical Fact Sheet: Diabetes, a Silent Killer in Africa, Integrated Africa Health Observatory (iAHO) - World Health Organization (WHO) African Region. Accessed: Apr. 01, 2025. [Online]. Available: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://web.archive.org/web/20250110032137/https://files.aho.afro.who.int/afahobckpcontainer/production/files/iAHO_Diabetes_Regional_Factsheet.pdf\u003c/span\u003e\u003cspan address=\"https://web.archive.org/web/20250110032137/https://files.aho.afro.who.int/afahobckpcontainer/production/files/iAHO_Diabetes_Regional_Factsheet.pdf\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNational Institute of Diabetes and Digestive and, Diseases K Diabetes | NIDDK, National Institute of Diabetes and Digestive and Kidney Diseases. Accessed: Apr. 30, 2025. [Online]. Available: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.niddk.nih.gov/health-information/diabetes\u003c/span\u003e\u003cspan address=\"https://www.niddk.nih.gov/health-information/diabetes\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFederation ID, South Africa Diabetes Country Report 2000 \u0026mdash; 2050, diabetesatlas.org. Accessed: Apr. 30, 2025. [Online]. Available: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://diabetesatlas.org/data-by-location/country/south-africa/\u003c/span\u003e\u003cspan address=\"https://diabetesatlas.org/data-by-location/country/south-africa/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWorld Bank Group Physicians (per 1,000 people) - South Africa, World Bank Group. Accessed: Apr. 01, 2025. [Online]. Available: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://data.worldbank.org/indicator/SH.MED.PHYS.ZS?locations=ZA\u003c/span\u003e\u003cspan address=\"https://data.worldbank.org/indicator/SH.MED.PHYS.ZS?locations=ZA\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRispel L (2016) Analysing the progress and fault lines of health sector transformation in South Africa, S Afr Health Rev, vol. no. 1, Jan. 2016, Accessed: Jun. 08, 2025. [Online]. Available: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://journals.co.za/doi/abs/\u003c/span\u003e\u003cspan address=\"https://journals.co.za/doi/abs/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.10520/EJC189322\u003c/span\u003e\u003cspan address=\"10.10520/EJC189322\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRamaphosa C One step closer to affordable quality health care for all, The Presidency Republic of South Africa. Accessed: Apr. 30, 2025. [Online]. Available: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.thepresidency.gov.za/node/8149\u003c/span\u003e\u003cspan address=\"https://www.thepresidency.gov.za/node/8149\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMayosi BM, Benatar SR (2014) Health and Health Care in South Africa \u0026mdash; 20 Years after Mandela, New England Journal of Medicine, vol. 371, no. 14, pp. 1344\u0026ndash;1353, Oct. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1056/NEJMsr1405012\u003c/span\u003e\u003cspan address=\"10.1056/NEJMsr1405012\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDEPARTMENT OF HEALTH DAG, DIRECTORATE: CHRONIC DISEASES, DEPARTMENT OF HEALTH, Type 2 Diabetes Management in Adults at Primary Care Level - January 2014, Accessed: Apr. 30, 2025. [Online]. Available: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.health.gov.za/wp-content/uploads/2021/09/Management-of-type-2-Diabetes-Electronic-copy-2014_compresseda.pdf\u003c/span\u003e\u003cspan address=\"https://www.health.gov.za/wp-content/uploads/2021/09/Management-of-type-2-Diabetes-Electronic-copy-2014_compresseda.pdf\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWorldometer South Africa Population (LIVE), Worldometers. Accessed: Apr. 30, 2025. [Online]. Available: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://www.worldometers.info/world-population/south-africa-population/\u003c/span\u003e\u003cspan address=\"https://www.worldometers.info/world-population/south-africa-population/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGijs E, Zuercher E, Henry V, Morin D, Bize R, Peytremann-Bridevaux I (Aug. 2017) Diabetes care: Comparison of patients\u0026rsquo; and healthcare professionals\u0026rsquo; assessment using the PACIC instrument. J Eval Clin Pract 23(4):803\u0026ndash;811. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1111/jep.12720\u003c/span\u003e\u003cspan address=\"10.1111/jep.12720\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJeong C (2024) Fine-tuning and Utilization Methods of Domain-specific LLMs. ArXiv Jan. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.13088/jiis.2024.30.1.093\u003c/span\u003e\u003cspan address=\"10.13088/jiis.2024.30.1.093\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eH. and O. O. Langa Kelly and Wang, Parameter-Efficient Fine-Tuning of Pre-trained Large Language Models for Financial Text Analysis, in Artificial Intelligence Research, J. and P. A. W. Gerber Aurona and Maritz, Ed., Cham: Springer Nature Switzerland, (2025) pp. 3\u0026ndash;20\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBuckley TA, Crowe B, Abdulnour R-EE, Rodman A, Manrai AK (Mar. 2025) Comparison of Frontier Open-Source and Proprietary Large Language Models for Complex Diagnoses. JAMA Health Forum 6(3):e250040. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1001/jamahealthforum.2025.0040\u003c/span\u003e\u003cspan address=\"10.1001/jamahealthforum.2025.0040\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eParthasarathy VB, Zafar A, Khan A, Shahid A (2024) The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities, Aug\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eNexla LLM Fine-Tuning\u0026mdash;Overview with Code Example, Nexla. Accessed: Aug. 19, 2025. [Online]. Available: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://nexla.com/enterprise-ai/llm-fine-tuning/\u003c/span\u003e\u003cspan address=\"https://nexla.com/enterprise-ai/llm-fine-tuning/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSheng B et al (Aug. 2024) Artificial intelligence for diabetes care: current and future prospects. Lancet Diabetes Endocrinol 12(8):569\u0026ndash;595. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/S2213-8587(24)00154-2\u003c/span\u003e\u003cspan address=\"10.1016/S2213-8587(24)00154-2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSheng B et al (2024) Large language models for diabetes care: Potentials and prospects, Sci Bull (Beijing), vol. 69, no. 5, pp. 583\u0026ndash;588, Mar. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.scib.2024.01.004\u003c/span\u003e\u003cspan address=\"10.1016/j.scib.2024.01.004\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMoodley K (Dec. 2023) Artificial intelligence (AI) or augmented intelligence? How big data and AI are transforming healthcare: Challenges and opportunities. South Afr Med J 114(1):22\u0026ndash;26. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.7196/SAMJ.2024.v114i1.1631\u003c/span\u003e\u003cspan address=\"10.7196/SAMJ.2024.v114i1.1631\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePavon JM, Schlientz D, Maciejewski ML, Economou-Zavlanos N, Lee RH (2025) Large Language Models in Diabetes Management: The Need for Human and Artificial Intelligence Collaboration, Diabetes Care, vol. 48, no. 2, pp. 182\u0026ndash;184, Feb. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.2337/dci24-0079\u003c/span\u003e\u003cspan address=\"10.2337/dci24-0079\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHealey E, Tan ALM, Flint KL, Ruiz JL, Kohane I (Jan. 2025) A case study on using a large language model to analyze continuous glucose monitoring data. Sci Rep 15(1):1143. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1038/s41598-024-84003-0\u003c/span\u003e\u003cspan address=\"10.1038/s41598-024-84003-0\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSharma S et al (2023) A Critical Review of ChatGPT as a Potential Substitute for Diabetes Educators. Cureus May. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.7759/cureus.38380\u003c/span\u003e\u003cspan address=\"10.7759/cureus.38380\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMashatian S et al (2024) Building Trustworthy Generative Artificial Intelligence for Diabetes Care and Limb Preservation: A Medical Knowledge Extraction Case. J Diabetes Sci Technol May. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1177/19322968241253568\u003c/span\u003e\u003cspan address=\"10.1177/19322968241253568\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWei L et al (Sep. 2024) Diabetica: Adapting Large Language Model to Enhance Multiple Medical Tasks in Diabetes Care and Management. ArXiv\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDao D, Teo JYC, Wang W, Nguyen HD (2024) LLM-Powered Multimodal AI Conversations for Diabetes Prevention, in Proceedings of the 1st ACM Workshop on AI-Powered Q\u0026amp;A Systems for Multimedia, New York, NY, USA: ACM, Jun. pp. 1\u0026ndash;6. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1145/3643479.3662049\u003c/span\u003e\u003cspan address=\"10.1145/3643479.3662049\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLi J et al (Oct. 2024) Integrated image-based deep learning and language models for primary diabetes care. Nat Med 30(10):2886\u0026ndash;2896. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1038/s41591-024-03139-8\u003c/span\u003e\u003cspan address=\"10.1038/s41591-024-03139-8\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLi H et al (2025) Large language models for diabetes training: a prospective study, Sci Bull (Beijing), vol. 70, no. 6, pp. 934\u0026ndash;942, Mar. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.scib.2025.01.034\u003c/span\u003e\u003cspan address=\"10.1016/j.scib.2025.01.034\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYang H et al (2023) Exploring the Potential of Large Language Models in Personalized Diabetes Treatment Strategies. medRxiv Jul. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1101/2023.06.30.23292034\u003c/span\u003e\u003cspan address=\"10.1101/2023.06.30.23292034\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAbbasian M et al (2024) Knowledge-Infused LLM-Powered Conversational Health Agent: A Case Study for Diabetes Patients. Feb\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLabrak Y, Bazoge A, Morin E, Gourraud P-A, Rouvier M, Dufour R (2024) BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains, Jul\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChen Z et al (2023) MEDITRON-70B: Scaling Medical Pretraining for Large Language Models. Nov\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLuo R et al (Nov. 2022) BioGPT: generative pre-trained transformer for biomedical text generation and mining. Brief Bioinform 23(6). \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1093/bib/bbac409\u003c/span\u003e\u003cspan address=\"10.1093/bib/bbac409\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBolton E et al (2024) BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text. Mar\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang K et al (2024) A generalist vision\u0026ndash;language foundation model for diverse biomedical tasks, Nat Med, vol. 30, no. 11, pp. 3129\u0026ndash;3141, Nov. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1038/s41591-024-03185-2\u003c/span\u003e\u003cspan address=\"10.1038/s41591-024-03185-2\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eZhang D et al (2024) A survey of datasets in medicine for large language models, Intelligence \u0026amp; Robotics, vol. 4, no. 4, pp. 457\u0026ndash;78, Dec. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.20517/ir.2024.27\u003c/span\u003e\u003cspan address=\"10.20517/ir.2024.27\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYang A et al (2025) Qwen3 Technical Report, May\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBolton E et al (2024) BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text. Mar\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Diabetes care and management, Domain-Specific LLM, Fine-tuning, South Africa","lastPublishedDoi":"10.21203/rs.3.rs-9563605/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9563605/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eThe prevalence of diabetes is rapidly increasing in low- and middle-income countries (LMIC), making it one of the fastest-growing global health emergencies of the modern era. Despite efforts by healthcare practitioners, the government, and communities to minimise the associated complications and mortality, there are significant challenges that can be potentially alleviated through innovative digital health technologies. This study developed a domain-specific large language model (DS-LLM) aimed at improving diabetes care and management using a case study of South Africa. To achieve this, local data was collected and supplemented with benchmark and medical Hugging Face datasets. Medical pre-trained large language models (LLMs): \u003cem\u003eBioMedLM (2.7B)\u003c/em\u003e and \u003cem\u003eBioMistral-7B\u003c/em\u003e were selected as base models, along with \u003cem\u003eQwen3-8B\u003c/em\u003e (a non-specialised LLM). Two Parameter-Efficient Fine-Tuning (PEFT) techniques: prompt tuning and Quantised Low-Rank Adaptation (QLoRA) were applied, with Retrieval-Augmented Generation (RAG) applied on the best-performing LLM. The fine-tuned LLMs were evaluated by comparing their respective performance with \u003cem\u003eDiabetica-7B\u003c/em\u003e, a specialised diabetes LLM. The final dataset comprised 18,079 processed question-answer pairs (14% artificially generated for the South African context) and 1,596 documents, covering medication, management, diagnosis, screening, and general diabetes topics that pertain to South Africa. For fill-in-the-blank and multiple-choice questions formats, \u003cem\u003eQwen QLoRA\u003c/em\u003e outperformed all LLMs (ROUGE-1 = 0.793, ROUGE-L= 0.792, and BERTScore F1 = 0.940). \u003cem\u003eDiabetica\u003c/em\u003e had the highest BLEU score (0.465), while \u003cem\u003eQwen3-8B\u003c/em\u003e had 0.365. For multiple-choice questions only, \u003cem\u003eQwen3-8B QLoRA\u003c/em\u003e achieved a top accuracy of 80.7%. For short and long answers, \u003cem\u003eBioMistral-7B\u003c/em\u003eQLoRA performed slightly better, with all models scoring above 0.800. These findings highlight the promising use of LLMs for diabetes care.\u003c/p\u003e","manuscriptTitle":"A Domain-specific Large Language Model for Diabetes Care and Management","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-05-04 10:02:43","doi":"10.21203/rs.3.rs-9563605/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"19304688-5c14-42b1-bf42-528acc1419f7","owner":[],"postedDate":"May 4th, 2026","published":true,"recentEditorialEvents":[{"type":"decision","content":"Rejected","date":"2026-05-03T16:53:51+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-05-03T16:53:03+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-05-01T09:53:01+00:00","index":"","fulltext":""}],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2026-05-05T09:58:44+00:00","versionOfRecord":[],"versionCreatedAt":"2026-05-04 10:02:43","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9563605","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9563605","identity":"rs-9563605","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}