What Patients Say About Reimbursable Digital Therapeutics in Germany: A Large-Scale App Store Review Analysis Using a Large Language Model

What Patients Say About Reimbursable Digital Therapeutics in Germany: A Large-Scale App Store Review Analysis Using a Large Language Model | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article What Patients Say About Reimbursable Digital Therapeutics in Germany: A Large-Scale App Store Review Analysis Using a Large Language Model Benjamin Kinast, Henrik Rohde, Björn Schreiweis, Hannes Ulrich This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-7922462/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract In 2019, Germany has introduced a unique regulatory framework for Digital Therapeutics (DTx) known as DiGAs, with the goal of integrating evidence-based DTx into statutory healthcare. DTx are approved for statutory health insurance reimbursement if the manufacturers can show evidence of health improvement, better coordination of care process, easier access to healthcare services, or the promotion of health literacy in a controlled study setting. Systematic investigations have revealed shortcomings in the evidence provided by manufacturers. The study examines patients experience and evaluate DTx in public App Stores by analyzing sentiments and thematic categories. In addition, it explores if the use of large language model (LLM) can provide effectively support the categorization and sentiment analysis of the patients’ reviews. First, a list of approved DTx is collected and limited to those with mobile apps. Patients’ reviews were extracted from the public app stores using a tailored Python script. In the second step, the sentiments and topics of the patients’ reviews were categorized into ten predefined categories using ChatGPT-4o. To ensure the quality, the LLM-based analysis was verified through manual validation. In total, 44 mobile DiGAs were included and were analyzed. After data extraction and cleansing, the final dataset comprises 4,328 patients' reviews containing at least one interpretable statement, resulting in 9,439 valid and interpretable statements. A systematic validation of the automated classification demonstrated exceptionally high model performance with 99% accuracy for sentiment classification (F1-scores of 1.00 for positive and 0.99 for negative categories) and 95% accuracy for category classification with an average F1-score of 0.95. While the categories Overall Impression and Effectiveness scored particularly well, patients were most negative for topics related to the login and registration process as well as technical malfunctions. The findings highlight both the potential and current limitations of reimbursable DTx from the patients’ perspective. While patients value the therapeutic benefits and content quality of many DTx, technical functionality and usability, particularly during login and registration, are frequently criticized. The study also demonstrates that LLM-assisted analysis combined with human-in-the-loop validation offers an efficient approach for structuring patient feedback at scale. This combined approach could serve as a valuable complement to traditional clinical evaluation in future assessments of digital health applications. Biological sciences/Computational biology and bioinformatics Health sciences/Health care Physical sciences/Mathematics and computing Health sciences/Medical research mHealth eHealth Apps Digital Therapeutics LLM Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Introduction In 2019, Germany has introduced a unique regulatory framework for Digital Therapeutics (DTx) known as DiGAs (ger.: Digitale Gesundheitsanwendungen/ eng.: Digital Health Applications). with the goal of integrating evidence-based DTx into routine care [ 1 , 2 ]. Under the Digital Healthcare Act ( Digitale-Versorgung-Gesetz ), DiGAs can be prescribed by physicians and reimbursed by statutory health insurance ( Gesetzliche Krankenversicherung ) if they are listed in an official directory maintained by the German Federal Institute for Drugs and Medical Devices (BfArM) [ 3 ]. The directory lists DTx with a variety of indications, such as mental and behavioral disorders, musculoskeletal disorders, metabolic disorders, and disorders of the nervous system. This initiative aims to improve access to innovative healthcare technologies while ensuring their safety, quality, and effectiveness. To qualify for inclusion, DTx must meet clinical, legal and technical requirements, including demonstrable benefits to patient- and/or structural-care as defined in § 139e SGB V (German Social Code, Book V) [ 4 ]. These so-called positive healthcare effects refer either to a medical benefit or to patient-relevant improvements in structure and process. In this context, a medical benefit is defined as an effect such as improvement of health status, reduction of disease duration, a prolonged survival or an increased quality of life. Patient relevant structural or procedural improvements may include for example better coordination of care process, easier access to healthcare services, or the promotion of health literacy [ 5 ]. To facilitate timely integration into routine care, the law introduced a dedicated admission pathway. This includes a fast-track procedure that allows the provisional listing for up to twelve months while the manufacturers provide evidence of positive healthcare effects. Although this pathway facilitates market entry, it has raised criticism regarding the rigor and consistency of supporting evidence [ 6 ]. The manufacturers, on the other hand, are free to choose the study design and type of effect they aim to demonstrate, meaning that for example retrospective designs and sole evidence of structural improvements can qualify for listing [ 6 , 7 ]. A systematic review by Dittrich et al. identified limitations in the validity of the studies conducted by the manufacturers, such as considerable variation in study quality, frequent methodological flaws, limited generalizability, and a lack of standardized tools for evaluating DTx in Germany [ 8 ]. Lantzsch et al. further highlighted procedural and methodological shortcomings in the underlying evidence submitted by DTx manufacturers, emphasizing the limited robustness of many clinical evaluations [ 9 ]. More recently, Sippli et al. reviewed all DiGA approval studies published by 15th March 2024 and found a high risk of bias and methodological weaknesses, even though the reported studies were mostly positive. This further strengthens concerns about the robustness of the current evidence base [ 10 ]. While clinical trials provide the basis of regulatory evaluation, they cannot adequately capture the patient-centered aspects of digital applications [ 11 , 12 ]. As emphasized by several studies, patient reviews of mobile health (mHealth) applications can reveal valuable insights into real-world usability, user satisfaction, and recurring quality concerns, all dimensions that are directly relevant to the DiGA approval framework [ 13 – 17 ]. Therefore, analyzing such patient-generated feedback can offer a scalable, real-world supplement to clinical evidence and may highlight practical barriers to adoption and sustained engagement. Importantly, § 139e of the German Social Code, Book V (SGB V) requires that the outcomes of digital health applications include patient-reported health status during usage as part of mandatory success monitoring [ 18 ]. This regulatory emphasis on patient experience is reflected in the BfArM guideline ( DiGA-Leitfaden ), which explicitly allows manufacturers to use comparative quantitative studies based on methods from fields such as social sciences or behavioral sciences to demonstrate positive healthcare effects as valid evidence of a positive healthcare effect, provided that validated instruments are used [ 19 ]. This institutional acknowledgment underscores the regulatory relevance of patient experience and aligns with the central hypothesis of this study: that large-scale patient feedback can serve as an important indicator of e.g., perceived effectiveness and quality. Despite the growing importance of DiGAs, only one study has analyzed patient feedback on German DiGAs including both regulated and non-regulated mHealth application, while covering only 15 DiGAs [ 17 ]. Our study addresses this gap by systematically analyzing patient reviews from the German Apple App Store (iOS) and Google Play Store exclusively for all approved mobile DiGAs. Using a classification framework based on regulatory requirements, we examine how patient report on key categories. As a secondary objective, this study investigates how large language models can be utilized as a foundational methodological approach to support mHealth analysis by reliably categorizing patient feedback according to sentiment and regulatory-based categories. Results Overview of the Dataset As of 25th February 2025, a total of 69 DiGAs were listed in the official directory maintained by the German BfArM. Of these, 45 DiGAs were available as mobile applications (iOS and Android), and 34 offered a web-based version, with several products available in both formats. Since patient reviews are only available for mobile applications distributed through public app stores, the analysis was limited to the mobile app versions of DiGAs. Web-based applications were excluded due to the lack of publicly accessible patient feedback. Although 45 DiGA had a mobile app, one DiGA had no publicly available user reviews and could therefore not be analyzed. Thus, 44 DiGA were included in the analysis. Among them, 29 (65,9%) had been permanently approved, and 15 (34,1%) were listed on provisional basis at the time of data collection. It is important to note that several DTx share a common mobile app. For example, the HelloBetter app includes six distinct DiGAs addressing different indications (e.g., chronic pain, diabetes, sleep disorders, panic, stress, and vaginismus), all delivered through the same app infrastructure. Similarly, the Selfapy app delivers five approved DiGAs (e.g., for depression, bulimia nervosa, generalized anxiety disorder, and chronic pain) through a single mobile application. Accordingly, the 44 DiGAs available as mobile apps are represented by 35 mobile apps. Consequently, the app-based analysis cannot fully differentiate between patient feedback for each individual DiGA when multiple products are bundled within one app. Nevertheless, all reviews associated with such apps were included in the analysis, as they reflect real-world patient experiences with the respective DiGA ecosystem. From these applications, 4,410 publicly accessible user reviews were extracted via the browser-based versions of the German Apple App Store and the German Google Play Store. For the Apple App Store, only up to ten written reviews are publicly visible per app when accessed via browser. During data cleaning and preprocessing, 30 reviews could not be processed further due to incomplete content, parsing errors, or technical parsing errors. These entries were excluded from the analysis. The remaining dataset of 4,380 was fully included in the qualitative and sentiment-based evaluation. Validation of Model Output From the 4,380 included user reviews, a total of 9,494 individual statements were extracted systematically categorized. Each statement was assigned both a sentiment label (positive, neutral, or negative) and a thematic category. The initial classification was conducted by a large language model, followed by the manual validation. During this review, 28 statements were excluded by the reviewers due to irrelevance or ambiguous content. These primarily included of off-topic comments, incomplete sentences, and non-informative content unrelated to the DTx itself – such as emoji-only responses, promotional slogans, or vague remarks like “I want to be a hero. Greatings Josef” , or “just installed” or “don’t know yet” . In addition, 27 statements were flagged as technically erroneous and removed. Together, these exclusions affected a total of 52 user reviews that were not included in the final dataset. As a result, the final dataset comprises 4,328 user reviews containing at least one interpretable statement, and a total of 9,439 valid and interpretable statements. The complete dataset can be found in Appendix II . These form the basis form the basis for all subsequent quantitative and qualitative analyses presented in this study. On average, a review consisted of 213.9 characters and 2.18 relevant statements. A detailed overview of the distribution of valid reviews and statements across individual apps can be found in the Appendix I. Validation of Sentiment and Category Classification To assess the reliability of the automated sentiment and category classification, a manual systematic validation was performed. Quantitative validation demonstrated high classification performance. For sentiment classification, the model achieved an overall accuracy of 99%, excellent precision and recall for positive and negative statements (F1 score: 1.00 and 0.99) and slightly lower performance for neutral statements (F1 = 0.92). The confusion matrix (Fig. 1 ) shows minimal misclassifications, primarily between neutral and negative labels. For category classification, the model reached an overall accuracy of 95%, with an average F1 score of 0.95. Most thematic assignments were consistent with human reviewers’ assessment, particularly for frequently occurring categories such as Overall Impression (F1 = 0.96), Content (F = 0.95), Effectiveness (F1 = 0.94), and Support (F1 = 0.98). However, lower agreement rates were observed for categories with fewer labeled examples, such as Prescription/Approval (F1 = 0.89) and Login/Registration (F1 = 0.92). Figure 2 presents the corresponding confusion matrix, illustrating that most discrepancies occurred within conceptually related categories, for example between UX/Design and Overall Impression or between Technology and Support . Distribution of Sentiments and Categories In total, 6,491 statements (68.7% out of 9,439) were classified as positive, 474 statements (5.0% out of 9,439) as neutral, and 2,474 statements (26.2% out of 9,439) as negative. The majority of positive statements were assigned to the category Content (N = 2,001, 30.8%), followed by Effectiveness (N = 1,440, 22.2%) and Overall Impression (N = 1,233, 18.9%). Among the 474 neutral statements, most were related to Content (N = 145, 30.6%), UX/Design (N = 83, 17.5%), and Effectiveness (N = 68, 14.3%). Negative classifications (N = 2,478) occurred most frequently in categories Technology (N = 564, 22.8%), UX/Design (N = 341, 13.8%), and Login/Registration (N = 265, 10.7%) (Fig. 3 - left ). Due to an exceptionally high number of user reviews for the DTx Zanadio (1,433/4,328, 33.1%), this application was excluded from the following subgroup analysis to avoid potential bias caused by its disproportionate weight in the dataset. Excluding Zanadio , a total of 6,098 valid and interpretable statements remained in the dataset (Fig. 3 - right ). Of these, 3,963 statements (64.9%) were classified as positive , 339 (5.5%) as neutral , and 1,796 (29.4%) as negative . The majority of positive statements referred to Content (N = 1,229, 24.8%), followed by Effectiveness (N = 1,045; 21.0%) and Overall Impression (N = 807, 16.3%). Among the 339 neutral statements, most were assigned to Content (N = 109, 32.2%), UX/Design (N = 58, 17.1%), and Effectiveness (N = 49, 14.4%). The largest share of negative classifications (N = 1,796) where related to Technology (N = 418, 23.3%), UX/Design (N = 249, 13.9%), and Login/Registration (N = 189, 10.6%). In addition to the aggregated results, sentiment distributions were also analyzed separately for each individual DTx included in the dataset. The detailed graphical breakdown of sentiment and thematic categories per app is provided in the Appendix III for reference. This allows a more granular exploration of potential differences in patient feedback across specific DTx products, beyond the aggregated trends presented in the result section. To further investigate potential quality changes over time, we compared user feedback before and after official listing in the BfArM directory for the ten most reviewed DiGAs. For each app and category, chi-squared test was conducted. No significant difference in sentiment distributions were identified, indicating no substantial shifts in patient-perceived app quality following formal inclusion. Categories The following sections provide a detailed overview of patient feedback for three selected categories: Overall impression , Effectiveness , and Login/ Registration . These categories were chosen due to their high frequency and relevance across the dataset. Analyses of additional categories, such as UX/Design , Tracking/ Documentation , Technology , Support , Prescription/ Approval , Cost/ Reimbursement , and Content are provided in Appendix II for reference. All referenced user statements were translated into English using DeepL (July 2025) , preserving their original meaning. Overall impression A total of 1,431 user statements were categorized under Overall impression, providing general patient feedback regarding the DTx experience. Positive reviews frequently emphasized the usefulness, clarity, and motivational aspects of the applications. One patient stated: “A great program. Motivating. Can be very well adapted to personal needs. Definitely recommendable!” (Sebastian W., Kaia Rückenschmerzen, Google Play Store). Another patient commented: “Digital health applications to understand and ultimately get a grip on men’s health issues described in the app. […]. The app is intuitive and clearly structured, I can recommend it 100%. Thumbs up!” (Nkgvfjkbhh, Kranus Edera, Apple App Store). Another patient stated: “I can highly recommend this app to anyone with migraines! Investigating the connection between migraines and blood sugar was very interesting. The app is clear and easy to use” (Marcellaelena, sinCephalea, Apple App Store). Neutral sentiments reflected initial or uncertain patient experiences, as in the following statement: “The start is okay” (Gerhard K., NichtraucherHelden, Google Play Store) and “I’m not really convinced yet” (RügenVan, Meine Tinnitus App, Google Play Store). Reviews with negative statements often criticized technical problems or unmet expectations, for instance: “Unfortunately, the app is disastrous. Not only does it run poorly, but it also fails to address personal problems because the algorithm cannot be adjusted. So far, I can only advise against it” (Paule M, Cara Care, Google Play Store). Another patient wrote “The app provides common knowledge and helps document sleep patterns. But it does not replace therapy. It didn’t help me” (Jochen K., somnio, Google Play Store). Login/Registration In total, 280 user statements addressed the login and registration process. Positive feedback frequently highlighted an easy onboarding experience and simple account handling. For example, a patient stated: “Great app for endometriosis beginners” I’ve only just started using it, but the Endo-App has already impressed me. The simple registration and clear profile make getting started easy” (André K., Endo-App, Google Play Store). Similarly, a patient emphasized “Simple handling, quick login, attentive team. Good exercises that can be done anywhere – whether at home, in the gym, or at the office” (Biscuit de l’empereur, ViViRA bei Rückenschmerzen, Google Play Store). Neutral reviews often reflected suggestions for improvement alongside generally positive impressions. One patient reported: “Very helpful! The app’s exercises clearly help. But you have to spend a lot of time reading through everything at the beginning. Suggestion: save login data, at least the email address” Martin K., Orthopy, Google Play Store). Negative experiences primarily focused on technical login issues or complex authentication procedures. One patient described: “I used to like the app. Unfortunately, after recent changes, I can’t access my account at all. Constantly having to log in again is annoying, and now I can’t access my data at all” (Nadine A., Cara Care, Google Play Store). Another patient reported: ”I also can’t log in, seems to be a known problem. Regardless of the email provider, I’ve never had such issues before. Maybe one day I’ll be able to use it” (Hanna K, Endo-App, Google Play Store). Further negative statements assigned to this category criticized: “The information content and exercises are very good. What annoys me is the two-step login process. Totally unnecessary in my opinion – it’s not a bank account” (Stefan C., Kranus Lutera, Google Play Store). Effectiveness The category Effectiveness received 1,609 user statements, making it one of the most mentioned aspects of the DTx evaluations. Many patients reported positive effects on their health effects, highlighting improved well-being, symptom relief, or successful integration of the app into daily routines. For instance, one patient stated: “I’m really satisfied and positively surprised. The app offers so many features – recipes, different sports options from easy to challenging, advisors you can contact anytime. Overall, it really helped me. Thank you!” (Andrea Z., zanadio, Google Play Store). Similarly, patients emphasized positive feedback like the following statement: “I struggled for a long time with falling and staying asleep. This app really helped. I fall asleep faster and stay asleep more often. It’s well structured, and everything can be analyzed. Really Great” (Peter H., somnio, Google Play Store). Neutral reviews often reflected mixed impressions or external limitations affecting the expected effectiveness. For example, a patient commented: “The reminders can be a bit annoying but are necessary. Good exercises and food for thought… though sometimes I lack the motivation to follow through” (Familie S., HelloBetter, Google Play Store). Similarly, a patient described his DTx as supportive but limited: “The app helps to better understand depressions and take first steps. But it requires discipline and cannot replace processional medical help” (Thom-05, MindDoc, Apple Store). Reviews with negative statements often expressed disappointment regarding insufficient health improvements. One patient stated: “Great tips, but unfortunately it didn’t help me. Still, thanks” (Jasmin M., NichtraucherHelden, Google Play Store). A more critical assessment mentioned: “Not recommended and extremely expensive. Despite completing all exercises, my condition hasn’t improved. The health insurance paid over 600€ for this. A physiotherapist would have explained everything better and been cheaper. Plus, the app is poorly designed and buggy” (Skippi, Kranus Lutera, Google Play Store). Discussion In this study, we conducted a qualitative large-scale analysis of user reviews related to digital therapeutics (DTx) using a large language model for sentiment analysis and thematic category classification. To our knowledge, this represents the largest qualitative analysis of patient feedback on reimbursable DTx. The final dataset comprised 4,328 user reviews covering 44 DTx and 35 mobile applications listed in the official German DiGA directory. These reviews were extracted from the German Apple (iOS) and Android App Stores and processed using a structured prompt-based approach. In total, the model identified and classified 9,439 individual statements across ten predefined categories derived from legal and regulatory criteria for DiGAs in Germany. The model achieved 99% sentiment accuracy, with slightly lower performance for neutral statements (F = 0.92), which often contained mixed tones like polite suggestions or critical requests. For category classification, the LLM reached an overall accuracy of 95% with an average F1 score of 0.95. While categories such as Overall Impression (F1 = 0.96) or Content (F = 0.95) showed high agreement with human validation, categories such as Prescription/ Approval (F1 = 0.89) and Login/Registration (F1 = 0.92) showed lower agreement rates. This may be the case as the prescription and approval processes are directly linked to the registration and onboarding process, as the patients receive a registration code for their prescribed DTx from their health insurer. From the patient’s perspective, this might be interpreted as a single process, as articulated accordingly in many reviews, leading to misclassifications. In only four cases did the model exceed the specified five statements per review, which is negligible given the number of reviews and could not lead to bias nor affect reproducibility. The manual validation ensured consistency but may have introduced an acceptance bias toward plausible model predictions: a lack of blinding during initial validation steps could have reinforced the model’s suggestions. Future evaluations should consider blinded or disagreement-focused review procedures to limit such a bias. The sentiment analysis showed that with 6,491 positive statements (68.7% out of 9,439) the majority of the statements were in favor of the DTx, showing a broad acceptance of the digital therapeutic approach. Despite generally positive sentiment, full-text reviewers tended to give lower star ratings ( Appendix I ). With a total of 2,474 (26.2% out of 9,439) negative statements, those patients not only use their reviews to praise the DTx, but also to criticize the applications and report frustration with processes, bugs or general dissatisfaction. This was particularly evident in the categories Technology , Login/Registration and Prescription/Approval . For instance, patients tended to report frustration if the registration process was prune to bugs or when daily logins required multiple authentication steps that hindered the integration into their daily routine. Complex multifactor authentication can contradict low-threshold access as required by national digital health regulations with respect to the average age of DiGA users (Ø 55–60 [ 6 ]). While these authentication requirements are aligned with national cybersecurity standards defined by BSI (German Federal Office for Information Security), frequent user criticism suggests a need to balance regulatory security demands with usability considerations tailored to the target demographic. With 564 of 669 (84.3%) technology-related statements being negative, patients frequently expressed frustration about bugs and app malfunctions. As DTx are regulated medical devices, such deficits raise concerns about quality assurance, suggesting the need for improved quality or release management by the vendors [ 20 ]. Conversely, patients tend to be more motivated to share negative experiences, particularly when they encounter disappointment or frustration [ 13 , 21 ]. This study did not examine when negative reviews on the subject of technology were written, e.g., during the launch phase, post-launch, or maturity phase [ 22 ]. At the same time, basic functions are often an implicit requirement from the patient's perspective, which is why they do not receive any specific praise. Requirements Engineering theory classifies such functionalities as "basic requirements" or "must-be" features, whose absence or malfunction commonly leads to negative reviews [ 23 ]. Another category predominantly characterized by negative feedback is the Approval or Prescription process. While the prescription is issued by a physician, approval is granted by health insurance providers based on corresponding medical indications. Many patients criticize the length of the approval process by health insurers. Additionally, patients report that even with valid prescriptions, digital therapy may be denied due to additional criteria (such as BMI thresholds for weight loss applications), which patients frequently describe as frustrating. Furthermore, patients frequently mention the standard prescription period of 90 days. They express that the recurring prescription process for DTx is perceived as burdensome. On the other hand, some patients report disappointment because they cannot access the app without a valid prescription. Additionally, patients frequently comment that they would like to "try out" the app before applying for a prescription. This aligns with medical practices where trial periods help assess treatment suitability. Such trial periods could help both patients and healthcare professionals assess whether the chosen therapy is suitable and worth continuing. Accordingly, a limited trial version could facilitate more informed decisions, improve acceptance, and prevent frustration or early discontinuation, resulting in more efficient use of healthcare resources. Ultimately, patients criticized the high costs and questioned the DTx cost-benefit ratio. In contrast, the categories Overall Impression , Effectiveness , Support and Content were characterized by predominantly positive statements. In the category Overall Impression (1,233/1,431, 86.0% positive), patients frequently praise their DTx in general, highlight high satisfaction with the apps or recommend it to patients with similar indications. Considering that patients are generally more likely to leave feedback when they encounter problems or frustration [ 24 ], the high involvement in positive evaluations of the Overall Impression can be interpreted as patient reported real-world evidence of user satisfaction – an essential quality indicator required by national digital health regulations [ 25 ]. Furthermore, the category Effectiveness is rated positively by the majority (89.4%). Patients report a noticeable improvement in symptoms, an improvement of their health literacy and better management of their illness resulting in improvements of their quality in life, as expected by effectiveness monitoring frameworks in DTx regulation. These findings align closely with the legal requirements in § 139e Abs. 13 SGB V, which highlights that results from accompanying effectiveness measurements should particularly include the patient-reported health status during the use of DTx [ 26 ]. Lastly, the positive statements regarding the category Content also showed an overall high patient satisfaction (2,002/2,566, 78.0%). Patients frequently valued the diverse and well-prepared presentation of content, while only a few commented on its scientific grounding. The predominantly positive feedback nevertheless suggests that patients perceive the content as credible and effective, indicating that the applications mostly meet the quality requirements for evidence-based, safe, and user-appropriate health information, as mandated by national regulatory frameworks [ 26 , 27 ]. The overall findings demonstrate that proposed category scheme is well-suited for structuring patient-reported feedback on DTx for content-related analyses and interpretation. A comparable study conducted by Uncovska et al. analyzed both 15 regulated DiGAs and non-prescription mHealth apps from public app stores, using BERTopic for unsupervised topic modelling on 17,588 user reviews [ 17 ]. Their findings showed that DiGAs had higher patient ratings, with positive themes around customer service and ease of use, while technical issues like registration challenges were also prominent. In contrast, our study focuses solely on patient feedback for 44 officially listed DiGAs, using a classification framework directly conceptualized based on legal and regulatory criteria, and manual validation. This allows for a more targeted evaluation of patient experience as it specifically relates to DiGA requirements. Our method thus complements and extends the insights of Uncovska et al. by providing regulatory alignment, also covering a broader sample of DiGAs. Due to the user-friendliness of ChatGPT 4-o, large-scale evaluation is possible with comparatively low technical barriers and resource requirements. Given the high classification accuracy of the model, further investigations using a more fine-grained category scheme are recommended, to gain nuanced insights that remain undetected by the current classification setup. A follow-up study will explore patient loyalty based on usage duration and prescription frequency. In addition, we aim to differentiate between technical and content-related aspects of user support and expand our category scheme by incorporating aspects identified by Haggag et al., such as reasons for uninstallation or statements related to data privacy and data security [ 13 ]. Our study evaluated patients’ experience with the DTx using a Large Language Models to scale the analysis. We could show that large language models paired with human review, can efficiently process and categorize large volumes of unstructured patient feedback. This approach may provide regulators, manufacturers, and researchers with timely, patient-centered insights that are otherwise difficult to capture through clinical trials alone. From a content perspective, our findings highlight both the potential and the current limitations of reimbursable DTx from the patient’s perspective. While patients value the therapeutic benefits and content quality of many DTx, basic technical functionality and usability, particularly during login and registration, are frequently criticized. These processes are perceived as fundamental requirements for medical-grade products and if unmet, lead to rejection and ultimately to a negative therapeutic outcome. To ensure equitable access and sustained engagement, registration procedures should be standardized across applications, and login processes must be truly barrier-free. Furthermore, patients demonstrate awareness of healthcare spending and critically assess the value provided by reimbursed applications, especially when functionality or therapeutic benefit appear limited. As suggested by some patients, offering limited trial version prior to prescription could support more informed decision-making and potentially increase patient acceptance. Additionally, we recommend that regulatory bodies implement structured, scenario-based usability testing with representative patient groups during both pre-approval phases and ongoing post-market surveillance. This approach could help identify and address technical or accessibility barriers proactively, ensuring that reimbursable DTx meet the regulatory and practical needs of diverse patient groups throughout their lifecycle. Future efforts should therefore focus on improving technical reliability, streamlining access procedures and incorporating patient preferences into both design and pricing strategies to enhance both acceptance as well as long-term engagement with DiGA. Methods Data Identification and Extraction Data collection involved two stages: first, we identified all mobile DTx listed by the Federal Institute for Drugs and Medical Devices as of February 25th 2025. Second, we used Selenium-based Python scripts (v4.29.0) to extract user reviews from the German Apple App Store and Google Play Store. Extracted data included review content, ratings, dates, developer responses, and app metadata, stored in structured JSON format (Fig. 4 ). Review Classification and Sentiment Framework Our classification framework was based on legal and regulatory requirements for DiGAs in Germany, including SGB V § 139e, the Digital Health Applications Ordinance (DiGAV), and the official DiGA Guide published by BfArM. The derived categories are: Overall Impression, Prescription/Approval, Content, Cost/Reimbursement, Login/Registration, Support, Technology, Tracking/Documentation, UX/Design, Effectiveness. The category Tracking/Documentation was included despite the regulation not yet being in force, as patient-reported outcomes (e.g., § 139e (13) SGB V) are anticipated to become increasingly relevant for DiGA assessment. A detailed explanation of the categorization framework, including corresponding DiGA criteria and legal references in German can be found in Appendix I . Each user review was analyzed for up to five thematically distinct statements. Model-Based Sentiment and Category Assignment To systematically assess user sentiment, we developed a custom Python script to automate the classification process using the GPT-4o API provided by OpenAI (Step 1–3). Each user review was processed individually using a carefully structured prompt designed to guide the model toward consistent and domain-specific classification (see Appendix IV ). The model was instructed to adopt the role of an expert in evaluating user experiences with digital health applications (DiGAs). Within this role, it was given a multi-step task: Extract up to five core statements from each user review Assign each statement a sentiment label (positive, neutral, or negative) Assign each statement to exactly one of ten predefined thematic categories derived from the legal and quality criteria for DiGAs (see Appendix I ) Merge statements that refer to the same theme and sentiment into a single, summarized expression Avoid repeating partial aspects of the same issue across multiple statements Limit each category to one statement per review, unless clearly distinct subtopics are present Exclude overall evaluations if the review indicates the app could not be used (e.g., due to failed registration or missing insurance approval) The model was provided with a fixed list of ten valid categories, accompanied by specific guidance on ambiguous cases. For example, access issues after prescription or insurer approval were to be categorized as Login/Registration, not Prescription/Approval. Finally, the model was instructed to return its output in a standardized, structured format with numbered statements, each containing three elements: the extracted statement , sentiment label , and assigned category . A representative example was included in the prompt to enforce consistent formatting and interpretation. Statements were required to be thematically distinct and clearly assigned. The script processed each review individually and parsed the model's structured response into a table of statements with their associated sentiment and category (Step 4–5). All results were exported in a standardized CSV format for further manual validation (Step 6). By assigning the model a clearly defined expert perspective and domain logic, we ensured that the sentiment analysis aligned with both linguistic consistency and health service evaluation standards. A translated example of this classification process in visualized in Fig. 5 . Human Validation of Model Output To evaluate the accuracy and conceptual validity of the model-generated classifications, we conducted a structured, validation process as shown in Fig. 6 . The full set of reviews, including original review, extracted statement, sentiment label, and categories, was evenly divided among three authors (Step 7), each of whom independently reviewed one third of the dataset (Step 8). The validation process followed a four-step protocol: Independent review : Each model-generated statement and its assigned sentiment and category were assessed in relation to the original review. Reviewers evaluated whether the extracted statement captured a meaningful aspect of the review and if the sentiment and category were accurate according to the predefined classification framework. Flagging of inconsistencies : Statements that were incomplete, overly generic, redundant, or incorrectly classified (e.g., in sentiment or category) were flagged. Each flagged item included a brief comment and a suggested correction, if appropriate. Triangulated reassessment : The two remaining authors independently evaluated the flagged items, providing their own classifications without being influenced by the previous assessments (Step 9). This resulted in three independent judgments per flagged entry. Consensus resolution : In cases of disagreement, the authors discussed the item until a final classification was agreed upon (Step 10). Validated statements were finalized and included in the analysis as a consolidated CSV dataset (Step 11). Structurally flawed outputs (e.g., empty responses or parsing errors) were reprocessed. The manual validation ensured consistency with the classification framework and enabled direct evaluation of the model on real-world data. Abbreviations BfArM : German Federal Institute for Drugs and Medical Devices DiGA: ger: Digitale Gesundheitsanwendung/ eng: Digital Health Application DiGAV : DiGA Verordnung DTx : Digital Therapeutics LLM: Large Language Model SGB V : German Social Code, Book V UX: User Experience Declarations Conflict of Interest The authors declare that they have no competing interests. This research was conducted independently and the analysis and interpretation of data were performed solely by the authors. Acknowledgemaents The authors would like to thank the colleagues involved in the discussions on digital health regulation and evaluation for their valuable insights. Funding Statement This research was supported by internal funds from the Medical Faculty of Kiel University. No specific grant number is associated with this support. Author Contributions BK developed the methodological framework, created the category scheme, conducted the data extraction and analysis, participated in the human validation of LLM output provided and the initial and final manuscript. HU contributed expertise in large language models (LLMs), selected the validation method, participated in the human validation of LLM output and revised the manuscript. HR participated in the human validation of LLM output. BS contributed expertise in mHealth and reviewed the initial and final manuscript. All authors approved the final manuscript. Data Availability All review data, the category scheme, detailed per-app analyses, and the initial LLM prompt are available in the supplementary information files. Multimedia Appendix Appendix I: Legal Basis and Results for User Feedback Categories in DiGA App Store Reviews Appendix II: App Store User Reviews Appendix III: Category and Sentiment Analysis per App Appendix IV: LLM-Prompt References Driving the digital transformation of Germany’s healthcare system for the good of patients, (n.d.). https://www.bundesgesundheitsministerium.de/en/digital-healthcare-act.html (accessed August 11, 2025). L. Schmidt, M. Pawlitzki, B.Y. Renard, S.G. Meuth, and L. Masanneck, The three-year evolution of Germany’s Digital Therapeutics reimbursement program and its path forward, Npj Digit. Med. 7 (2024) 139. doi:10.1038/s41746-024-01137-1. Bundesinstitut für Arzneimittel und Medizinprodukte (BfArM), DiGA-Verzeichnis – Digitale Gesundheitsanwendungen, BfArM – Bundesinstitut für Arzneimittel und Medizinprodukte . (n.d.). https://diga.bfarm.de/de/verzeichnis (accessed March 15, 2025). § 139e SGB V – Verzeichnis digitaler Gesundheitsanwendungen, 2019. https://www.gesetze-im-internet.de/sgb_5/__139e.html. Bundesministerium für Gesundheit, Digitale-Gesundheitsanwendungen-Verordnung (DiGAV), 2020. https://www.gesetze-im-internet.de/digav/__8.html (accessed May 10, 2025). GKV-Spitzenverband, Bericht nach § 139e Absatz 10 SGB V zur Nutzung, Akzeptanz und Wirkung digitaler Gesundheitsanwendungen – DiGA-Bericht 2024, GKV-Spitzenverband, Berlin, 2025. https://www.gkv-spitzenverband.de/media/dokumente/krankenversicherung_1/telematik/digitales/2024_DiGA-Bericht_final.pdf (accessed July 22, 2025). M. Mäder, P. Timpel, T. Schönfelder, C. Militzer-Horstmann, S. Scheibe, R. Heinrich, and D. Häckl, Evidence requirements of permanently listed digital health applications (DiGA) and their implementation in the German DiGA directory: an analysis, BMC Health Serv Res . 23 (2023) 369. doi:10.1186/s12913-023-09287-w. F. Dittrich, A. Mielitz, E. Pustozerov, D. Lawin, U. Von Jan, and U.-V. Albrecht, Digital health applications from a government-regulated directory of reimbursable health apps in Germany—a systematic review for evidence and bias, mHealth . 9 (2023) 35–35. doi:10.21037/mhealth-23-17. H. Lantzsch, H. Eckhardt, A. Campione, R. Busse, and C. Henschke, Digital health applications and the fast-track pathway to public health coverage in Germany: challenges and opportunities based on first results, BMC Health Serv Res . 22 (2022) 1182. doi:10.1186/s12913-022-08500-6. K. Sippli, S. Deckert, J. Schmitt, and M. Scheibe, Healthcare effects and evidence robustness of reimbursable digital health applications in Germany: a systematic review, Npj Digit. Med. 8 (2025) 495. doi:10.1038/s41746-025-01879-6. E. Bolinger, and B. Tyl, Key Considerations for Designing Clinical Studies to Evaluate Digital Health Solutions, J Med Internet Res . 26 (2024) e54518. doi:10.2196/54518. C. Guo, H. Ashrafian, S. Ghafur, G. Fontana, C. Gardner, and M. Prime, Challenges for the evaluation of digital health solutions—A call for innovative evidence generation approaches, Npj Digit. Med. 3 (2020) 110. doi:10.1038/s41746-020-00314-2. O. Haggag, J. Grundy, M. Abdelrazek, and S. Haggag, A large scale analysis of mHealth app user reviews, Empir Software Eng . 27 (2022) 196. doi:10.1007/s10664-022-10222-6. R. Vasa, L. Hoon, K. Mouzakis, and A. Noguchi, A preliminary analysis of mobile app user reviews, in: Proceedings of the 24th Australian Computer-Human Interaction Conference, ACM, Melbourne Australia, 2012: pp. 241–244. doi:10.1145/2414536.2414577. Li Xiaozhou, Zhang Zheying, and Stefanidis Kostas, Mobile App Evolution Analysis Based on User Reviews, in: Frontiers in Artificial Intelligence and Applications, IOS Press, 2018. doi:10.3233/978-1-61499-900-3-773. P.L. Kolominsky-Rabas, M. Tauscher, R. Gerlach, M. Perleth, and N. Dietzel, Wie belastbar sind Studien der aktuell dauerhaft aufgenommenen digitalen Gesundheitsanwendungen (DiGA)? Methodische Qualität der Studien zum Nachweis positiver Versorgungseffekte von DiGA, ZEFQ . 175 (2022) 1–16. doi:10.1016/j.zefq.2022.09.008. M. Uncovska, B. Freitag, S. Meister, and L. Fehring, Rating analysis and BERTopic modeling of consumer versus regulated mHealth app reviews in Germany, Npj Digit. Med. 6 (2023) 115. doi:10.1038/s41746-023-00862-3. § 139e Abs. 13 Nr. 3 Fünftes Sozialgesetzbuch (SGB V), 2019. https://www.gesetze-im-internet.de/sgb_5/__139e.html (accessed August 10, 2025). Bundesinstitut für Arzneimittel und Medizinprodukte (BfArM), DiGA-Leitfaden. Das Fast-Track-Verfahren für digitale Gesundheitsanwendungen nach § 139e SGB V, BfArM, Bonn, 2023. https://www.bfarm.de/SharedDocs/Downloads/DE/Medizinprodukte/diga_leitfaden.html?nn=597198 (accessed July 7, 2025). L. Schramm, and C.-C. Carbon, Critical success factors for creating sustainable digital health applications: A systematic review of the German case, DIGITAL HEALTH . 10 (2024) 20552076241249604. doi:10.1177/20552076241249604. D. Pagano, and W. Maalej, User feedback in the appstore: An empirical study, in: 2013 21st IEEE International Requirements Engineering Conference (RE), IEEE, Rio de Janeiro-RJ, Brazil, 2013: pp. 125–134. doi:10.1109/RE.2013.6636712. ISO/IEC/IEEE International Standard - Systems and software engineering – Life cycle processes – Requirements engineering, ISO/IEC/IEEE 29148:2018(E) . (2018) 1–104. doi:10.1109/IEEESTD.2018.8559686. E. Hull, K. Jackson, and J. Dick, Requirements engineering, 2nd ed., Springer, London, 2005. http://gso.gbv.de/DB=2.1/PPNSET?PPN=589178075. E.W. Anderson, Customer Satisfaction and Word of Mouth, Journal of Service Research . 1 (1998) 5–17. doi:10.1177/109467059800100102. § 139e Abs. 13 Nr. 2 Fünftes Sozialgesetzbuch (SGB V), 2019. https://www.gesetze-im-internet.de/sgb_5/__139e.html (accessed August 10, 2025). § 5 Abs. 8 Satz 2 Digitale Gesundheitsanwendungen-Verordnung (DiGAV), 2020. https://www.gesetze-im-internet.de/digav/__5.html (accessed July 7, 2025). § 139e Abs. 2 Fünftes Sozialgesetzbuch (SGB V), 2019. https://www.gesetze-im-internet.de/sgb_5/__139e.html (accessed July 7, 2025). Additional Declarations No competing interests reported. Supplementary Files AppendixICategoriesComparisonofRatingsLegalBasisV02.pdf Appendix I: Legal Basis and Results for User Feedback Categories in DiGA App Store Reviews AppendixIIUserReviewsV02.xlsx Appendix II: App Store User Reviews AppendixIIICategoryandSentimentAnalysisperApp.pdf Appendix III: Category and Sentiment Analysis per App AppendixIVPromptv02.pdf Appendix IV: LLM-Prompt Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-7922462","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":542500874,"identity":"e72ff747-2bcb-4c94-9f11-99a639a4410f","order_by":0,"name":"Benjamin Kinast","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABBElEQVRIiWNgGAWjYBACAwiWYGDgYWwACcgx8DCwMYAQsVoMjInSAgE8EH5iAyEt5uy9Bwp+5ljkMfAcbnvw48+f9A1njj97wFBmg1OLZc+5BMPebRLFDLyN7Ya9bQa5G872mBswnEvD7bAbOQYGvNskEhv4GdskeBuAWs7zsEkwth3GreX+GwPDv1Atkn/+GKQbnGd/BtTyH48tPAbGYFt4G9ukedgMEgzONpgBtRzA45ccA2NZoJY2noNt0rJtxoYzz5wxk0g4l4xTizn7GTPDt9vqEvt50p9JvvkjJ893Jv2ZxIcyO5xagIANHDeoEZGATwMDA/MD/PKjYBSMglEw4gEAzT1QoRMU0GgAAAAASUVORK5CYII=","orcid":"","institution":"Kiel University and University Hospital Schleswig-Holstein","correspondingAuthor":true,"prefix":"","firstName":"Benjamin","middleName":"","lastName":"Kinast","suffix":""},{"id":542500876,"identity":"bee58409-be97-4fd7-a654-05ce91261007","order_by":1,"name":"Henrik Rohde","email":"","orcid":"","institution":"Kiel University and University Hospital Schleswig-Holstein","correspondingAuthor":false,"prefix":"","firstName":"Henrik","middleName":"","lastName":"Rohde","suffix":""},{"id":542500877,"identity":"59590ace-5f56-4d2d-90a0-f0c90c2a03c6","order_by":2,"name":"Björn Schreiweis","email":"","orcid":"","institution":"Kiel University and University Hospital Schleswig-Holstein","correspondingAuthor":false,"prefix":"","firstName":"Björn","middleName":"","lastName":"Schreiweis","suffix":""},{"id":542500878,"identity":"575981bd-54d7-46c4-bbac-fb3a891f3b05","order_by":3,"name":"Hannes Ulrich","email":"","orcid":"","institution":"Kiel University and University Hospital Schleswig-Holstein","correspondingAuthor":false,"prefix":"","firstName":"Hannes","middleName":"","lastName":"Ulrich","suffix":""}],"badges":[],"createdAt":"2025-10-22 10:23:38","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-7922462/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-7922462/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":95667336,"identity":"dedcf98e-4de8-4061-aed4-68b08df6dac0","added_by":"auto","created_at":"2025-11-11 16:55:04","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1675335,"visible":true,"origin":"","legend":"","description":"","filename":"DiGASentimentmanuscriptV15.7.docx","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/9475cbd97531c9a8f23ceb5e.docx"},{"id":95667333,"identity":"2b35ab49-85e2-4aa0-a757-b4ca60e25837","added_by":"auto","created_at":"2025-11-11 16:55:04","extension":"json","order_by":1,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":7458,"visible":true,"origin":"","legend":"","description":"","filename":"f0c14465d65a47fa92c7b31ad20bc1b8.json","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/c6f187e60e43b365e3cbe7a7.json"},{"id":95667338,"identity":"9ca3176d-049d-4fb1-adfe-62b33d285c24","added_by":"auto","created_at":"2025-11-11 16:55:04","extension":"pdf","order_by":2,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":105504,"visible":true,"origin":"","legend":"","description":"","filename":"AppendixIVPromptv02.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/17b6046767542473f0d78788.pdf"},{"id":95667343,"identity":"7cdb150b-f521-42a5-9e77-64fcab652313","added_by":"auto","created_at":"2025-11-11 16:55:04","extension":"pdf","order_by":3,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":104616,"visible":true,"origin":"","legend":"","description":"","filename":"AppendixIIICategoryandSentimentAnalysisperApp.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/4a3b8f67437444af10a7e716.pdf"},{"id":95797826,"identity":"952aaf06-be94-4f46-907d-b1778fda8f0b","added_by":"auto","created_at":"2025-11-13 08:11:52","extension":"xlsx","order_by":4,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1206773,"visible":true,"origin":"","legend":"","description":"","filename":"AppendixIIUserReviewsV02.xlsx","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/58325f4d33224cf37beb3707.xlsx"},{"id":95667344,"identity":"09a461d5-b502-4267-99cf-dac3f1a6b8b7","added_by":"auto","created_at":"2025-11-11 16:55:04","extension":"pdf","order_by":5,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":1033875,"visible":true,"origin":"","legend":"","description":"","filename":"AppendixICategoriesComparisonofRatingsLegalBasisV02.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/b86144b7dd5493f318ba33c2.pdf"},{"id":95799133,"identity":"f3ca94ba-9fdf-4c28-bc4a-6663cb513e1b","added_by":"auto","created_at":"2025-11-13 08:18:52","extension":"xml","order_by":6,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":91938,"visible":true,"origin":"","legend":"","description":"","filename":"f0c14465d65a47fa92c7b31ad20bc1b81enriched.xml","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/594225f37c78d2bd308d36d3.xml"},{"id":95667362,"identity":"a3f48026-0b2e-436b-bed2-7de65083590f","added_by":"auto","created_at":"2025-11-11 16:55:05","extension":"png","order_by":7,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":45647,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/5340b4deb73ada71274eaf70.png"},{"id":95667345,"identity":"e693a07c-11c0-4c93-b0b1-aed09769baea","added_by":"auto","created_at":"2025-11-11 16:55:04","extension":"png","order_by":8,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":145506,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/0298aa5f824baf8838f32e4c.png"},{"id":95667341,"identity":"509a86f6-1341-4d48-94d1-95bbdf22bcd5","added_by":"auto","created_at":"2025-11-11 16:55:04","extension":"png","order_by":9,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":124439,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/33914e2f1ee1b20f7f4ca5df.png"},{"id":95667346,"identity":"f03574dd-2e9a-4955-b6d6-9e0e392d6966","added_by":"auto","created_at":"2025-11-11 16:55:04","extension":"png","order_by":10,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":112108,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/e01718036766c242e737cbd9.png"},{"id":95667359,"identity":"16dd09e9-7141-4ae6-ab51-189927e5b587","added_by":"auto","created_at":"2025-11-11 16:55:04","extension":"png","order_by":11,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":275788,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/70d96bb8a2a7287939f11efc.png"},{"id":95798424,"identity":"fa125dc7-b7f5-402b-9cb5-82ce7b3a7c07","added_by":"auto","created_at":"2025-11-13 08:16:47","extension":"png","order_by":12,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":122482,"visible":true,"origin":"","legend":"","description":"","filename":"floatimage6.png","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/f652e79504cb523a2dfd133d.png"},{"id":95798354,"identity":"856d681e-7524-4db3-99bb-ed3d52e33ec7","added_by":"auto","created_at":"2025-11-13 08:16:31","extension":"png","order_by":13,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":10912,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/4a88524230344f89cf3afa36.png"},{"id":95667360,"identity":"a244309c-6089-4ab7-8b54-b269069feed6","added_by":"auto","created_at":"2025-11-11 16:55:05","extension":"png","order_by":14,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":30956,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/56981f4a9da6baa9439e753d.png"},{"id":95667353,"identity":"a5e38161-7289-43a3-817d-210f43226e2a","added_by":"auto","created_at":"2025-11-11 16:55:04","extension":"png","order_by":15,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":29226,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage3.png","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/c0258001ab5689d8354342f4.png"},{"id":95667356,"identity":"8ab26b2c-90ab-46b7-a113-2321c9721afc","added_by":"auto","created_at":"2025-11-11 16:55:04","extension":"png","order_by":16,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":28247,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/7b77cf71d4f6bb7ebdddfbad.png"},{"id":95667348,"identity":"f9fd5747-bd27-446c-a0f0-2b6ebef8e898","added_by":"auto","created_at":"2025-11-11 16:55:04","extension":"png","order_by":17,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":66083,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage5.png","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/0fdb53e60cc79ecd6ff9dbe4.png"},{"id":95667347,"identity":"07a8e0ef-0fa9-4391-a521-8d84dc0030ff","added_by":"auto","created_at":"2025-11-11 16:55:04","extension":"png","order_by":18,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":51742,"visible":true,"origin":"","legend":"","description":"","filename":"Onlinefloatimage6.png","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/75ccd9d13e2fe2109929450b.png"},{"id":95667355,"identity":"dad6de47-ae8c-4f95-92ff-c8412fba2991","added_by":"auto","created_at":"2025-11-11 16:55:04","extension":"xml","order_by":19,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":92675,"visible":true,"origin":"","legend":"","description":"","filename":"f0c14465d65a47fa92c7b31ad20bc1b81structuring.xml","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/02fc11dc140735ce997820f9.xml"},{"id":95667364,"identity":"38f365a8-a8cc-42be-b383-a46755d35217","added_by":"auto","created_at":"2025-11-11 16:55:05","extension":"html","order_by":20,"title":"","display":"","copyAsset":false,"role":"acdc-reference","size":103570,"visible":true,"origin":"","legend":"","description":"","filename":"earlyproof.html","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/3f6810e66b540453c7c82c9f.html"},{"id":95667335,"identity":"57b637f8-4acc-4e97-a4bd-7cab3af9a163","added_by":"auto","created_at":"2025-11-11 16:55:04","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":52821,"visible":true,"origin":"","legend":"\u003cp\u003eConfusion Matrix - Sentiment\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/85491e4b921cdf1da30217e6.png"},{"id":95797226,"identity":"62b4a3e0-6c62-4c8a-b832-a0c4b494f657","added_by":"auto","created_at":"2025-11-13 08:01:59","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":116172,"visible":true,"origin":"","legend":"\u003cp\u003eConfusion Matrix - Categories\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/7a13b56764a50dac0c6d9a15.png"},{"id":95667340,"identity":"b92d4d1e-0dac-4121-8311-16d092015924","added_by":"auto","created_at":"2025-11-11 16:55:04","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":124439,"visible":true,"origin":"","legend":"\u003cp\u003eSentiment Distribution per Category across All Apps (left) and excluding Zanadio (right).\u003c/p\u003e","description":"","filename":"floatimage31.png","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/7984c6bbbceb4eb21abb3d4d.png"},{"id":95667351,"identity":"812c0e6a-319a-4708-8c45-06ad4db27c03","added_by":"auto","created_at":"2025-11-11 16:55:04","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":126536,"visible":true,"origin":"","legend":"\u003cp\u003eReimbursable Digital Therapeutics Identification\u003c/p\u003e","description":"","filename":"floatimage4.png","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/522c18d915553075cb59a6ed.png"},{"id":95667357,"identity":"69b6faf3-e8d6-409a-9491-75c64061056b","added_by":"auto","created_at":"2025-11-11 16:55:04","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":275788,"visible":true,"origin":"","legend":"\u003cp\u003eIllustrative Example of Review Classification Process\u003c/p\u003e","description":"","filename":"floatimage51.png","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/18e88a02f42447c419d001c9.png"},{"id":95667352,"identity":"944e6953-3d47-4277-8970-1c04d4f685be","added_by":"auto","created_at":"2025-11-11 16:55:04","extension":"png","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":142368,"visible":true,"origin":"","legend":"\u003cp\u003eOverview of the two-phase classification and sentiment analysis process, combining automated extraction and classification (Phase 1) with manual validation and consensus (Phase 2).\u003c/p\u003e","description":"","filename":"floatimage6.png","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/3630c770da7cb473e984e2ad.png"},{"id":97556691,"identity":"28a9682a-c274-4cef-8781-58dd0daac59b","added_by":"auto","created_at":"2025-12-05 18:53:18","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1313939,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/4c3c3a4c-77f5-4b6f-b9ed-81c6fc19cfdd.pdf"},{"id":95667339,"identity":"cdf58e6c-e61e-48c3-896c-25a65cc6142b","added_by":"auto","created_at":"2025-11-11 16:55:04","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":1033875,"visible":true,"origin":"","legend":"\u003cp\u003eAppendix I: \u0026nbsp;Legal Basis and Results for User Feedback Categories in DiGA App Store Reviews\u003c/p\u003e","description":"","filename":"AppendixICategoriesComparisonofRatingsLegalBasisV02.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/905ea9450fe572dd90c0aac7.pdf"},{"id":95667342,"identity":"b3e1ba84-4940-4f17-ae94-4a8f37d04a8b","added_by":"auto","created_at":"2025-11-11 16:55:04","extension":"xlsx","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":1206773,"visible":true,"origin":"","legend":"\u003cp\u003eAppendix II: App Store User Reviews\u003c/p\u003e","description":"","filename":"AppendixIIUserReviewsV02.xlsx","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/584d4a6ee4a4484a734850d2.xlsx"},{"id":95667349,"identity":"9a5cd090-4dcb-40da-8443-e97d58cc4c91","added_by":"auto","created_at":"2025-11-11 16:55:04","extension":"pdf","order_by":3,"title":"","display":"","copyAsset":false,"role":"supplement","size":104616,"visible":true,"origin":"","legend":"\u003cp\u003eAppendix III: Category and Sentiment Analysis per App\u003c/p\u003e","description":"","filename":"AppendixIIICategoryandSentimentAnalysisperApp.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/5892aff77c5e34adc37bd476.pdf"},{"id":95798484,"identity":"777695c4-ae4b-4d4f-b42e-e0172c7ac1f2","added_by":"auto","created_at":"2025-11-13 08:16:52","extension":"pdf","order_by":4,"title":"","display":"","copyAsset":false,"role":"supplement","size":105504,"visible":true,"origin":"","legend":"\u003cp\u003eAppendix IV: \u0026nbsp;LLM-Prompt\u003c/p\u003e","description":"","filename":"AppendixIVPromptv02.pdf","url":"https://assets-eu.researchsquare.com/files/rs-7922462/v1/e4ea6db0c1f85899030af2b5.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"What Patients Say About Reimbursable Digital Therapeutics in Germany: A Large-Scale App Store Review Analysis Using a Large Language Model","fulltext":[{"header":"Introduction","content":"\u003cp\u003eIn 2019, Germany has introduced a unique regulatory framework for Digital Therapeutics (DTx) known as DiGAs (ger.: Digitale Gesundheitsanwendungen/ eng.: Digital Health Applications). with the goal of integrating evidence-based DTx into routine care [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. Under the Digital Healthcare Act (\u003cem\u003eDigitale-Versorgung-Gesetz\u003c/em\u003e), DiGAs can be prescribed by physicians and reimbursed by statutory health insurance (\u003cem\u003eGesetzliche Krankenversicherung\u003c/em\u003e) if they are listed in an official directory maintained by the German Federal Institute for Drugs and Medical Devices (BfArM) [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. The directory lists DTx with a variety of indications, such as mental and behavioral disorders, musculoskeletal disorders, metabolic disorders, and disorders of the nervous system. This initiative aims to improve access to innovative healthcare technologies while ensuring their safety, quality, and effectiveness.\u003c/p\u003e\u003cp\u003eTo qualify for inclusion, DTx must meet clinical, legal and technical requirements, including demonstrable benefits to patient- and/or structural-care as defined in \u0026sect;\u0026nbsp;139e SGB V (German Social Code, Book V) [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. These so-called \u003cem\u003epositive healthcare effects\u003c/em\u003e refer either to a medical benefit or to patient-relevant improvements in structure and process. In this context, a medical benefit is defined as an effect such as improvement of health status, reduction of disease duration, a prolonged survival or an increased quality of life. Patient relevant structural or procedural improvements may include for example better coordination of care process, easier access to healthcare services, or the promotion of health literacy [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e].\u003c/p\u003e\u003cp\u003eTo facilitate timely integration into routine care, the law introduced a dedicated admission pathway. This includes a fast-track procedure that allows the provisional listing for up to twelve months while the manufacturers provide evidence of positive healthcare effects. Although this pathway facilitates market entry, it has raised criticism regarding the rigor and consistency of supporting evidence [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]. The manufacturers, on the other hand, are free to choose the study design and type of effect they aim to demonstrate, meaning that for example retrospective designs and sole evidence of structural improvements can qualify for listing [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e, \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. A systematic review by Dittrich et al. identified limitations in the validity of the studies conducted by the manufacturers, such as considerable variation in study quality, frequent methodological flaws, limited generalizability, and a lack of standardized tools for evaluating DTx in Germany [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. Lantzsch et al. further highlighted procedural and methodological shortcomings in the underlying evidence submitted by DTx manufacturers, emphasizing the limited robustness of many clinical evaluations [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. More recently, Sippli et al. reviewed all DiGA approval studies published by 15th March 2024 and found a high risk of bias and methodological weaknesses, even though the reported studies were mostly positive. This further strengthens concerns about the robustness of the current evidence base [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e].\u003c/p\u003e\u003cp\u003eWhile clinical trials provide the basis of regulatory evaluation, they cannot adequately capture the patient-centered aspects of digital applications [\u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e, \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. As emphasized by several studies, patient reviews of mobile health (mHealth) applications can reveal valuable insights into real-world usability, user satisfaction, and recurring quality concerns, all dimensions that are directly relevant to the DiGA approval framework [\u003cspan additionalcitationids=\"CR14 CR15 CR16\" citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. Therefore, analyzing such patient-generated feedback can offer a scalable, real-world supplement to clinical evidence and may highlight practical barriers to adoption and sustained engagement.\u003c/p\u003e\u003cp\u003eImportantly, \u0026sect;\u0026nbsp;139e of the German Social Code, Book V (SGB V) requires that the outcomes of digital health applications include patient-reported health status during usage as part of mandatory success monitoring [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e]. This regulatory emphasis on patient experience is reflected in the BfArM guideline (\u003cem\u003eDiGA-Leitfaden\u003c/em\u003e), which explicitly allows manufacturers to use comparative quantitative studies based on methods from fields such as social sciences or behavioral sciences to demonstrate positive healthcare effects as valid evidence of a positive healthcare effect, provided that validated instruments are used [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e]. This institutional acknowledgment underscores the regulatory relevance of patient experience and aligns with the central hypothesis of this study: that large-scale patient feedback can serve as an important indicator of e.g., perceived effectiveness and quality.\u003c/p\u003e\u003cp\u003eDespite the growing importance of DiGAs, only one study has analyzed patient feedback on German DiGAs including both regulated and non-regulated mHealth application, while covering only 15 DiGAs [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. Our study addresses this gap by systematically analyzing patient reviews from the German Apple App Store (iOS) and Google Play Store exclusively for all approved mobile DiGAs. Using a classification framework based on regulatory requirements, we examine how patient report on key categories. As a secondary objective, this study investigates how large language models can be utilized as a foundational methodological approach to support mHealth analysis by reliably categorizing patient feedback according to sentiment and regulatory-based categories.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003eOverview of the Dataset\u003c/p\u003e\u003cp\u003eAs of 25th February 2025, a total of 69 DiGAs were listed in the official directory maintained by the German BfArM. Of these, 45 DiGAs were available as mobile applications (iOS and Android), and 34 offered a web-based version, with several products available in both formats. Since patient reviews are only available for mobile applications distributed through public app stores, the analysis was limited to the mobile app versions of DiGAs. Web-based applications were excluded due to the lack of publicly accessible patient feedback.\u003c/p\u003e\u003cp\u003eAlthough 45 DiGA had a mobile app, one DiGA had no publicly available user reviews and could therefore not be analyzed. Thus, 44 DiGA were included in the analysis. Among them, 29 (65,9%) had been permanently approved, and 15 (34,1%) were listed on provisional basis at the time of data collection. It is important to note that several DTx share a common mobile app. For example, the \u003cem\u003eHelloBetter\u003c/em\u003e app includes six distinct DiGAs addressing different indications (e.g., chronic pain, diabetes, sleep disorders, panic, stress, and vaginismus), all delivered through the same app infrastructure. Similarly, the \u003cem\u003eSelfapy\u003c/em\u003e app delivers five approved DiGAs (e.g., for depression, bulimia nervosa, generalized anxiety disorder, and chronic pain) through a single mobile application. Accordingly, the 44 DiGAs available as mobile apps are represented by 35 mobile apps. Consequently, the app-based analysis cannot fully differentiate between patient feedback for each individual DiGA when multiple products are bundled within one app. Nevertheless, all reviews associated with such apps were included in the analysis, as they reflect real-world patient experiences with the respective DiGA ecosystem. From these applications, 4,410 publicly accessible user reviews were extracted via the browser-based versions of the German Apple App Store and the German Google Play Store. For the Apple App Store, only up to ten written reviews are publicly visible per app when accessed via browser. During data cleaning and preprocessing, 30 reviews could not be processed further due to incomplete content, parsing errors, or technical parsing errors. These entries were excluded from the analysis. The remaining dataset of 4,380 was fully included in the qualitative and sentiment-based evaluation.\u003c/p\u003e\u003cp\u003eValidation of Model Output\u003c/p\u003e\u003cp\u003eFrom the 4,380 included user reviews, a total of 9,494 individual statements were extracted systematically categorized. Each statement was assigned both a sentiment label (positive, neutral, or negative) and a thematic category. The initial classification was conducted by a large language model, followed by the manual validation. During this review, 28 statements were excluded by the reviewers due to irrelevance or ambiguous content. These primarily included of off-topic comments, incomplete sentences, and non-informative content unrelated to the DTx itself \u0026ndash; such as emoji-only responses, promotional slogans, or vague remarks like \u003cem\u003e\u0026ldquo;I want to be a hero. Greatings Josef\u0026rdquo;\u003c/em\u003e, or \u003cem\u003e\u0026ldquo;just installed\u0026rdquo;\u003c/em\u003e or \u003cem\u003e\u0026ldquo;don\u0026rsquo;t know yet\u0026rdquo;\u003c/em\u003e. In addition, 27 statements were flagged as technically erroneous and removed. Together, these exclusions affected a total of 52 user reviews that were not included in the final dataset. As a result, the final dataset comprises 4,328 user reviews containing at least one interpretable statement, and a total of 9,439 valid and interpretable statements. The complete dataset can be found in \u003cem\u003eAppendix II\u003c/em\u003e. These form the basis form the basis for all subsequent quantitative and qualitative analyses presented in this study. On average, a review consisted of 213.9 characters and 2.18 relevant statements. A detailed overview of the distribution of valid reviews and statements across individual apps can be found in the \u003cem\u003eAppendix I.\u003c/em\u003e\u003c/p\u003e\u003cp\u003eValidation of Sentiment and Category Classification\u003c/p\u003e\u003cp\u003eTo assess the reliability of the automated sentiment and category classification, a manual systematic validation was performed. Quantitative validation demonstrated high classification performance. For sentiment classification, the model achieved an overall accuracy of 99%, excellent precision and recall for positive and negative statements (F1 score: 1.00 and 0.99) and slightly lower performance for neutral statements (F1\u0026thinsp;=\u0026thinsp;0.92). The confusion matrix (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e) shows minimal misclassifications, primarily between neutral and negative labels.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eFor category classification, the model reached an overall accuracy of 95%, with an average F1 score of 0.95. Most thematic assignments were consistent with human reviewers\u0026rsquo; assessment, particularly for frequently occurring categories such as \u003cem\u003eOverall Impression\u003c/em\u003e (F1\u0026thinsp;=\u0026thinsp;0.96), \u003cem\u003eContent\u003c/em\u003e (F\u0026thinsp;=\u0026thinsp;0.95), \u003cem\u003eEffectiveness\u003c/em\u003e (F1\u0026thinsp;=\u0026thinsp;0.94), and \u003cem\u003eSupport\u003c/em\u003e (F1\u0026thinsp;=\u0026thinsp;0.98). However, lower agreement rates were observed for categories with fewer labeled examples, such as \u003cem\u003ePrescription/Approval\u003c/em\u003e (F1\u0026thinsp;=\u0026thinsp;0.89) and \u003cem\u003eLogin/Registration\u003c/em\u003e (F1\u0026thinsp;=\u0026thinsp;0.92). Figure\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e presents the corresponding confusion matrix, illustrating that most discrepancies occurred within conceptually related categories, for example between \u003cem\u003eUX/Design\u003c/em\u003e and \u003cem\u003eOverall Impression\u003c/em\u003e or between \u003cem\u003eTechnology\u003c/em\u003e and \u003cem\u003eSupport\u003c/em\u003e.\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eDistribution of Sentiments and Categories\u003c/p\u003e\u003cp\u003eIn total, 6,491 statements (68.7% out of 9,439) were classified as positive, 474 statements (5.0% out of 9,439) as neutral, and 2,474 statements (26.2% out of 9,439) as negative. The majority of positive statements were assigned to the category \u003cem\u003eContent\u003c/em\u003e (N\u0026thinsp;=\u0026thinsp;2,001, 30.8%), followed by \u003cem\u003eEffectiveness\u003c/em\u003e (N\u0026thinsp;=\u0026thinsp;1,440, 22.2%) and \u003cem\u003eOverall Impression\u003c/em\u003e (N\u0026thinsp;=\u0026thinsp;1,233, 18.9%). Among the 474 neutral statements, most were related to \u003cem\u003eContent\u003c/em\u003e (N\u0026thinsp;=\u0026thinsp;145, 30.6%), \u003cem\u003eUX/Design\u003c/em\u003e (N\u0026thinsp;=\u0026thinsp;83, 17.5%), and \u003cem\u003eEffectiveness\u003c/em\u003e (N\u0026thinsp;=\u0026thinsp;68, 14.3%). Negative classifications (N\u0026thinsp;=\u0026thinsp;2,478) occurred most frequently in categories \u003cem\u003eTechnology\u003c/em\u003e (N\u0026thinsp;=\u0026thinsp;564, 22.8%), \u003cem\u003eUX/Design\u003c/em\u003e (N\u0026thinsp;=\u0026thinsp;341, 13.8%), and \u003cem\u003eLogin/Registration\u003c/em\u003e (N\u0026thinsp;=\u0026thinsp;265, 10.7%) (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e \u003cem\u003e- left\u003c/em\u003e). Due to an exceptionally high number of user reviews for the DTx \u003cem\u003eZanadio\u003c/em\u003e (1,433/4,328, 33.1%), this application was excluded from the following subgroup analysis to avoid potential bias caused by its disproportionate weight in the dataset. Excluding \u003cem\u003eZanadio\u003c/em\u003e, a total of 6,098 valid and interpretable statements remained in the dataset (Fig.\u0026nbsp;\u003cspan refid=\"Fig3\" class=\"InternalRef\"\u003e3\u003c/span\u003e \u003cem\u003e- right\u003c/em\u003e). Of these, 3,963 statements (64.9%) were classified as \u003cem\u003epositive\u003c/em\u003e, 339 (5.5%) as \u003cem\u003eneutral\u003c/em\u003e, and 1,796 (29.4%) as \u003cem\u003enegative\u003c/em\u003e. The majority of \u003cem\u003epositive\u003c/em\u003e statements referred to \u003cem\u003eContent\u003c/em\u003e (N\u0026thinsp;=\u0026thinsp;1,229, 24.8%), followed by \u003cem\u003eEffectiveness\u003c/em\u003e (N\u0026thinsp;=\u0026thinsp;1,045; 21.0%) and \u003cem\u003eOverall Impression\u003c/em\u003e (N\u0026thinsp;=\u0026thinsp;807, 16.3%). Among the 339 \u003cem\u003eneutral\u003c/em\u003e statements, most were assigned to \u003cem\u003eContent\u003c/em\u003e (N\u0026thinsp;=\u0026thinsp;109, 32.2%), \u003cem\u003eUX/Design\u003c/em\u003e (N\u0026thinsp;=\u0026thinsp;58, 17.1%), and \u003cem\u003eEffectiveness\u003c/em\u003e (N\u0026thinsp;=\u0026thinsp;49, 14.4%). The largest share of \u003cem\u003enegative\u003c/em\u003e classifications (N\u0026thinsp;=\u0026thinsp;1,796) where related to \u003cem\u003eTechnology\u003c/em\u003e (N\u0026thinsp;=\u0026thinsp;418, 23.3%), \u003cem\u003eUX/Design\u003c/em\u003e (N\u0026thinsp;=\u0026thinsp;249, 13.9%), and \u003cem\u003eLogin/Registration\u003c/em\u003e (N\u0026thinsp;=\u0026thinsp;189, 10.6%).\u003c/p\u003e\u003cp\u003e\u003c/p\u003e\u003cp\u003eIn addition to the aggregated results, sentiment distributions were also analyzed separately for each individual DTx included in the dataset. The detailed graphical breakdown of sentiment and thematic categories per app is provided in the \u003cem\u003eAppendix III\u003c/em\u003e for reference. This allows a more granular exploration of potential differences in patient feedback across specific DTx products, beyond the aggregated trends presented in the result section. To further investigate potential quality changes over time, we compared user feedback before and after official listing in the BfArM directory for the ten most reviewed DiGAs. For each app and category, chi-squared test was conducted. No significant difference in sentiment distributions were identified, indicating no substantial shifts in patient-perceived app quality following formal inclusion.\u003c/p\u003e\u003cp\u003eCategories\u003c/p\u003e\u003cp\u003eThe following sections provide a detailed overview of patient feedback for three selected categories: \u003cem\u003eOverall impression\u003c/em\u003e, \u003cem\u003eEffectiveness\u003c/em\u003e, and \u003cem\u003eLogin/ Registration\u003c/em\u003e. These categories were chosen due to their high frequency and relevance across the dataset. Analyses of additional categories, such as \u003cem\u003eUX/Design\u003c/em\u003e, \u003cem\u003eTracking/ Documentation\u003c/em\u003e, \u003cem\u003eTechnology\u003c/em\u003e, \u003cem\u003eSupport\u003c/em\u003e, \u003cem\u003ePrescription/ Approval\u003c/em\u003e, \u003cem\u003eCost/ Reimbursement\u003c/em\u003e, and \u003cem\u003eContent\u003c/em\u003e are provided in \u003cem\u003eAppendix II\u003c/em\u003e for reference. All referenced user statements were translated into English using \u003cem\u003eDeepL (July 2025)\u003c/em\u003e, preserving their original meaning.\u003c/p\u003e\u003cp\u003eOverall impression\u003c/p\u003e\u003cp\u003eA total of 1,431 user statements were categorized under Overall impression, providing general patient feedback regarding the DTx experience. Positive reviews frequently emphasized the usefulness, clarity, and motivational aspects of the applications. One patient stated: \u003cem\u003e\u0026ldquo;A great program. Motivating. Can be very well adapted to personal needs. Definitely recommendable!\u0026rdquo;\u003c/em\u003e (Sebastian W., Kaia R\u0026uuml;ckenschmerzen, Google Play Store). Another patient commented: \u003cem\u003e\u0026ldquo;Digital health applications to understand and ultimately get a grip on men\u0026rsquo;s health issues described in the app. [\u0026hellip;]. The app is intuitive and clearly structured, I can recommend it 100%. Thumbs up!\u0026rdquo;\u003c/em\u003e (Nkgvfjkbhh, Kranus Edera, Apple App Store). Another patient stated: \u003cem\u003e\u0026ldquo;I can highly recommend this app to anyone with migraines! Investigating the connection between migraines and blood sugar was very interesting. The app is clear and easy to use\u0026rdquo;\u003c/em\u003e (Marcellaelena, sinCephalea, Apple App Store). Neutral sentiments reflected initial or uncertain patient experiences, as in the following statement: \u003cem\u003e\u0026ldquo;The start is okay\u0026rdquo;\u003c/em\u003e (Gerhard K., NichtraucherHelden, Google Play Store) and \u003cem\u003e\u0026ldquo;I\u0026rsquo;m not really convinced yet\u0026rdquo;\u003c/em\u003e (R\u0026uuml;genVan, Meine Tinnitus App, Google Play Store). Reviews with negative statements often criticized technical problems or unmet expectations, for instance: \u003cem\u003e\u0026ldquo;Unfortunately, the app is disastrous. Not only does it run poorly, but it also fails to address personal problems because the algorithm cannot be adjusted. So far, I can only advise against it\u0026rdquo;\u003c/em\u003e (Paule M, Cara Care, Google Play Store). Another patient wrote \u003cem\u003e\u0026ldquo;The app provides common knowledge and helps document sleep patterns. But it does not replace therapy. It didn\u0026rsquo;t help me\u0026rdquo;\u003c/em\u003e (Jochen K., somnio, Google Play Store).\u003c/p\u003e\u003cp\u003eLogin/Registration\u003c/p\u003e\u003cp\u003eIn total, 280 user statements addressed the login and registration process. Positive feedback frequently highlighted an easy onboarding experience and simple account handling. For example, a patient stated: \u003cem\u003e\u0026ldquo;Great app for endometriosis beginners\u0026rdquo; I\u0026rsquo;ve only just started using it, but the Endo-App has already impressed me. The simple registration and clear profile make getting started easy\u0026rdquo;\u003c/em\u003e (Andr\u0026eacute; K., Endo-App, Google Play Store). Similarly, a patient emphasized \u003cem\u003e\u0026ldquo;Simple handling, quick login, attentive team. Good exercises that can be done anywhere \u0026ndash; whether at home, in the gym, or at the office\u0026rdquo;\u003c/em\u003e (Biscuit de l\u0026rsquo;empereur, ViViRA bei R\u0026uuml;ckenschmerzen, Google Play Store). Neutral reviews often reflected suggestions for improvement alongside generally positive impressions. One patient reported: \u003cem\u003e\u0026ldquo;Very helpful! The app\u0026rsquo;s exercises clearly help. But you have to spend a lot of time reading through everything at the beginning. Suggestion: save login data, at least the email address\u0026rdquo;\u003c/em\u003e Martin K., Orthopy, Google Play Store). Negative experiences primarily focused on technical login issues or complex authentication procedures. One patient described: \u003cem\u003e\u0026ldquo;I used to like the app. Unfortunately, after recent changes, I can\u0026rsquo;t access my account at all. Constantly having to log in again is annoying, and now I can\u0026rsquo;t access my data at all\u0026rdquo;\u003c/em\u003e (Nadine A., Cara Care, Google Play Store). Another patient reported: \u003cem\u003e\u0026rdquo;I also can\u0026rsquo;t log in, seems to be a known problem. Regardless of the email provider, I\u0026rsquo;ve never had such issues before. Maybe one day I\u0026rsquo;ll be able to use it\u0026rdquo;\u003c/em\u003e (Hanna K, Endo-App, Google Play Store). Further negative statements assigned to this category criticized: \u003cem\u003e\u0026ldquo;The information content and exercises are very good. What annoys me is the two-step login process. Totally unnecessary in my opinion \u0026ndash; it\u0026rsquo;s not a bank account\u0026rdquo;\u003c/em\u003e (Stefan C., Kranus Lutera, Google Play Store).\u003c/p\u003e\u003cp\u003eEffectiveness\u003c/p\u003e\u003cp\u003eThe category Effectiveness received 1,609 user statements, making it one of the most mentioned aspects of the DTx evaluations. Many patients reported positive effects on their health effects, highlighting improved well-being, symptom relief, or successful integration of the app into daily routines. For instance, one patient stated: \u003cem\u003e\u0026ldquo;I\u0026rsquo;m really satisfied and positively surprised. The app offers so many features \u0026ndash; recipes, different sports options from easy to challenging, advisors you can contact anytime. Overall, it really helped me. Thank you!\u0026rdquo;\u003c/em\u003e (Andrea Z., zanadio, Google Play Store). Similarly, patients emphasized positive feedback like the following statement: \u003cem\u003e\u0026ldquo;I struggled for a long time with falling and staying asleep. This app really helped. I fall asleep faster and stay asleep more often. It\u0026rsquo;s well structured, and everything can be analyzed. Really Great\u0026rdquo;\u003c/em\u003e (Peter H., somnio, Google Play Store). Neutral reviews often reflected mixed impressions or external limitations affecting the expected effectiveness. For example, a patient commented: \u003cem\u003e\u0026ldquo;The reminders can be a bit annoying but are necessary. Good exercises and food for thought\u0026hellip; though sometimes I lack the motivation to follow through\u0026rdquo;\u003c/em\u003e (Familie S., HelloBetter, Google Play Store). Similarly, a patient described his DTx as supportive but limited: \u003cem\u003e\u0026ldquo;The app helps to better understand depressions and take first steps. But it requires discipline and cannot replace processional medical help\u0026rdquo;\u003c/em\u003e (Thom-05, MindDoc, Apple Store). Reviews with negative statements often expressed disappointment regarding insufficient health improvements. One patient stated: \u003cem\u003e\u0026ldquo;Great tips, but unfortunately it didn\u0026rsquo;t help me. Still, thanks\u0026rdquo;\u003c/em\u003e (Jasmin M., NichtraucherHelden, Google Play Store). A more critical assessment mentioned: \u003cem\u003e\u0026ldquo;Not recommended and extremely expensive. Despite completing all exercises, my condition hasn\u0026rsquo;t improved. The health insurance paid over 600\u0026euro; for this. A physiotherapist would have explained everything better and been cheaper. Plus, the app is poorly designed and buggy\u0026rdquo;\u003c/em\u003e (Skippi, Kranus Lutera, Google Play Store).\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eIn this study, we conducted a qualitative large-scale analysis of user reviews related to digital therapeutics (DTx) using a large language model for sentiment analysis and thematic category classification. To our knowledge, this represents the largest qualitative analysis of patient feedback on reimbursable DTx. The final dataset comprised 4,328 user reviews covering 44 DTx and 35 mobile applications listed in the official German DiGA directory. These reviews were extracted from the German Apple (iOS) and Android App Stores and processed using a structured prompt-based approach. In total, the model identified and classified 9,439 individual statements across ten predefined categories derived from legal and regulatory criteria for DiGAs in Germany. The model achieved 99% sentiment accuracy, with slightly lower performance for neutral statements (F = 0.92), which often contained mixed tones like polite suggestions or critical requests. For category classification, the LLM reached an overall accuracy of 95% with an average F1 score of 0.95. While categories such as \u003cem\u003eOverall Impression\u003c/em\u003e (F1 = 0.96) or \u003cem\u003eContent\u003c/em\u003e (F = 0.95) showed high agreement with human validation, categories such as \u003cem\u003ePrescription/ Approval\u003c/em\u003e (F1 = 0.89) and \u003cem\u003eLogin/Registration\u003c/em\u003e (F1 = 0.92) showed lower agreement rates. This may be the case as the prescription and approval processes are directly linked to the registration and onboarding process, as the patients receive a registration code for their prescribed DTx from their health insurer. From the patient’s perspective, this might be interpreted as a single process, as articulated accordingly in many reviews, leading to misclassifications.\u003c/p\u003e\u003cp\u003eIn only four cases did the model exceed the specified five statements per review, which is negligible given the number of reviews and could not lead to bias nor affect reproducibility. The manual validation ensured consistency but may have introduced an acceptance bias toward plausible model predictions: a lack of blinding during initial validation steps could have reinforced the model’s suggestions. Future evaluations should consider blinded or disagreement-focused review procedures to limit such a bias. The sentiment analysis showed that with 6,491 positive statements (68.7% out of 9,439) the majority of the statements were in favor of the DTx, showing a broad acceptance of the digital therapeutic approach. Despite generally positive sentiment, full-text reviewers tended to give lower star ratings (\u003cem\u003eAppendix I\u003c/em\u003e). With a total of 2,474 (26.2% out of 9,439) negative statements, those patients not only use their reviews to praise the DTx, but also to criticize the applications and report frustration with processes, bugs or general dissatisfaction. This was particularly evident in the categories \u003cem\u003eTechnology\u003c/em\u003e, \u003cem\u003eLogin/Registration\u003c/em\u003e and \u003cem\u003ePrescription/Approval\u003c/em\u003e. For instance, patients tended to report frustration if the registration process was prune to bugs or when daily logins required multiple authentication steps that hindered the integration into their daily routine. Complex multifactor authentication can contradict low-threshold access as required by national digital health regulations with respect to the average age of DiGA users (Ø 55–60 [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e]). While these authentication requirements are aligned with national cybersecurity standards defined by BSI (German Federal Office for Information Security), frequent user criticism suggests a need to balance regulatory security demands with usability considerations tailored to the target demographic. With 564 of 669 (84.3%) technology-related statements being negative, patients frequently expressed frustration about bugs and app malfunctions. As DTx are regulated medical devices, such deficits raise concerns about quality assurance, suggesting the need for improved quality or release management by the vendors [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e]. Conversely, patients tend to be more motivated to share negative experiences, particularly when they encounter disappointment or frustration [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e, \u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]. This study did not examine when negative reviews on the subject of technology were written, e.g., during the launch phase, post-launch, or maturity phase [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]. At the same time, basic functions are often an implicit requirement from the patient's perspective, which is why they do not receive any specific praise. Requirements Engineering theory classifies such functionalities as \"basic requirements\" or \"must-be\" features, whose absence or malfunction commonly leads to negative reviews [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e]. Another category predominantly characterized by negative feedback is the \u003cem\u003eApproval\u003c/em\u003e or \u003cem\u003ePrescription\u003c/em\u003e process. While the prescription is issued by a physician, approval is granted by health insurance providers based on corresponding medical indications. Many patients criticize the length of the approval process by health insurers. Additionally, patients report that even with valid prescriptions, digital therapy may be denied due to additional criteria (such as BMI thresholds for weight loss applications), which patients frequently describe as frustrating. Furthermore, patients frequently mention the standard prescription period of 90 days. They express that the recurring prescription process for DTx is perceived as burdensome. On the other hand, some patients report disappointment because they cannot access the app without a valid prescription. Additionally, patients frequently comment that they would like to \"try out\" the app before applying for a prescription. This aligns with medical practices where trial periods help assess treatment suitability. Such trial periods could help both patients and healthcare professionals assess whether the chosen therapy is suitable and worth continuing. Accordingly, a limited trial version could facilitate more informed decisions, improve acceptance, and prevent frustration or early discontinuation, resulting in more efficient use of healthcare resources. Ultimately, patients criticized the high costs and questioned the DTx cost-benefit ratio. In contrast, the categories \u003cem\u003eOverall Impression\u003c/em\u003e, \u003cem\u003eEffectiveness\u003c/em\u003e, \u003cem\u003eSupport\u003c/em\u003e and \u003cem\u003eContent\u003c/em\u003e were characterized by predominantly positive statements. In the category \u003cem\u003eOverall Impression\u003c/em\u003e (1,233/1,431, 86.0% positive), patients frequently praise their DTx in general, highlight high satisfaction with the apps or recommend it to patients with similar indications. Considering that patients are generally more likely to leave feedback when they encounter problems or frustration [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e], the high involvement in positive evaluations of the \u003cem\u003eOverall Impression\u003c/em\u003e can be interpreted as patient reported real-world evidence of user satisfaction – an essential quality indicator required by national digital health regulations [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]. Furthermore, the category \u003cem\u003eEffectiveness\u003c/em\u003e is rated positively by the majority (89.4%). Patients report a noticeable improvement in symptoms, an improvement of their health literacy and better management of their illness resulting in improvements of their quality in life, as expected by effectiveness monitoring frameworks in DTx regulation. These findings align closely with the legal requirements in §\u0026nbsp;139e Abs. 13 SGB V, which highlights that results from accompanying effectiveness measurements should particularly include the patient-reported health status during the use of DTx [\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e]. Lastly, the positive statements regarding the category \u003cem\u003eContent\u003c/em\u003e also showed an overall high patient satisfaction (2,002/2,566, 78.0%). Patients frequently valued the diverse and well-prepared presentation of content, while only a few commented on its scientific grounding. The predominantly positive feedback nevertheless suggests that patients perceive the content as credible and effective, indicating that the applications mostly meet the quality requirements for evidence-based, safe, and user-appropriate health information, as mandated by national regulatory frameworks [\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e].\u003c/p\u003e\u003cp\u003eThe overall findings demonstrate that proposed category scheme is well-suited for structuring patient-reported feedback on DTx for content-related analyses and interpretation. A comparable study conducted by Uncovska et al. analyzed both 15 regulated DiGAs and non-prescription mHealth apps from public app stores, using BERTopic for unsupervised topic modelling on 17,588 user reviews [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. Their findings showed that DiGAs had higher patient ratings, with positive themes around customer service and ease of use, while technical issues like registration challenges were also prominent. In contrast, our study focuses solely on patient feedback for 44 officially listed DiGAs, using a classification framework directly conceptualized based on legal and regulatory criteria, and manual validation. This allows for a more targeted evaluation of patient experience as it specifically relates to DiGA requirements. Our method thus complements and extends the insights of Uncovska et al. by providing regulatory alignment, also covering a broader sample of DiGAs. Due to the user-friendliness of ChatGPT 4-o, large-scale evaluation is possible with comparatively low technical barriers and resource requirements. Given the high classification accuracy of the model, further investigations using a more fine-grained category scheme are recommended, to gain nuanced insights that remain undetected by the current classification setup. A follow-up study will explore patient loyalty based on usage duration and prescription frequency. In addition, we aim to differentiate between technical and content-related aspects of user support and expand our category scheme by incorporating aspects identified by Haggag et al., such as reasons for uninstallation or statements related to data privacy and data security [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e].\u003c/p\u003e\u003cp\u003eOur study evaluated patients’ experience with the DTx using a Large Language Models to scale the analysis. We could show that large language models paired with human review, can efficiently process and categorize large volumes of unstructured patient feedback. This approach may provide regulators, manufacturers, and researchers with timely, patient-centered insights that are otherwise difficult to capture through clinical trials alone.\u003c/p\u003e\u003cp\u003eFrom a content perspective, our findings highlight both the potential and the current limitations of reimbursable DTx from the patient’s perspective. While patients value the therapeutic benefits and content quality of many DTx, basic technical functionality and usability, particularly during login and registration, are frequently criticized. These processes are perceived as fundamental requirements for medical-grade products and if unmet, lead to rejection and ultimately to a negative therapeutic outcome. To ensure equitable access and sustained engagement, registration procedures should be standardized across applications, and login processes must be truly barrier-free. Furthermore, patients demonstrate awareness of healthcare spending and critically assess the value provided by reimbursed applications, especially when functionality or therapeutic benefit appear limited. As suggested by some patients, offering limited trial version prior to prescription could support more informed decision-making and potentially increase patient acceptance.\u003c/p\u003e\u003cp\u003eAdditionally, we recommend that regulatory bodies implement structured, scenario-based usability testing with representative patient groups during both pre-approval phases and ongoing post-market surveillance. This approach could help identify and address technical or accessibility barriers proactively, ensuring that reimbursable DTx meet the regulatory and practical needs of diverse patient groups throughout their lifecycle. Future efforts should therefore focus on improving technical reliability, streamlining access procedures and incorporating patient preferences into both design and pricing strategies to enhance both acceptance as well as long-term engagement with DiGA.\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003eData Identification and Extraction\u003c/p\u003e\u003cp\u003eData collection involved two stages: first, we identified all mobile DTx listed by the Federal Institute for Drugs and Medical Devices as of February 25th 2025. Second, we used Selenium-based Python scripts (v4.29.0) to extract user reviews from the German Apple App Store and Google Play Store. Extracted data included review content, ratings, dates, developer responses, and app metadata, stored in structured JSON format (Fig.\u0026nbsp;\u003cspan refid=\"Fig4\" class=\"InternalRef\"\u003e4\u003c/span\u003e).\u003c/p\u003e\u003cp\u003eReview Classification and Sentiment Framework\u003c/p\u003e\u003cp\u003eOur classification framework was based on legal and regulatory requirements for DiGAs in Germany, including SGB V §\u0026nbsp;139e, the Digital Health Applications Ordinance (DiGAV), and the official DiGA Guide published by BfArM. The derived categories are: Overall Impression, Prescription/Approval, Content, Cost/Reimbursement, Login/Registration, Support, Technology, Tracking/Documentation, UX/Design, Effectiveness. The category Tracking/Documentation was included despite the regulation not yet being in force, as patient-reported outcomes (e.g., §\u0026nbsp;139e (13) SGB V) are anticipated to become increasingly relevant for DiGA assessment. A detailed explanation of the categorization framework, including corresponding DiGA criteria and legal references in German can be found in \u003cem\u003eAppendix I\u003c/em\u003e. Each user review was analyzed for up to five thematically distinct statements.\u003c/p\u003e\u003cp\u003eModel-Based Sentiment and Category Assignment\u003c/p\u003e\u003cp\u003eTo systematically assess user sentiment, we developed a custom Python script to automate the classification process using the GPT-4o API provided by OpenAI (Step 1–3). Each user review was processed individually using a carefully structured prompt designed to guide the model toward consistent and domain-specific classification (see \u003cem\u003eAppendix IV\u003c/em\u003e). The model was instructed to adopt the role of an expert in evaluating user experiences with digital health applications (DiGAs). Within this role, it was given a multi-step task:\u003c/p\u003e\u003cul\u003e\u003cli\u003e\u003cp\u003eExtract up to five core statements from each user review\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eAssign each statement a sentiment label (positive, neutral, or negative)\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eAssign each statement to exactly one of ten predefined thematic categories derived from the legal and quality criteria for DiGAs (see \u003cem\u003eAppendix I\u003c/em\u003e)\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eMerge statements that refer to the same theme and sentiment into a single, summarized expression\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eAvoid repeating partial aspects of the same issue across multiple statements\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eLimit each category to one statement per review, unless clearly distinct subtopics are present\u003c/p\u003e\u003c/li\u003e\u003cli\u003e\u003cp\u003eExclude overall evaluations if the review indicates the app could not be used (e.g., due to failed registration or missing insurance approval)\u003c/p\u003e\u003c/li\u003e\u003c/ul\u003e\u003cp\u003eThe model was provided with a fixed list of ten valid categories, accompanied by specific guidance on ambiguous cases. For example, access issues after prescription or insurer approval were to be categorized as Login/Registration, not Prescription/Approval. Finally, the model was instructed to return its output in a standardized, structured format with numbered statements, each containing three elements: the \u003cem\u003eextracted statement\u003c/em\u003e, \u003cem\u003esentiment label\u003c/em\u003e, and \u003cem\u003eassigned category\u003c/em\u003e. A representative example was included in the prompt to enforce consistent formatting and interpretation. Statements were required to be thematically distinct and clearly assigned. The script processed each review individually and parsed the model's structured response into a table of statements with their associated sentiment and category (Step 4–5). All results were exported in a standardized CSV format for further manual validation (Step 6). By assigning the model a clearly defined expert perspective and domain logic, we ensured that the sentiment analysis aligned with both linguistic consistency and health service evaluation standards. A translated example of this classification process in visualized in Fig.\u0026nbsp;\u003cspan refid=\"Fig5\" class=\"InternalRef\"\u003e5\u003c/span\u003e.\u003c/p\u003e\u003cp\u003eHuman Validation of Model Output\u003c/p\u003e\u003cp\u003eTo evaluate the accuracy and conceptual validity of the model-generated classifications, we conducted a structured, validation process as shown in Fig.\u0026nbsp;\u003cspan refid=\"Fig6\" class=\"InternalRef\"\u003e6\u003c/span\u003e. The full set of reviews, including original review, extracted statement, sentiment label, and categories, was evenly divided among three authors (Step 7), each of whom independently reviewed one third of the dataset (Step 8). The validation process followed a four-step protocol:\u003c/p\u003e\u003col\u003e\u003cspan\u003e\u003cli\u003e\u003cp\u003e\u003cb\u003eIndependent review\u003c/b\u003e: Each model-generated statement and its assigned sentiment and category were assessed in relation to the original review. Reviewers evaluated whether the extracted statement captured a meaningful aspect of the review and if the sentiment and category were accurate according to the predefined classification framework.\u003c/p\u003e\u003c/li\u003e\u003c/span\u003e\u003cspan\u003e\u003cli\u003e\u003cp\u003e\u003cb\u003eFlagging of inconsistencies\u003c/b\u003e: Statements that were incomplete, overly generic, redundant, or incorrectly classified (e.g., in sentiment or category) were flagged. Each flagged item included a brief comment and a suggested correction, if appropriate.\u003c/p\u003e\u003c/li\u003e\u003c/span\u003e\u003cspan\u003e\u003cli\u003e\u003cp\u003e\u003cb\u003eTriangulated reassessment\u003c/b\u003e: The two remaining authors independently evaluated the flagged items, providing their own classifications without being influenced by the previous assessments (Step 9). This resulted in three independent judgments per flagged entry.\u003c/p\u003e\u003c/li\u003e\u003c/span\u003e\u003cspan\u003e\u003cli\u003e\u003cp\u003e\u003cb\u003eConsensus resolution\u003c/b\u003e: In cases of disagreement, the authors discussed the item until a final classification was agreed upon (Step 10).\u003c/p\u003e\u003c/li\u003e\u003c/span\u003e\u003c/ol\u003e\u003cp\u003eValidated statements were finalized and included in the analysis as a consolidated CSV dataset (Step 11). Structurally flawed outputs (e.g., empty responses or parsing errors) were reprocessed. The manual validation ensured consistency with the classification framework and enabled direct evaluation of the model on real-world data.\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cp\u003e\u003cstrong\u003eBfArM\u003c/strong\u003e: German Federal Institute for Drugs and Medical Devices\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDiGA:\u003c/strong\u003e\u0026nbsp; ger: Digitale Gesundheitsanwendung/ eng: Digital Health Application\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDiGAV\u003c/strong\u003e: DiGA Verordnung\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDTx\u003c/strong\u003e: Digital Therapeutics\u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eLLM:\u003c/strong\u003e Large Language Model\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eSGB V\u003c/strong\u003e: German Social Code, Book V\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eUX:\u003c/strong\u003e User Experience\u003c/p\u003e\n"},{"header":"Declarations","content":"\u003cp\u003eConflict of Interest\u003c/p\u003e\n\u003cp\u003eThe authors declare that they have no competing interests. This research was conducted independently and the analysis and interpretation of data were performed solely by the authors.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eAcknowledgemaents\u003c/p\u003e\n\u003cp\u003eThe authors would like to thank the colleagues involved in the discussions on digital health regulation and evaluation for their valuable insights.\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eFunding Statement\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eThis research was supported by internal funds from the Medical Faculty of Kiel University. No specific grant number is associated with this support.\u003c/p\u003e\n\u003cp\u003eAuthor Contributions\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eBK developed the methodological framework, created the category scheme, conducted the data extraction and analysis, participated in the human validation of LLM output provided and the initial and final manuscript. HU contributed expertise in large language models (LLMs), selected the validation method, participated in the human validation of LLM output and revised the manuscript. HR participated in the human validation of LLM output. BS contributed expertise in mHealth and reviewed the initial and final manuscript. All authors approved the final manuscript.\u003c/p\u003e\u003cp\u003eData Availability\u003c/p\u003e\n\u003cp\u003eAll review data, the category scheme, detailed per-app analyses, and the initial LLM prompt are available in the supplementary information files.\u003c/p\u003e\n\u003cp\u003eMultimedia Appendix\u003c/p\u003e\n\u003cp\u003eAppendix I: \u0026nbsp; \u0026nbsp;Legal Basis and Results for User Feedback Categories in DiGA App Store Reviews\u003c/p\u003e\n\u003cp\u003eAppendix II:\u0026nbsp; \u0026nbsp;\u0026nbsp;App Store User Reviews\u003c/p\u003e\n\u003cp\u003eAppendix III: \u0026nbsp;Category and Sentiment Analysis per App\u003c/p\u003e\n\u003cp\u003eAppendix IV: \u0026nbsp;LLM-Prompt\u003c/p\u003e\n"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eDriving the digital transformation of Germany\u0026rsquo;s healthcare system for the good of patients, (n.d.). https://www.bundesgesundheitsministerium.de/en/digital-healthcare-act.html (accessed August 11, 2025).\u003c/li\u003e\n\u003cli\u003eL. Schmidt, M. Pawlitzki, B.Y. Renard, S.G. Meuth, and L. Masanneck, The three-year evolution of Germany\u0026rsquo;s Digital Therapeutics reimbursement program and its path forward, \u003cem\u003eNpj Digit. \u003c/em\u003e\u003cem\u003eMed.\u003c/em\u003e \u003cstrong\u003e7\u003c/strong\u003e (2024) 139. doi:10.1038/s41746-024-01137-1.\u003c/li\u003e\n\u003cli\u003eBundesinstitut f\u0026uuml;r Arzneimittel und Medizinprodukte (BfArM), DiGA-Verzeichnis \u0026ndash; Digitale Gesundheitsanwendungen, \u003cem\u003eBfArM \u0026ndash; Bundesinstitut f\u0026uuml;r Arzneimittel und Medizinprodukte\u003c/em\u003e. (n.d.). https://diga.bfarm.de/de/verzeichnis (accessed March 15, 2025).\u003c/li\u003e\n\u003cli\u003e\u0026sect; 139e SGB V \u0026ndash; Verzeichnis digitaler Gesundheitsanwendungen, 2019. https://www.gesetze-im-internet.de/sgb_5/__139e.html.\u003c/li\u003e\n\u003cli\u003eBundesministerium f\u0026uuml;r Gesundheit, Digitale-Gesundheitsanwendungen-Verordnung (DiGAV), 2020. https://www.gesetze-im-internet.de/digav/__8.html (accessed May 10, 2025).\u003c/li\u003e\n\u003cli\u003eGKV-Spitzenverband, Bericht nach \u0026sect; 139e Absatz 10 SGB V zur Nutzung, Akzeptanz und Wirkung digitaler Gesundheitsanwendungen \u0026ndash; DiGA-Bericht 2024, GKV-Spitzenverband, Berlin, 2025. https://www.gkv-spitzenverband.de/media/dokumente/krankenversicherung_1/telematik/digitales/2024_DiGA-Bericht_final.pdf (accessed July 22, 2025).\u003c/li\u003e\n\u003cli\u003eM. M\u0026auml;der, P. Timpel, T. Sch\u0026ouml;nfelder, C. Militzer-Horstmann, S. Scheibe, R. Heinrich, and D. H\u0026auml;ckl, Evidence requirements of permanently listed digital health applications (DiGA) and their implementation in the German DiGA directory: an analysis, \u003cem\u003eBMC Health Serv Res\u003c/em\u003e. \u003cstrong\u003e23\u003c/strong\u003e (2023) 369. doi:10.1186/s12913-023-09287-w.\u003c/li\u003e\n\u003cli\u003eF. Dittrich, A. Mielitz, E. Pustozerov, D. Lawin, U. Von Jan, and U.-V. Albrecht, Digital health applications from a government-regulated directory of reimbursable health apps in Germany\u0026mdash;a systematic review for evidence and bias, \u003cem\u003emHealth\u003c/em\u003e. \u003cstrong\u003e9\u003c/strong\u003e (2023) 35\u0026ndash;35. doi:10.21037/mhealth-23-17.\u003c/li\u003e\n\u003cli\u003eH. Lantzsch, H. Eckhardt, A. Campione, R. Busse, and C. Henschke, Digital health applications and the fast-track pathway to public health coverage in Germany: challenges and opportunities based on first results, \u003cem\u003eBMC Health Serv Res\u003c/em\u003e. \u003cstrong\u003e22\u003c/strong\u003e (2022) 1182. doi:10.1186/s12913-022-08500-6.\u003c/li\u003e\n\u003cli\u003eK. Sippli, S. Deckert, J. Schmitt, and M. Scheibe, Healthcare effects and evidence robustness of reimbursable digital health applications in Germany: a systematic review, \u003cem\u003eNpj Digit. Med.\u003c/em\u003e \u003cstrong\u003e8\u003c/strong\u003e (2025) 495. doi:10.1038/s41746-025-01879-6.\u003c/li\u003e\n\u003cli\u003eE. Bolinger, and B. Tyl, Key Considerations for Designing Clinical Studies to Evaluate Digital Health Solutions, \u003cem\u003eJ Med Internet Res\u003c/em\u003e. \u003cstrong\u003e26\u003c/strong\u003e (2024) e54518. doi:10.2196/54518.\u003c/li\u003e\n\u003cli\u003eC. Guo, H. Ashrafian, S. Ghafur, G. Fontana, C. Gardner, and M. Prime, Challenges for the evaluation of digital health solutions\u0026mdash;A call for innovative evidence generation approaches, \u003cem\u003eNpj Digit. Med.\u003c/em\u003e \u003cstrong\u003e3\u003c/strong\u003e (2020) 110. doi:10.1038/s41746-020-00314-2.\u003c/li\u003e\n\u003cli\u003eO. Haggag, J. Grundy, M. Abdelrazek, and S. Haggag, A large scale analysis of mHealth app user reviews, \u003cem\u003eEmpir Software Eng\u003c/em\u003e. \u003cstrong\u003e27\u003c/strong\u003e (2022) 196. doi:10.1007/s10664-022-10222-6.\u003c/li\u003e\n\u003cli\u003eR. Vasa, L. Hoon, K. Mouzakis, and A. Noguchi, A preliminary analysis of mobile app user reviews, in: Proceedings of the 24th Australian Computer-Human Interaction Conference, ACM, Melbourne Australia, 2012: pp. 241\u0026ndash;244. doi:10.1145/2414536.2414577.\u003c/li\u003e\n\u003cli\u003eLi Xiaozhou, Zhang Zheying, and Stefanidis Kostas, Mobile App Evolution Analysis Based on User Reviews, in: Frontiers in Artificial Intelligence and Applications, IOS Press, 2018. doi:10.3233/978-1-61499-900-3-773.\u003c/li\u003e\n\u003cli\u003eP.L. Kolominsky-Rabas, M. Tauscher, R. Gerlach, M. Perleth, and N. Dietzel, Wie belastbar sind Studien der aktuell dauerhaft aufgenommenen digitalen Gesundheitsanwendungen (DiGA)? Methodische Qualit\u0026auml;t der Studien zum Nachweis positiver Versorgungseffekte von DiGA, \u003cem\u003eZEFQ\u003c/em\u003e. \u003cstrong\u003e175\u003c/strong\u003e (2022) 1\u0026ndash;16. doi:10.1016/j.zefq.2022.09.008.\u003c/li\u003e\n\u003cli\u003eM. Uncovska, B. Freitag, S. Meister, and L. Fehring, Rating analysis and BERTopic modeling of consumer versus regulated mHealth app reviews in Germany, \u003cem\u003eNpj Digit. \u003c/em\u003e\u003cem\u003eMed.\u003c/em\u003e \u003cstrong\u003e6\u003c/strong\u003e (2023) 115. doi:10.1038/s41746-023-00862-3.\u003c/li\u003e\n\u003cli\u003e\u0026sect; 139e Abs. 13 Nr. 3 F\u0026uuml;nftes Sozialgesetzbuch (SGB V), 2019. https://www.gesetze-im-internet.de/sgb_5/__139e.html (accessed August 10, 2025).\u003c/li\u003e\n\u003cli\u003eBundesinstitut f\u0026uuml;r Arzneimittel und Medizinprodukte (BfArM), DiGA-Leitfaden. Das Fast-Track-Verfahren f\u0026uuml;r digitale Gesundheitsanwendungen nach \u0026sect; 139e SGB V, BfArM, Bonn, 2023. https://www.bfarm.de/SharedDocs/Downloads/DE/Medizinprodukte/diga_leitfaden.html?nn=597198 (accessed July 7, 2025).\u003c/li\u003e\n\u003cli\u003eL. Schramm, and C.-C. Carbon, Critical success factors for creating sustainable digital health applications: A systematic review of the German case, \u003cem\u003eDIGITAL HEALTH\u003c/em\u003e. \u003cstrong\u003e10\u003c/strong\u003e (2024) 20552076241249604. doi:10.1177/20552076241249604.\u003c/li\u003e\n\u003cli\u003eD. Pagano, and W. Maalej, User feedback in the appstore: An empirical study, in: 2013 21st IEEE International Requirements Engineering Conference (RE), IEEE, Rio de Janeiro-RJ, Brazil, 2013: pp. 125\u0026ndash;134. doi:10.1109/RE.2013.6636712.\u003c/li\u003e\n\u003cli\u003eISO/IEC/IEEE International Standard - Systems and software engineering \u0026ndash; Life cycle processes \u0026ndash; Requirements engineering, \u003cem\u003eISO/IEC/IEEE 29148:2018(E)\u003c/em\u003e. (2018) 1\u0026ndash;104. doi:10.1109/IEEESTD.2018.8559686.\u003c/li\u003e\n\u003cli\u003eE. Hull, K. Jackson, and J. Dick, Requirements engineering, 2nd ed., Springer, London, 2005. http://gso.gbv.de/DB=2.1/PPNSET?PPN=589178075.\u003c/li\u003e\n\u003cli\u003eE.W. Anderson, Customer Satisfaction and Word of Mouth, \u003cem\u003eJournal of Service Research\u003c/em\u003e. \u003cstrong\u003e1\u003c/strong\u003e (1998) 5\u0026ndash;17. doi:10.1177/109467059800100102.\u003c/li\u003e\n\u003cli\u003e\u0026sect; 139e Abs. 13 Nr. 2 F\u0026uuml;nftes Sozialgesetzbuch (SGB V), 2019. https://www.gesetze-im-internet.de/sgb_5/__139e.html (accessed August 10, 2025).\u003c/li\u003e\n\u003cli\u003e\u0026sect; 5 Abs. 8 Satz 2 Digitale Gesundheitsanwendungen-Verordnung (DiGAV), 2020. https://www.gesetze-im-internet.de/digav/__5.html (accessed July 7, 2025).\u003c/li\u003e\n\u003cli\u003e\u0026sect; 139e Abs. 2 F\u0026uuml;nftes Sozialgesetzbuch (SGB V), 2019. https://www.gesetze-im-internet.de/sgb_5/__139e.html (accessed July 7, 2025).\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"mHealth, eHealth, Apps, Digital Therapeutics, LLM","lastPublishedDoi":"10.21203/rs.3.rs-7922462/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-7922462/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eIn 2019, Germany has introduced a unique regulatory framework for Digital Therapeutics (DTx) known as DiGAs, with the goal of integrating evidence-based DTx into statutory healthcare. DTx are approved for statutory health insurance reimbursement if the manufacturers can show evidence of health improvement, better coordination of care process, easier access to healthcare services, or the promotion of health literacy in a controlled study setting. Systematic investigations have revealed shortcomings in the evidence provided by manufacturers.\u003c/p\u003e\u003cp\u003eThe study examines patients experience and evaluate DTx in public App Stores by analyzing sentiments and thematic categories. In addition, it explores if the use of large language model (LLM) can provide effectively support the categorization and sentiment analysis of the patients\u0026rsquo; reviews.\u003c/p\u003e\u003cp\u003eFirst, a list of approved DTx is collected and limited to those with mobile apps. Patients\u0026rsquo; reviews were extracted from the public app stores using a tailored Python script. In the second step, the sentiments and topics of the patients\u0026rsquo; reviews were categorized into ten predefined categories using ChatGPT-4o. To ensure the quality, the LLM-based analysis was verified through manual validation.\u003c/p\u003e\u003cp\u003eIn total, 44 mobile DiGAs were included and were analyzed. After data extraction and cleansing, the final dataset comprises 4,328 patients' reviews containing at least one interpretable statement, resulting in 9,439 valid and interpretable statements. A systematic validation of the automated classification demonstrated exceptionally high model performance with 99% accuracy for sentiment classification (F1-scores of 1.00 for positive and 0.99 for negative categories) and 95% accuracy for category classification with an average F1-score of 0.95. While the categories \u003cem\u003eOverall Impression\u003c/em\u003e and \u003cem\u003eEffectiveness\u003c/em\u003e scored particularly well, patients were most negative for topics related to the login and registration process as well as technical malfunctions.\u003c/p\u003e\u003cp\u003eThe findings highlight both the potential and current limitations of reimbursable DTx from the patients\u0026rsquo; perspective. While patients value the therapeutic benefits and content quality of many DTx, technical functionality and usability, particularly during login and registration, are frequently criticized. The study also demonstrates that LLM-assisted analysis combined with human-in-the-loop validation offers an efficient approach for structuring patient feedback at scale. This combined approach could serve as a valuable complement to traditional clinical evaluation in future assessments of digital health applications.\u003c/p\u003e","manuscriptTitle":"What Patients Say About Reimbursable Digital Therapeutics in Germany: A Large-Scale App Store Review Analysis Using a Large Language Model","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-11-11 16:54:59","doi":"10.21203/rs.3.rs-7922462/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"d97d84cb-c4d7-4572-8a3f-7a1877f3656d","owner":[],"postedDate":"November 11th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":57709634,"name":"Biological sciences/Computational biology and bioinformatics"},{"id":57709635,"name":"Health sciences/Health care"},{"id":57709636,"name":"Physical sciences/Mathematics and computing"},{"id":57709637,"name":"Health sciences/Medical research"}],"tags":[],"updatedAt":"2025-12-05T18:53:09+00:00","versionOfRecord":[],"versionCreatedAt":"2025-11-11 16:54:59","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-7922462","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-7922462","identity":"rs-7922462","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}