Methodologies for the benefit-risk analysis of medical devices: A systematic review

Methodologies for the benefit-risk analysis of medical devices: A systematic review | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Systematic Review Methodologies for the benefit-risk analysis of medical devices: A systematic review Oscar Freyer, Fatemeh Jahed, Max Ostermann, Mirko Feig, Stephen Gilbert This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4832842/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Introduction : The use of medical devices (MDs) by patients carries both benefits and risks. Regulatory frameworks such as the EU Medical Device Regulation (MDR) and the US Federal Food, Drug, and Cosmetic Act (FD&C Act) mandate the systematic weighing of these benefits and risks through a benefit-risk analysis (BRA). This systematic review aims to identify existing BRA methodologies for MDs, and evaluate their strengths and weaknesses. Methods : A literature search was conducted in PubMed, Scopus, Web of Science, Google Scholar, and Semantic Scholar, covering publications from 2000 onwards using a search string that contained the search terms (1) methodologies AND (2) benefit-risk analysis AND (3) medical device OR in vitro diagnostic medical device. Peer-reviewed publications were included when they described BRA methodologies for MDs. The exclusion criteria included records on unrelated technologies, pharmaceuticals, non-English publications, and insufficient descriptions. The quality assessment was performed using a method proposed by Hawker et al. (2002). The methods and their characteristics were narratively summarised. Each method was assessed for its degree of objectivity and for using numerical calculations on a scale from 0 to 5. Results : The search identified 622 records, with six meeting the inclusion criteria. The included studies described ten BRA methodologies: Multicriteria Decision Analysis (MCDA), Health Outcomes Modelling (HOM), Stated-Choice Surveys (SCS), Composite Net Clinical Outcome (CNCO), Pairwise Comparisons (PC), Complete Profile of Benefits and Risks (CPBR), Net Benefit Score and Benefit-Risk Ratio (NBS & BRR), Quantitative Benefit-Risk Assessment (qBRA), Quantitative Benefit-Risk Determination (QBRD), and the FDA’s Benefit-Risk Framework (BRF). Methods varied in their reliance on numerical calculations and their degree of objectivity, with a significant correlation between both. The average level of objectivity was assessed as medium (2.1/5). Discussion : This review describes the ongoing debate between qualitative and quantitative BRA methods. While qualitative methods are often criticised for being subjective and biased, our findings reveal that quantitative methods, though more objective, still exhibit subjectivity, especially in endpoint identification and relevance assignment. Despite the rise of quantitative methods, often originating from the pharmaceutical industry, qualitative methods are often still used by MD decision-makers. Additionally, several quantitative methods are unsuited for certain difficult-to-quantify risks of MDs. Integrating both qualitative and quantitative elements may offer a more comprehensive BRA framework for MDs. Translational Medicine Benefit-Risk Analysis Medical Devices Systematic Review Methodologies PRISMA Regulation Risk Management Figures Figure 1 Introduction The use of medical devices (MDs) inherently carries risks in addition to the intended benefits for patients. One main objective of quality processes that are used in the development of MDs and their related regulatory approval processes is to identify and assess these risks and then mitigate them in a manner which brings them under acceptable control [ 1 – 3 ]. While some risks can be completely eliminated, others are only reduced in likelihood or severity, referred to as ‘residual risks’[ 3 ]. To determine whether a MD is safe and effective and should be made available on the market, many international MD legal frameworks, such as the EU’s Medical Device Regulation (MDR) [ 1 ] and the Federal Food, Drug & Cosmetic Act in the US (FD&C Act) [ 4 ] require manufacturers and regulators to systematically weigh the remaining risks against the benefits of the MD. This ‘benefit-risk-analysis’ (BRA), also referred to as ‘benefit-risk assessment’ or ‘benefit-risk determination’, is therefore a crucial step in the MD approval process and throughout the MDs entire lifecycle [ 1 – 3 , 5 ]. Both the FD&C Act in the US as well as the MDR in the EU state that BRA is required and define high-level principles of how it should be conducted. Neither legislation sets out specific methods [ 1 , 4 ]. These regulations are defined in more detail in official guidelines or by reference to recognised international standards [ 1 , 4 ]. For the US, multiple guidance documents issued by the Federal Drug and Cosmetic Agency (FDA) describe the concept, goal, and factors that should be considered within a BRA and define a method known as the FDA’s benefit-risk framework (BRF) [ 2 ]. Additionally, those guidance documents also indicate the applicability of ISO 14971:2019, the recognised standard for the risk management of MDs, which describes the BRA as part of the risk management process [ 3 ]. In the EU, reference is instead made to the harmonised version EN ISO 14971:2019 [ 6 ], which is identical in terms of content. The technical report ISO TR 24971:2020, which describes the implementation of the framework provided in ISO 14971:2019, provides further guidance and examples [ 7 ]. However, although these documents provide a general framework for the BRA, they do not specify a detailed method. It is, therefore, likely that different methods, qualitative or quantitative nature, are used in the approval processes of MDs both in the EU and the US. In recent years, the methods used for BRA in the pharmaceutical sector moved from qualitative and subjective approaches to more structured, quantitative and objective frameworks [ 8 , 9 ]. This shift is primarily enabled by the fact that the risks and benefits of pharmaceuticals are comparatively easy to quantify, as they usually have a narrow indication with an intended benefit that could serve as a clinical endpoint in a trial, making statistical evaluation more straightforward [ 10 ]. This similarly applies to the risks associated with a medication, as adverse events and negative effects are registered in clinical trials and also evaluated. For MDs, however, implementing such quantitative methods is considerably challenging since they often have a broader intended use (an example of this is an ultrasound scanner that can be used for multiple purposes in multiple conditions and on multiple body regions), and require considering stakeholder perspectives, accounting uncertainty, and the crucial role of usability, training and experience to assess benefits and risks accurately [ 3 , 10 , 11 ]. Additionally, innovative MDs often pose risks that were previously unknown and difficult to quantify, e.g., risks associated with cybersecurity [ 12 ], gamification of interfaces [ 13 ], artificial intelligence (AI) [ 14 , 15 ], or virtual reality [ 16 , 17 ]. Current guidance documents acknowledge this difficulty and the complexity of assessing such risks quantitatively [ 2 , 3 , 7 , 11 , 18 ]. While the relevance of the BRA is acknowledged, and several methods for the conduct of a BRA have been published [ 5 , 10 ], there is currently no definitive standard that defines how the BRA should be conducted, and that is applicable to developers and regulators [ 10 , 19 ]. This is paradoxical, as the market approval and surveillance approaches of MDs heavily rely on the concept of BRA [ 1 , 4 , 5 , 10 ]. To address the existing ambiguities around the methods on how to conduct a BRA, this systematic review primarily aims to (1) identify existing BRA methodologies for MDs and (2) describe the weaknesses and strengths of those methods. Methods To answer its primary research questions, this systematic review was conducted according to the Preferred Reporting Items for Systematic Reviews (PRISMA) reporting guidelines [ 20 ] and was pre-registered on February 22, 2024, with Open Science Framework Registries under the following DOI: 10.17605/OSF.IO/RT4SV . The completed PRISMA checklist can be found in Supplemental Material 1 . The protocol has been updated twice. Updates are accessible in the registry. Additional deviations to the protocol are listed in Supplemental Material 1. Search strategy The literature search was conducted from February 27, 2024, to April 05, 2024, using the following electronic databases: PubMed, Scopus, WebOfScience, Google Scholar, and Semantic Scholar, covering articles published after the year 2000 due to major changes in the MD regulations in the US and EU in recent years and since the BRA concept is comparatively novel [ 8 ]. A standardised search term was developed consisting of three main sub-term categories: (1) methodologies AND (2) Benefit-risk analysis AND (3) MDs or IVDs. A description of the complete search strings, adapted for each database, can be found in Supplemental Material 1 . Additionally, the reference lists of included articles were screened to identify additional relevant records. The bibliographic data of all identified publications was exported and gathered in an electronic database. Duplications were removed semi-automatic by a human reviewer comparing the bibliographic metadata of a record. Eligibility criteria Records were eligible for inclusion if they met the following inclusion criteria: (1) The record addressed benefits, risks, performance, and/or safety in MDs; (2) The record describes and/or discusses BRA methodologies for MDs, (3) the record was peer-reviewed. Records were excluded: (1) if they were focussed on unrelated technologies or products; (2) when the information provided was insufficient to undertake a scientific analysis; (3) when they were not available in English; (4) when the full text was not available; (5) when they were published before 2000. Study selection and data extraction After the removal of duplicates, the titles and abstracts of the identified articles were screened by two independent researchers to evaluate whether they met the criteria for inclusion. In the case of disagreements, a third independent reviewer was consulted. The screening process was performed by using rayyan.ai [ 21 ] between April 5th, 2024, and May 31, 2024. After the initial screening, two reviewers screened the full text of the included publications for suitability. One reviewer extracted the data of all included records using the software MAXQDA 2022 [ 22 ]. The extracted data included a description of the method, context, rationale, procedures, type of device or technology, and the methodological strengths and limitations identified by the authors of each record. At this stage, the references cited by each record were screened for relevance, adhering to the same inclusion and exclusion criteria. Quality assessment As a risk of bias analysis was not feasible for this type of evidence, the quality assessment method ‘Appraising the Evidence’ for qualitative health research, initially proposed by Hawker et al. [ 23 ], was used. The detailed quality assessment checklist for each included record can be found in Supplemental Material 1 . Data synthesis and analysis For the synthesis and analysis, one reviewer categorised the content of the records into broad methodological themes, considering the extracted data. The methods and their characteristics were then narratively summarised using figures, tables and narrative summaries. Additionally, an assessment of each method was conducted to identify the type and degree of both numerical and subjective components involved. This assessment used a two-dimensional framework, with one axis representing the extent to which a method relies on mathematical calculations (numerical vs. non-numerical) and the other axis evaluating the degree of human judgment involved (subjective vs. objective). Methods that heavily utilised mathematical calculations to determine the relevance and relation of benefits and risks were categorised as highly numerical, while those that did not employ mathematical calculations were classified as non-numerical. Concurrently, methods that applied stringent, automatic criteria for the identification and weight assignment without human judgment were considered highly objective, whereas those that depended solely on human judgment without any formalised system were deemed highly subjective. Each reviewer had to rate the objectivity and use of numerical calculations on a scale from 0 (lowest) to 5 (highest) based on predefined criteria. A complete description of the assessment framework can be found in Supplemental Material 1 . The assessment was conducted by three independent reviewers (authors F.J. and M.O.), with a third reviewer as the referee in case of deviations (Author O.F.). The statistical analysis was conducted using Python version 3.11.6 and the following libraries: pandas, and SciPy. Spearman's rank correlation coefficient and significance were calculated to test the correlation between the use of numerical methods and objectivity [ 24 ]. Results Search results The systematic search of five databases identified 622 records, from which 137 duplicates were removed. Of the remaining 485 records, 460 were excluded based on the screening of the title and abstract, and another 21 were excluded based on the assessment of the full text. In total, four records were included in the analysis through the systematic search process, an additional two were included via citation search. Figure 1 presents the selection process of the records as a PRISMA Flowchart. Characteristics of included papers and Quality Assessment The included records describe one or more BRA methods for MDs each. Four documents describe multiple methods [ 10 , 25 – 27 ]. A large percentage of the authors come from the US (4/6) [ 10 , 25 – 27 ]. All papers were published since 2014, with four of them published after 2020. The quality of the included records was mostly rated as ‘Fair’, with one record being ranked as ‘Good’. Most of the included records had weaknesses in their sampling strategy and transferability. Some records describe methods but only provide vague definitions of them [ 10 , 25 ]. This is, to some degree, reflected in the quality assessment of the paper describing the methods. Table 1 summarises the key characteristics of these studies, including their authors, year of publication, country, described methods, and quality assessment. Table 1 Characteristics of included reports and studies, ordered by the year of publication. Study Year Origin Described Methods Quality Rating Agapova et al. [ 10 ] 2014 USA Multicriteria Decision Analysis (MCDA); Health outcomes modelling (HOM); Stated-choice surveys (SCS) Fair Colopy et al. [ 25 ] 2015 USA Composite net clinical outcome (CNCO); Pairwise Comparisons (PC); Complete profile of benefits and risks (CPBR) Fair Sun et al. [ 26 ] 2020 USA Net Benefit Score and Benefit-Risk Ratio (NBS & BRR); Good Fu et al. [ 27 ] 2020 USA Quantitative Benefit-Risk Assessment (qBRA), MCDA Fair Su & Deng [ 5 ] 2023 China MCDA; NBS & BRR; Quantitative Benefit-Risk Determination (QBRD); FDA’s Benefit–Risk Framework (BRF) Fair Tervonen et al. [ 28 ] 2023 Switzerland MCDA Fair BRA Methods for Medical Devices The six documents included in our study described ten individual BRA methods for MDs: Multicriteria Decision Analysis (MCDA) [ 5 , 10 , 27 , 28 ], Health Outcomes Modelling (HOM) [ 10 ], Stated-Choice Surveys (SCS) [ 10 ], Composite Net Clinical Outcome (CNCO) [ 25 ], Pairwise Comparisons (PC) [ 25 ], Complete Profile of Benefits and Risks (CPBR) [ 25 ], Net Benefit Score and Benefit-Risk Ratio (NBS & BRR) [ 5 , 26 ], Quantitative Benefit-Risk Assessment (qBRA) [ 27 ], Quantitative Benefit-Risk Determination (QBRD) [ 5 ], and the FDA’s Benefit-Risk Framework (BRF) [ 5 ]. MDCA was discussed in four documents [ 5 , 10 , 27 , 28 ], while NBS & BRR was discussed in two documents [ 5 , 26 ]. Eight methods (MCDA, NBS & BRR, HOM, SCS, CNCO, CBPR, qBRA, PC) originate from the pharma sector and were initially designed to assess the benefits and risks of pharmaceuticals but could extended to MDs [ 5 , 10 , 25 – 28 ]. BRF was initially proposed by the FDA and is used for their regulatory decisions [ 2 , 5 ]. NBS & BRR is based on a similar FDA framework for drugs and biological products [ 29 ] and describes itself as adding quantitative aspects to it [ 26 ]. Three methods are further developments of existing ones, which either add quantifiable elements (NBS & BRR, qBRA) [ 26 , 27 ] or allow an evaluation of MDs in the first place (MCDA) [ 5 ]. MCDA is a structured and transparent method, allowing for the comparison of multiple alternatives by evaluating conflicting criteria [ 5 , 10 , 28 ]. However, it is complex to implement, relies on subjective weight assignments, and requires comprehensive data sources [ 5 , 10 ]. NBS & BRR provides a calculated score for a product’s benefit-risk profile but has a potential for oversimplification of complex benefits and assumes equal weights for all benefits and risks [ 5 , 26 ]. QBRD is a highly numerical method that calculates the benefit-risk profile based on pairwise comparisons [ 5 , 30 ]. Despite reducing subjectivity, it requires comprehensive data collection and struggles with non-quantifiable data [ 5 ]. HOM aggregates multiple benefits and risks into a single metric, such as quality-adjusted life years (QALY), which helps in understanding the long-term effects of a drug or device. Its limitations include the requirement for extensive quantitative data, the complexity of computations, and less transparency [ 10 ]. SCS assesses stakeholder perspectives on the benefit-risk profile of a product based on choice experiments but faces challenges with sample size requirements and potential discrepancies between experimental choices and real-life decisions [ 10 ]. CNCO combines multiple endpoints into a single metric but assumes clinical equivalence of endpoints and does not consider multiple risks for one patient [ 25 ]. Similar to QBRD, PC compares a single benefit with a single risk endpoint [ 5 , 25 ]. However, it is not as structured as QBRD, is limited when multiple endpoints are involved, and provides no clear judgment of the benefit-risk profile [ 25 ]. CPBR provides a transparent overview of all relevant endpoints by comparing them between a test device and a control group. However, it can be complex due to the high number of endpoints and assumes equivalence of all endpoints [ 25 ]. qBRA uses functions and statistical analyses for a structured and transparent assessment but requires subjective identification of relevant endpoints and involves complicated calculations [ 27 ]. BRF, provided and used by the FDA, is a qualitative method with a systematic approach for evaluating MDs benefits and risks BRF [ 2 , 5 ], but it could be subjective and untransparent [ 5 , 26 ]. Table 2 provides an overview of the included methods, their strength and limitations. Table 2 Methods for the benefit-risk analysis of medical devices. This table provides an overview of the methods identified through the systematic search. For each method, a summary and description, and their limitations and strengths, as described by the authors, are provided. Name Summary Description Limitations Strengths MCDA [ 5 , 10 , 26 , 28 ] MCDA is a structured framework that can calculate and compare BRA scores of multiple alternatives. For each alternative, benefit and risk criteria are defined. Each criterion is given a score based on performance data and a predefined performance matrix. The performance scores are multiplied with a relative weight for the criteria, predefined based on stakeholder perspectives. The weighted scores are summed. This total score can be compared with the total score of the alternatives. • Criteria are treated as independent • Complex to implement due to diverse data sources • Subjectivity in assigning weights • Subjectivity in defining benefit and risk criteria • Data from post-market surveillance and literature may vary in quality and reliability • Requires a control group/alternatives for comparison • Structured presentation • Transparency in assessing weights and in presenting clinical-related benefit and risk indicators • Supports subsequent sensitivity analysis to test the weight dependence • A single score for each alternative allows comparison • Flexible and adaptable to various scenarios • Considers uncertainty NBS & BRR [ 5 , 26 ] NBS & BRR is a quantitative version of the FDA’s BRF. It could be used to calculate a Net Benefit Score (NBS) and a Benefit-Risk Ratio (BRR) for a MD. This score shows whether the benefits outweigh the risks. All benefits, risks, and risk mitigations are rated with a score based on the severity and prevalence of the condition, the availability of alternatives, and the robustness of the data. The NBS is calculated by the sum of the benefit scores minus the sum of the risk scores (taking the mitigation of each risk into account). The BRR is the sum of the benefit scores divided by the sum of the risk scores (taking the mitigation of each risk into account). • May oversimplify complex benefits. • May not capture all risk factors. • Assumes equal weight for all benefits and risks. • Provides a numerical value for the benefits, risks and how they relate to each other. • Considers risk mitigation. • Allows to predict FDA decisions. QBRD [ 5 , 30 ] QBRD identifies specific benefits and risks, assigns values, calculates benefit-risk ratios, and compares them to the state of the art. Quantifiable risks and benefits are identified and get a numerical value assigned (magnitude for benefits, severity for risks) based on a predefined matrix. The benefit and risk values are then calculated by multiplying the numerical value with the frequency of occurrence. Each benefit will be pairwise compared with each risk by calculating a pair-specific benefit-risk ratio. This benefit-risk ration will be compared with the state-of-the-art and with pre-defined acceptance criteria. • Requires comprehensive data collection. • Subjectivity in determining magnitude values. • Complex for devices with multiple benefits and risks. • Does not integrate all benefits and risks. • Reduces subjectivity • Ensures clarity and specificity in the analysis. • Allows a quantitative comparison. • Provides a clear metric for decision-making. HOM [ 10 ] HOM uses a disease modelling approach where the benefit-risk profile is determined based on a single metric that describes the impact of a device on this disease. A measure that is used for disease modelling must be chosen, e.g., quality-adjusted life years (QALY). This measure is then adjusted by the benefits and risks (e.g., QALYs gained by the benefits and lost through the risks) based on quantitative and qualitative data. The impact of the device could then be expressed through the net health benefit. • Data may be limited for specific health states. • Requires extensive quantitative data. • Computationally intensive. • Utility weights may not always be available. • Untransparent results due to the combination of multiple benefits and risks into one metric. • Regulators were not very receptive to epidemiological methods such as QALYs. • Can incorporate multiple benefits and risks. • Aggregates outcomes over long periods. • Incorporates uncertainty in parameter values. • Helps decision-makers understand long-term outcomes. SCS [ 10 ] This method quantifies stakeholder preferences by presenting choices between different attributes of treatments. After the test device is defined, a number of choices, which represent the risk profile in abstract form, is created. Multiple stakeholders are presented with this choice experiment. From the survey data, weights and preferences for the benefit-risk profile could be calculated using statistical methods • Respondents may struggle with the understanding of rare risks. • Sample size requirements can be large. • Combining the results across multiple criteria is complex and requires advanced statistical techniques. • Subjects could decide differently in the experiment than they would in real life. • Captures preferences for specific attributes. • Quantifies stakeholder preferences for benefits and risks. • Estimates willingness to accept risks for benefits. • Provides clear metrics for decision-making. • Considers multiple stakeholder perspectives. • Considers uncertainty. CNCO [ 25 ] This method combines multiple benefit-risk endpoints into a single metric. Relevant endpoints are identified and combined into a single endpoint. This endpoint could then be measured, e.g., with the time-to-first event method. • Endpoints are considered as clinically equivalent. • Does not consider multiple risks for one patient. • Is relatively simple to calculate. PC [ 25 ] This method compares a single endpoint related to a benefit to a single endpoint related to a risk. A benefit and a risk related endpoint are chosen. A numerical value is assigned to each endpoint, e.g., based on a clinical trial or stakeholder interviews. Both values could be compared with each other to assess whether one outweighs the other. • Limited when trade-offs involve multiple endpoints. • Treats separate events as clinically equivalent. • Provides clear values for a benefit and risk pair • Could use different comparison metrics. • Could be used for several benefit and risk types. • Easy to understand and communicate. CPBR [ 25 ] This method provides an overview of all benefit- and risk-related endpoints. In this method, the benefit-risk profile of a test device is compared with a control group. All relevant benefit- and risk-related endpoints are defined. The probability of the occurrence of those endpoints is compared between the two groups. The risk difference between both groups could be shown in a forest plot. • Can be complex to assess due to the high number of endpoints. • Needs a comparison/control group. • Deciding to favour one device could be difficult if some endpoints favouring one device, and the rest the control. • Considers all endpoints as equivalent as a default. • Provides a transparent overview of all endpoints. • Allows an individualised assessment. qBRA [ 27 ] This method calculates a device’s benefit-risk profile through complex functions and statistical analyses. Favourable (benefits) and unfavourable (risks) endpoints are defined by a multi-functional team. A standardised score (SC) is calculated with a value function for each endpoint. Based on stakeholder preferences, the SCs are altered with weights. The overall benefit-risk value is then calculated with a utility function. To obtain numerical results, stochastic methods can be applied. • Subjective identification of relevant endpoints. • Needs comparison/control groups. • Involves complex calculations • Is structured and transparent. • Considers uncertainty. • Could be used for multiple use cases by choosing value and utility functions. BRF [ 2 , 5 ] This qualitative method is used by the FDA to determine whether the benefits of a device outweigh its risks. The BRF provides a worksheet and examples that could be followed. Initially, the type, magnitude, probability, and duration of the benefits are listed, and the uncertainty is assessed. Then, the risks and their severity, probability and duration are assessed. Based on this, it is determined, whether the benefits outweigh the risks? • Is qualitative and subjective. • Could be hard to predict and untransparent. • Provides a structured and systematic approach for FDA staff. Assessment of Quantitative and Qualitative Methods A majority of 70% (7/10) of the methods are described by their authors as quantitative (NBS & BRR, QBRD, HOM, SCS, CNCO, PC, and qBRA) [ 5 , 10 , 25 – 27 ], 20% (2/10) of described methods are described as both qualitative and quantitative (MCDA and CPBR) [ 5 , 10 , 25 , 27 , 28 ], and 10% (1/10) of the methods are described as qualitative only (BRF) [ 5 ]. While all included papers use this classification, only one provides a clear definition, as follows: qualitative BRA methods are defined as descriptive approaches that focus on individual benefits and risks, including their frequency and duration, and describe how this information influences the decision of whether the benefits outweigh the risks, while quantitative BRA methods integrate endpoints by quantifying their clinical relevance and trade-offs, which usually requires judgment in assigning numerical weights to benefits and risks [ 25 ]. This definition was used within the analysis of this review. Qualitative methods are described as inferior in effectively assessing trade-offs between different outcomes [ 25 , 28 ], more subjective [ 5 , 26 ], untransparent [ 26 ], and inferior in providing clear results [ 27 , 28 ]. In contrast, quantitative methods are described as more transparent, rational, and consistent for decision-making [ 5 , 10 , 26 , 27 ] by quantifying clinical relevance and trade-offs [ 5 , 25 ] but can be mathematically complex and potentially subjective in weight assignments [ 25 ]. We conducted an assessment to determine whether the methods described used numerical calculations and the degree of objectivity, regardless of how they were categorised by their authors. The use of numerical methods and calculations to determine a device’s benefit-risk profile ranged from not numerical (0) to completely numerical (5). At the same time, the objectivity for identifying benefits, risks, and their relevance ranged from not objective and thus, subjective (0) to nearly objective (4), with a mean value of 2.1 for all methods, and 2.2 for methods classified as quantitative. Spearman's rank correlation coefficient was 0.70 with a p-value of < 0.05, which is interpreted as a strong and significant positive correlation. An outlier here is the SCS, which relies heavily on numerical calculations based on statistical analyses but makes these on the basis of highly subjective stakeholder statements [ 10 ]. Table 3 provides an overview of the assessment. Table 3 Assessment Results. This table shows the results of the assessment of whether the methods described used numerical calculations and the degree of objectivity. Name Self-Classification Use of numerical calculations (0–5) Objectivity (0–5) MCDA [ 5 , 10 , 26 , 28 ] Qualitative/ Quantitative 4 3 NBS & BRR [ 5 , 26 ] Quantitative 5 3 QBRD [ 5 , 30 ] Quantitative 5 4 HOM [ 10 ] Quantitative 3 2 SCS [ 10 ] Quantitative 4 0 CNCO [ 25 ] Quantitative 3 2 PC [ 25 ] Quantitative 3 2 CPBR [ 25 ] Qualitative/ Quantitative 3 1 qBRA [ 27 ] Quantitative 5 3 BRF [ 2 , 5 ] Qualitative 0 1 Discussion Principal Results The first objective of this systematic review was to identify existing BRA methods for medical devices described in the scientific literature. We analysed six studies, revealing ten individual BRA methods for MDs. A majority (7) of those methods are described as quantitative, two as both qualitative and quantitative, and one as qualitative only. The quality of the included papers was mixed, ranging from x to y. Several papers describe methods which, however, are only insufficiently defined within those papers (HOM, CNCO, PC, CPBR) [ 10 , 25 ]. Our assessment of the methods shows that, even when being described as quantitative, the use of mathematical calculations to determine the variables within the BRA ranges from not numerical (0) to completely numerical (5). At the same time, the methods range from nearly subjective (1) to nearly objective (4). We identified a strong positive and significant correlation between objectivity and the use of numerical calculations. The second objective was to describe the weaknesses and strengths of the included methods as reported in the literature. For example, the MCDA method was noted for its structured approach to evaluating multiple conflicting criteria but was criticised for its reliance on point estimates and the complexity of implementation [ 5 , 10 , 27 , 28 ]. The NBS & BRR method provided clear numerical values for benefits and risks but was criticised for potentially oversimplifying complex benefits and risks [ 5 , 26 ]. Other methods, like HOM, SCS, and qBRA, offered comprehensive approaches but required extensive data and sophisticated analysis techniques [ 10 , 27 ]. Comparing Qualitative and Quantitative BRA Methods Multiple authors criticise the qualitative nature of some current methods, such as the BRF, as being subjective, susceptible to bias and therefore inferior to quantitative methods [ 5 , 26 – 28 ]. Our findings partially support this critique since the use of numeric methods correlated with the objectiveness of a method, although there are some outliers. Due to this assumption, the majority (70%) of the methods identified in this systematic review were developed to be quantitative in nature. However, other studies indicate that the most common BRA methods used by MD decision-makers in real-world assessments are qualitative [ 19 , 31 – 33 ]. One potential reason for this preference could be that many of the quantitative methods originate from the pharmaceutical industry, where such approaches are well-established and useful [ 5 , 8 ]. Multiple studies are attempts to adapt these methods for MDs [ 5 , 26 , 27 ], yet some may not be a perfect fit due to fundamental differences between pharmaceuticals and MDs described above [ 3 , 5 , 10 , 11 , 27 ]. Especially purely quantitative methods, such as PC and QBRD, could only process quantitative data, making them unsuitable for hard-to-quantifiable risks, such as those related to new technologies [ 12 – 17 ]. This limitation is acknowledged in the most recent version of applicable standards [ 3 , 7 ]. In our assessment of the use of numerical calculations and objectivity within each method, we observed that the extent to which numerical methods were used varied from three to five in the subgroup of quantitative methods. This categorisation as ‘quantitative’ was derived from the self-classification of the authors of the individual methods. However, the strong correlation between numerical methods and the objectivity shown in our analysis and expected by other authors [ 5 , 26 – 28 ] does not apply universally; there are methods that, despite being highly numerical, remain very subjective (SCS [ 10 ]). Additionally, our assessment of the overall level of objectivity was medium-low, with a mean value of 2.2 for methods labelled as quantitative. The finding, that even methods described as highly quantitative could, in fact, be subjective in their practical application, is supported by other researchers [ 25 ]. This could be due to the fact that most methods for the BRA of MDs include some degree of subjectiveness, at least in the identification of endpoints and the assignment of relevance, making complete objectivity potentially impossible. Practical Implications and Current Gaps To our knowledge, this is the first review that refers exclusively to BRA methods for MDs. Previous reviews have primarily focused on pharmaceuticals, resulting in a higher number of methods overall, including a greater variety of qualitative BRA methods [ 9 ]. However, due to their unique characteristics, not all of those methods could be applied to MDs [ 3 , 5 , 10 , 11 , 27 ]. Additionally, the BRA of pharmaceuticals relies on the comparison of effects between different groups, e.g., intervention against placebo [ 9 , 25 ], which is not always possible for MD testing scenarios. In this case, using the current state of the art (SOTA) as a base for comparison is recommended by methods and guidelines [ 1 , 5 , 30 ]. The importance of conducting a BRA is widely recognised and mandated by regulatory requirements and standards [ 1 – 4 , 7 , 34 ]. In the EU, there is currently no standardised approach or guideline for BRA, creating a considerable gap [ 5 , 10 , 19 ]. In contrast, in the US, the FDA has issued comprehensive guidance documents that describe the qualitative BRF and provide practical examples but leaves open whether quantitative methods may also be used [ 2 ]. However, both US and EU regulations recognise the BRA is an integral step for the approval process, in which all risks of a device are weighted against the clinical benefits to determine whether is should be made available on the market [ 1 , 2 , 4 ]. Thus, in standards and guidance, two levels of the BRA exist: first, an individual BRA for each unacceptable risk, and second, an overall BRA [ 1 – 3 , 7 , 11 , 18 , 35 ]. Some BRA methods, such as PC and QBRD, rely on pairwise comparisons of benefits and risks, which have some benefits (granularity, transparency) [ 5 , 25 , 30 ] but fall short in several critical aspects. Firstly, MD benefits and risks are not necessarily linked in pairs, making such comparisons impractical in many cases. Secondly, these methods rely on quantitative data, which could be difficult to collect for some risks. Thirdly, conducting the overall BRA required by standards and guidelines [ 1 – 3 , 7 , 11 , 18 ], could be difficult. Lastly, as the number of risks and benefits increases, the complexity of pairwise comparison methods like QBRD becomes challenging. With new categories of digital and connected MDs, new challenges that might be difficult to assess by some of the methods described in the review arise. It is already recognised that finding numerical data about risks connected to, for example, cybersecurity, user interfaces, or MDs integrated into IT systems could be hard to find [ 3 , 7 ] and that innovative MDs often pose risks that were previously unknown and are difficult to quantify [ 12 – 17 ]. This is a gap that current BRA methods described in the literature do not address adequately. While addressing this gap might be difficult, multiple authors state that there is no “one size fits all solution” [ 9 ] which covers all types of MDs, all stages of the product life cycle, and all cases in which a BRA might be necessary [ 9 , 10 , 19 , 25 ]. Therefore, different methods could be used depending on the level and type of risk. One approach would be to conduct a qualitative BRA first and then assess whether a quantitative BRA should be conducted [ 28 ]. Alternatively, combining qualitative and quantitative approaches into one method could be effective. This can be achieved with methods such as MCDA [ 5 , 28 ] or the BRF [ 2 , 5 ]. The benefit of such a combined approach is that these frameworks provide a holistic overview of MDs and are capable of incorporating new risks, making it a comprehensive tool for evaluating the safety and efficacy of especially innovative MDs. A benefit of the BRF, in particular, is that it can incorporate quantitative aspects through improvements proposed by various authors [ 26 ], thus overcoming the limitations of qualitative approaches (subjectivity and lack of transparency) by introducing consistency and objectivity to benefit evaluations [ 5 , 26 ]. We suggest that future research should empirically evaluate the methods identified in this review through applying them to a range of example devices (either real devices or realistic and detailed models of devices), as has already been done in part for pharmaceuticals [ 36 ]. This would enable the direct analysis and comparison of the strengths and weaknesses of the different methods and would furthermore allow the assessment of which methods should be excluded from application, which should be retained as reliable. It would also identify how the methods could be further refined and developed to be better fit for purpose. Limitations This systematic review is subject to several limitations. First, our findings may be influenced by a form of publication bias, where MD manufacturers do not disclose their used BRA method to the public. Second, we included only reports written in English. Although the majority of scientific research is published in English, this may have resulted in the exclusion of pertinent studies in other languages. Third, we limited our review to studies published after 2000. This restriction was necessary due to the major updates to regulations in recent years, but it may exclude valuable historical insights. Fourth, there is a scarcity of methods specifically designed for medical devices, with most methodologies being developed for pharmaceuticals. The significant differences between these fields mean that many pharmaceutical methods may not be applicable to MDs. Fifth, our review focused exclusively on the EU and the US. This geographic limitation may reduce the applicability of our findings to other regions. Sixth, while research indicates that companies often employ qualitative approaches [ 32 ], these are infrequently detailed in the scientific literature, potentially limiting our understanding of their use in practice. Seventh, the lack of standardisation of the terminology in the field may have affected the comprehensiveness of our search terms, possibly leading to the omission of relevant reports. These limitations suggest that our findings should be interpreted with caution and highlight the need for further research to gain a complete understanding. Conclusion Despite the critical role of the BRA in the regulatory processes of MD approval, there is currently no generally recognised method or standard. Multiple methods have been proposed in the literature, ranging from qualitative to quantitative. While traditionally, mostly qualitative methods have been used for the BRA, the number of quantitative methods, most of which originate from the pharmaceutical sector, has increased. Although the use of these methods correlates with increased objectivity, this was not particularly high overall for the methods analysed. Additionally, the specific aspects of MDs in terms of their intended use and particular risks that are not quantifiable make it difficult to use exclusively quantitative methods. Future approaches may benefit from integrating qualitative and quantitative elements to enhance comprehensiveness and objectivity. Addressing the gaps in current methodologies is essential for advancing the safety and efficacy evaluation of medical devices. Declarations Data availability Data collected and used in this review is available within the paper and its supplementary information files. Acknowledgements Funding This work was supported by the European Commission under the Horizon Europe Program, as part of the project CYMEDSEC (101094218). Views and opinions expressed are, however, those of the author(s) only and do not necessarily reflect those of the European Union. Neither the European Union, nor the granting authorities, can be held responsible for them. Responsibility for the information and views expressed therein lies entirely with the authors. Acknowledgement of AI Use During the preparation of this work the author(s) used DeepL, Grammarly, and ChatGPT (in versions GPT-3.5, GPT-4, GPT-4o) in order to improve grammar, spelling, and readability of the manuscript. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the content of the publication. Author information Contributions O.F. and F.J.contributed equally to this paper. O.F., S.G., and F.J. developed the design of the study. F.J. created the protocol. F.J. conducted the search. F.J., M.O., and O.F. conducted the screening. O.F. and F.J., conducted the data extraction. F.J., M.O. and O.F., conducted the analysis. F.J. and O.F. wrote the first draft of the manuscript. O.F., and F.J. conducted the quality assessment. F.J., M.O., and O.F., conducted the assessment of the methods. F.J., O.F., M.O., M.F., and S.G. contributed to the writing, interpretation of the content, and editing of the manuscript, revising it critically for important intellectual content. All authors had final approval of the completed version and take accountability for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. Corresponding author Correspondence to : Oscar Freyer, [email protected] , TUD Dresden University of Technology, Else Kröner Fresenius Center for Digital Health, Fetscherstr. 74, 01307 Dresden Competing interests S.G. is an Advisory Group member of the EY-coordinated ‘Study on Regulatory Governance and Innovation in the field of Medical Devices’ conducted on behalf of the DG SANTE of the European Commission. Authors declare the following competing interests: S.G. has or has had consulting relationships with Una Health GmbH, Lindus Health Ltd.; Flo Ltd, Thymia Ltd., FORUM Institut für Management GmbH, High-Tech Gründerfonds Management GmbH, Ada Health GmbH, and he holds share options in Ada Health GmbH. O.F. has worked as a freelance doctor for CRS Clinical Research Services Berlin GmbH, and has a leadership role and holds stock in WhalesDontFly GmbH. M.F. is an employee of Carl ZEISS Digital Innovation GmbH. F.J. and M.O. declare no competing interests. References Parliament E, European Council (2017). Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) 178/2002 and Regulation (EC) 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EEC (Text with EEA relevance)Text with EEA relevance U.S. Food and Drug Administration (FDA) (2019) Factors to Consider When Making Benefit-Risk Determinations in Medical Device Premarket Approval and De Novo Classifications International Organization for Standardization (2019) ISO 14971:2019 U.S (1938) Congress. United States Code: Federal Food, Drug, and Cosmetic Act Su G, Deng D (2023) Regulatory requirements and optimization of multiple criteria decision analysis to quantify the benefit-risk assessment of medical devices. Expert Rev Med Devices 20:273–281. 10.1080/17434440.2023.2190021 Deutsches Institut für Normung DIN EN ISO 14971:2022-04 International Organization for Standardization (2020) ISO/TR 24971:2020 Juhaeri J (2019) Benefit–risk evaluation: the past, present and future. Ther Adv Drug Saf 10:2042098619871180. 10.1177/2042098619871180 Mt-Isa S, Hallgreen CE, Wang N et al (2014) Balancing benefit and risk of medicines: a systematic review and classification of available methodologies. Pharmacoepidemiol Drug Saf 23:667–678. 10.1002/pds.3636 Agapova M, Devine EB, Bresnahan BW et al (2014) Applying Quantitative Benefit–Risk Analysis to Aid Regulatory Decision Making in Diagnostic Imaging: Methods, Challenges, and Opportunities. Acad Radiol 21:1138–1143. 10.1016/j.acra.2014.05.006 Food US, Drug Administration (FDA) (2019) Consideration of Uncertainty in Making Benefit-Risk Determinations in Medical Device Premarket Approvals, De Novo Classifications, and Humanitarian. Device Exemptions - Guidance for Industry and Food and Drug Administration Staff Perakslis E, Ginsburg GS (2021) Digital Health—The Need to Assess Benefits, Risks, and Value. JAMA 325:127–128. 10.1001/jama.2020.22919 Gilbert S, Freyer O, Wondra K (2024) The regulatory status of health apps that employ serious games and gamification Freyer O, Wiest I, Kather JN et al A future role for large language model health applications depends on regulators being brave enough to enforce safety standards (In Press). Lancet Digit Health Meskó B, Topol EJ (2023) The imperative for regulatory oversight of large language models (or generative AI) in healthcare. Npj Digit Med 6:1–6. 10.1038/s41746-023-00873-0 Freyer O, Gilbert S (2023) Bridging between hype and implementation in medical extended reality. Npj Digit Med 6:1–3. 10.1038/s41746-023-00972-y Souchet AD, Lourdeaux D, Pagani A et al (2023) A narrative review of immersive virtual reality’s ergonomics and risks at the workplace: cybersickness, visual fatigue, muscular fatigue, acute stress, and mental overload. Virtual Real 27:19–50. 10.1007/s10055-022-00672-0 U.S. Food and Drug Administration (FDA) (2016) Factors to Consider Regarding Benefit-Risk in Medical Device Product Availability, Compliance, and Enforcement Decisions -. Guidance for Industry and Food and Drug Administration Staff Smith MY, van Til J, DiSantostefano RL et al (2021) Quantitative Benefit–Risk Assessment: State of the Practice Within Industry. Ther Innov Regul Sci 55:415–425. 10.1007/s43441-020-00230-3 Page MJ, McKenzie JE, Bossuyt PM et al (2021) The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372:n71. 10.1136/bmj.n71 Rayyan (2021) Rayyan – Intelligent Systematic Review - Rayyan MAXQDA (2022) MAXQDA | All-In-One Qualitative & Mixed Methods Data Analysis Tool Hawker S, Payne S, Kerr C et al (2002) Appraising the Evidence: Reviewing Disparate Data Systematically. Qual Health Res 12:1284–1299. 10.1177/1049732302238251 Spearman C (1904) The Proof and Measurement of Association between Two Things. Am J Psychol 15:72–101. 10.2307/1412159 Colopy MW, Damaraju CV, He W et al (2015) Benefit-Risk Evaluation and Decision Making: Some Practical Insights. Ther Innov Regul Sci 49:425–433. 10.1177/2168479014565469 Sun S, Heske S, Mercadel M et al (2021) Predicting Regulatory Product Approvals Using a Proposed Quantitative Version of FDA’s Benefit–Risk Framework to Calculate Net-Benefit Score and Benefit–Risk Ratio. Ther Innov Regul Sci 55:129–137. 10.1007/s43441-020-00197-1 Fu B, Li X, Scott J et al (2020) A new framework to address challenges in quantitative benefit-risk assessment for medical products. Contemp Clin Trials 95:106073. 10.1016/j.cct.2020.106073 Tervonen T, Veldwijk J, Payne K et al (2023) Quantitative Benefit-Risk Assessment in Medical Product Decision Making: A Good Practices Report of an ISPOR Task Force. Value Health 26:449–460. 10.1016/j.jval.2022.12.006 U.S. Food and Drug Administration (FDA). Benefit-Risk Assessment for New Drug and Biological Products. (2023) Chung B, Kutty J (2022) Benefit-Risk Determination: A Quantitative Approach Kheir O, Jacoby A, Verwulgen S (2022) Risk Identification and Analysis in the Development of Medical Devices Among Start-Ups: Towards a Broader Risk Management Framework. Med Devices Auckl NZ 15:349–363. 10.2147/MDER.S375977 Gebel M, Renz C, Rodriguez L et al (2024) A Survey to Assess the Current Status of Structured Benefit-Risk Assessment in the Global Drug and Medical Device Industry. Ther Innov Regul Sci 58:756–765. 10.1007/s43441-024-00650-5 Leong J, McAuslane N, Walker S et al (2013) Is there a need for a universal benefit-risk assessment framework for medicines? Regulatory and industry perspectives. Pharmacoepidemiol Drug Saf 22:1004–1012. 10.1002/pds.3464 Medical Device Coordination Group (MDCG) (2019) MDCG 2019-16 Guidance on Cybersecurity for medical devices International Electrotechnical Commission (IEC) (2021) IEC 80001-1:2021 Hughes D, Waddingham E, Mt-Isa S et al (2016) Recommendations for benefit–risk assessment methodologies and visual representations. Pharmacoepidemiol Drug Saf 25:251–262. 10.1002/pds.3958 Additional Declarations The authors declare potential competing interests as follows: S.G. is an Advisory Group member of the EY-coordinated ‘Study on Regulatory Governance and Innovation in the field of Medical Devices’ conducted on behalf of the DG SANTE of the European Commission. Authors declare the following competing interests: S.G. has or has had consulting relationships with Una Health GmbH, Lindus Health Ltd.; Flo Ltd, Thymia Ltd., FORUM Institut für Management GmbH, High-Tech Gründerfonds Management GmbH, Ada Health GmbH, and he holds share options in Ada Health GmbH. O.F. has worked as a freelance doctor for CRS Clinical Research Services Berlin GmbH, and has a leadership role and holds stock in WhalesDontFly GmbH. M.F. is an employee of Carl ZEISS Digital Innovation GmbH. F.J. and M.O. declare no competing interests. Supplementary Files SupplementalMaterial.pdf Supplemental Material Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4832842","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Systematic Review","associatedPublications":[],"authors":[{"id":334060186,"identity":"9de7ea1e-53fc-4c4b-893a-3bc460552799","order_by":0,"name":"Oscar Freyer","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABFElEQVRIiWNgGAWjYPACOQj1AUQcAGIeGAM3MAaTjDMSSNXCzEOMFn7+ww8/VzAY5PNLJB/7bPvjMAPf7cOHP7ypuMPAd7wBqxbJGWnGkmcYDCxnzkhLnp2TcJhB8lxamuScM88YJM9gt8bgBg+DZAPDHwODGznGzEAt9RvO8Jgx87YdBkolYNdy/gzzzwYGAwP7G/mfmS2Athic4f/8mfcfkHH/AXYtB3LYJEFaDCRymJkZwFp4GKR5G0C2YPc+0C9mlg0gHWeeGTP2pKUDvcBmJjnn2GEeyTPYHQYMscc3GyoMDPjbkx8z/LCxZuA7w/z4w5uaw3J8x7F7H+o8IBZAM5MHj3qYffjMHAWjYBSMghENAEPWXcyKiGeYAAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0003-3323-2492","institution":"TUD Dresden University of Technology","correspondingAuthor":true,"prefix":"","firstName":"Oscar","middleName":"","lastName":"Freyer","suffix":""},{"id":334061805,"identity":"f15199be-0eaa-400b-93fe-ccf8dba18415","order_by":1,"name":"Fatemeh Jahed","email":"","orcid":"","institution":"TUD Dresden University of Technology","correspondingAuthor":false,"prefix":"","firstName":"Fatemeh","middleName":"","lastName":"Jahed","suffix":""},{"id":334061806,"identity":"5c112a81-1a22-4d50-a249-8625485ba532","order_by":2,"name":"Max Ostermann","email":"","orcid":"https://orcid.org/0009-0004-7808-2701","institution":"TUD Dresden University of Technology","correspondingAuthor":false,"prefix":"","firstName":"Max","middleName":"","lastName":"Ostermann","suffix":""},{"id":334061807,"identity":"c990c6c6-e477-4c9c-9586-17d7426aa28c","order_by":3,"name":"Mirko Feig","email":"","orcid":"","institution":"Carl ZEISS Digital Innovation","correspondingAuthor":false,"prefix":"","firstName":"Mirko","middleName":"","lastName":"Feig","suffix":""},{"id":334061808,"identity":"28c25ebf-4fe9-44ca-b9d0-3d8100c3d9e0","order_by":4,"name":"Stephen Gilbert","email":"","orcid":"https://orcid.org/0000-0002-1997-1689","institution":"TUD Dresden University of Technology","correspondingAuthor":false,"prefix":"","firstName":"Stephen","middleName":"","lastName":"Gilbert","suffix":""}],"badges":[],"createdAt":"2024-07-31 06:25:00","currentVersionCode":1,"declarations":{"humanSubjects":false,"vertebrateSubjects":false,"conflictsOfInterestStatement":true,"humanSubjectEthicalGuidelines":false,"humanSubjectConsent":false,"humanSubjectClinicalTrial":false,"humanSubjectCaseReport":false,"vertebrateSubjectEthicalGuidelines":false},"doi":"10.21203/rs.3.rs-4832842/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4832842/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":61811249,"identity":"94a87296-e51f-4c01-aa14-3f3da00a3a8e","added_by":"auto","created_at":"2024-08-05 20:24:36","extension":"jpeg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":388033,"visible":true,"origin":"","legend":"\u003cp\u003ePRISMA Flowchart. This flowchart follows the template provided by Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews [19].\u003c/p\u003e","description":"","filename":"floatimage1.jpeg","url":"https://assets-eu.researchsquare.com/files/rs-4832842/v1/a01d31b5cfb020b570e13d16.jpeg"},{"id":61811810,"identity":"69f6b648-bf8d-4bc8-bd99-8b5aa5b86500","added_by":"auto","created_at":"2024-08-05 20:32:36","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1174404,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4832842/v1/b39b6b29-6f8c-4acc-990f-bc75e6253321.pdf"},{"id":61809780,"identity":"106d6090-e4f6-4f57-87b6-f6ff51ac60d1","added_by":"auto","created_at":"2024-08-05 20:16:36","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":1371861,"visible":true,"origin":"","legend":"\u003cp\u003eSupplemental Material\u003c/p\u003e","description":"","filename":"SupplementalMaterial.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4832842/v1/affc8e5081adc69bb83ee158.pdf"}],"financialInterests":"The authors declare potential competing interests as follows: S.G. is an Advisory Group member of the EY-coordinated ‘Study on Regulatory Governance and Innovation in the field of Medical Devices’ conducted on behalf of the DG SANTE of the European Commission. Authors declare the following competing interests: S.G. has or has had consulting relationships with Una Health GmbH, Lindus Health Ltd.; Flo Ltd, Thymia Ltd., FORUM Institut für Management GmbH, High-Tech Gründerfonds Management GmbH, Ada Health GmbH, and he holds share options in Ada Health GmbH. O.F. has worked as a freelance doctor for CRS Clinical Research Services Berlin GmbH, and has a leadership role and holds stock in WhalesDontFly GmbH. M.F. is an employee of Carl ZEISS Digital Innovation GmbH. F.J. and M.O. declare no competing interests. ","formattedTitle":"\u003cp\u003e\u003cstrong\u003eMethodologies for the benefit-risk analysis of medical devices: A systematic review\u003c/strong\u003e\u003c/p\u003e","fulltext":[{"header":"Introduction","content":"\u003cp\u003eThe use of medical devices (MDs) inherently carries risks in addition to the intended benefits for patients. One main objective of quality processes that are used in the development of MDs and their related regulatory approval processes is to identify and assess these risks and then mitigate them in a manner which brings them under acceptable control [\u003cspan additionalcitationids=\"CR2\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. While some risks can be completely eliminated, others are only reduced in likelihood or severity, referred to as \u0026lsquo;residual risks\u0026rsquo;[\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. To determine whether a MD is safe and effective and should be made available on the market, many international MD legal frameworks, such as the EU\u0026rsquo;s Medical Device Regulation (MDR) [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e] and the Federal Food, Drug \u0026amp; Cosmetic Act in the US (FD\u0026amp;C Act) [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e] require manufacturers and regulators to systematically weigh the remaining risks against the benefits of the MD. This \u0026lsquo;benefit-risk-analysis\u0026rsquo; (BRA), also referred to as \u0026lsquo;benefit-risk assessment\u0026rsquo; or \u0026lsquo;benefit-risk determination\u0026rsquo;, is therefore a crucial step in the MD approval process and throughout the MDs entire lifecycle [\u003cspan additionalcitationids=\"CR2\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eBoth the FD\u0026amp;C Act in the US as well as the MDR in the EU state that BRA is required and define high-level principles of how it should be conducted. Neither legislation sets out specific methods [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. These regulations are defined in more detail in official guidelines or by reference to recognised international standards [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. For the US, multiple guidance documents issued by the Federal Drug and Cosmetic Agency (FDA) describe the concept, goal, and factors that should be considered within a BRA and define a method known as the FDA\u0026rsquo;s benefit-risk framework (BRF) [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. Additionally, those guidance documents also indicate the applicability of ISO 14971:2019, the recognised standard for the risk management of MDs, which describes the BRA as part of the risk management process [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. In the EU, reference is instead made to the harmonised version EN ISO 14971:2019 [\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e], which is identical in terms of content. The technical report ISO TR 24971:2020, which describes the implementation of the framework provided in ISO 14971:2019, provides further guidance and examples [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. However, although these documents provide a general framework for the BRA, they do not specify a detailed method. It is, therefore, likely that different methods, qualitative or quantitative nature, are used in the approval processes of MDs both in the EU and the US.\u003c/p\u003e \u003cp\u003eIn recent years, the methods used for BRA in the pharmaceutical sector moved from qualitative and subjective approaches to more structured, quantitative and objective frameworks [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e, \u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. This shift is primarily enabled by the fact that the risks and benefits of pharmaceuticals are comparatively easy to quantify, as they usually have a narrow indication with an intended benefit that could serve as a clinical endpoint in a trial, making statistical evaluation more straightforward [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. This similarly applies to the risks associated with a medication, as adverse events and negative effects are registered in clinical trials and also evaluated. For MDs, however, implementing such quantitative methods is considerably challenging since they often have a broader intended use (an example of this is an ultrasound scanner that can be used for multiple purposes in multiple conditions and on multiple body regions), and require considering stakeholder perspectives, accounting uncertainty, and the crucial role of usability, training and experience to assess benefits and risks accurately [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e]. Additionally, innovative MDs often pose risks that were previously unknown and difficult to quantify, e.g., risks associated with cybersecurity [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e], gamification of interfaces [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e], artificial intelligence (AI) [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e], or virtual reality [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e, \u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. Current guidance documents acknowledge this difficulty and the complexity of assessing such risks quantitatively [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e, \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e, \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eWhile the relevance of the BRA is acknowledged, and several methods for the conduct of a BRA have been published [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e], there is currently no definitive standard that defines how the BRA should be conducted, and that is applicable to developers and regulators [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e]. This is paradoxical, as the market approval and surveillance approaches of MDs heavily rely on the concept of BRA [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. To address the existing ambiguities around the methods on how to conduct a BRA, this systematic review primarily aims to (1) identify existing BRA methodologies for MDs and (2) describe the weaknesses and strengths of those methods.\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003eTo answer its primary research questions, this systematic review was conducted according to the Preferred Reporting Items for Systematic Reviews (PRISMA) reporting guidelines [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e] and was pre-registered on February 22, 2024, with Open Science Framework Registries under the following DOI: \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.17605/OSF.IO/RT4SV\u003c/span\u003e\u003cspan address=\"10.17605/OSF.IO/RT4SV\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e. The completed PRISMA checklist can be found in \u003cb\u003eSupplemental Material 1\u003c/b\u003e. The protocol has been updated twice. Updates are accessible in the registry. Additional deviations to the protocol are listed in \u003cb\u003eSupplemental Material 1.\u003c/b\u003e\u003c/p\u003e \u003cdiv id=\"Sec3\" class=\"Section2\"\u003e \u003ch2\u003eSearch strategy\u003c/h2\u003e \u003cp\u003eThe literature search was conducted from February 27, 2024, to April 05, 2024, using the following electronic databases: PubMed, Scopus, WebOfScience, Google Scholar, and Semantic Scholar, covering articles published after the year 2000 due to major changes in the MD regulations in the US and EU in recent years and since the BRA concept is comparatively novel [\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. A standardised search term was developed consisting of three main sub-term categories: (1) methodologies AND (2) Benefit-risk analysis AND (3) MDs or IVDs. A description of the complete search strings, adapted for each database, can be found in \u003cb\u003eSupplemental Material 1\u003c/b\u003e. Additionally, the reference lists of included articles were screened to identify additional relevant records. The bibliographic data of all identified publications was exported and gathered in an electronic database. Duplications were removed semi-automatic by a human reviewer comparing the bibliographic metadata of a record.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec4\" class=\"Section2\"\u003e \u003ch2\u003eEligibility criteria\u003c/h2\u003e \u003cp\u003eRecords were eligible for inclusion if they met the following inclusion criteria: (1) The record addressed benefits, risks, performance, and/or safety in MDs; (2) The record describes and/or discusses BRA methodologies for MDs, (3) the record was peer-reviewed. Records were excluded: (1) if they were focussed on unrelated technologies or products; (2) when the information provided was insufficient to undertake a scientific analysis; (3) when they were not available in English; (4) when the full text was not available; (5) when they were published before 2000.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec5\" class=\"Section2\"\u003e \u003ch2\u003eStudy selection and data extraction\u003c/h2\u003e \u003cp\u003eAfter the removal of duplicates, the titles and abstracts of the identified articles were screened by two independent researchers to evaluate whether they met the criteria for inclusion. In the case of disagreements, a third independent reviewer was consulted. The screening process was performed by using rayyan.ai [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e] between April 5th, 2024, and May 31, 2024. After the initial screening, two reviewers screened the full text of the included publications for suitability.\u003c/p\u003e \u003cp\u003eOne reviewer extracted the data of all included records using the software MAXQDA 2022 [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]. The extracted data included a description of the method, context, rationale, procedures, type of device or technology, and the methodological strengths and limitations identified by the authors of each record. At this stage, the references cited by each record were screened for relevance, adhering to the same inclusion and exclusion criteria.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec6\" class=\"Section2\"\u003e \u003ch2\u003eQuality assessment\u003c/h2\u003e \u003cp\u003eAs a risk of bias analysis was not feasible for this type of evidence, the quality assessment method \u0026lsquo;Appraising the Evidence\u0026rsquo; for qualitative health research, initially proposed by Hawker et al. [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e], was used. The detailed quality assessment checklist for each included record can be found in \u003cb\u003eSupplemental Material 1\u003c/b\u003e.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec7\" class=\"Section2\"\u003e \u003ch2\u003eData synthesis and analysis\u003c/h2\u003e \u003cp\u003eFor the synthesis and analysis, one reviewer categorised the content of the records into broad methodological themes, considering the extracted data. The methods and their characteristics were then narratively summarised using figures, tables and narrative summaries. Additionally, an assessment of each method was conducted to identify the type and degree of both numerical and subjective components involved. This assessment used a two-dimensional framework, with one axis representing the extent to which a method relies on mathematical calculations (numerical vs. non-numerical) and the other axis evaluating the degree of human judgment involved (subjective vs. objective). Methods that heavily utilised mathematical calculations to determine the relevance and relation of benefits and risks were categorised as highly numerical, while those that did not employ mathematical calculations were classified as non-numerical. Concurrently, methods that applied stringent, automatic criteria for the identification and weight assignment without human judgment were considered highly objective, whereas those that depended solely on human judgment without any formalised system were deemed highly subjective. Each reviewer had to rate the objectivity and use of numerical calculations on a scale from 0 (lowest) to 5 (highest) based on predefined criteria. A complete description of the assessment framework can be found in \u003cb\u003eSupplemental Material 1\u003c/b\u003e. The assessment was conducted by three independent reviewers (authors F.J. and M.O.), with a third reviewer as the referee in case of deviations (Author O.F.). The statistical analysis was conducted using Python version 3.11.6 and the following libraries: pandas, and SciPy. Spearman's rank correlation coefficient and significance were calculated to test the correlation between the use of numerical methods and objectivity [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e].\u003c/p\u003e \u003c/div\u003e"},{"header":"Results","content":"\u003cdiv id=\"Sec9\" class=\"Section2\"\u003e \u003ch2\u003eSearch results\u003c/h2\u003e \u003cp\u003eThe systematic search of five databases identified 622 records, from which 137 duplicates were removed. Of the remaining 485 records, 460 were excluded based on the screening of the title and abstract, and another 21 were excluded based on the assessment of the full text. In total, four records were included in the analysis through the systematic search process, an additional two were included via citation search. Figure\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e presents the selection process of the records as a PRISMA Flowchart.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec10\" class=\"Section2\"\u003e \u003ch2\u003eCharacteristics of included papers and Quality Assessment\u003c/h2\u003e \u003cp\u003eThe included records describe one or more BRA methods for MDs each. Four documents describe multiple methods [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan additionalcitationids=\"CR26\" citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e]. A large percentage of the authors come from the US (4/6) [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan additionalcitationids=\"CR26\" citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e]. All papers were published since 2014, with four of them published after 2020. The quality of the included records was mostly rated as \u0026lsquo;Fair\u0026rsquo;, with one record being ranked as \u0026lsquo;Good\u0026rsquo;. Most of the included records had weaknesses in their sampling strategy and transferability. Some records describe methods but only provide vague definitions of them [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]. This is, to some degree, reflected in the quality assessment of the paper describing the methods. Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e summarises the key characteristics of these studies, including their authors, year of publication, country, described methods, and quality assessment.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eCharacteristics of included reports and studies, ordered by the year of publication.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eStudy\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eYear\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eOrigin\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eDescribed Methods\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eQuality Rating\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eAgapova et al. [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2014\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eUSA\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eMulticriteria Decision Analysis (MCDA); Health outcomes modelling (HOM); Stated-choice surveys (SCS)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eFair\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eColopy et al. [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2015\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eUSA\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eComposite net clinical outcome (CNCO);\u003c/p\u003e \u003cp\u003ePairwise Comparisons (PC); Complete profile of benefits\u003c/p\u003e \u003cp\u003eand risks (CPBR)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eFair\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSun et al. [\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2020\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eUSA\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eNet Benefit Score\u0026nbsp; and Benefit-Risk Ratio (NBS \u0026amp; BRR);\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eGood\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFu et al. [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2020\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eUSA\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eQuantitative Benefit-Risk Assessment (qBRA), MCDA\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eFair\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSu \u0026amp; Deng [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2023\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eChina\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eMCDA; NBS \u0026amp; BRR; Quantitative Benefit-Risk Determination (QBRD); FDA\u0026rsquo;s Benefit\u0026ndash;Risk Framework (BRF)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eFair\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTervonen et al. [\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e2023\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSwitzerland\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eMCDA\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003eFair\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec11\" class=\"Section2\"\u003e \u003ch2\u003eBRA Methods for Medical Devices\u003c/h2\u003e \u003cp\u003eThe six documents included in our study described ten individual BRA methods for MDs: Multicriteria Decision Analysis (MCDA) [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e, \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e], Health Outcomes Modelling (HOM) [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e], Stated-Choice Surveys (SCS) [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e], Composite Net Clinical Outcome (CNCO) [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e], Pairwise Comparisons (PC) [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e], Complete Profile of Benefits and Risks (CPBR) [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e], Net Benefit Score and Benefit-Risk Ratio (NBS \u0026amp; BRR) [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e], Quantitative Benefit-Risk Assessment (qBRA) [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e], Quantitative Benefit-Risk Determination (QBRD) [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e], and the FDA\u0026rsquo;s Benefit-Risk Framework (BRF) [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. MDCA was discussed in four documents [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e, \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e], while NBS \u0026amp; BRR was discussed in two documents [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e]. Eight methods (MCDA, NBS \u0026amp; BRR, HOM, SCS, CNCO, CBPR, qBRA, PC) originate from the pharma sector and were initially designed to assess the benefits and risks of pharmaceuticals but could extended to MDs [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan additionalcitationids=\"CR26 CR27\" citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]. BRF was initially proposed by the FDA and is used for their regulatory decisions [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. NBS \u0026amp; BRR is based on a similar FDA framework for drugs and biological products [\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e] and describes itself as adding quantitative aspects to it [\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e]. Three methods are further developments of existing ones, which either add quantifiable elements (NBS \u0026amp; BRR, qBRA) [\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e] or allow an evaluation of MDs in the first place (MCDA) [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eMCDA is a structured and transparent method, allowing for the comparison of multiple alternatives by evaluating conflicting criteria [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]. However, it is complex to implement, relies on subjective weight assignments, and requires comprehensive data sources [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. NBS \u0026amp; BRR provides a calculated score for a product\u0026rsquo;s benefit-risk profile but has a potential for oversimplification of complex benefits and assumes equal weights for all benefits and risks [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e]. QBRD is a highly numerical method that calculates the benefit-risk profile based on pairwise comparisons [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e]. Despite reducing subjectivity, it requires comprehensive data collection and struggles with non-quantifiable data [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. HOM aggregates multiple benefits and risks into a single metric, such as quality-adjusted life years (QALY), which helps in understanding the long-term effects of a drug or device. Its limitations include the requirement for extensive quantitative data, the complexity of computations, and less transparency [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. SCS assesses stakeholder perspectives on the benefit-risk profile of a product based on choice experiments but faces challenges with sample size requirements and potential discrepancies between experimental choices and real-life decisions [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. CNCO combines multiple endpoints into a single metric but assumes clinical equivalence of endpoints and does not consider multiple risks for one patient [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]. Similar to QBRD, PC compares a single benefit with a single risk endpoint [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]. However, it is not as structured as QBRD, is limited when multiple endpoints are involved, and provides no clear judgment of the benefit-risk profile [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]. CPBR provides a transparent overview of all relevant endpoints by comparing them between a test device and a control group. However, it can be complex due to the high number of endpoints and assumes equivalence of all endpoints [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]. qBRA uses functions and statistical analyses for a structured and transparent assessment but requires subjective identification of relevant endpoints and involves complicated calculations [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e]. BRF, provided and used by the FDA, is a qualitative method with a systematic approach for evaluating MDs benefits and risks BRF [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e], but it could be subjective and untransparent [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e]. Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e provides an overview of the included methods, their strength and limitations.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003e\u003cb\u003eMethods for the benefit-risk analysis of medical devices.\u003c/b\u003e This table provides an overview of the methods identified through the systematic search. For each method, a summary and description, and their limitations and strengths, as described by the authors, are provided.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"5\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c5\" colnum=\"5\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eName\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSummary\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eDescription\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eLimitations\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c5\"\u003e \u003cp\u003eStrengths\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMCDA [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e, \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eMCDA is a structured framework that can calculate and compare BRA scores of multiple alternatives.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eFor each alternative, benefit and risk criteria are defined. Each criterion is given a score based on performance data and a predefined performance matrix. The performance scores are multiplied with a relative weight for the criteria, predefined based on stakeholder perspectives. The weighted scores are summed. This total score can be compared with the total score of the alternatives.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u0026bull; Criteria are treated as independent\u003c/p\u003e \u003cp\u003e\u0026bull; Complex to implement due to diverse data sources\u003c/p\u003e \u003cp\u003e\u0026bull; Subjectivity in assigning weights\u003c/p\u003e \u003cp\u003e\u0026bull; Subjectivity in defining benefit and risk criteria\u003c/p\u003e \u003cp\u003e\u0026bull; Data from post-market surveillance and literature may vary in quality and reliability\u003c/p\u003e \u003cp\u003e\u0026bull; Requires a control group/alternatives for comparison\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u0026bull; Structured presentation\u003c/p\u003e \u003cp\u003e\u0026bull; Transparency in assessing weights and in presenting clinical-related benefit and risk indicators\u003c/p\u003e \u003cp\u003e\u0026bull; Supports subsequent sensitivity analysis to test the weight dependence\u003c/p\u003e \u003cp\u003e\u0026bull; A single score for each alternative allows comparison\u003c/p\u003e \u003cp\u003e\u0026bull; Flexible and adaptable to various scenarios\u003c/p\u003e \u003cp\u003e\u0026bull; Considers\u0026nbsp; uncertainty\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNBS \u0026amp; BRR [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eNBS \u0026amp; BRR is a quantitative version of the FDA\u0026rsquo;s BRF. It could be used to calculate a Net Benefit Score (NBS) and a Benefit-Risk Ratio (BRR) for a MD. This score shows whether the benefits outweigh the risks.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAll benefits, risks, and risk mitigations are rated with a score based on the severity and prevalence of the condition, the availability of alternatives, and the robustness of the data. The NBS is calculated by the sum of the benefit scores minus the sum of the risk scores (taking the mitigation of each risk into account). The BRR is the sum of the benefit scores divided by the sum of the risk scores (taking the mitigation of each risk into account).\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u0026bull; May oversimplify complex benefits.\u003c/p\u003e \u003cp\u003e\u0026bull; May not capture all risk factors.\u003c/p\u003e \u003cp\u003e\u0026bull; Assumes equal weight for all benefits and risks.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u0026bull; Provides a numerical value for the benefits, risks and how they relate to each other.\u003c/p\u003e \u003cp\u003e\u0026bull; Considers risk mitigation.\u003c/p\u003e \u003cp\u003e\u0026bull; Allows to predict FDA decisions.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eQBRD [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eQBRD identifies specific benefits and risks, assigns values, calculates benefit-risk ratios, and compares them to the state of the art.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eQuantifiable risks and benefits are identified and get a numerical value assigned (magnitude for benefits, severity for risks)\u0026nbsp; based on a predefined matrix. The benefit and risk values are then calculated by multiplying the numerical value with the frequency of occurrence. Each benefit will be pairwise compared with each risk by calculating a pair-specific benefit-risk ratio. This benefit-risk ration will be compared with the state-of-the-art and with pre-defined acceptance criteria.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u0026bull; Requires comprehensive data collection.\u003c/p\u003e \u003cp\u003e\u0026bull; Subjectivity in determining magnitude values.\u003c/p\u003e \u003cp\u003e\u0026bull; Complex for devices with multiple benefits and risks.\u003c/p\u003e \u003cp\u003e\u0026bull; Does not integrate all benefits and risks.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u0026bull; Reduces subjectivity\u003c/p\u003e \u003cp\u003e\u0026bull; Ensures clarity and specificity in the analysis.\u003c/p\u003e \u003cp\u003e\u0026bull; Allows a quantitative comparison.\u003c/p\u003e \u003cp\u003e\u0026bull; Provides a clear metric for decision-making.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHOM [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eHOM uses a disease modelling approach where the benefit-risk profile is determined based on a single metric that describes the impact of a device on this disease.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eA measure that is used for disease modelling must be chosen, e.g., quality-adjusted life years (QALY). This measure is then adjusted by the benefits and risks (e.g., QALYs gained by the benefits and lost through the risks) based on quantitative and qualitative data. The impact of the device could then be expressed through the net health benefit.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u0026bull; Data may be limited for specific health states.\u003c/p\u003e \u003cp\u003e\u0026bull; Requires extensive quantitative data.\u003c/p\u003e \u003cp\u003e\u0026bull; Computationally intensive.\u003c/p\u003e \u003cp\u003e\u0026bull; Utility weights may not always be available.\u003c/p\u003e \u003cp\u003e\u0026bull; Untransparent results due to the combination of multiple benefits and risks into one metric.\u003c/p\u003e \u003cp\u003e\u0026bull; Regulators were not very receptive to epidemiological methods such as QALYs.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u0026bull; Can incorporate multiple benefits and risks.\u003c/p\u003e \u003cp\u003e\u0026bull; Aggregates outcomes over long periods.\u003c/p\u003e \u003cp\u003e\u0026bull; Incorporates uncertainty in parameter values.\u003c/p\u003e \u003cp\u003e\u0026bull; Helps decision-makers understand long-term outcomes.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSCS [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThis method quantifies stakeholder preferences by presenting choices between different attributes of treatments.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAfter the test device is defined, a number of choices, which represent the risk profile in abstract form, is created. Multiple stakeholders are presented with this choice experiment. From the survey data, weights and preferences for the benefit-risk profile could be calculated using statistical methods\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u0026bull; Respondents may struggle with the understanding of rare risks.\u003c/p\u003e \u003cp\u003e\u0026bull; Sample size requirements can be large.\u003c/p\u003e \u003cp\u003e\u0026bull; Combining the results across multiple criteria is complex and requires advanced statistical techniques.\u003c/p\u003e \u003cp\u003e\u0026bull; Subjects could decide differently in the experiment than they would in real life.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u0026bull; Captures preferences for specific attributes.\u003c/p\u003e \u003cp\u003e\u0026bull; Quantifies stakeholder preferences for benefits and risks.\u003c/p\u003e \u003cp\u003e\u0026bull; Estimates willingness to accept risks for benefits.\u003c/p\u003e \u003cp\u003e\u0026bull; Provides clear metrics for decision-making.\u003c/p\u003e \u003cp\u003e\u0026bull; Considers multiple stakeholder perspectives.\u003c/p\u003e \u003cp\u003e\u0026bull; Considers uncertainty.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCNCO [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThis method combines multiple benefit-risk endpoints into a single metric.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eRelevant endpoints are identified and combined into a single endpoint. This endpoint could then be measured, e.g., with the time-to-first event method.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u0026bull; Endpoints are considered as clinically equivalent.\u003c/p\u003e \u003cp\u003e\u0026bull; Does not consider multiple risks for one patient.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u0026bull; Is relatively simple to calculate.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePC [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThis method compares a single endpoint related to a benefit to a single endpoint related to a risk.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eA benefit and a risk related endpoint are chosen. A numerical value is assigned to each endpoint, e.g., based on a clinical trial or stakeholder interviews. Both values could be compared with each other to assess\u0026nbsp; whether one outweighs the other.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u0026bull; Limited when trade-offs involve multiple endpoints.\u003c/p\u003e \u003cp\u003e\u0026bull; Treats separate events as clinically equivalent.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u0026bull; Provides clear values for a benefit and risk pair\u003c/p\u003e \u003cp\u003e\u0026bull; Could use different comparison metrics.\u003c/p\u003e \u003cp\u003e\u0026bull; Could be used for several benefit and risk types.\u003c/p\u003e \u003cp\u003e\u0026bull; Easy to understand and communicate.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCPBR [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThis method provides an overview of all benefit- and risk-related endpoints.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eIn this method, the benefit-risk profile of a test device is compared with a control group.\u003c/p\u003e \u003cp\u003eAll relevant benefit- and risk-related endpoints are defined. The probability of the occurrence of those endpoints is compared between the two groups. The risk difference between both groups could be shown in a forest plot.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u0026bull; Can be complex to assess due to the high number of endpoints.\u003c/p\u003e \u003cp\u003e\u0026bull; Needs a comparison/control group.\u003c/p\u003e \u003cp\u003e\u0026bull; Deciding to favour one device could be difficult if some endpoints favouring one device, and the rest the control.\u003c/p\u003e \u003cp\u003e\u0026bull; Considers all endpoints as equivalent as a default.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u0026bull; Provides a transparent overview of all endpoints.\u003c/p\u003e \u003cp\u003e\u0026bull; Allows an individualised assessment.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eqBRA [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThis method calculates a device\u0026rsquo;s benefit-risk profile through complex functions and statistical analyses.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eFavourable (benefits) and unfavourable (risks) endpoints are defined by a multi-functional team. A standardised score (SC) is calculated with a value function for each endpoint. Based on stakeholder preferences, the SCs are altered with weights. The overall benefit-risk value is then calculated with a utility function. To obtain numerical results, stochastic methods can be applied.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u0026bull; Subjective identification of relevant endpoints.\u003c/p\u003e \u003cp\u003e\u0026bull; Needs comparison/control groups.\u003c/p\u003e \u003cp\u003e\u0026bull; Involves complex calculations\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u0026bull; Is structured and transparent.\u003c/p\u003e \u003cp\u003e\u0026bull; Considers uncertainty.\u003c/p\u003e \u003cp\u003e\u0026bull; Could be used for multiple use cases by choosing value and utility functions.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBRF [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eThis qualitative method is used by the FDA to determine whether the benefits of a device outweigh its risks.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eThe BRF provides a worksheet and examples that could be followed. Initially, the type, magnitude, probability, and duration of the benefits are listed, and the uncertainty is assessed. Then, the risks and their severity, probability and duration are assessed. Based on this, it is determined, whether the benefits outweigh the risks?\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e\u0026bull; Is qualitative and subjective.\u003c/p\u003e \u003cp\u003e\u0026bull; Could be hard to predict and untransparent.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c5\"\u003e \u003cp\u003e\u0026bull; Provides a structured and systematic approach for FDA staff.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec12\" class=\"Section2\"\u003e \u003ch2\u003eAssessment of Quantitative and Qualitative Methods\u003c/h2\u003e \u003cp\u003eA majority of 70% (7/10) of the methods are described by their authors as quantitative (NBS \u0026amp; BRR, QBRD, HOM, SCS, CNCO, PC, and qBRA) [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan additionalcitationids=\"CR26\" citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e], 20% (2/10) of described methods are described as both qualitative and quantitative (MCDA and CPBR) [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e, \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e], and 10% (1/10) of the methods are described as qualitative only (BRF) [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. While all included papers use this classification, only one provides a clear definition, as follows: qualitative BRA methods are defined as descriptive approaches that focus on individual benefits and risks, including their frequency and duration, and describe how this information influences the decision of whether the benefits outweigh the risks, while quantitative BRA methods integrate endpoints by quantifying their clinical relevance and trade-offs, which usually requires judgment in assigning numerical weights to benefits and risks [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]. This definition was used within the analysis of this review.\u003c/p\u003e \u003cp\u003eQualitative methods are described as inferior in effectively assessing trade-offs between different outcomes [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e, \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e], more subjective [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e], untransparent [\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e], and inferior in providing clear results [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e, \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]. In contrast, quantitative methods are described as more transparent, rational, and consistent for decision-making [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e] by quantifying clinical relevance and trade-offs [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e] but can be mathematically complex and potentially subjective in weight assignments [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eWe conducted an assessment to determine whether the methods described used numerical calculations and the degree of objectivity, regardless of how they were categorised by their authors. The use of numerical methods and calculations to determine a device\u0026rsquo;s benefit-risk profile ranged from not numerical (0) to completely numerical (5). At the same time, the objectivity for identifying benefits, risks, and their relevance ranged from not objective and thus, subjective (0) to nearly objective (4), with a mean value of 2.1 for all methods, and 2.2 for methods classified as quantitative. Spearman's rank correlation coefficient was 0.70 with a p-value of \u0026lt;\u0026thinsp;0.05, which is interpreted as a strong and significant positive correlation. An outlier here is the SCS, which relies heavily on numerical calculations based on statistical analyses but makes these on the basis of highly subjective stakeholder statements [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e provides an overview of the assessment.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003e\u003cb\u003eAssessment Results.\u003c/b\u003e This table shows the results of the assessment of whether the methods described used numerical calculations and the degree of objectivity.\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eName\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSelf-Classification\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eUse of numerical calculations (0\u0026ndash;5)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eObjectivity (0\u0026ndash;5)\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eMCDA [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e, \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eQualitative/ Quantitative\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eNBS \u0026amp; BRR [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eQuantitative\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eQBRD [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eQuantitative\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHOM [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eQuantitative\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSCS [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eQuantitative\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCNCO [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eQuantitative\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePC [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eQuantitative\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e2\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eCPBR [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eQualitative/ Quantitative\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eqBRA [\u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eQuantitative\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eBRF [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eQualitative\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e0\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e1\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003c/div\u003e"},{"header":"Discussion","content":"\u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003ePrincipal Results\u003c/h2\u003e \u003cp\u003eThe first objective of this systematic review was to identify existing BRA methods for medical devices described in the scientific literature. We analysed six studies, revealing ten individual BRA methods for MDs. A majority (7) of those methods are described as quantitative, two as both qualitative and quantitative, and one as qualitative only. The quality of the included papers was mixed, ranging from x to y. Several papers describe methods which, however, are only insufficiently defined within those papers (HOM, CNCO, PC, CPBR) [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]. Our assessment of the methods shows that, even when being described as quantitative, the use of mathematical calculations to determine the variables within the BRA ranges from not numerical (0) to completely numerical (5). At the same time, the methods range from nearly subjective (1) to nearly objective (4). We identified a strong positive and significant correlation between objectivity and the use of numerical calculations.\u003c/p\u003e \u003cp\u003eThe second objective was to describe the weaknesses and strengths of the included methods as reported in the literature. For example, the MCDA method was noted for its structured approach to evaluating multiple conflicting criteria but was criticised for its reliance on point estimates and the complexity of implementation [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e, \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]. The NBS \u0026amp; BRR method provided clear numerical values for benefits and risks but was criticised for potentially oversimplifying complex benefits and risks [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e]. Other methods, like HOM, SCS, and qBRA, offered comprehensive approaches but required extensive data and sophisticated analysis techniques [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e].\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec15\" class=\"Section2\"\u003e \u003ch2\u003eComparing Qualitative and Quantitative BRA Methods\u003c/h2\u003e \u003cp\u003eMultiple authors criticise the qualitative nature of some current methods, such as the BRF, as being subjective, susceptible to bias and therefore inferior to quantitative methods [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan additionalcitationids=\"CR27\" citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]. Our findings partially support this critique since the use of numeric methods correlated with the objectiveness of a method, although there are some outliers. Due to this assumption, the majority (70%) of the methods identified in this systematic review were developed to be quantitative in nature. However, other studies indicate that the most common BRA methods used by MD decision-makers in real-world assessments are qualitative [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e, \u003cspan additionalcitationids=\"CR32\" citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e]. One potential reason for this preference could be that many of the quantitative methods originate from the pharmaceutical industry, where such approaches are well-established and useful [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e]. Multiple studies are attempts to adapt these methods for MDs [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e], yet some may not be a perfect fit due to fundamental differences between pharmaceuticals and MDs described above [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e]. Especially purely quantitative methods, such as PC and QBRD, could only process quantitative data, making them unsuitable for hard-to-quantifiable risks, such as those related to new technologies [\u003cspan additionalcitationids=\"CR13 CR14 CR15 CR16\" citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. This limitation is acknowledged in the most recent version of applicable standards [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eIn our assessment of the use of numerical calculations and objectivity within each method, we observed that the extent to which numerical methods were used varied from three to five in the subgroup of quantitative methods. This categorisation as \u0026lsquo;quantitative\u0026rsquo; was derived from the self-classification of the authors of the individual methods. However, the strong correlation between numerical methods and the objectivity shown in our analysis and expected by other authors [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan additionalcitationids=\"CR27\" citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e] does not apply universally; there are methods that, despite being highly numerical, remain very subjective (SCS [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]). Additionally, our assessment of the overall level of objectivity was medium-low, with a mean value of 2.2 for methods labelled as quantitative. The finding, that even methods described as highly quantitative could, in fact, be subjective in their practical application, is supported by other researchers [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]. This could be due to the fact that most methods for the BRA of MDs include some degree of subjectiveness, at least in the identification of endpoints and the assignment of relevance, making complete objectivity potentially impossible.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec16\" class=\"Section2\"\u003e \u003ch2\u003ePractical Implications and Current Gaps\u003c/h2\u003e \u003cp\u003eTo our knowledge, this is the first review that refers exclusively to BRA methods for MDs. Previous reviews have primarily focused on pharmaceuticals, resulting in a higher number of methods overall, including a greater variety of qualitative BRA methods [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. However, due to their unique characteristics, not all of those methods could be applied to MDs [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e]. Additionally, the BRA of pharmaceuticals relies on the comparison of effects between different groups, e.g., intervention against placebo [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e, \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e], which is not always possible for MD testing scenarios. In this case, using the current state of the art (SOTA) as a base for comparison is recommended by methods and guidelines [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe importance of conducting a BRA is widely recognised and mandated by regulatory requirements and standards [\u003cspan additionalcitationids=\"CR2 CR3\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e, \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e, \u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e]. In the EU, there is currently no standardised approach or guideline for BRA, creating a considerable gap [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e]. In contrast, in the US, the FDA has issued comprehensive guidance documents that describe the qualitative BRF and provide practical examples but leaves open whether quantitative methods may also be used [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e]. However, both US and EU regulations recognise the BRA is an integral step for the approval process, in which all risks of a device are weighted against the clinical benefits to determine whether is should be made available on the market [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e, \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. Thus, in standards and guidance, two levels of the BRA exist: first, an individual BRA for each unacceptable risk, and second, an overall BRA [\u003cspan additionalcitationids=\"CR2\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e, \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e, \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e, \u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e35\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eSome BRA methods, such as PC and QBRD, rely on pairwise comparisons of benefits and risks, which have some benefits (granularity, transparency) [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e, \u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e] but fall short in several critical aspects. Firstly, MD benefits and risks are not necessarily linked in pairs, making such comparisons impractical in many cases. Secondly, these methods rely on quantitative data, which could be difficult to collect for some risks. Thirdly, conducting the overall BRA required by standards and guidelines [\u003cspan additionalcitationids=\"CR2\" citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e, \u003cspan citationid=\"CR11\" class=\"CitationRef\"\u003e11\u003c/span\u003e, \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e], could be difficult. Lastly, as the number of risks and benefits increases, the complexity of pairwise comparison methods like QBRD becomes challenging.\u003c/p\u003e \u003cp\u003eWith new categories of digital and connected MDs, new challenges that might be difficult to assess by some of the methods described in the review arise. It is already recognised that finding numerical data about risks connected to, for example, cybersecurity, user interfaces, or MDs integrated into IT systems could be hard to find [\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e, \u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e] and that innovative MDs often pose risks that were previously unknown and are difficult to quantify [\u003cspan additionalcitationids=\"CR13 CR14 CR15 CR16\" citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e]. This is a gap that current BRA methods described in the literature do not address adequately.\u003c/p\u003e \u003cp\u003eWhile addressing this gap might be difficult, multiple authors state that there is no \u0026ldquo;one size fits all solution\u0026rdquo; [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e] which covers all types of MDs, all stages of the product life cycle, and all cases in which a BRA might be necessary [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e, \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e, \u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e, \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]. Therefore, different methods could be used depending on the level and type of risk. One approach would be to conduct a qualitative BRA first and then assess whether a quantitative BRA should be conducted [\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e]. Alternatively, combining qualitative and quantitative approaches into one method could be effective. This can be achieved with methods such as MCDA [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e] or the BRF [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. The benefit of such a combined approach is that these frameworks provide a holistic overview of MDs and are capable of incorporating new risks, making it a comprehensive tool for evaluating the safety and efficacy of especially innovative MDs. A benefit of the BRF, in particular, is that it can incorporate quantitative aspects through improvements proposed by various authors [\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e], thus overcoming the limitations of qualitative approaches (subjectivity and lack of transparency) by introducing consistency and objectivity to benefit evaluations [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e, \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eWe suggest that future research should empirically evaluate the methods identified in this review through applying them to a range of example devices (either real devices or realistic and detailed models of devices), as has already been done in part for pharmaceuticals [\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e]. This would enable the direct analysis and comparison of the strengths and weaknesses of the different methods and would furthermore allow the assessment of which methods should be excluded from application, which should be retained as reliable. It would also identify how the methods could be further refined and developed to be better fit for purpose.\u003c/p\u003e \u003c/div\u003e \u003cdiv id=\"Sec17\" class=\"Section2\"\u003e \u003ch2\u003eLimitations\u003c/h2\u003e \u003cp\u003eThis systematic review is subject to several limitations. First, our findings may be influenced by a form of publication bias, where MD manufacturers do not disclose their used BRA method to the public. Second, we included only reports written in English. Although the majority of scientific research is published in English, this may have resulted in the exclusion of pertinent studies in other languages. Third, we limited our review to studies published after 2000. This restriction was necessary due to the major updates to regulations in recent years, but it may exclude valuable historical insights. Fourth, there is a scarcity of methods specifically designed for medical devices, with most methodologies being developed for pharmaceuticals. The significant differences between these fields mean that many pharmaceutical methods may not be applicable to MDs. Fifth, our review focused exclusively on the EU and the US. This geographic limitation may reduce the applicability of our findings to other regions. Sixth, while research indicates that companies often employ qualitative approaches [\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e], these are infrequently detailed in the scientific literature, potentially limiting our understanding of their use in practice. Seventh, the lack of standardisation of the terminology in the field may have affected the comprehensiveness of our search terms, possibly leading to the omission of relevant reports. These limitations suggest that our findings should be interpreted with caution and highlight the need for further research to gain a complete understanding.\u003c/p\u003e \u003c/div\u003e"},{"header":"Conclusion","content":"\u003cp\u003eDespite the critical role of the BRA in the regulatory processes of MD approval, there is currently no generally recognised method or standard. Multiple methods have been proposed in the literature, ranging from qualitative to quantitative. While traditionally, mostly qualitative methods have been used for the BRA, the number of quantitative methods, most of which originate from the pharmaceutical sector, has increased. Although the use of these methods correlates with increased objectivity, this was not particularly high overall for the methods analysed. Additionally, the specific aspects of MDs in terms of their intended use and particular risks that are not quantifiable make it difficult to use exclusively quantitative methods. Future approaches may benefit from integrating qualitative and quantitative elements to enhance comprehensiveness and objectivity. Addressing the gaps in current methodologies is essential for advancing the safety and efficacy evaluation of medical devices.\u003c/p\u003e "},{"header":"Declarations","content":"\u003cdiv id=\"Sec19\" class=\"Section2\"\u003e \u003ch2\u003eData availability\u003c/h2\u003e \u003cp\u003eData collected and used in this review is available within the paper and its supplementary information files.\u003c/p\u003e \u003c/div\u003e\n\u003ch2\u003eAcknowledgements\u003c/h2\u003e\n\u003ch2\u003eFunding\u003c/h2\u003e\n\u003cp\u003eThis work was supported by the European Commission under the Horizon Europe Program, as part of the project CYMEDSEC (101094218). Views and opinions expressed are, however, those of the author(s) only and do not necessarily reflect those of the European Union. Neither the European Union, nor the granting authorities, can be held responsible for them. Responsibility for the information and views expressed therein lies entirely with the authors.\u0026nbsp;\u003c/p\u003e\n\u003ch2\u003eAcknowledgement of AI Use\u003c/h2\u003e\n\u003cp\u003eDuring the preparation of this work the author(s) used DeepL, Grammarly, and ChatGPT (in versions GPT-3.5, GPT-4, GPT-4o) in order to improve grammar, spelling, and readability of the manuscript. After using this tool/service, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the content of the publication.\u003c/p\u003e\n\u003ch2\u003eAuthor information\u003c/h2\u003e\n\u003ch2\u003eContributions\u003c/h2\u003e\n\u003cp\u003eO.F. and F.J.contributed equally to this paper. O.F., S.G., and F.J. developed the design of the study. F.J. created the protocol. F.J. conducted the search. F.J., M.O., and O.F. conducted the screening. O.F. and F.J., conducted the data extraction. F.J., M.O. and O.F., conducted the analysis. F.J. and O.F. wrote the first draft of the manuscript. O.F., and F.J. conducted the quality assessment. F.J., M.O., and O.F., conducted the assessment of the methods. F.J., O.F., M.O., M.F., and S.G. contributed to the writing, interpretation of the content, and editing of the manuscript, revising it critically for important intellectual content. All authors had final approval of the completed version and take accountability for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.\u003c/p\u003e\n\u003ch2\u003eCorresponding author\u003c/h2\u003e\n\u003cp\u003eCorrespondence to : Oscar Freyer, [email protected], TUD Dresden University of Technology, Else Kr\u0026ouml;ner Fresenius Center for Digital Health, Fetscherstr. 74, 01307 Dresden\u003c/p\u003e\n\u003ch2\u003eCompeting interests\u003c/h2\u003e\n\u003cp\u003eS.G. is an Advisory Group member of the EY-coordinated \u0026lsquo;Study on Regulatory Governance and Innovation in the field of Medical Devices\u0026rsquo; conducted on behalf of the DG SANTE of the European Commission. Authors declare the following competing interests: S.G. has or has had consulting relationships with Una Health GmbH, Lindus Health Ltd.; Flo Ltd, Thymia Ltd., FORUM Institut f\u0026uuml;r Management GmbH, High-Tech Gr\u0026uuml;nderfonds Management GmbH, Ada Health GmbH, and he holds share options in Ada Health GmbH. O.F. has worked as a freelance doctor for CRS Clinical Research Services Berlin GmbH, and has a leadership role and holds stock in WhalesDontFly GmbH. M.F. is an employee of Carl ZEISS Digital Innovation GmbH. F.J. and M.O. declare no competing interests.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003eParliament E, European Council (2017). Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) 178/2002 and Regulation (EC) 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EEC (Text with EEA relevance)Text with EEA relevance\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eU.S. Food and Drug Administration (FDA) (2019) Factors to Consider When Making Benefit-Risk Determinations in Medical Device Premarket Approval and De Novo Classifications\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eInternational Organization for Standardization (2019) ISO 14971:2019\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eU.S (1938) Congress. United States Code: Federal Food, Drug, and Cosmetic Act\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSu G, Deng D (2023) Regulatory requirements and optimization of multiple criteria decision analysis to quantify the benefit-risk assessment of medical devices. Expert Rev Med Devices 20:273\u0026ndash;281. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1080/17434440.2023.2190021\u003c/span\u003e\u003cspan address=\"10.1080/17434440.2023.2190021\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDeutsches Institut f\u0026uuml;r Normung DIN EN ISO 14971:2022-04\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eInternational Organization for Standardization (2020) ISO/TR 24971:2020\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eJuhaeri J (2019) Benefit\u0026ndash;risk evaluation: the past, present and future. Ther Adv Drug Saf 10:2042098619871180. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1177/2042098619871180\u003c/span\u003e\u003cspan address=\"10.1177/2042098619871180\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMt-Isa S, Hallgreen CE, Wang N et al (2014) Balancing benefit and risk of medicines: a systematic review and classification of available methodologies. Pharmacoepidemiol Drug Saf 23:667\u0026ndash;678. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1002/pds.3636\u003c/span\u003e\u003cspan address=\"10.1002/pds.3636\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eAgapova M, Devine EB, Bresnahan BW et al (2014) Applying Quantitative Benefit\u0026ndash;Risk Analysis to Aid Regulatory Decision Making in Diagnostic Imaging: Methods, Challenges, and Opportunities. Acad Radiol 21:1138\u0026ndash;1143. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.acra.2014.05.006\u003c/span\u003e\u003cspan address=\"10.1016/j.acra.2014.05.006\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFood US, Drug Administration (FDA) (2019) Consideration of Uncertainty in Making Benefit-Risk Determinations in Medical Device Premarket Approvals, De Novo Classifications, and Humanitarian. Device Exemptions - Guidance for Industry and Food and Drug Administration Staff\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePerakslis E, Ginsburg GS (2021) Digital Health\u0026mdash;The Need to Assess Benefits, Risks, and Value. JAMA 325:127\u0026ndash;128. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1001/jama.2020.22919\u003c/span\u003e\u003cspan address=\"10.1001/jama.2020.22919\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGilbert S, Freyer O, Wondra K (2024) The regulatory status of health apps that employ serious games and gamification\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFreyer O, Wiest I, Kather JN et al A future role for large language model health applications depends on regulators being brave enough to enforce safety standards (In Press). \u003cem\u003eLancet Digit Health\u003c/em\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMesk\u0026oacute; B, Topol EJ (2023) The imperative for regulatory oversight of large language models (or generative AI) in healthcare. Npj Digit Med 6:1\u0026ndash;6. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1038/s41746-023-00873-0\u003c/span\u003e\u003cspan address=\"10.1038/s41746-023-00873-0\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFreyer O, Gilbert S (2023) Bridging between hype and implementation in medical extended reality. Npj Digit Med 6:1\u0026ndash;3. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1038/s41746-023-00972-y\u003c/span\u003e\u003cspan address=\"10.1038/s41746-023-00972-y\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSouchet AD, Lourdeaux D, Pagani A et al (2023) A narrative review of immersive virtual reality\u0026rsquo;s ergonomics and risks at the workplace: cybersickness, visual fatigue, muscular fatigue, acute stress, and mental overload. Virtual Real 27:19\u0026ndash;50. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/s10055-022-00672-0\u003c/span\u003e\u003cspan address=\"10.1007/s10055-022-00672-0\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eU.S. Food and Drug Administration (FDA) (2016) Factors to Consider Regarding Benefit-Risk in Medical Device Product Availability, Compliance, and Enforcement Decisions -. Guidance for Industry and Food and Drug Administration Staff\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSmith MY, van Til J, DiSantostefano RL et al (2021) Quantitative Benefit\u0026ndash;Risk Assessment: State of the Practice Within Industry. Ther Innov Regul Sci 55:415\u0026ndash;425. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/s43441-020-00230-3\u003c/span\u003e\u003cspan address=\"10.1007/s43441-020-00230-3\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePage MJ, McKenzie JE, Bossuyt PM et al (2021) The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372:n71. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1136/bmj.n71\u003c/span\u003e\u003cspan address=\"10.1136/bmj.n71\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRayyan (2021) Rayyan \u0026ndash; Intelligent Systematic Review - Rayyan\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMAXQDA (2022) MAXQDA | All-In-One Qualitative \u0026amp; Mixed Methods Data Analysis Tool\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHawker S, Payne S, Kerr C et al (2002) Appraising the Evidence: Reviewing Disparate Data Systematically. Qual Health Res 12:1284\u0026ndash;1299. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1177/1049732302238251\u003c/span\u003e\u003cspan address=\"10.1177/1049732302238251\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSpearman C (1904) The Proof and Measurement of Association between Two Things. Am J Psychol 15:72\u0026ndash;101. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.2307/1412159\u003c/span\u003e\u003cspan address=\"10.2307/1412159\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eColopy MW, Damaraju CV, He W et al (2015) Benefit-Risk Evaluation and Decision Making: Some Practical Insights. Ther Innov Regul Sci 49:425\u0026ndash;433. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1177/2168479014565469\u003c/span\u003e\u003cspan address=\"10.1177/2168479014565469\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSun S, Heske S, Mercadel M et al (2021) Predicting Regulatory Product Approvals Using a Proposed Quantitative Version of FDA\u0026rsquo;s Benefit\u0026ndash;Risk Framework to Calculate Net-Benefit Score and Benefit\u0026ndash;Risk Ratio. Ther Innov Regul Sci 55:129\u0026ndash;137. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/s43441-020-00197-1\u003c/span\u003e\u003cspan address=\"10.1007/s43441-020-00197-1\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eFu B, Li X, Scott J et al (2020) A new framework to address challenges in quantitative benefit-risk assessment for medical products. Contemp Clin Trials 95:106073. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.cct.2020.106073\u003c/span\u003e\u003cspan address=\"10.1016/j.cct.2020.106073\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eTervonen T, Veldwijk J, Payne K et al (2023) Quantitative Benefit-Risk Assessment in Medical Product Decision Making: A Good Practices Report of an ISPOR Task Force. Value Health 26:449\u0026ndash;460. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1016/j.jval.2022.12.006\u003c/span\u003e\u003cspan address=\"10.1016/j.jval.2022.12.006\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eU.S. Food and Drug Administration (FDA). Benefit-Risk Assessment for New Drug and Biological Products. (2023)\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChung B, Kutty J (2022) Benefit-Risk Determination: A Quantitative Approach\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKheir O, Jacoby A, Verwulgen S (2022) Risk Identification and Analysis in the Development of Medical Devices Among Start-Ups: Towards a Broader Risk Management Framework. Med Devices Auckl NZ 15:349\u0026ndash;363. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.2147/MDER.S375977\u003c/span\u003e\u003cspan address=\"10.2147/MDER.S375977\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGebel M, Renz C, Rodriguez L et al (2024) A Survey to Assess the Current Status of Structured Benefit-Risk Assessment in the Global Drug and Medical Device Industry. Ther Innov Regul Sci 58:756\u0026ndash;765. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1007/s43441-024-00650-5\u003c/span\u003e\u003cspan address=\"10.1007/s43441-024-00650-5\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLeong J, McAuslane N, Walker S et al (2013) Is there a need for a universal benefit-risk assessment framework for medicines? Regulatory and industry perspectives. Pharmacoepidemiol Drug Saf 22:1004\u0026ndash;1012. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1002/pds.3464\u003c/span\u003e\u003cspan address=\"10.1002/pds.3464\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMedical Device Coordination Group (MDCG) (2019) MDCG 2019-16 Guidance on Cybersecurity for medical devices\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eInternational Electrotechnical Commission (IEC) (2021) IEC 80001-1:2021\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHughes D, Waddingham E, Mt-Isa S et al (2016) Recommendations for benefit\u0026ndash;risk assessment methodologies and visual representations. Pharmacoepidemiol Drug Saf 25:251\u0026ndash;262. \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003e10.1002/pds.3958\u003c/span\u003e\u003cspan address=\"10.1002/pds.3958\" targettype=\"DOI\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[{"identity":"b535a299-54a5-4008-902d-e8dd2d3644d9","identifier":"10.13039/501100000780","name":"European Commission","awardNumber":"101094218","order_by":0}],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":true,"hideJournal":true,"highlight":"","institution":"TU Dresden","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"Benefit-Risk Analysis, Medical Devices, Systematic Review, Methodologies, PRISMA, Regulation, Risk Management","lastPublishedDoi":"10.21203/rs.3.rs-4832842/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4832842/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eIntroduction\u003c/strong\u003e: The use of medical devices (MDs) by patients carries both benefits and risks. Regulatory frameworks such as the EU Medical Device Regulation (MDR) and the US Federal Food, Drug, and Cosmetic Act (FD\u0026amp;C Act) mandate the systematic weighing of these benefits and risks through a benefit-risk analysis (BRA). This systematic review aims to identify existing BRA methodologies for MDs, and evaluate their strengths and weaknesses.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMethods\u003c/strong\u003e: A literature search was conducted in PubMed, Scopus, Web of Science, Google Scholar, and Semantic Scholar, covering publications from 2000 onwards using a search string that contained the search terms (1) methodologies AND (2) benefit-risk analysis AND (3) medical device OR in vitro diagnostic medical device. Peer-reviewed publications were included when they described BRA methodologies for MDs. The exclusion criteria included records on unrelated technologies, pharmaceuticals, \u0026nbsp;non-English publications, and insufficient descriptions. The quality assessment was performed using a method proposed by Hawker et al. (2002). The methods and their characteristics were narratively summarised. Each method was assessed for its degree of objectivity and for using numerical calculations on a scale from 0 to 5.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults\u003c/strong\u003e: The search identified 622 records, with six meeting the inclusion criteria. The included studies described ten BRA methodologies: Multicriteria Decision Analysis (MCDA), Health Outcomes Modelling (HOM), Stated-Choice Surveys (SCS), Composite Net Clinical Outcome (CNCO), Pairwise Comparisons (PC), Complete Profile of Benefits and Risks (CPBR), Net Benefit Score and Benefit-Risk Ratio (NBS \u0026amp; BRR), Quantitative Benefit-Risk Assessment (qBRA), Quantitative Benefit-Risk Determination (QBRD), and the FDA’s Benefit-Risk Framework (BRF). Methods varied in their reliance on numerical calculations and their degree of objectivity, with a significant correlation between both. The average level of objectivity was assessed as medium (2.1/5).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eDiscussion\u003c/strong\u003e: This review describes the ongoing debate between qualitative and quantitative BRA methods. While qualitative methods are often criticised for being subjective and biased, our findings reveal that quantitative methods, though more objective, still exhibit subjectivity, especially in endpoint identification and relevance assignment. Despite the rise of quantitative methods, often originating from the pharmaceutical industry, qualitative methods are often still used by MD decision-makers. Additionally, several quantitative methods are unsuited for certain difficult-to-quantify risks of MDs. Integrating both qualitative and quantitative elements may offer a more comprehensive BRA framework for MDs.\u003c/p\u003e","manuscriptTitle":"Methodologies for the benefit-risk analysis of medical devices: A systematic review","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-08-05 20:16:31","doi":"10.21203/rs.3.rs-4832842/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"68654fa9-6a4c-4562-9fff-39a60ba48e03","owner":[],"postedDate":"August 5th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[{"id":35375521,"name":"Translational Medicine"}],"tags":[],"updatedAt":"2024-08-05T20:16:31+00:00","versionOfRecord":[],"versionCreatedAt":"2024-08-05 20:16:31","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4832842","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4832842","identity":"rs-4832842","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}