Achieving Appropriate Model Transparency: Challenges and Potential Solutions for Making Value-Based Decisions in the United States.

Greater transparency in economic models can help improve and ensure accuracy and relevance in modeling efforts. Transparency requires effort by modelers and the maintaining of a balance between protection of intellectual property and assessment of model validity and reliability. The Institute for Clinical and Economic Review has piloted structured mechanisms to allow for model validation efforts while protecting the work product of the modeling groups involved, which can serve as a springboard for future innovations to increase model transparency.

To alleviate some of the concerns and risks of open access to models, innovative data sharing agreements between research groups can be used to establish acceptable parameters for model sharing. Through contracts and financial incentives, interested parties can be allowed access to the model for understanding and technical review but with restrictions on future use of the model or the intellectual property contained therein. In fact, the ISPOR good practices for outcomes research paper on model transparency and validity suggests the use of formal data use agreements to facilitate model transparency efforts [ 4 ]. In addition to improved rules and agreements via contracts, portals for sharing models could be designed better to help foster transparency while protecting against copying valuable aspects of the model. The most feasible type of model-sharing agreement will depend on multiple factors, such as incentives for scholarly activities, future funding, ownership of the model, and jurisdiction of model development. A key component of the discussion will naturally be how the work involved in model sharing will be funded. As with most aspects of transparency initiatives generally, the possible range exists on a broad spectrum between cost recovery and the typically six-figure contracts to develop models for commercial clients. Model-sharing agreements can range from relatively simple versions such as confidentiality agreements or creative commons licenses to more advanced licensing or data use agreements. The benefits of confidentiality agreements include relatively quick execution between parties along with a relatively low but still relevant protection of intellectual property. Creative commons licenses provide free open access while also allowing model developers to further specify the use of their model by outside parties [ 21 ]. With creative commons licensing, modelers can allow or deny commercial use of their model and specify whether new users can adapt the model for other applications. However, creative commons licenses may be difficult to enforce [ 17 ]. More detailed and robust licensing agreements through universities or other entities can allow access under more specific rules that provide further protection and consequences for licensing infractions and the means to support the extra effort required to share models through licensing fees. Each of these arrangements require different levels of effort to design and execute, so their use should be aligned with the model-sharing goals, parties involved, and decision-making context.

With the above-mentioned potential benefits to direct model access come several challenges. A fundamental barrier stems from the currently low levels of non-industry funding in the USA for model development, which typically only covers time related to specific project objectives, leaving little room for other activities, even if deemed meritorious or considered a public good. In this context, the potentially unfunded time and effort required to share models with external groups, who often require a detailed technical guide and/or a simplified user interface, may not be viewed as a priority. The amount of extra effort required increases for stakeholders lacking modeling expertise. Even with National Institutes of Health-funded models, which are required to be made public, it is seldom if ever that transparency processes are followed in a decision-relevant timeframe [ 12 ], an issue that also persists with publicly funded clinical trial data [ 13 ]. Models funded by industry come with a unique set of issues, as contractual arrangements with academic researchers may limit the ability to share the model with external audiences. Another concern is that allowing direct access can lead to delays, or even bias, created by input on model assumptions and parameters from groups with different sets of incentives. Allowing external groups to access a model during the development process opens the process to external influence. Much of the input received would likely consist of valid critiques and suggestions for improvement. However, groups such as manufacturers of the products in question often prefer modeling techniques and interpretations of data that favor their products and may attempt to steer the model development process toward a set of assumptions or inputs that match their interests. Reviews of the cost-effectiveness literature support such findings [ 14 – 16 ]. In the extreme and given the amount of potential profits in play for the developers and manufacturers of the interventions in question, especially when the model results have a potential impact on subsequent approval and pricing of the technology, some may view transparency initiatives as an opportunity to undermine the model development process altogether. This potential for delays and introduction of bias is directly at odds with the goals of providing timely and valid results and increases the resources required to develop models. Finally, a specific challenge when models are developed to support HTA decisions relates to the acceptability of prepublication or otherwise confidential data from manufacturers, which are often redacted in public documents. The presence of these data will naturally limit the ability to replicate or fully interrogate models, even if released publicly. A general reluctance about calls for free open source models within academia stems from several contextual considerations. Model-building activities in academic settings are considered research, and the standard for reporting research findings is to describe the process sufficiently to allow replication. This allows for confirmation of the research findings when another research group sets about answering the same research question but using their own processes and resources and with sufficient technical details regarding the methodology. Healthcare research dollars are a limited resource, and researchers and their institutions compete for research funding. Therefore, academic researchers and related institutions consider model development to be a research investment that can yield short- and long-term returns in the form of future research funding, collaborations, and publications. Career prospects are directly linked to the ability to attain research funding and produce scholarly works. Therefore, especially as a voluntary effort, there is little enthusiasm for options such as making models completely open to the public if that could decrease funding and/or publication opportunities. For example, other research groups competing for the same pool of research funds could provide a competitive research proposal because less resources will be required if a robust, validated model is freely available. An analogy can be made to laboratory-based research. If a research group spent time and effort developing a microbial strain, giving the strain away for free may allow other groups far greater opportunity to leverage the innovation and potentially outcompete the innovator group for future research dollars. Scientific progress is made through incremental improvements on previous researchers’ works but also through competition and incentives for innovators. The key to producing short- and long-term scientific gains is to find the appropriate balance.

A prominent real-world example of providing direct model access to interested stakeholders is a transparency project launched by ICER in conjunction with the academic collaborators who develop and specify economic models to inform the cost-effectiveness and budget impact evaluations contained in ICER appraisals of both new and established technologies. The initial pilots were associated with migraine [ 18 ] and endometriosis [ 19 ] reviews that were ongoing at the time. Rather than a focus on broad public release for its own sake, this transparency effort was intended to answer questions about whether the models were fit for purpose to inform the policy decisions of interest for the specific ICER review. The pilots involved direct contracting between the academic groups developing the models and the manufacturers of the products under review, given that the intellectual property being shared resided with the collaborators. Under these agreements, manufacturers paid a small fee to the relevant academic institution to cover the added costs of preparing the models for review, including the development of user documentation. Manufacturers were also asked to sign confidentiality and/or licensing agreements that prevented copying and/or distributing of the models. Access was time limited and targeted to fall within the 4-week public comment period following ICER’s posting of its draft reports. Results from the pilots were disparate. The endometriosis pilot involved a single manufacturer whose general engagement during the review was limited, and the invitation to participate in the pilot was declined. The migraine pilot involved three manufacturers and featured an Excel-based model that was released on a Box [ 22 ] platform to allow controlled access from any location. All calculations and formulas were available to reviewers. Overall, this pilot was deemed a success. The manufacturers considered the contracting and model release process to be relatively smooth and described communication about the release as clear and consistent. In addition, stakeholders identified minor but relevant errors in the model that could be corrected in time for the final report. Still, there were several logistical challenges worth mentioning. First, company firewalls created problems with access to Box in some cases, requiring individuals to work outside of their preferred and secured information technology environment. In addition, the migraine model was populated in part by data submitted as “in confidence” by the manufacturers, which necessitated data redaction along with a concurrent worry that back calculation of confidential results, while prohibited, was nonetheless potentially feasible. Manufacturers requested customized versions of the model with their own data unredacted, but the nominal fee charged would not have covered the additional effort required for this change. Finally, the manufacturers were interested in more detail in the technical documentation and greater opportunity for interaction with the modelers. However, they also reported that the model structure, estimation, and documentation was reasonably straightforward; the benefit that additional interaction could provide was therefore unclear. Subsequent reviews have produced similar outcomes. ICER’s review of treatments for hereditary angioedema involved two manufacturers, both of whom declined to participate in a model release. However, the organization developed an internal model of medication-assisted treatments for opioid use disorder, which was shared with three participating manufacturers as a “live” web app on heRo3 sm [ 23 ], an online modeling tool that works with a cloud-based, open-source health economics modeling package in the programming language R. In this case, authorized users were sent a secure weblink to access the model in the hero3 environment. Users could modify certain parameters to assess changes in model results but could not make any permanent model changes. One manufacturer submitted confidential data and had exclusive access to a separate version of the model populated with the confidential information. As with the migraine pilot, feedback from the manufacturers focused primarily on logistical issues. First and foremost, while the use of R-based modeling is increasing, the participating manufacturers had limited exposure to R, making a technical evaluation and understanding of the available code challenging. Some manufacturers also reported difficulty in identifying certain parameters and reviewing sensitivity analyses, citing a lack of familiarity with the platform. While the hero3 vendor did offer a tutorial session for reviewers, this did not appear to mitigate all concerns. Moving forward, ICER intends to work with its collaborators to routinely offer the opportunity for model examination to manufacturers of the products under review for every future topic. The hope is that, given the nominal fee and the fact that manufacturers already devote substantial resources to the review of ICER models, this will become a more predictable and consistent exercise. ICER is also willing to extend invitations for model review to patient advocacy groups, payers, and other stakeholders relevant to the model review process; to date, such interest has been limited.

Overall, model access during development, if viewed primarily in the contexts of efficiency and validation that have driven the ICER model transparency initiative, seems feasible and could be quite beneficial to all involved parties. Indeed, the importance of this discussion is international in scope, given that nearly all mature HTA bodies develop or critique manufacturer-submitted models to assess the value of new health interventions. Direct and openly public access after model development may be more difficult to resolve because of the lack of funding and incentives. Certainly, achieving model transparency will require improved stakeholder engagement, increased funding by interested parties, and further development of legal assurances to protect intellectual property. The ongoing debate about model transparency is important as we collectively work to improve the development and use of economic evidence to support healthcare decision making. As this process evolves, there is a strong impetus to work together with the interests and constraints of all stakeholders considered. As all health economists—formal and amateur—know, incentives matter. Hence, the key to moving forward is to develop a sustainable approach to reap the benefits of transparency that is robust, objective, and responsive to the various needs of the involved stakeholders.

Transparency in decision modeling remains a topic of rigorous debate among healthcare stakeholders, given tensions between the potential benefits of external access during model development and the need to protect intellectual property and reward research investments [ 1 – 3 ]. Recognizing that decision modeling is conducted by various organizations, this article focuses on issues in transparency from the perspective of university-based researchers and academic institutions and recent experience in conducting collaborative research with the Institute for Clinical and Economic Review (ICER), a US-based health technology assessment organization that is actively engaged in model development for multiple audiences in the USA. Importantly, some level of transparency already exists in terms of methods and other technical specifications for published models; ICER public reports also include a technical appendix with details on model structure, parameter estimates, risk equations, and syntheses of clinical data that inform the model. Therefore, increased transparency in decision modeling is taken here to mean mechanisms to allow direct external access to a model’s structure, source code, and data. This is similar to the definition provided by Eddy et al. [ 4 ] in an International Society for Pharmacoeconomics and Outcomes Research (ISPOR) good practices report, in which it is stated that transparency “… refers to the extent to which interested parties can review a model’s structure, equations, parameter values, and assumptions.” As such, transparency is separated from the process of model building, although a process of transparency may interact with and itself be a step in the process of building the model. The current debate on transparency has arisen in the USA at a time when the use of decision models is becoming more prominent in healthcare decision making through the emergence of value-based formularies and the efforts of groups such as ICER [ 5 ]. Strategies to allow direct external access to models can take on many forms but are bounded between the status quo (e.g., detailed methods reporting) and free, publicly available open-source models. Balancing transparency with practicality and the interests of all involved parties is a daunting task with many ethical, legal, and infrastructure-related hurdles and is the subject of intense ongoing debate [ 2 , 6 – 8 ]. To establish a suitable level of transparency, there is a need to balance the pursuit of model validity and reliability with protecting intellectual property rights and allowing for rewards for research investments. As with many such situations, a practical approach that meets the goals of the activity and balances the incentives and constraints of the interested parties is the ultimate goal. A variety of different stakeholders are engaged in this topic, including model developers (both the developers of a specific model and the larger modeling community), model commissioners or funders, model users (i.e., healthcare decision makers, including healthcare payers and clinical guidelines groups), developers of modeling methods and supportive software, pharmaceutical and medical device manufacturers, and patients and other healthcare consumers. Although each group will have its own perspective, there is typically a shared goal of producing timely, accurate, valid, and reliable evidence about the comparative clinical and economic impact of healthcare interventions. The primary audience for most decision models is healthcare payers and, perhaps to a lesser extent, clinical guidelines groups. The process of developing and validating the relevant decision models should ensure that these audiences trust the results of the decision model. We note that transparency in the development process is a separate consideration from the subsequent activities of accessing the model to produce custom results, update model inputs, train future modelers, and repurpose models for a different research question. Although these activities have merit, they are not the primary purpose of developing a de novo decision model to inform decision makers about a specific research question. The ISPOR–SMDM (Society for Medical Decision Making) guidance appears to promote model transparency for the purpose of public validation rather than a broader set of purposes [ 4 ]. This encapsulates a central tension in the debate: Opponents of broader release worry about model appropriation or misuse for nefarious purposes as well as a detrimental effect on future funding prospects, whereas proponents argue that limited release undercuts the potential for further innovation and improvement through experimentation by others [ 1 , 3 , 9 , 10 ]. The purpose of this paper is to provide an overview of the issues surrounding transparency from the perspective of university-based researchers and academic institutions, informed by recent joint experiences with ICER, and to offer key considerations for the field of health economics and outcomes research in the future. We believe these issues apply regardless of whether the work is being funded by governments, payers, health technology assessment (HTA) bodies, or industry. Our focus is on strategies for providing “direct model access”—in other words, allowing one or more interested party to obtain and review the source code, data, and technical documentation of a model developed by another party. Acknowledging existing standards for the description of methods and inputs used in academic models, this paper seeks to delineate potential strategies to help move toward increased transparency as well as challenges related to balancing the needs and interests of the academic model developers and other parties with interests in model outputs. Finally, the paper also details an initiative to increase the transparency of models that is being developed to support ICER reviews of new and emerging technologies. In the spirit of the ISPOR–SMDM guidance [ 4 ], this initiative has a narrow scope, focusing on model development, with the goal of providing direct access to a working version of the model for the stakeholders most qualified to review the model structure, key assumptions, parameter estimates, and other features.