Comparing cost estimates of implementation strategies using Cost-IS: a collective case study

Comparing cost estimates of implementation strategies using Cost-IS: a collective case study | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Comparing cost estimates of implementation strategies using Cost-IS: a collective case study Thomasina Donovan, Bridget Abell, Steven M. McPhail, Hannah E. Carter This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-4873079/v1 This work is licensed under a CC BY 4.0 License Status: Posted Version 1 posted You are reading this latest preprint version Abstract Background There is a lack of pragmatic approaches that standardise the cost estimation of implementation strategies. An instrument for data collection and cost estimation of implementation strategies for digital health innovations (Cost-IS) was recently developed. This study aimed to compare the nature of implementation cost estimates across three case studies. Secondary aims were to investigate the perceived value of these cost estimates to project leads and stakeholders, and to assess the feasibility and generalisability of Cost-IS as a standardised tool to compare the cost estimates of implementation strategies across three case studies. Methods An evaluative and descriptive collective case study was conducted to estimate the cost of implementation strategies associated with the introduction of healthcare interventions using the Cost-IS instrument. Eligible case studies included completed implementation projects that had planned to cost implementation and had prospectively collected data on implementation-related resource use throughout the project. Implementation data was separately analysed for each case study. Adaptations to the instrument were made where appropriate and documented. The Cost-IS instrument was tailored in a way that best allowed for meaningful outputs from each case study. The feasibility, generalisability and limitations of Cost-IS were observed by the analyst through the costing process. Feedback on the outputs were obtained from project leads, via questionnaires. Results The Cost-IS instrument enabled comparisons of cost estimates of implementation strategies between three real-world implementation case studies. Labour accounted for the majority of implementation costs. Adaptions to Cost-IS during the costing process were made to improve data capture granularity, meaningfully categorise the data, and enhance the instrument’s flexibility for analysis. Implementation cost findings were perceived by project leads to be valuable for reporting and evaluating, should be customised to project context, and include relevant contextual information. Conclusions Cost-IS was found to be a flexible instrument that can be tailored to meet the requirements of a range of implementation projects. Challenges during this process included separating implementation-related costs from intervention costs and research-related costs. Cost-IS is limited to costing implementation strategies and thus far has only captured tangible resources. implementation costs digital health implementation science health economics Figures Figure 1 Figure 2 Contributions To The Literature Cost-IS is feasible to use as a standardised approach to estimate the cost of implementation strategies. The implementation cost findings from Cost-IS demonstrated value for reporting and evaluating implementation projects. Our findings further highlighted that implementation cost findings should be customised to project context and include relevant contextual information. Challenges during the costing of implementation included separating implementation from intervention costs and research-related costs. These decisions will generally be context specific. However, it is important that assumptions are clear, transparent, and applied consistently throughout the project. 2 INTRODUCTION Theory-informed implementation strategies are increasingly being adopted by health services to promote successful and sustainable implementation efforts [ 1 ]. While additional resourcing is generally required to support implementation strategies, these costs have been under reported in the literature, and often excluded from economic evaluations [ 2 , 3 ]. This is problematic as the appropriate allocation of healthcare-related resources requires an ability to accurately estimate the cost of healthcare models, services, or interventions, which should also include their associated implementation activities [ 4 ]. Noted barriers to costing implementation include inconsistencies in the definition of implementation-related costs, and a lack of methodological guidance [ 5 ]. Current literature highlights key considerations for costing implementation and sources for those costs [ 6 – 8 ] but there remains a lack of pragmatic approaches to standardise the costing of implementation strategies [ 9 ]. Standardised methods are important to promote transparency and confidence in cost estimates [ 10 ]. While some standardised approaches have been proposed, there remains a need for pragmatic, flexible and user-friendly tools to collect and estimate implementation costs [ 9 ]. We recently developed an instrument to collect and quantify the cost of implementation strategies, with a focus on digital health innovations: Cost-IS [ 12 ]. Cost-IS was iteratively developed from findings of a literature review [ 5 ], qualitative interviews [ 9 ] and a modified e-Delphi process [ 12 ]. Cost-IS is intended to provide a flexible, pragmatic approach for data collection and cost estimation of the resources associated with implementation strategies in health services research. While the instrument was originally developed for use in digital health settings, it may have practical application in broader implementation contexts. Health services research requires methodological approaches that can attend to the complexity of the health system [ 13 ]. The case study approach ‘investigates contemporary phenomena within its real-life context’ [ 14 ] and therefore is useful in complex health service research [ 15 ]. Case study research can provide insights into the contextual factors and identify the conditions necessary for interventions to be used successful in practice. Case studies can inform assessments of where, when, how and for whom interventions (or in our case the Cost-IS instrument) may be successful by highlighting the necessary and sufficient conditions under which there are favourable effects [ 13 ]. This study aimed to compare the nature of implementation cost estimates across three case studies. Secondary aims were to investigate the perceived value of these cost estimates to project leads and stakeholders, and to assess the feasibility and generalisability of Cost-IS as a standardised tool to compare the cost estimates of implementation strategies across three case studies. 3 METHODS Study design We report on an evaluative and descriptive collective case study [ 16 ] to retrospectively cost the implementation associated with the introduction of healthcare interventions using the newly developed Cost-IS instrument [ 12 ]. The collective case study approach involves studying multiple cases to generate a broader appreciation of a particular issue [ 15 ]. Retrospectively obtained implementation data was separately analysed for each case study, and adaptations to the instrument were made where appropriate. The case study’s findings were reviewed collectively to gain a comprehensive understanding of implementation costing. This produced insights to inform adaptions to Cost-IS and assist future users of the instrument [ 13 ]. This study is guided by a scientific realist perspective [ 17 ]. Cost-IS The scope of Cost-IS is to cost implementation strategies for digital health innovations. Here, an implementation strategy is defined as a method or technique used to enhance the adoption, implementation, and sustainability of an intervention. This is separate to what may be considered an intervention or technology related cost, which is defined as the evidence-based practice, programme, policy, process, or guideline recommendation that is being implemented [ 18 ]. Cost-IS comprises of three data collection templates. The first template is used to plan and identify the implementation costs, with the remaining two templates (labour and non-labour resources) used to collect the data necessary to quantify implementation costs. Summary figures and tables can be created from the data in the completed templates in a meaningful way as determined by the analyst. Case studies Eligible case studies were completed implementation projects that had planned to cost implementation and had prospectively collected data on implementation-related resource use throughout the project. The implementation resource use data needed to be available for use by researcher TD in the current study and was supported by amendments to original study ethical approvals where required. Case studies were identified through existing collaborative research networks. Project leads of potential case studies were contacted to explain the aims of this study and seek their consent to cost their study’s implementation using Cost-IS. Data collection All relevant implementation resource use data were obtained from the project leads for each case study. Data was retrieved from multiple sources including project data files with details of implementation activities and/or time log of personnel during implementation activities. Feedback on the outputs of implementation costing were obtained from project leads in questionnaires created in Qualtrics©. Questions were either formatted as 9-point Likert scale (where 1 = strongly disagree to 9 = strongly agree) or free text spaces. Questions related to the stimulus material provided to project leads. The stimulus material included the implementation costing relating to project leads’ respective case studies. The implementation cost findings were presented in various ways and participants were asked to consider the value and usefulness of these findings, the most useful output of the findings, and how the findings output could be improved. The questionnaires can be found in Additional File 1. Data analysis Data were analysed by tailoring the Cost-IS instrument in a way that best allowed for meaningful outputs from each case study. This is consistent with the intended use of the instrument, with the ability to add or remove data collection fields as relevant. Decisions on how to tailor the instrument were determined by the nature of the study, the type of implementation resource use data available, and through discussion with project leads for each case study. One researcher, TD, conducted all data analysis. Any changes to the Cost-IS instrument were discussed with the wider research team. Adaptions and limitations of the Cost-IS instrument were documented. The ability of Cost-IS to cost implementation strategies within a case study and across the case studies informed its feasibility and generalisability, respectively. Data from the project lead feedback questionnaires were analysed using a thematic content analysis approach. A Qualtrics© report was exported to a PDF for analysis. Similar suggestions and comments in the free text questions were combined and collapsed into a single suggestion or comment. 4 RESULTS Characteristics of the case studies Three multi-site case study studies were identified from existing collaborative research networks. All case studies had completed the implementation of their respective projects. Projects were conducted between 2019 and 2023, with timelines ranging from 1 to 4 years. All case studies were conducted in Queensland, Australia. Case study A was conducted in eleven residential aged care homes and aimed to reduce unnecessary hospital admissions. Case study B was conducted across three acute tertiary hospitals with the aim of increasing appropriate end-of-life care and included a digital health solution. Case study C was conducted within one tertiary hospital across three wards and aimed to improve the quality of hospital care at the end of life. Characteristics of the case studies can be found in Table 1 . Table 1 Case study characteristics Case study characteristics Case study A Case study B Case study C Year of implementation project 2019–2023 2019–2021 2022–2023 Location Queensland, Australia Queensland, Australia Queensland, Australia Health service area Residential aged care homes Acute tertiary hospitals Wards within a single tertiary hospital Number of sites 11 3 3 Aim To implement an early detection program to reduce hospital admissions and length of hospital stay by residents in aged care homes. To implement a prospective feedback loop intervention in three acute hospitals to increase appropriate care and treatment decisions and pathways for older patient populations at the end-of-life. To improve the quality of hospital care for people with serious illness through patient experience measurement and feedback informing facilitated ward-based improvement. Intervention A tailored hospital avoidance program that enhances the ability of residential aged care home staff to respond appropriately to early signs of resident deterioration. A prospective feedback loop, on outcomes related to appropriate care and treatment at the end-of-life, and a tailored clinical response. Co-design and pilot test pragmatic and innovative solutions to effect sustainable changes in care delivery for people with serious illness in the inpatient setting. Digital health No Yes No Implementation clusters and strategies * Provide interactive assistance • Facilitation Use evaluative and iterative strategies • Context assessment Provide interactive assistance • Facilitation Develop stakeholder interrelationships • Executive advisory group • Clinical champion Adapt and tailor to context • Adapt and tailor to context Provide interactive assistance • Facilitation Develop stakeholder interrelationships • Use advisory boards and workgroups Use evaluative and iterative strategies • Conduct local need assessment Adapt and tailor to context • Adapt and tailor to context Resources costed Site team • Clinical facilitator • Clinical care manager • Administration staff • Residential manger • Registered nurse Project team • Project implementation facilitator • Nurse educator • Materials for clinical facilitators • Travel Site team • Auditor • Administration staff • Clinician (nurse, medical) • Executive advisory group member • Health information analytics team • Implementation facilitator • Participating clinician • Site lead Project team • Chief Investigator • Research project manager • Printing and materials for facilitator • Travel Site team • Clinician (nurse, medical, allied health) Project team • Consumer • Implementation facilitator • Project facilitator (research nurse) • Research team (Professor, research fellow) * The Expert Recommendations for Implementing Change (ERIC) framework [ 19 ] was used to classify strategies where appropriate. Table 1 . Case study characteristics Comparison of implementation cost estimates The Cost-IS instrument was used to estimate the costs associated with implementation strategies in each case study. Figure 1 illustrates the proportion of implementation cost estimates from individual case studies and a combined proportion from all case studies by implementation strategies (Fig. 1 -A) and by resources (Fig. 1 -B). Implementation strategies were organised into clusters from the Expert Recommendations for Implementing Change (ERIC) framework to enable comparison between the case studies [ 19 ]. Implementation costs in the case studies covered four of the nine ERIC clusters. The majority of the combined implementation costs (56%) were in the provide interactive assistance cluster. All case studies recorded implementation costs in this cluster, unlike the other clusters. Facilitation was the only implementation strategy within this cluster. Facilitation contributed to most of the implementation costs within each individual case study, except for case study C. The strategies within the d evelop stakeholder interrelationships cluster accounted for 17% of combined implementation costs. Most of the implementation costs within this cluster related to the use of advisory boards and workgroups strategies. The remaining implementation costs in this cluster were related to the strategy clinical champions. A similar proportion of combined implementation costs (16%) was attributed to the use evaluative and iterative strategies cluster. The strategies within this cluster related to context and local needs assessments. Most of the implementation costs for case study C (38%) were found within this cluster, whereas it only contributed to 13% of implementation costs for case study B. The final cluster adapt and tailor to context accounted for 10% combined implementation costs. Implementation cost estimates were analysed by resource category (site or project team) and type (labour or non-labour), as seen in Fig. 1 -B. The majority of the combined implementation costs (63%) were associated with site team resources. Site teams were involved in local implementation at the site. In the case studies, only labour resources contributed to the implementation costs within the site team. The number and type of roles in the site teams of each case study was different (Table 1 ). Site team resources contributed to the majority of implementation costs within each individual case study (case study A = 70%, case study B = 85%), except for case study C (30%). The other resource category, project team, was comprised of labour and non-labour resources and accounted for 37% of combined implementation costs. The project teams were responsible for facilitating the implementation project in each case study. Similar to the site team, the number and types of resources comprising the project teams in each case study was different (Table 1 ). Labour resources contributed to most of the combined implementation costs within this category (29%). Labour resources overall also contributed the majority of implementation costs, within each case study (case study A = 85%, case study B = 90%, case study C = 100%) and combined (92%). Non-labour resources only accounted for 8% of combined implementation costs and ranged between 10% and 15% within individual case studies. Case study C did not have any non-labour resources attributed to implementation costs. Figure 1 . Proportion of individual and combined case studies’ implementation costs by implementation strategies, organised into clusters from the Expert Recommendations for Implementing Change (ERIC) framework, and resource, category (site or project team) and type (labour or non-labour). Feedback from project leads Analyses were conducted using the Cost-IS instrument to present findings in a form that was meaningful for decision making. This was determined on a case-by-case basis by the analyst in consultation with project leads. As a result, the cost of implementation strategies was analysed by strategy, activity, resource, labour and non-labour, personnel team, site, and/or phase. Additional File 2 contains the summary output of the implementation costing analysis for each case study. Implementation costs in case study C could not be analysed per week across phases and sites because the number of weeks per phase was not consistent across the sites. Apart from the previous exception, all case studies underwent the same analyses to provide to the project leads for feedback. Seven project leads and stakeholders, at least one from each case study, were invited and all responded to the questionnaire and provided feedback on the outputs of implementation costing (Table 2 ). All participants agreed (five strongly agreed and two somewhat agreed) that the implementation cost findings provide value for reporting or evaluating the project. Participants expanded saying it is, “essential for replication, decision making, understanding good/bad implementers, for layering into the process evaluation, for 'valuing' facilitation or other activities of the implementation focus and role/s” [P01: case study B] . All participants agreed (four strongly agreed and three somewhat) that the findings are presented in a way that is useful for reporting or evaluating the project. Participants liked how the, “data [was] broken down in a multitude of ways. It allows us to consider the reasons behind certain results and to triangulate this data with other contextual and qualitative information” [P07: case study A] . Some participants found the findings were presented in a way that was, “very clear and easy to follow” [P05: case study A] . Other participants wanted “more explanation about what is included in each cost” [P06: case study B] or “a little more contextual info” [P03: case study A] . Table 2 Project lead feedback Feedback statement from questionnaire Participants responses Supporting quote/s The implementation cost findings provide value for reporting or evaluating the project. Strongly Agree (n = 5), Agree (n = 2) “essential for replication, decision making, understanding good/bad implementers, for layering into the process evaluation, for 'valuing' facilitation or other activities of the implementation focus and role/s” [P01: case study B] “implementation costs are often overlooked. The tool was valuable in providing a mechanism for capturing and reporting this information. This allows for a more complete picture to be articulated in our results papers” [P07: case study A] The findings are presented in a way that is useful for reporting or evaluating the project. Strongly Agree (n = 4), Agree (n = 3) “the data [was] broken down in a multitude of ways. It allows us to consider the reasons behind certain results and to triangulate this data with other contextual and qualitative information” [P07: case study A] “very clear and easy to follow” [P05: case study A] “more explanation about what is included in each cost” [P06: case study B] “a little more contextual info” [P03: case study A] Table 2 . Project lead feedback Participants reported that an analysis of implementation costs presented by site and strategy was the most useful output for reporting or evaluating the project (Fig. 2 and Table 3 ). The least useful output was implementation costs presented per week across phases and sites, one participant stated this was because it was “not clear for slide 3 how that is calculated” [P02: case study A] . Participants preferred tables to figures because they are, “easier to read and work through” [P01: case study B] . Some participants chose graphs that, “have more detailed information which I found easier to understand” [P06: case study B] . Whereas others, “picked the ones that provide a high-level summary and complement the reporting of the process evaluation” [P05: case study A] . Preference for how implementation costs were presented varied across the case studies. For example, in case study A project leads found the output of actual implementation cost findings across phases and site to be useful, whereas leads from the other case studies did not. This may be because case study A had 11 sites whereas the other two case studies only had 3 sites. Table 3 Example of the most useful implementation cost output for reporting or evaluating the project- case study A implementation costs presented by site and strategy (table) Home Context assessment Facilitation Grand Total Home 01 1,090.75 9,912.76 11,003.51 Home 02 938.93 18,694.48 19,633.41 Home 03 801.46 7,929.99 8,731.45 Home 04 1,277.52 4,515.78 5,793.30 Home 05 977.26 1,820.32 2,797.58 Home 06 435.34 7,451.81 7,887.15 Home 07 606.78 6,182.35 6,789.13 Home 08 929.44 9,921.32 10,850.76 Home 09 1,173.33 4,151.79 5,325.12 Home 10 1,733.17 1,978.61 3,711.78 Home 11 1,372.01 3,010.82 4,382.83 Grand Total 11,335.99 75,570.04 86,906.03 Figure 2 . Example of the most useful implementation cost output for reporting or evaluating the project- case study A implementation costs presented by site and strategy (graph) Table 3 . Example of the most useful implementation cost output for reporting or evaluating the project- case study A implementation costs presented by site and strategy (table) Participants suggested other ways the results could be presented including, “some averages or box plots so you get a sense of the range and relationality a bit more” [P07: case study A] and consolidating the phases because too many phases, “are less meaningful” [P01: case study B] . Participants thought contextual and analysis information was missing from the findings which, if included, would be useful for reporting or evaluating the project. Contextual information related to, “the demographic differences between sites” [P05: case study A] and a “breakdown by bed numbers, average across all sites and the range” [P03: case study A] . Including context information would help “understand why there were such big discrepancies in implementation costs between sites” [P03: case study A] . Analysis information included, “what is in vs out (implementation vs intervention costs)” [P07: case study A] and “explain how the costs were calculated, what was included in each bar on the graphs” [P06: case study B] . Using Cost-IS to cost implementation Costing implementation strategies using the Cost-IS instrument was feasible and enabled the comparison of implementation cost estimates across case studies. The flexibility of Cost-IS was beneficial because it was tailored to meet the needs of each case study. For example, in case study C, a ‘Phase’ column was added to Template 2 in order to describe what phase of the project the implementation activity occurred in. Case studies A and B were also analysed by phase, but this was determined using the ‘Date’ data. We found it was more straightforward to analyse by phase using the dedicated ‘Phase’ column. Case study C- Template 2 did not contain a ‘Labour/ Non-labour’ column because no non-labour resources were consumed. The Cost-IS instrument performed in a similar manner across all the case studies, indicating the generalisability of this instrument beyond the digital health setting. Additional File 3 outlines how the Cost-IS instrument was used and may serve as a guide to future users. Additional File 4 contains the completed Cost-IS instrument for each case study. Limitations of Cost-IS were highlighted when applying it to the three case studies reported here, as a result adaptions were made to improve the utility of the Cost-IS instrument. The adaptations are summarised in Table 4 . The previously published Cost-IS instrument has been updated with these adaptions; blank templates and a worked example can be found online at https://cost-is.github.io/instrument/ . Table 4 Summary of the updates made to the Cost-IS instrument after trialling in the case study Instrument section Cost-IS updates Template 1 • Non-labour resources were added to both ‘Activity’ and ‘Resource’ columns Template 2 (formerly 2A) • Added the following columns: o Site o Date o Resource- Category o Labour/ Non-labour • Changed ‘Role’ column to ‘Resource’ • Changed ‘Labour cost ( $ )’ to ‘Cost ( $ )’ Template 2B • Removed Template 3 • Table added. It contains salaries of personnel (with source information), and classification of role categories. Summary tables • Automated summary tables have been replaced with recommended use of Excel’s Pivot Table function, or other data analysis program, to summarise data as appropriate for a given study. Table 4 . Summary of the updates made to the Cost-IS instrument after trialling in the case study Template 1 was updated to include non-labour resources under both activity and resource columns. This allowed for non-labour resources to be captured in Template 2 for an enhanced flexibility of analysis. Other adaptions to Template 2 which allowed labour and non-labour resources to be captured included changing columns ‘labour cost ( $ )’ to ‘cost ( $ ) and ‘role’ column to ‘resource’, and the addition of column ‘labour/ non-labour’ where entries were categorised as either a labour or non-labour resources. Template 2B was removed because non-labour resources were captured in Template 2. The purpose of having all cost captured in one table (Template 2) was to provide enhanced flexibility for reporting findings to best inform decision making for a particular study. The remaining adaptions to Template 2 included the addition of a ‘site’ and ‘date’ column, these adaptions enabled greater depth of analysis. Template 3 was added to the instrument which allowed the valuation of resources to be clear and easily updated. Template 3 informs Template 2’s columns ‘hourly wage rate’ and ‘resource- category’. The addition of the ‘resource- category’ column was an adaption to Template 2. It improved analysis by allowed roles to be categorised in a meaningful way to the project leads. 5 DISCUSSION The Cost-IS instrument was applied to data from three real-world implementation case studies demonstrating the usability of this instrument and enabling comparisons between the cost estimates of implementation strategies. The Cost-IS instrument performed in a similar manner across all the case studies including a digital health setting and two health service settings more broadly. All relevant costs and resources were captured in Cost-IS templates, without the exclusion of important costs and erroneous inclusion of irrelevant costs. Cost-IS was initially developed for data collection and cost estimation of implementation strategies for digital health innovations [ 12 ], but these findings indicate the instrument is likely to have generalisability beyond the digital health setting. The majority of implementation costs in the present case studies were attributed to labour resources which is consistent with a recent review on implementation costs within the digital health setting [ 5 ]. We assume this is likely to be similar in other settings which utilise the same implementation strategies, yet this is unable to be confirmed without further research. However, similar findings were seen in a practice change intervention to increase routine provision of antenatal care addressing maternal alcohol consumption reported that labour costs comprised 70% of the total intervention delivery costs (akin to implementation costs) [ 20 ]. The implementation strategies adopted within the included case studies covered only four of the nine ERIC clusters. Although the entire spectrum of implementation strategies within ERIC was not explored in this study, recent reviews indicate that these four clusters are frequently used for the implementation of digital health interventions[ 5 ], as well as community, public health, and health service research settings broadly [ 7 ]. Another ERIC cluster that is often utilised within [ 5 ] and beyond the digital health setting [ 7 ] is the train and educate stakeholder. Within our case studies, these strategies were present but were considered integral to the intervention and were therefore costed as intervention costs in the separate project evaluation. Project leads found that the implementation cost findings provided value for reporting or evaluating their project. The inclusion of contextual and analysis information was suggested by project leads to strengthen implementation cost findings. There was variation in preference for how implementation costs were presented, based on this implementation cost findings should be presented in a way that is meaningful toto the project. These findings reflect the intended use of Cost-IS as a flexible, pragmatic yet structured instrument for collecting and estimating costs of implementation strategies, and makes an important contribution to emerging research efforts in this field [ 9 ]. The Cost-IS instrument was tailored and adapted during the costing process to improve the analysis of implementation cost data. The adaptions included: improved data capture granularity (addition of date, site, phase columns); meaningful categorisation (addition of resource- category column); and enhanced flexibility for analysis (combining labour and non-labour resources in one template). A key challenge in estimating the costs of implementation strategies relates to the understanding and definition of ‘implementation’ as a distinct activity, separate to the design and conduct of an intervention (more detail provided in Additional File 3). The challenge of separating implementation from intervention activities was most pronounced in the case study which involved a digital health solution. Information systems frameworks, including the systems development life cycle (SDLC), have long recognised the importance of including project management principles from systems development to delivery [ 21 ]. The phases of SDLC include requirement analysis, design, development and testing, implementation, documentation, and evaluation [ 22 ]. These phases and the underlying principles of SDLC overlap with common frameworks used in implementation science research, including the plan–do–study–act (PDSA) cycle [ 23 ]. As a result of this similarity, it was challenging to identify whether some project activities were related to the development and delivery of a digital health solution intervention, or if these related to an implementation strategy. A single activity could also have a dual purpose, for example co-design may be used for both intervention development and for implementation buy-in, making it more challenging to separate between implementation and intervention costs. Decisions around separating implementation and intervention activities will generally be context specific. However, it is important that assumptions are clear, transparent, and applied consistently throughout the project. In the case studies we report on here, it was important to separate implementation costs from intervention costs to avoid duplication when both were later used in each case study’s analysis of the overall project cost (outside the scope of this study). The consolidated framework for implementation research (CFIR) supports the identification of implementation strategies by encouraging users to describe their overall approach, or implementation process framework. This serves to guide implementation efforts, and may further help distinguish the innovation from the implementation process and accompanying implementation strategies [ 24 ]. CFIR distinguishes the implementation process used to implement the innovation as activities that end after implementation is complete, from the innovation as the “thing” that continues when implementation is complete [ 24 ]. We took a similar approach in our study with the assumption that implementation activities supported the intervention, therefore we assumed that the intervention was able to function without the implementation activities. This study demonstrates the ability of Cost-IS to retrospectively estimate the costs of implementation strategies using existing project documentation (Additional File 3 outlines data sources used). This may be useful to help reduce the burden of data collection when conducting implementation research, a challenge that has been highlighted previously [ 9 ]. However, the multiple case study design using project records to apply the costing tool was associated with some limitations. Where data were not available, the costing analysis relied on assumption. For example, if time estimates were not available from data sources, assumptions were made on the likely time taken for given activities, with input from project leads. In case study B, an implementation activity (travel) could not be separately allocated to each site due to the nature of its documentation. Also in case study B, chief investigator meetings were considered research costs and not implementation costs as the investigator group was comprised of researchers who were focussed on evaluating the program. It was likely that some implementation was discussed in the meetings but we were unable to determine what proportion. The project leads were consulted to deal with these challenges, particularly because the projects in each case study were unfamiliar to the analyst costing the implementation using Cost-IS. The use of Cost-IS to guide prospective data collection would likely allow more comprehensive and accurate data to be collected throughout the project and may reduce the reliance on informed assumptions and recall. This study’s evaluation of Cost-IS is dependent on the included case studies. The results of this study will likely be most applicable to the Australian context because all case studies were located in Australia. Opportunity costs were not present in any of the included case studies. Opportunity costs are defined as the benefits foregone because the resources were not used in the next best alternative [ 25 ]. The use of physical space, such as training rooms or meeting rooms, is an example of an opportunity cost. Non-labour costs should be noted as either an opportunity or monetary cost in the relevant column in Template 3 of Cost-IS because it could affect analysis. The analysis of opportunity costs is dependent on the time-horizon and perspective of the analysis. Implementation science research typically has short time-horizons for analysis (1–3 years), and in these cases it has been recommended to exclude costs that are fixed in that time period, including physical space [ 25 ]. Digital health interventions can have short lifecycles (3–5 years) and it has been recommended that cost assessments should be an iterative process where frequent and rapid cost estimations are conducted throughout the technology’s lifecycle [ 26 , 27 ]. Currently, Cost-IS has been developed and tested for costing implementation strategies, which limits the scope of overall implementation costing in this study. Using the ERIC framework to categorise implementation strategies was not necessary but may allow for easier comparison of strategies and costs across projects. Only tangible costs could be captured in the Cost-IS instrument. Intangible costs including soft skills, personal reflection time, existing relationships, level of authority, and mental load were not costed but have been highlighted as contributing to implementation [ 9 ]. 6 CONCLUSION The use of Cost-IS facilitated comparisons of cost estimates of implementation strategies between three real-world implementation case studies. Project leads found that the implementation cost findings provided value for reporting or evaluating their project. Challenges during this process included separating implementation from intervention costs and research-related costs. The dependence on existing documentation for case study implementation contributed to this challenge as well as other challenges including key data inputs being unavailable, and aggregate documentation of some resources. Nonetheless, Cost-IS was shown to be a flexible instrument that can be tailored to meet the requirements of a range of implementation projects. Abbreviations Cost-IS Costing Implementation Strategies instrument CFIR Consolidated Framework for Implementation Research ERIC the Expert Recommendations for Implementing Change framework SDLC Systems Development Life Cycle PDSA Plan–Do–Study–Act Declarations Ethics approval and consent to participate Ethical approval was obtained from Bolton Clarke Human Research Ethics Committee (ref 170031), Metro North Health Human Research Ethics Committee (HREC/2019/QRBW/51606), and Townsville Hospital and Health Service Human Research Ethics Committee (HREC/2022/QTHS/84709). Consent for publication Not applicable. Availability of data and materials All data generated or analysed during this study are included in this published article and its supplementary files. Competing interests The authors declare that they have no competing interests. Funding This work was supported by the Digital Health CRC Limited (DHCRC). DHCRC is funded under the Australian Commonwealth’s Cooperative Research Centres (CRC) Program. The funders had no role in study design or decision to submit for publication. Author Contributions TD, HC, BA, and SM conceived the study. TD costed implementation in each case study and drafted the manuscript. All authors reviewed the analysed data, and revised and edited the manuscript. All authors read and approved the final manuscript. Acknowledgments The authors gratefully thank the Queensland University of Technology and Australian Government Research Training Program Scholarship for supporting this work. References Pieterse M, Kip H, Cruz-Martínez RR. The complexity of ehealth implementation: a theoretical and practical perspective. Theory and Development: A Multi-Disciplinary Approach London: Routledge.: eHealth Research; 2018. pp. 247–70. Roberts SLE, Healey A, Sevdalis N. Use of health economic evaluation in the implementation and improvement science fields—a systematic literature review. Implement Sci. 2019;14(1):72. Bowser DM, Henry BF, McCollister KE. Cost analysis in implementation studies of evidence-based practices for mental health and substance use disorders: a systematic review. Implement Sci. 2021;16(1):26. Drummond MF, Sculpher MJ, Torrance GW, Stoddart GL. Methods for the economic evaluation of healthcare programs. 3 ed. USA: Oxford University Press; 2005. Donovan T, Abell B, Fernando M, McPhail SM, Carter HE. Implementation costs of hospital-based computerised decision support systems: a systematic review. Implement Sci. 2023;18(1):7. Gold HT, McDermott C, Hoomans T, Wagner TH. Cost data in implementation science: categories and approaches to costing. Implement Sci. 2022;17(1):11. Michaud TL, Pereira E, Porter G, Golden C, Hill J, Kim J, et al. Scoping review of costs of implementation strategies in community, public health and healthcare settings. BMJ Open. 2022;12(6):e060785. Ritzwoller DP, Sukhanova A, Gaglio B, Glasgow RE. Costing behavioral interventions: a practical guide to enhance translation. Ann Behav Med. 2009;37(2):218–27. Donovan T, Carter HE, McPhail SM, Abell B. A qualitative interview study to explore costing of implementation strategies to support digital health adoption it’s the difference between success and failure. 01 February 2024, PREPRINT (Version 1) available at Research Square [ https://doiorg/1021203/rs3rs-3828958/v1] Chapel JM, Wang G. Understanding cost data collection tools to improve economic evaluations of health interventions. Stroke Vasc Neurol. 2019;4(4):214–22. Hoeft TJ, Wilcox H, Hinton L, Unutzer J. Costs of implementing and sustaining enhanced collaborative care programs involving community partners. Implement Sci. 2019;14(1):37. Donovan T, Abell B, McPhail SM, Carter HE. May. Development of Cost-IS (costing implementation strategies) instrument for digital health solutions: a modified e-Delphi study. 16 2024, PREPRINT (Version 1) available at Research Square [ https://doiorg/1021203/rs3rs-4229774/v1] Paparini S, Green J, Papoutsi C, Murdoch J, Petticrew M, Greenhalgh T, et al. Case study research for better evaluations of complex interventions: rationale and challenges. BMC Med. 2020;18(1):301. Yin RK. Case study research: Design and methods. sage; 2009. Crowe S, Cresswell K, Robertson A, Huby G, Avery A, Sheikh A. The case study approach. BMC Med Res Methodol. 2011;11:100. Thomas G. A typology for the case study in social science following a review of definition, discourse, and structure. Qualitative Inq. 2011;17(6):511–21. Murray M, Sargeant S, Harper D, Thompson A. Qualitative Research Methods in Mental Health and Psychotherapy: An Introduction for Students and Practitioners. Chapter:, Publisher: Sage, Editors: David Harper and Andrew Thompson. 2011:163 – 75. Pinnock H, Barwick M, Carpenter CR, Eldridge S, Grandes G, Griffiths CJ, et al. Standards for Reporting Implementation Studies (StaRI) Statement. BMJ. 2017;356:i6795. Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, Chinman MJ, Smith JL, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci. 2015;10(1):109. Szewczyk Z, Reeves P, Kingsland M, Doherty E, Elliott E, Wolfenden L, et al. Cost, cost-consequence and cost-effectiveness evaluation of a practice change intervention to increase routine provision of antenatal care addressing maternal alcohol consumption. Implement Sci. 2022;17(1):14. Curry JM, McGregor C, Tracy S. A systems development life cycle approach to patient journey modeling projects. Stud Health Technol Inf. 2007;129(Pt 2):905–9. Satzinger JW, Jackson RB, Burd SD. Systems Analysis and Design in a Changing World. Boston, MA, UNITED STATES: Cengage Learning; 2015. Langley GJ. The improvement guide a practical approach to enhancing organizational performance. 2nd ed. ed. San Francisco, California: Jossey-Bass; 2009. Damschroder LJ, Reardon CM, Widerquist MAO, Lowery J. The updated Consolidated Framework for Implementation Research based on user feedback. Implement Sci. 2022;17(1):75. Wagner TH. Rethinking How We Measure Costs in Implementation Research. J Gen Intern Med. 2020;35(Suppl 2):870–4. Khan ZA, Kidholm K, Pedersen SA, Haga SM, Drozd F, Sundrehagen T, et al. Developing a Program Costs Checklist of Digital Health Interventions: A Scoping Review and Empirical Case Study. PharmacoEconomics. 2024;42(6):663–78. Huter K, Krick T, Rothgang H. Health economic evaluation of digital nursing technologies: a review of methodological recommendations. Health Econ Rev. 2022;12(1):35. Supplementary Files AF1Questionnairev0.docx AF2StimulusMaterialv0.pdf AF3CostISUserGuideV0.docx AF4CompletedCostISv2.pptx.xlsx Cite Share Download PDF Status: Posted Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-4873079","acceptedTermsAndConditions":true,"allowDirectSubmit":true,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":346295634,"identity":"21febf48-fec3-47eb-bbcb-a3ce4e0b6e58","order_by":0,"name":"Thomasina Donovan","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABLElEQVRIie3PsUrDQBjA8ZODdLnS9UoNvkJKIBpS7bMcgXQJRegSEPQgkC5xLyj6CpfRyZbDZrkHSMgkXesqDhkMJLY2qXUVvP9wcN/ld0cAkMn+Yni7auWkFVbDzeluqEmQKLfoMAHfCHYPk2HPf2Ef+dm4Q2G0uvT4yWn3bZF5wCLPAC4yBC4Irb1yvHTS2wBP8FyZ6DPB+093Y9sUYERCoNgWAnaDYNdI2hQTOkdGrx04RyxzjS4FvCDFBAG4j6R5jsljRYYsFV+k816Qm30kQwomrCQDwhK0eUUpCG+QxHEyNcAk4sW/IDGwmXB1k2ojPeSKbt5rsV4jrZm9TNf5NXmI/WiFPHzOYtFPqWep06n/mqy9K7VGtsGdnQarifbT97/dIJPJZP+6T+gqag3Pdae1AAAAAElFTkSuQmCC","orcid":"https://orcid.org/0000-0002-0127-0091","institution":"QUT Health: Queensland University of Technology Faculty of Health","correspondingAuthor":true,"prefix":"","firstName":"Thomasina","middleName":"","lastName":"Donovan","suffix":""},{"id":346295635,"identity":"e67d5bc7-b32f-474e-9d31-1e061d03915d","order_by":1,"name":"Bridget Abell","email":"","orcid":"","institution":"Queensland University of Technology","correspondingAuthor":false,"prefix":"","firstName":"Bridget","middleName":"","lastName":"Abell","suffix":""},{"id":346295636,"identity":"aef11366-2c44-4c53-adac-b8f36c27b7b7","order_by":2,"name":"Steven M. McPhail","email":"","orcid":"","institution":"Queensland University of Technology","correspondingAuthor":false,"prefix":"","firstName":"Steven","middleName":"M.","lastName":"McPhail","suffix":""},{"id":346295637,"identity":"4a7b61e6-add9-45e7-b046-b108632cd648","order_by":3,"name":"Hannah E. Carter","email":"","orcid":"","institution":"Queensland University of Technology","correspondingAuthor":false,"prefix":"","firstName":"Hannah","middleName":"E.","lastName":"Carter","suffix":""}],"badges":[],"createdAt":"2024-08-07 08:24:05","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-4873079/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-4873079/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":66894787,"identity":"37ecbc4e-8236-4f13-a51b-097c181f3997","added_by":"auto","created_at":"2024-10-17 15:07:02","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":160422,"visible":true,"origin":"","legend":"\u003cp\u003eProportion of individual and combined case studies’ implementation costs by implementation strategies, organised into clusters from the Expert Recommendations for Implementing Change (ERIC) framework, and resource, category (site or project team) and type (labour or non-labour).\u003c/p\u003e","description":"","filename":"Figure1v0.png","url":"https://assets-eu.researchsquare.com/files/rs-4873079/v1/315d5e646e6c235bde732990.png"},{"id":66895577,"identity":"0a3f1949-0b65-4b20-8474-2236ad645cbb","added_by":"auto","created_at":"2024-10-17 15:15:02","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":71280,"visible":true,"origin":"","legend":"\u003cp\u003eExample of the most useful implementation cost output for reporting or evaluating the project- case study A implementation costs presented by site and strategy (graph)\u003c/p\u003e","description":"","filename":"Figure2v0.png","url":"https://assets-eu.researchsquare.com/files/rs-4873079/v1/3f97f1b45b38a0e494665267.png"},{"id":70715240,"identity":"0e019b16-0141-4d7e-b23a-d2fdb041c5fc","added_by":"auto","created_at":"2024-12-06 01:37:01","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":807694,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4873079/v1/939ceb1e-3b47-42dd-95cc-a65b5b687a4a.pdf"},{"id":66894788,"identity":"430d5da5-ae1c-4700-8ab0-21dc7d16caa3","added_by":"auto","created_at":"2024-10-17 15:07:02","extension":"docx","order_by":1,"title":"","display":"","copyAsset":false,"role":"supplement","size":26473,"visible":true,"origin":"","legend":"","description":"","filename":"AF1Questionnairev0.docx","url":"https://assets-eu.researchsquare.com/files/rs-4873079/v1/93d8ce14506fa245dbf18e6f.docx"},{"id":66894793,"identity":"ff869463-0de6-4a2f-906a-6a2668a1ea19","added_by":"auto","created_at":"2024-10-17 15:07:02","extension":"pdf","order_by":2,"title":"","display":"","copyAsset":false,"role":"supplement","size":273757,"visible":true,"origin":"","legend":"","description":"","filename":"AF2StimulusMaterialv0.pdf","url":"https://assets-eu.researchsquare.com/files/rs-4873079/v1/c6418736612873a7d49f7795.pdf"},{"id":66894789,"identity":"559380f7-36ea-43e0-b835-a21502f95ab0","added_by":"auto","created_at":"2024-10-17 15:07:02","extension":"docx","order_by":3,"title":"","display":"","copyAsset":false,"role":"supplement","size":89453,"visible":true,"origin":"","legend":"","description":"","filename":"AF3CostISUserGuideV0.docx","url":"https://assets-eu.researchsquare.com/files/rs-4873079/v1/b7550b92cd940b5d17463520.docx"},{"id":66895578,"identity":"0db2efb1-6ea1-400e-b019-113adfd0cf52","added_by":"auto","created_at":"2024-10-17 15:15:02","extension":"xlsx","order_by":4,"title":"","display":"","copyAsset":false,"role":"supplement","size":301351,"visible":true,"origin":"","legend":"","description":"","filename":"AF4CompletedCostISv2.pptx.xlsx","url":"https://assets-eu.researchsquare.com/files/rs-4873079/v1/cf30f622ccdfa5c42f8ca865.xlsx"}],"financialInterests":"","formattedTitle":"Comparing cost estimates of implementation strategies using Cost-IS: a collective case study","fulltext":[{"header":"Contributions To The Literature","content":"\u003cp\u003e\u003col\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eCost-IS is feasible to use as a standardised approach to estimate the cost of implementation strategies.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eThe implementation cost findings from Cost-IS demonstrated value for reporting and evaluating implementation projects. Our findings further highlighted that implementation cost findings should be customised to project context and include relevant contextual information.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003cspan\u003e \u003cli\u003e \u003cp\u003eChallenges during the costing of implementation included separating implementation from intervention costs and research-related costs. These decisions will generally be context specific. However, it is important that assumptions are clear, transparent, and applied consistently throughout the project.\u003c/p\u003e \u003c/li\u003e \u003c/span\u003e \u003c/ol\u003e\u003c/p\u003e"},{"header":"2 INTRODUCTION","content":"\u003cp\u003eTheory-informed implementation strategies are increasingly being adopted by health services to promote successful and sustainable implementation efforts [\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e]. While additional resourcing is generally required to support implementation strategies, these costs have been under reported in the literature, and often excluded from economic evaluations [\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e, \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e]. This is problematic as the appropriate allocation of healthcare-related resources requires an ability to accurately estimate the cost of healthcare models, services, or interventions, which should also include their associated implementation activities [\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e]. Noted barriers to costing implementation include inconsistencies in the definition of implementation-related costs, and a lack of methodological guidance [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eCurrent literature highlights key considerations for costing implementation and sources for those costs [\u003cspan additionalcitationids=\"CR7\" citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e] but there remains a lack of pragmatic approaches to standardise the costing of implementation strategies [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. Standardised methods are important to promote transparency and confidence in cost estimates [\u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e10\u003c/span\u003e]. While some standardised approaches have been proposed, there remains a need for pragmatic, flexible and user-friendly tools to collect and estimate implementation costs [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. We recently developed an instrument to collect and quantify the cost of implementation strategies, with a focus on digital health innovations: Cost-IS [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. Cost-IS was iteratively developed from findings of a literature review [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e], qualitative interviews [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e] and a modified e-Delphi process [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. Cost-IS is intended to provide a flexible, pragmatic approach for data collection and cost estimation of the resources associated with implementation strategies in health services research. While the instrument was originally developed for use in digital health settings, it may have practical application in broader implementation contexts.\u003c/p\u003e \u003cp\u003eHealth services research requires methodological approaches that can attend to the complexity of the health system [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]. The case study approach \u003cem\u003e\u0026lsquo;investigates contemporary phenomena within its real-life context\u0026rsquo;\u003c/em\u003e [\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e] and therefore is useful in complex health service research [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. Case study research can provide insights into the contextual factors and identify the conditions necessary for interventions to be used successful in practice. Case studies can inform assessments of where, when, how and for whom interventions (or in our case the Cost-IS instrument) may be successful by highlighting the necessary and sufficient conditions under which there are favourable effects [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThis study aimed to compare the nature of implementation cost estimates across three case studies. Secondary aims were to investigate the perceived value of these cost estimates to project leads and stakeholders, and to assess the feasibility and generalisability of Cost-IS as a standardised tool to compare the cost estimates of implementation strategies across three case studies.\u003c/p\u003e"},{"header":"3 METHODS","content":"\u003cp\u003eStudy design\u003c/p\u003e \u003cp\u003eWe report on an evaluative and descriptive collective case study [\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e] to retrospectively cost the implementation associated with the introduction of healthcare interventions using the newly developed Cost-IS instrument [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e]. The collective case study approach involves studying multiple cases to generate a broader appreciation of a particular issue [\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e]. Retrospectively obtained implementation data was separately analysed for each case study, and adaptations to the instrument were made where appropriate. The case study\u0026rsquo;s findings were reviewed collectively to gain a comprehensive understanding of implementation costing. This produced insights to inform adaptions to Cost-IS and assist future users of the instrument [\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e]. This study is guided by a scientific realist perspective [\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eCost-IS\u003c/p\u003e \u003cp\u003eThe scope of Cost-IS is to cost implementation strategies for digital health innovations. Here, an implementation strategy is defined as a method or technique used to enhance the adoption, implementation, and sustainability of an intervention. This is separate to what may be considered an intervention or technology related cost, which is defined as the evidence-based practice, programme, policy, process, or guideline recommendation that is being implemented [\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e]. Cost-IS comprises of three data collection templates. The first template is used to plan and identify the implementation costs, with the remaining two templates (labour and non-labour resources) used to collect the data necessary to quantify implementation costs. Summary figures and tables can be created from the data in the completed templates in a meaningful way as determined by the analyst.\u003c/p\u003e \u003cp\u003eCase studies\u003c/p\u003e \u003cp\u003eEligible case studies were completed implementation projects that had planned to cost implementation and had prospectively collected data on implementation-related resource use throughout the project. The implementation resource use data needed to be available for use by researcher TD in the current study and was supported by amendments to original study ethical approvals where required. Case studies were identified through existing collaborative research networks. Project leads of potential case studies were contacted to explain the aims of this study and seek their consent to cost their study\u0026rsquo;s implementation using Cost-IS.\u003c/p\u003e \u003cp\u003eData collection\u003c/p\u003e \u003cp\u003eAll relevant implementation resource use data were obtained from the project leads for each case study. Data was retrieved from multiple sources including project data files with details of implementation activities and/or time log of personnel during implementation activities. Feedback on the outputs of implementation costing were obtained from project leads in questionnaires created in Qualtrics\u0026copy;. Questions were either formatted as 9-point Likert scale (where 1\u0026thinsp;=\u0026thinsp;strongly disagree to 9\u0026thinsp;=\u0026thinsp;strongly agree) or free text spaces. Questions related to the stimulus material provided to project leads. The stimulus material included the implementation costing relating to project leads\u0026rsquo; respective case studies. The implementation cost findings were presented in various ways and participants were asked to consider the value and usefulness of these findings, the most useful output of the findings, and how the findings output could be improved. The questionnaires can be found in Additional File 1.\u003c/p\u003e \u003cp\u003eData analysis\u003c/p\u003e \u003cp\u003eData were analysed by tailoring the Cost-IS instrument in a way that best allowed for meaningful outputs from each case study. This is consistent with the intended use of the instrument, with the ability to add or remove data collection fields as relevant. Decisions on how to tailor the instrument were determined by the nature of the study, the type of implementation resource use data available, and through discussion with project leads for each case study. One researcher, TD, conducted all data analysis. Any changes to the Cost-IS instrument were discussed with the wider research team. Adaptions and limitations of the Cost-IS instrument were documented. The ability of Cost-IS to cost implementation strategies within a case study and across the case studies informed its feasibility and generalisability, respectively. Data from the project lead feedback questionnaires were analysed using a thematic content analysis approach. A Qualtrics\u0026copy; report was exported to a PDF for analysis. Similar suggestions and comments in the free text questions were combined and collapsed into a single suggestion or comment.\u003c/p\u003e"},{"header":"4 RESULTS","content":"\u003cp\u003eCharacteristics of the case studies\u003c/p\u003e \u003cp\u003eThree multi-site case study studies were identified from existing collaborative research networks. All case studies had completed the implementation of their respective projects. Projects were conducted between 2019 and 2023, with timelines ranging from 1 to 4 years. All case studies were conducted in Queensland, Australia. Case study A was conducted in eleven residential aged care homes and aimed to reduce unnecessary hospital admissions. Case study B was conducted across three acute tertiary hospitals with the aim of increasing appropriate end-of-life care and included a digital health solution. Case study C was conducted within one tertiary hospital across three wards and aimed to improve the quality of hospital care at the end of life. Characteristics of the case studies can be found in Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 1\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eCase study characteristics\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cem\u003eCase study characteristics\u003c/em\u003e\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCase study A\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eCase study B\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eCase study C\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cem\u003eYear of implementation project\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e2019\u0026ndash;2023\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e2019\u0026ndash;2021\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e2022\u0026ndash;2023\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cem\u003eLocation\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eQueensland, Australia\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eQueensland, Australia\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eQueensland, Australia\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cem\u003eHealth service area\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eResidential aged care homes\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAcute tertiary hospitals\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eWards within a single tertiary hospital\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cem\u003eNumber of sites\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e3\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cem\u003eAim\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eTo implement an early detection program to reduce hospital admissions and length of hospital stay by residents in aged care homes.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eTo implement a prospective feedback loop intervention in three acute hospitals to increase appropriate care and treatment decisions and pathways for older patient populations at the end-of-life.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eTo improve the quality of hospital care for people with serious illness through patient experience measurement and feedback informing facilitated ward-based improvement.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cem\u003eIntervention\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eA tailored hospital avoidance program that enhances the ability of residential aged care home staff to respond appropriately to early signs of resident deterioration.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eA prospective feedback loop, on outcomes related to appropriate care and treatment at the end-of-life, and a tailored clinical response.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eCo-design and pilot test pragmatic and innovative solutions to effect sustainable changes in care delivery for people with serious illness in the inpatient setting.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cem\u003eDigital health\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eNo\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eYes\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eNo\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cem\u003eImplementation clusters and strategies *\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eProvide interactive assistance\u003c/p\u003e \u003cp\u003e\u0026bull; Facilitation\u003c/p\u003e \u003cp\u003eUse evaluative and iterative strategies\u003c/p\u003e \u003cp\u003e\u0026bull; Context assessment\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eProvide interactive assistance\u003c/p\u003e \u003cp\u003e\u0026bull; Facilitation\u003c/p\u003e \u003cp\u003eDevelop stakeholder interrelationships\u003c/p\u003e \u003cp\u003e\u0026bull; Executive advisory group\u003c/p\u003e \u003cp\u003e\u0026bull; Clinical champion\u003c/p\u003e \u003cp\u003eAdapt and tailor to context\u003c/p\u003e \u003cp\u003e\u0026bull; Adapt and tailor to context\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eProvide interactive assistance\u003c/p\u003e \u003cp\u003e\u0026bull; Facilitation\u003c/p\u003e \u003cp\u003eDevelop stakeholder interrelationships\u003c/p\u003e \u003cp\u003e\u0026bull; Use advisory boards and workgroups\u003c/p\u003e \u003cp\u003eUse evaluative and iterative strategies\u003c/p\u003e \u003cp\u003e\u0026bull; Conduct local need assessment\u003c/p\u003e \u003cp\u003eAdapt and tailor to context\u003c/p\u003e \u003cp\u003e\u0026bull; Adapt and tailor to context\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cem\u003eResources costed\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eSite team\u003c/p\u003e \u003cp\u003e\u0026bull; Clinical facilitator\u003c/p\u003e \u003cp\u003e\u0026bull; Clinical care manager\u003c/p\u003e \u003cp\u003e\u0026bull; Administration staff\u003c/p\u003e \u003cp\u003e\u0026bull; Residential manger\u003c/p\u003e \u003cp\u003e\u0026bull; Registered nurse\u003c/p\u003e \u003cp\u003eProject team\u003c/p\u003e \u003cp\u003e\u0026bull; Project implementation facilitator\u003c/p\u003e \u003cp\u003e\u0026bull; Nurse educator\u003c/p\u003e \u003cp\u003e\u0026bull; Materials for clinical facilitators\u003c/p\u003e \u003cp\u003e\u0026bull; Travel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSite team\u003c/p\u003e \u003cp\u003e\u0026bull; Auditor\u003c/p\u003e \u003cp\u003e\u0026bull; Administration staff\u003c/p\u003e \u003cp\u003e\u0026bull; Clinician (nurse, medical)\u003c/p\u003e \u003cp\u003e\u0026bull; Executive advisory group member\u003c/p\u003e \u003cp\u003e\u0026bull; Health information analytics team\u003c/p\u003e \u003cp\u003e\u0026bull; Implementation facilitator\u003c/p\u003e \u003cp\u003e\u0026bull; Participating clinician\u003c/p\u003e \u003cp\u003e\u0026bull; Site lead\u003c/p\u003e \u003cp\u003eProject team\u003c/p\u003e \u003cp\u003e\u0026bull; Chief Investigator\u003c/p\u003e \u003cp\u003e\u0026bull; Research project manager\u003c/p\u003e \u003cp\u003e\u0026bull; Printing and materials for facilitator\u003c/p\u003e \u003cp\u003e\u0026bull; Travel\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSite team\u003c/p\u003e \u003cp\u003e\u0026bull; Clinician (nurse, medical, allied health)\u003c/p\u003e \u003cp\u003eProject team\u003c/p\u003e \u003cp\u003e\u0026bull; Consumer\u003c/p\u003e \u003cp\u003e\u0026bull; Implementation facilitator\u003c/p\u003e \u003cp\u003e\u0026bull; Project facilitator (research nurse)\u003c/p\u003e \u003cp\u003e\u0026bull; Research team (Professor, research fellow)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003ctfoot\u003e \u003ctr\u003e\u003ctd colspan=\"4\"\u003e* The Expert Recommendations for Implementing Change (ERIC) framework [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e] was used to classify strategies where appropriate.\u003c/td\u003e\u003c/tr\u003e \u003c/tfoot\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. Case study characteristics\u003c/p\u003e \u003cp\u003eComparison of implementation cost estimates\u003c/p\u003e \u003cp\u003eThe Cost-IS instrument was used to estimate the costs associated with implementation strategies in each case study. Figure\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e illustrates the proportion of implementation cost estimates from individual case studies and a combined proportion from all case studies by implementation strategies (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e-A) and by resources (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e-B). Implementation strategies were organised into clusters from the Expert Recommendations for Implementing Change (ERIC) framework to enable comparison between the case studies [\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e]. Implementation costs in the case studies covered four of the nine ERIC clusters. The majority of the combined implementation costs (56%) were in the \u003cem\u003eprovide interactive assistance\u003c/em\u003e cluster. All case studies recorded implementation costs in this cluster, unlike the other clusters. Facilitation was the only implementation strategy within this cluster. Facilitation contributed to most of the implementation costs within each individual case study, except for case study C. The strategies within the d\u003cem\u003eevelop stakeholder interrelationships\u003c/em\u003e cluster accounted for 17% of combined implementation costs. Most of the implementation costs within this cluster related to the use of advisory boards and workgroups strategies. The remaining implementation costs in this cluster were related to the strategy clinical champions. A similar proportion of combined implementation costs (16%) was attributed to the \u003cem\u003euse evaluative and iterative strategies\u003c/em\u003e cluster. The strategies within this cluster related to context and local needs assessments. Most of the implementation costs for case study C (38%) were found within this cluster, whereas it only contributed to 13% of implementation costs for case study B. The final cluster \u003cem\u003eadapt and tailor to context\u003c/em\u003e accounted for 10% combined implementation costs.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eImplementation cost estimates were analysed by resource category (site or project team) and type (labour or non-labour), as seen in Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e-B. The majority of the combined implementation costs (63%) were associated with site team resources. Site teams were involved in local implementation at the site. In the case studies, only labour resources contributed to the implementation costs within the site team. The number and type of roles in the site teams of each case study was different (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). Site team resources contributed to the majority of implementation costs within each individual case study (case study A\u0026thinsp;=\u0026thinsp;70%, case study B\u0026thinsp;=\u0026thinsp;85%), except for case study C (30%). The other resource category, project team, was comprised of labour and non-labour resources and accounted for 37% of combined implementation costs. The project teams were responsible for facilitating the implementation project in each case study. Similar to the site team, the number and types of resources comprising the project teams in each case study was different (Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e1\u003c/span\u003e). Labour resources contributed to most of the combined implementation costs within this category (29%). Labour resources overall also contributed the majority of implementation costs, within each case study (case study A\u0026thinsp;=\u0026thinsp;85%, case study B\u0026thinsp;=\u0026thinsp;90%, case study C\u0026thinsp;=\u0026thinsp;100%) and combined (92%). Non-labour resources only accounted for 8% of combined implementation costs and ranged between 10% and 15% within individual case studies. Case study C did not have any non-labour resources attributed to implementation costs.\u003c/p\u003e \u003cp\u003eFigure \u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e. Proportion of individual and combined case studies\u0026rsquo; implementation costs by implementation strategies, organised into clusters from the Expert Recommendations for Implementing Change (ERIC) framework, and resource, category (site or project team) and type (labour or non-labour).\u003c/p\u003e \u003cp\u003eFeedback from project leads\u003c/p\u003e \u003cp\u003eAnalyses were conducted using the Cost-IS instrument to present findings in a form that was meaningful for decision making. This was determined on a case-by-case basis by the analyst in consultation with project leads. As a result, the cost of implementation strategies was analysed by strategy, activity, resource, labour and non-labour, personnel team, site, and/or phase. Additional File 2 contains the summary output of the implementation costing analysis for each case study. Implementation costs in case study C could not be analysed per week across phases and sites because the number of weeks per phase was not consistent across the sites. Apart from the previous exception, all case studies underwent the same analyses to provide to the project leads for feedback.\u003c/p\u003e \u003cp\u003eSeven project leads and stakeholders, at least one from each case study, were invited and all responded to the questionnaire and provided feedback on the outputs of implementation costing (Table\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e). All participants agreed (five strongly agreed and two somewhat agreed) that the implementation cost findings provide value for reporting or evaluating the project. Participants expanded saying it is, \u003cem\u003e\u0026ldquo;essential for replication, decision making, understanding good/bad implementers, for layering into the process evaluation, for 'valuing' facilitation or other activities of the implementation focus and role/s\u0026rdquo; [P01: case study B]\u003c/em\u003e. All participants agreed (four strongly agreed and three somewhat) that the findings are presented in a way that is useful for reporting or evaluating the project. Participants liked how the, \u003cem\u003e\u0026ldquo;data [was] broken down in a multitude of ways. It allows us to consider the reasons behind certain results and to triangulate this data with other contextual and qualitative information\u0026rdquo; [P07: case study A]\u003c/em\u003e. Some participants found the findings were presented in a way that was, \u003cem\u003e\u0026ldquo;very clear and easy to follow\u0026rdquo; [P05: case study A]\u003c/em\u003e. Other participants wanted \u003cem\u003e\u0026ldquo;more explanation about what is included in each cost\u0026rdquo; [P06: case study B]\u003c/em\u003e or \u003cem\u003e\u0026ldquo;a little more contextual info\u0026rdquo; [P03: case study A]\u003c/em\u003e.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab2\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eProject lead feedback\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"3\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eFeedback statement from questionnaire\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eParticipants responses\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSupporting quote/s\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eThe implementation cost findings provide value for reporting or evaluating the project.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eStrongly Agree (n\u0026thinsp;=\u0026thinsp;5), Agree (n\u0026thinsp;=\u0026thinsp;2)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003e\u0026ldquo;essential for replication, decision making, understanding good/bad implementers, for layering into the process evaluation, for 'valuing' facilitation or other activities of the implementation focus and role/s\u0026rdquo; [P01: case study B]\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003e\u0026ldquo;implementation costs are often overlooked. The tool was valuable in providing a mechanism for capturing and reporting this information. This allows for a more complete picture to be articulated in our results papers\u0026rdquo; [P07: case study A]\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eThe findings are presented in a way that is useful for reporting or evaluating the project.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eStrongly Agree (n\u0026thinsp;=\u0026thinsp;4), Agree (n\u0026thinsp;=\u0026thinsp;3)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003e\u003cem\u003e\u0026ldquo;the data [was] broken down in a multitude of ways. It allows us to consider the reasons behind certain results and to triangulate this data with other contextual and qualitative information\u0026rdquo; [P07: case study A]\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003e\u0026ldquo;very clear and easy to follow\u0026rdquo; [P05: case study A]\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003e\u0026ldquo;more explanation about what is included in each cost\u0026rdquo; [P06: case study B]\u003c/em\u003e\u003c/p\u003e \u003cp\u003e\u003cem\u003e\u0026ldquo;a little more contextual info\u0026rdquo; [P03: case study A]\u003c/em\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab2\" class=\"InternalRef\"\u003e2\u003c/span\u003e. Project lead feedback\u003c/p\u003e \u003cp\u003eParticipants reported that an analysis of implementation costs presented by site and strategy was the most useful output for reporting or evaluating the project (Fig.\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e and Table\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e). The least useful output was implementation costs presented per week across phases and sites, one participant stated this was because it was \u003cem\u003e\u0026ldquo;not clear for slide 3 how that is calculated\u0026rdquo; [P02: case study A]\u003c/em\u003e. Participants preferred tables to figures because they are, \u003cem\u003e\u0026ldquo;easier to read and work through\u0026rdquo; [P01: case study B]\u003c/em\u003e. Some participants chose graphs that, \u003cem\u003e\u0026ldquo;have more detailed information which I found easier to understand\u0026rdquo; [P06: case study B]\u003c/em\u003e. Whereas others, \u003cem\u003e\u0026ldquo;picked the ones that provide a high-level summary and complement the reporting of the process evaluation\u0026rdquo; [P05: case study A]\u003c/em\u003e. Preference for how implementation costs were presented varied across the case studies. For example, in case study A project leads found the output of actual implementation cost findings across phases and site to be useful, whereas leads from the other case studies did not. This may be because case study A had 11 sites whereas the other two case studies only had 3 sites.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab3\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 3\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eExample of the most useful implementation cost output for reporting or evaluating the project- case study A implementation costs presented by site and strategy (table)\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"char\" char=\".\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHome\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eContext assessment\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eFacilitation\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eGrand Total\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHome 01\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1,090.75\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e9,912.76\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e11,003.51\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHome 02\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e938.93\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e18,694.48\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e19,633.41\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHome 03\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e801.46\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e7,929.99\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e8,731.45\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHome 04\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1,277.52\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e4,515.78\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e5,793.30\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHome 05\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e977.26\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e1,820.32\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e2,797.58\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHome 06\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e435.34\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e7,451.81\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e7,887.15\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHome 07\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e606.78\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e6,182.35\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e6,789.13\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHome 08\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e929.44\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e9,921.32\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e10,850.76\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHome 09\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1,173.33\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e4,151.79\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e5,325.12\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHome 10\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1,733.17\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e1,978.61\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e3,711.78\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eHome 11\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e1,372.01\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e3,010.82\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e4,382.83\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003e\u003cb\u003eGrand Total\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c2\"\u003e \u003cp\u003e\u003cb\u003e11,335.99\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c3\"\u003e \u003cp\u003e\u003cb\u003e75,570.04\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"char\" char=\".\" colname=\"c4\"\u003e \u003cp\u003e\u003cb\u003e86,906.03\u003c/b\u003e\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eFigure \u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e2\u003c/span\u003e. Example of the most useful implementation cost output for reporting or evaluating the project- case study A implementation costs presented by site and strategy (graph)\u003c/p\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab3\" class=\"InternalRef\"\u003e3\u003c/span\u003e. Example of the most useful implementation cost output for reporting or evaluating the project- case study A implementation costs presented by site and strategy (table)\u003c/p\u003e \u003cp\u003eParticipants suggested other ways the results could be presented including, \u003cem\u003e\u0026ldquo;some averages or box plots so you get a sense of the range and relationality a bit more\u0026rdquo; [P07: case study A]\u003c/em\u003e and consolidating the phases because too many phases, \u003cem\u003e\u0026ldquo;are less meaningful\u0026rdquo; [P01: case study B]\u003c/em\u003e. Participants thought contextual and analysis information was missing from the findings which, if included, would be useful for reporting or evaluating the project. Contextual information related to, \u003cem\u003e\u0026ldquo;the demographic differences between sites\u0026rdquo; [P05: case study A]\u003c/em\u003e and a \u003cem\u003e\u0026ldquo;breakdown by bed numbers, average across all sites and the range\u0026rdquo; [P03: case study A]\u003c/em\u003e. Including context information would help \u003cem\u003e\u0026ldquo;understand why there were such big discrepancies in implementation costs between sites\u0026rdquo; [P03: case study A]\u003c/em\u003e. Analysis information included, \u003cem\u003e\u0026ldquo;what is in vs out (implementation vs intervention costs)\u0026rdquo; [P07: case study A]\u003c/em\u003e and \u003cem\u003e\u0026ldquo;explain how the costs were calculated, what was included in each bar on the graphs\u0026rdquo; [P06: case study B]\u003c/em\u003e.\u003c/p\u003e \u003cp\u003eUsing Cost-IS to cost implementation\u003c/p\u003e \u003cp\u003eCosting implementation strategies using the Cost-IS instrument was feasible and enabled the comparison of implementation cost estimates across case studies. The flexibility of Cost-IS was beneficial because it was tailored to meet the needs of each case study. For example, in case study C, a \u0026lsquo;Phase\u0026rsquo; column was added to Template 2 in order to describe what phase of the project the implementation activity occurred in. Case studies A and B were also analysed by phase, but this was determined using the \u0026lsquo;Date\u0026rsquo; data. We found it was more straightforward to analyse by phase using the dedicated \u0026lsquo;Phase\u0026rsquo; column. Case study C- Template 2 did not contain a \u0026lsquo;Labour/ Non-labour\u0026rsquo; column because no non-labour resources were consumed. The Cost-IS instrument performed in a similar manner across all the case studies, indicating the generalisability of this instrument beyond the digital health setting. Additional File 3 outlines how the Cost-IS instrument was used and may serve as a guide to future users. Additional File 4 contains the completed Cost-IS instrument for each case study.\u003c/p\u003e \u003cp\u003eLimitations of Cost-IS were highlighted when applying it to the three case studies reported here, as a result adaptions were made to improve the utility of the Cost-IS instrument. The adaptations are summarised in Table\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e. The previously published Cost-IS instrument has been updated with these adaptions; blank templates and a worked example can be found online at \u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://cost-is.github.io/instrument/\u003c/span\u003e\u003cspan address=\"https://cost-is.github.io/instrument/\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e.\u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab4\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 4\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eSummary of the updates made to the Cost-IS instrument after trialling in the case study\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"2\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\"\u003e \u003cp\u003eInstrument section\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCost-IS updates\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTemplate 1\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026bull; Non-labour resources were added to both \u0026lsquo;Activity\u0026rsquo; and \u0026lsquo;Resource\u0026rsquo; columns\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTemplate 2\u003c/p\u003e \u003cp\u003e(formerly 2A)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026bull; Added the following columns:\u003c/p\u003e \u003cp\u003eo Site\u003c/p\u003e \u003cp\u003eo Date\u003c/p\u003e \u003cp\u003eo Resource- Category\u003c/p\u003e \u003cp\u003eo Labour/ Non-labour\u003c/p\u003e \u003cp\u003e\u0026bull; Changed \u0026lsquo;Role\u0026rsquo; column to \u0026lsquo;Resource\u0026rsquo;\u003c/p\u003e \u003cp\u003e\u0026bull; Changed \u0026lsquo;Labour cost (\u003cspan\u003e$\u003c/span\u003e)\u0026rsquo; to \u0026lsquo;Cost (\u003cspan\u003e$\u003c/span\u003e)\u0026rsquo;\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTemplate 2B\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026bull; Removed\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eTemplate 3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026bull; Table added. It contains salaries of personnel (with source information), and classification of role categories.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003eSummary tables\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003e\u0026bull; Automated summary tables have been replaced with recommended use of Excel\u0026rsquo;s Pivot Table function, or other data analysis program, to summarise data as appropriate for a given study.\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003eTable\u0026nbsp;\u003cspan refid=\"Tab4\" class=\"InternalRef\"\u003e4\u003c/span\u003e. Summary of the updates made to the Cost-IS instrument after trialling in the case study\u003c/p\u003e \u003cp\u003eTemplate 1 was updated to include non-labour resources under both activity and resource columns. This allowed for non-labour resources to be captured in Template 2 for an enhanced flexibility of analysis. Other adaptions to Template 2 which allowed labour and non-labour resources to be captured included changing columns \u0026lsquo;labour cost (\u003cspan\u003e$\u003c/span\u003e)\u0026rsquo; to \u0026lsquo;cost (\u003cspan\u003e$\u003c/span\u003e) and \u0026lsquo;role\u0026rsquo; column to \u0026lsquo;resource\u0026rsquo;, and the addition of column \u0026lsquo;labour/ non-labour\u0026rsquo; where entries were categorised as either a labour or non-labour resources. Template 2B was removed because non-labour resources were captured in Template 2. The purpose of having all cost captured in one table (Template 2) was to provide enhanced flexibility for reporting findings to best inform decision making for a particular study. The remaining adaptions to Template 2 included the addition of a \u0026lsquo;site\u0026rsquo; and \u0026lsquo;date\u0026rsquo; column, these adaptions enabled greater depth of analysis. Template 3 was added to the instrument which allowed the valuation of resources to be clear and easily updated. Template 3 informs Template 2\u0026rsquo;s columns \u0026lsquo;hourly wage rate\u0026rsquo; and \u0026lsquo;resource- category\u0026rsquo;. The addition of the \u0026lsquo;resource- category\u0026rsquo; column was an adaption to Template 2. It improved analysis by allowed roles to be categorised in a meaningful way to the project leads.\u003c/p\u003e"},{"header":"5 DISCUSSION","content":"\u003cp\u003eThe Cost-IS instrument was applied to data from three real-world implementation case studies demonstrating the usability of this instrument and enabling comparisons between the cost estimates of implementation strategies. The Cost-IS instrument performed in a similar manner across all the case studies including a digital health setting and two health service settings more broadly. All relevant costs and resources were captured in Cost-IS templates, without the exclusion of important costs and erroneous inclusion of irrelevant costs. Cost-IS was initially developed for data collection and cost estimation of implementation strategies for digital health innovations [\u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e12\u003c/span\u003e], but these findings indicate the instrument is likely to have generalisability beyond the digital health setting.\u003c/p\u003e \u003cp\u003eThe majority of implementation costs in the present case studies were attributed to labour resources which is consistent with a recent review on implementation costs within the digital health setting [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e]. We assume this is likely to be similar in other settings which utilise the same implementation strategies, yet this is unable to be confirmed without further research. However, similar findings were seen in a practice change intervention to increase routine provision of antenatal care addressing maternal alcohol consumption reported that labour costs comprised 70% of the total intervention delivery costs (akin to implementation costs) [\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eThe implementation strategies adopted within the included case studies covered only four of the nine ERIC clusters. Although the entire spectrum of implementation strategies within ERIC was not explored in this study, recent reviews indicate that these four clusters are frequently used for the implementation of digital health interventions[\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e], as well as community, public health, and health service research settings broadly [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e]. Another ERIC cluster that is often utilised within [\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e] and beyond the digital health setting [\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e] is the \u003cem\u003etrain and educate stakeholder.\u003c/em\u003e Within our case studies, these strategies were present but were considered integral to the intervention and were therefore costed as intervention costs in the separate project evaluation.\u003c/p\u003e \u003cp\u003eProject leads found that the implementation cost findings provided value for reporting or evaluating their project. The inclusion of contextual and analysis information was suggested by project leads to strengthen implementation cost findings. There was variation in preference for how implementation costs were presented, based on this implementation cost findings should be presented in a way that is meaningful toto the project. These findings reflect the intended use of Cost-IS as a flexible, pragmatic yet structured instrument for collecting and estimating costs of implementation strategies, and makes an important contribution to emerging research efforts in this field [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. The Cost-IS instrument was tailored and adapted during the costing process to improve the analysis of implementation cost data. The adaptions included: improved data capture granularity (addition of date, site, phase columns); meaningful categorisation (addition of resource- category column); and enhanced flexibility for analysis (combining labour and non-labour resources in one template).\u003c/p\u003e \u003cp\u003eA key challenge in estimating the costs of implementation strategies relates to the understanding and definition of \u0026lsquo;implementation\u0026rsquo; as a distinct activity, separate to the design and conduct of an intervention (more detail provided in Additional File 3). The challenge of separating implementation from intervention activities was most pronounced in the case study which involved a digital health solution. Information systems frameworks, including the systems development life cycle (SDLC), have long recognised the importance of including project management principles from systems development to delivery [\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e]. The phases of SDLC include requirement analysis, design, development and testing, implementation, documentation, and evaluation [\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e]. These phases and the underlying principles of SDLC overlap with common frameworks used in implementation science research, including the plan\u0026ndash;do\u0026ndash;study\u0026ndash;act (PDSA) cycle [\u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e]. As a result of this similarity, it was challenging to identify whether some project activities were related to the development and delivery of a digital health solution intervention, or if these related to an implementation strategy. A single activity could also have a dual purpose, for example co-design may be used for both intervention development and for implementation buy-in, making it more challenging to separate between implementation and intervention costs. Decisions around separating implementation and intervention activities will generally be context specific. However, it is important that assumptions are clear, transparent, and applied consistently throughout the project. In the case studies we report on here, it was important to separate implementation costs from intervention costs to avoid duplication when both were later used in each case study\u0026rsquo;s analysis of the overall project cost (outside the scope of this study).\u003c/p\u003e \u003cp\u003eThe consolidated framework for implementation research (CFIR) supports the identification of implementation strategies by encouraging users to describe their overall approach, or implementation process framework. This serves to guide implementation efforts, and may further help distinguish the innovation from the implementation process and accompanying implementation strategies [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e]. CFIR distinguishes the implementation process used to implement the innovation as activities that end after implementation is complete, from the innovation as the \u0026ldquo;thing\u0026rdquo; that continues when implementation is complete [\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e]. We took a similar approach in our study with the assumption that implementation activities supported the intervention, therefore we assumed that the intervention was able to function without the implementation activities.\u003c/p\u003e \u003cp\u003eThis study demonstrates the ability of Cost-IS to retrospectively estimate the costs of implementation strategies using existing project documentation (Additional File 3 outlines data sources used). This may be useful to help reduce the burden of data collection when conducting implementation research, a challenge that has been highlighted previously [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e]. However, the multiple case study design using project records to apply the costing tool was associated with some limitations. Where data were not available, the costing analysis relied on assumption. For example, if time estimates were not available from data sources, assumptions were made on the likely time taken for given activities, with input from project leads. In case study B, an implementation activity (travel) could not be separately allocated to each site due to the nature of its documentation. Also in case study B, chief investigator meetings were considered research costs and not implementation costs as the investigator group was comprised of researchers who were focussed on evaluating the program. It was likely that some implementation was discussed in the meetings but we were unable to determine what proportion. The project leads were consulted to deal with these challenges, particularly because the projects in each case study were unfamiliar to the analyst costing the implementation using Cost-IS. The use of Cost-IS to guide prospective data collection would likely allow more comprehensive and accurate data to be collected throughout the project and may reduce the reliance on informed assumptions and recall.\u003c/p\u003e \u003cp\u003eThis study\u0026rsquo;s evaluation of Cost-IS is dependent on the included case studies. The results of this study will likely be most applicable to the Australian context because all case studies were located in Australia. Opportunity costs were not present in any of the included case studies. Opportunity costs are defined as the benefits foregone because the resources were not used in the next best alternative [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]. The use of physical space, such as training rooms or meeting rooms, is an example of an opportunity cost. Non-labour costs should be noted as either an opportunity or monetary cost in the relevant column in Template 3 of Cost-IS because it could affect analysis. The analysis of opportunity costs is dependent on the time-horizon and perspective of the analysis. Implementation science research typically has short time-horizons for analysis (1\u0026ndash;3 years), and in these cases it has been recommended to exclude costs that are fixed in that time period, including physical space [\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e]. Digital health interventions can have short lifecycles (3\u0026ndash;5 years) and it has been recommended that cost assessments should be an iterative process where frequent and rapid cost estimations are conducted throughout the technology\u0026rsquo;s lifecycle [\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e].\u003c/p\u003e \u003cp\u003eCurrently, Cost-IS has been developed and tested for costing implementation strategies, which limits the scope of overall implementation costing in this study. Using the ERIC framework to categorise implementation strategies was not necessary but may allow for easier comparison of strategies and costs across projects. Only tangible costs could be captured in the Cost-IS instrument. Intangible costs including soft skills, personal reflection time, existing relationships, level of authority, and mental load were not costed but have been highlighted as contributing to implementation [\u003cspan citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e].\u003c/p\u003e"},{"header":"6 CONCLUSION","content":"\u003cp\u003eThe use of Cost-IS facilitated comparisons of cost estimates of implementation strategies between three real-world implementation case studies. Project leads found that the implementation cost findings provided value for reporting or evaluating their project. Challenges during this process included separating implementation from intervention costs and research-related costs. The dependence on existing documentation for case study implementation contributed to this challenge as well as other challenges including key data inputs being unavailable, and aggregate documentation of some resources. Nonetheless, Cost-IS was shown to be a flexible instrument that can be tailored to meet the requirements of a range of implementation projects.\u003c/p\u003e"},{"header":"Abbreviations","content":"\u003cdiv class=\"DefinitionList\"\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eCost-IS\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eCosting Implementation Strategies instrument\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eCFIR\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eConsolidated Framework for Implementation Research\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eERIC\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003ethe Expert Recommendations for Implementing Change framework\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003eSDLC\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003eSystems Development Life Cycle\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003cdiv class=\"DefinitionListEntry\"\u003e \u003cdiv class=\"Term\"\u003ePDSA\u003c/div\u003e \u003cdiv class=\"Description\"\u003e \u003cp\u003ePlan\u0026ndash;Do\u0026ndash;Study\u0026ndash;Act\u003c/p\u003e \u003c/div\u003e \u003c/div\u003e \u003c/div\u003e"},{"header":"Declarations","content":"\u003ch2\u003eEthics approval and consent to participate\u003c/h2\u003e\n\u003cp\u003eEthical approval was obtained from Bolton Clarke Human Research Ethics Committee (ref 170031), Metro North Health Human Research Ethics Committee (HREC/2019/QRBW/51606), and Townsville Hospital and Health Service Human Research Ethics Committee (HREC/2022/QTHS/84709).\u003c/p\u003e\n\u003ch2\u003eConsent for publication\u003c/h2\u003e\n\u003cp\u003eNot applicable.\u003c/p\u003e\n\u003ch2\u003eAvailability of data and materials\u003c/h2\u003e\n\u003cp\u003eAll data generated or analysed during this study are included in this published article and its supplementary files.\u0026nbsp;\u003c/p\u003e\n\u003ch2\u003eCompeting interests\u003c/h2\u003e\n\u003cp\u003eThe authors declare that they have no competing interests.\u003c/p\u003e\n\u003ch2\u003eFunding\u003c/h2\u003e\n\u003cp\u003eThis work was supported by the Digital Health CRC Limited (DHCRC). DHCRC is funded under the Australian Commonwealth\u0026rsquo;s Cooperative Research Centres (CRC) Program. The funders had no role in study design or decision to submit for publication.\u003c/p\u003e\n\u003ch2\u003eAuthor Contributions\u003c/h2\u003e\n\u003cp\u003eTD, HC, BA, and SM conceived the study. TD costed implementation in each case study and drafted the manuscript. All authors reviewed the analysed data, and revised and edited the manuscript. All authors read and approved the final manuscript.\u003c/p\u003e\n\u003ch2\u003eAcknowledgments\u003c/h2\u003e\n\u003cp\u003eThe authors gratefully thank the Queensland University of Technology and Australian Government Research Training Program Scholarship for supporting this work.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\u003cli\u003e\u003cspan\u003ePieterse M, Kip H, Cruz-Mart\u0026iacute;nez RR. The complexity of ehealth implementation: a theoretical and practical perspective. Theory and Development: A Multi-Disciplinary Approach London: Routledge.: eHealth Research; 2018. pp. 247\u0026ndash;70.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRoberts SLE, Healey A, Sevdalis N. Use of health economic evaluation in the implementation and improvement science fields\u0026mdash;a systematic literature review. Implement Sci. 2019;14(1):72.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eBowser DM, Henry BF, McCollister KE. Cost analysis in implementation studies of evidence-based practices for mental health and substance use disorders: a systematic review. Implement Sci. 2021;16(1):26.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDrummond MF, Sculpher MJ, Torrance GW, Stoddart GL. Methods for the economic evaluation of healthcare programs. 3 ed. USA: Oxford University Press; 2005.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDonovan T, Abell B, Fernando M, McPhail SM, Carter HE. Implementation costs of hospital-based computerised decision support systems: a systematic review. Implement Sci. 2023;18(1):7.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eGold HT, McDermott C, Hoomans T, Wagner TH. Cost data in implementation science: categories and approaches to costing. Implement Sci. 2022;17(1):11.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMichaud TL, Pereira E, Porter G, Golden C, Hill J, Kim J, et al. Scoping review of costs of implementation strategies in community, public health and healthcare settings. BMJ Open. 2022;12(6):e060785.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eRitzwoller DP, Sukhanova A, Gaglio B, Glasgow RE. Costing behavioral interventions: a practical guide to enhance translation. Ann Behav Med. 2009;37(2):218\u0026ndash;27.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDonovan T, Carter HE, McPhail SM, Abell B. A qualitative interview study to explore costing of implementation strategies to support digital health adoption it\u0026rsquo;s the difference between success and failure. 01 February 2024, PREPRINT (Version 1) available at Research Square [\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doiorg/1021203/rs3rs-3828958/v1]\u003c/span\u003e\u003cspan address=\"https://doiorg/1021203/rs3rs-3828958/v1]\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eChapel JM, Wang G. Understanding cost data collection tools to improve economic evaluations of health interventions. Stroke Vasc Neurol. 2019;4(4):214\u0026ndash;22.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHoeft TJ, Wilcox H, Hinton L, Unutzer J. Costs of implementing and sustaining enhanced collaborative care programs involving community partners. Implement Sci. 2019;14(1):37.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDonovan T, Abell B, McPhail SM, Carter HE. May. Development of Cost-IS (costing implementation strategies) instrument for digital health solutions: a modified e-Delphi study. 16 2024, PREPRINT (Version 1) available at Research Square [\u003cspan class=\"ExternalRef\"\u003e\u003cspan class=\"RefSource\"\u003ehttps://doiorg/1021203/rs3rs-4229774/v1]\u003c/span\u003e\u003cspan address=\"https://doiorg/1021203/rs3rs-4229774/v1]\" targettype=\"URL\" class=\"RefTarget\"\u003e\u003c/span\u003e\u003c/span\u003e\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePaparini S, Green J, Papoutsi C, Murdoch J, Petticrew M, Greenhalgh T, et al. Case study research for better evaluations of complex interventions: rationale and challenges. BMC Med. 2020;18(1):301.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eYin RK. Case study research: Design and methods. sage; 2009.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCrowe S, Cresswell K, Robertson A, Huby G, Avery A, Sheikh A. The case study approach. BMC Med Res Methodol. 2011;11:100.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eThomas G. A typology for the case study in social science following a review of definition, discourse, and structure. Qualitative Inq. 2011;17(6):511\u0026ndash;21.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eMurray M, Sargeant S, Harper D, Thompson A. Qualitative Research Methods in Mental Health and Psychotherapy: An Introduction for Students and Practitioners. Chapter:, Publisher: Sage, Editors: David Harper and Andrew Thompson. 2011:163\u0026thinsp;\u0026ndash;\u0026thinsp;75.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003ePinnock H, Barwick M, Carpenter CR, Eldridge S, Grandes G, Griffiths CJ, et al. Standards for Reporting Implementation Studies (StaRI) Statement. BMJ. 2017;356:i6795.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWaltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, Chinman MJ, Smith JL, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci. 2015;10(1):109.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSzewczyk Z, Reeves P, Kingsland M, Doherty E, Elliott E, Wolfenden L, et al. Cost, cost-consequence and cost-effectiveness evaluation of a practice change intervention to increase routine provision of antenatal care addressing maternal alcohol consumption. Implement Sci. 2022;17(1):14.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eCurry JM, McGregor C, Tracy S. A systems development life cycle approach to patient journey modeling projects. Stud Health Technol Inf. 2007;129(Pt 2):905\u0026ndash;9.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eSatzinger JW, Jackson RB, Burd SD. Systems Analysis and Design in a Changing World. Boston, MA, UNITED STATES: Cengage Learning; 2015.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eLangley GJ. The improvement guide a practical approach to enhancing organizational performance. 2nd ed. ed. San Francisco, California: Jossey-Bass; 2009.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eDamschroder LJ, Reardon CM, Widerquist MAO, Lowery J. The updated Consolidated Framework for Implementation Research based on user feedback. Implement Sci. 2022;17(1):75.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eWagner TH. Rethinking How We Measure Costs in Implementation Research. J Gen Intern Med. 2020;35(Suppl 2):870\u0026ndash;4.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eKhan ZA, Kidholm K, Pedersen SA, Haga SM, Drozd F, Sundrehagen T, et al. Developing a Program Costs Checklist of Digital Health Interventions: A Scoping Review and Empirical Case Study. PharmacoEconomics. 2024;42(6):663\u0026ndash;78.\u003c/span\u003e\u003c/li\u003e \u003cli\u003e\u003cspan\u003eHuter K, Krick T, Rothgang H. Health economic evaluation of digital nursing technologies: a review of methodological recommendations. Health Econ Rev. 2022;12(1):35.\u003c/span\u003e\u003c/li\u003e\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":true,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true},"keywords":"implementation costs, digital health, implementation science, health economics","lastPublishedDoi":"10.21203/rs.3.rs-4873079/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-4873079/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground\u003c/h2\u003e \u003cp\u003eThere is a lack of pragmatic approaches that standardise the cost estimation of implementation strategies. An instrument for data collection and cost estimation of implementation strategies for digital health innovations (Cost-IS) was recently developed. This study aimed to compare the nature of implementation cost estimates across three case studies. Secondary aims were to investigate the perceived value of these cost estimates to project leads and stakeholders, and to assess the feasibility and generalisability of Cost-IS as a standardised tool to compare the cost estimates of implementation strategies across three case studies.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e \u003cp\u003eAn evaluative and descriptive collective case study was conducted to estimate the cost of implementation strategies associated with the introduction of healthcare interventions using the Cost-IS instrument. Eligible case studies included completed implementation projects that had planned to cost implementation and had prospectively collected data on implementation-related resource use throughout the project. Implementation data was separately analysed for each case study. Adaptations to the instrument were made where appropriate and documented. The Cost-IS instrument was tailored in a way that best allowed for meaningful outputs from each case study. The feasibility, generalisability and limitations of Cost-IS were observed by the analyst through the costing process. Feedback on the outputs were obtained from project leads, via questionnaires.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003eThe Cost-IS instrument enabled comparisons of cost estimates of implementation strategies between three real-world implementation case studies. Labour accounted for the majority of implementation costs. Adaptions to Cost-IS during the costing process were made to improve data capture granularity, meaningfully categorise the data, and enhance the instrument\u0026rsquo;s flexibility for analysis. Implementation cost findings were perceived by project leads to be valuable for reporting and evaluating, should be customised to project context, and include relevant contextual information.\u003c/p\u003e\u003ch2\u003eConclusions\u003c/h2\u003e \u003cp\u003eCost-IS was found to be a flexible instrument that can be tailored to meet the requirements of a range of implementation projects. Challenges during this process included separating implementation-related costs from intervention costs and research-related costs. Cost-IS is limited to costing implementation strategies and thus far has only captured tangible resources.\u003c/p\u003e","manuscriptTitle":"Comparing cost estimates of implementation strategies using Cost-IS: a collective case study","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2024-10-17 15:06:57","doi":"10.21203/rs.3.rs-4873079/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"researchsquare","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":true,"externalIdentity":"","sideBox":"","snPcode":"","submissionUrl":"/submission","title":"Research Square","twitterHandle":"researchsquare","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"","reportingPortfolio":"","inReviewEnabled":false,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"73d321d9-daa1-45b0-a7a7-0168ed9708de","owner":[],"postedDate":"October 17th, 2024","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"posted","subjectAreas":[],"tags":[],"updatedAt":"2024-12-06T01:28:53+00:00","versionOfRecord":[],"versionCreatedAt":"2024-10-17 15:06:57","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-4873079","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-4873079","identity":"rs-4873079","version":["v1"]},"buildId":"qtupq5eGEP_6zYnWcrvyt","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}