Usability testing a digital web application for evidence-based commissioning decisions in the context of implementing Mobile Stroke Units in England

Usability testing a digital web application for evidence-based commissioning decisions in the context of implementing Mobile Stroke Units in England | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Article Usability testing a digital web application for evidence-based commissioning decisions in the context of implementing Mobile Stroke Units in England Lisa Moseley, Anna Laws, Michael Allen, Gary A Ford, Martin James, and 11 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5668150/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 09 May, 2025 Read the published version in npj Digital Medicine → Version 1 posted You are reading this latest preprint version Abstract Commissioning of innovations in healthcare is a complex socio-technical process, ideally informed by high quality evidence. However, evidence is not always prepared and presented in a format usable for commissioning decisions. Agile methodology, combined with qualitative co-design, were used to develop a digital web application incorporating machine learning models of stroke outcomes to inform commissioning decisions for the implementation of Mobile Stroke Units (MSUs) in England, followed by usability testing using Think Aloud methodology. Sixteen stakeholders involved in developing consensus on model parameters and pathways participated with data thematically analysed. Required improvements to the web application were identified and novel insights into the complexity of context-specific commissioning decisions were generated, which also informed participants’ views on the viability of MSUs. This study provides empirical evidence in support of developing innovative and accessible digital dissemination methods to engage with commissioning processes and prospectively understand commissioning challenges. Health sciences/Health care/Health policy Health sciences/Health care/Health services Health sciences/Neurology/Neurological disorders/Stroke Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Introduction Stroke is one of the leading causes of death and disability worldwide 1 . Strokes are caused either by a clot, or a bleed, in the brain and are characterised by rapid onset of symptoms, which range in severity, and can impact all areas of function including mobility and speech 2 . The long-term effects following a stroke can be severe, requiring ongoing health treatment as well as social care support 3 , and it is estimated that the number of strokes in the UK will rise by 59% by 2035, from the 2020 level of approximately 100,000 a year 4 . This makes stroke an important health concern due to the individual impact, alongside the economic burden. Treatment for ischaemic strokes, where the stroke is caused by a clot, can reduce, and even fully diminish, the impacts of a stroke. However, this treatment is time sensitive, making rapid diagnosis and intervention imperative 5 . Mobile Stroke Units (MSUs) have been growing in popularity as a means to diagnose and begin treatment of ischemic strokes, in pre-hospital settings, leading to the European Stroke Organisation (ESO) to recommend consideration of MSU deployment 6 . MSUs are specialist ambulances that have a computed tomography (CT) or computed tomography angiography (CT-A) scanner, capable of scanning the head with the latter using contrast to visualise blood vessels, and MSUs also have stroke specialist staffing and point of care testing 7 – 14 . MSUs have focused on the treatment of ischaemic stroke, and have shown that MSUs can speed up to time to treatment with intravenous thrombolysis (IVT), a medication used in some cases to dissolve clots 9 – 13 . An additional treatment, mechanical thrombectomy (MT), is used for some patients where a clot is occluding a large vessel, known as a large vessel occlusion (LVO). MT is performed by hospital-based interventional neuroradiologists who manually retrieve the clot to restore blood flow 15 . In the United Kingdom (UK), like most countries, MT is only available at certain specialist centres, comprehensive stroke centres (CSC), usually based in urban areas. However, patients with suspected stroke are currently often transported by the ambulance service to their nearest hospital with standard stroke services available, which can be an acute stroke unit (ASU) without MT capabilities, and thus requires an additional transfer of patients with LVO to CSCs. While initial trials of MSUs focused on the increased speed in which IVT was given, more recent evidence has found there is also the potential for MSUs to speed up access to MT, although this has not been consistently evidenced 7 , 8 , 14 . Despite the evidence favouring MSU in certain settings and healthcare systems, they are currently not commissioned in the NHS. Commissioning has been a part of the National Health Service (NHS) since its inception in 1948, with services being provided by Local Authorities, hospitals and independent practitioners (i.e. General Practitioners), each having to submit estimated costs, but largely managing their own budgets locally 16 . In the 1990s the NHS was changed to a quasi-market to encourage competition between providers, aiming to reduce costs and improve efficiencies 17 . Arguments for the quasi-marketisation championed the purchasing of services being separated from those providing the service, with the view that this would improve equitable access to services universally, without influence from those providing services, particularly clinicians 18 . Commissioning of services within the NHS remains complex, with decisions being made centrally and by local commissioners 19 . Increasingly there is also tension between cost-effectiveness commissioning compared to values-based commissioning which, while similar, differ in emphasis; the former focuses on societal benefits and the latter is intended to consider individual patient perspectives 20 . Evidence-based policy is key to all service planning within the NHS and is a key consideration in commissioning decisions 21 . The potential benefits of evidence-based commissioning decisions include a greater chance of a service being successfully implemented as well as more efficient use of public funds within the health service 22 , 23 . Despite the benefits of using data in commissioning decisions the reality of how commissioning decisions are made is much more complex. 24 , 25 Instead of being a transactional approach, commissioning involves close relational interaction between commissioners, providers of services and often patients and the public 26 , 27 . Numerous barriers exist to commissioners using evidence in their decision-making, including research articles being behind paywalls, research being clinically focused rather than commissioner focused, lack of localised knowledge or context in research, and the effort required to understand academic research 28 . A number of facilitators to data-informed commissioning have been identified including making data presented manipulatable, showing the data meaning at local levels, and making the data easy to interpret 29 . Alongside these complexities in the commissioning of services there is now an emphasis on the co-production of health and care services, requiring services to be informed by relevant stakeholders 30 . However the use of co-production has posed further challenges to commissioners with difficulties involving stakeholders in an already complex commissioning process, the work required for co-production being unfeasible within commissioner time constraints, and the inflexibility of the commissioning processes to respond to additional input 25 , 31 . To address these challenges, researchers are increasingly required to develop new approaches to sharing evidence created through research that can directly inform commissioning decisions, such as more accessible data summaries 32 . Web applications, as one such approach, are increasingly being used to provide end-user access to complex models (e.g. see Narayanan et al 33 ) and have key advantages for making models available. These advantages include reduced barriers to end-user engagement, no specialist software or hardware needs for the end-user, rapid prototyping and development, the ability to work across multiple platforms (e.g. PC, tablet, phone), centralised updates/maintenance, immediate deployment of fixes or enhanced functionality, and scalable infrastructure which can adapt to demand. Within healthcare informatics, there is now an increased emphasis on sharing the underlying code of models 34 . This paper describes the approach taken to developing a web application for modelling the effectiveness of MSUs in England and examines the usability of the web application for informing commissioning decisions. Methods Digital web application development We built models of clinical benefit of MSUs 35 , and we aimed to make those models available to potential end-users – including stakeholders involved in commissioning processes – as a digital web application. The chosen method of web application development was to follow an agile process 36 where iterative development, with user feedback, is used rather than a highly developed starting specification. User feedback was incorporated throughout the modelling and web application development via a process of qualitative co-design with stakeholders, recognised to be of importance for helping to ensure clinical relevance 37 . Model code development took place on GitHub (https://github.com/stroke-modelling/muster_workshop_web_app_1) and Streamlit (https://streamlit.io/) was chosen as the deployment platform, as this provides a platform designed for modellers and data scientists to easily implement web applications. Initial ranges and default values for the model parameters (which the end-user can change) were reached after examining published information on timings from MSU clinical trials (see Chen et al 38 and Fatima et al 39 for reviews). The choice of user-controlled input parameters and ranges for their values, and preferred model outputs, were then discussed with stakeholders using Nominal Group Technique 40 to examine their appropriateness and develop consensus. Some examples of the web application interface are provided in the findings. Study design The study tested the usability of the developed web application using qualitative Think Aloud methodology 41 . Think Aloud is a common methodology in usability testing and asks participants to use the web application while talking out loud about what actions they are performing and why, and to provide thoughts on usability. We expanded the usability testing to examine how the web application could inform commissioning decisions in the context of deploying MSUs in England. Participants, recruitment and sampling Participants were recruited via purposive, opportunistic and snowball sampling, as part of the wider research project in relation to MSUs 42 Stakeholders consisted of stroke clinicians (doctors and nurses), ambulance service staff, decision-makers involved in commissioning processes, and those with lived experience of stroke care (either as a stroke patient or supporting someone impacted by stroke). The research team used existing networks to recruit staff participants and the recruitment of patient and public involvement representatives (PPI) was supported by a national charity, the Stroke Association. PPI representatives who had expressed an interest to the Stroke Association in being part of research projects were contacted by the charity with an overview of the study and individuals responded if they wished to take part. The researchers then purposively sampled those who had responded, to ensure diverse representation, based on geographic location, age and type of stroke. Data collection All data were collected online via Microsoft Teams between September and October 2024 using interviews that consisted of a free-form Think-Aloud component, followed by semi-structured discussion. All discussions were audio recorded and transcribed verbatim. Prior to data collection the research team developed a topic guide (supplementary material) to cover key aspects of usability based on the major elements of the web application. These were used to facilitate reflection and discussion amongst participants during the Think Aloud component, particularly during periods of silence, and also to guide discussion during the semi-structured component of the interview. Prompts included: what they liked/disliked about the app; ease of use; any additional information they would like to see; how this might be used by commissioning and in wider practice and how the information influenced views on MSUs. Data collection was conducted by JS and LM. Participants were given a guided tutorial of the web application and the various functions by JS. Participants were then asked to access the web application themselves and share their screen with the researchers while using it. Both researchers observed how users engaged with the web application, whether certain parts of the application were used more than others, and areas that participants required additional guidance to utilise. To ensure participants were not digitally excluded from participating, they were also given the option for the interviewer to use the web application with the participant telling the interviewer what to do and giving their thoughts. Data analysis Following data collection all interviews were transcribed verbatim and analysed using Nvivo 12 (Lumivero) 43 by LM. To consider data from participants’ actions that were seen during the usability rather than verbally recorded, LM and JS met after each interview to compare observations on the participants’ use of the web application. These observations were incorporated into the analysis. Thematic analysis 44 was used by LM to create initial themes exploring commonalities in the data as well as contrasts, with additional input from JS. The themes were further scrutinised and refined with input from the wider research team resulting in the finalised themes presented. Findings Sixteen people participated in usability testing, consisting of patient and public involvement representatives (n=6, 38%), stroke or critical care physicians (n=6, 38%), and ambulance service staff with front-line clinical and senior management expertise (n=4, 25%). Interviews lasted between 38 minutes and 89 minutes (mean = 58 minutes). We identified where users were able to successfully navigate and operate the web application, as well as areas that its usability could be improved. Together, these form the first theme: Usability of the web application. Central to this theme were the importance of clear user instructions and participants’ preferences for data presentation. The second theme developed, Informing commissioning and implementation decision-making, examines how usability testing of the web application provided insight to the complexity of commissioning and implementation of MSUs, and allowed for a deeper exploration of how various stakeholders considered how the web application could be used. Related to this was a third theme, Informing views on intervention viability, that captures how usability testing allowed a further examination of whether MSUs could or should be commissioned. Finally, we developed a single cross-cutting theme that interacted with and was central to the second and third major themes; Trustworthy and robust data. This theme, particularly focusing on the need to include health economics data on cost-effectiveness of MSUs, reinforces the need for commissioning and implementation decisions to be data-driven and was seen as a requirement for participants to provide conclusive views on whether MSUs should be implemented. Figure 1 presents an overview of the thematic structure of the findings. Images of the web application are included to contextualise findings. Outcomes shown in images are for illustrative purposes only and not necessarily representative of the final models. Usability of the web application The overall view of the web application was positive, with most participants quickly able to use it themselves after a brief guided tutorial by researchers. However, participants highlighted that should the web application be distributed widely, it would not be feasible to give each individual a tutorial, and as such would benefit from a recorded video tutorial, as well as written instructions with screenshots. This concern was discussed by a stroke consultant: “You are here, you explained it to me, and I found it very easy. So the question is, somebody who has not seen this and is seeing them for the first time, how do you make it user-friendly for them?...People differ, some people like that written instruction…with some images attached…Others would like the video…I think you need to cater for those two groups…” (Participant 13, stroke physician). When using the web application, participants particularly found the selection of model parameters intuitive, where they were able to easily modify some time parameters used in the modelling (see figure 2). This allowed participants to tailor the models, for instance by amending the ambulance response time using publicly available data on response times to stroke calls within their region. Participants felt that the modifiable parameters needed a brief explanation of how the parameter interacted with the pathway and evidence-based justification for default values, such as whether previous stroke data had been used to obtain average values. These observations were explained by a stroke physician and an ambulance service staff member: “…where there’s this little question mark [above each parameter]…that’ll be nice if the question mark said, you know, the clarification of what it meant…” (participant 9, stroke or critical physician) “Is that just a random figure or is that based on data? Is that currently where we think we are [in relation to ambulance response times]?” (Participant 2, ambulance service staff) Observing participant interactions with the web application revealed that they sometimes struggled to select the base location for an MSU, having to manually deselect default sites from a list of 128 locations. Despite being shown how to do this as part of the tutorial, some participants were unsure which boxes to (un)tick, and it was time-consuming for people to deselect multiple default options (figure 3). Within the web application, data were presented in multiple formats including maps, graphs and tables (figure 4). Observing participant interactions with the web application, it was apparent that maps were the most accessible format, with participants paying little attention to the graphs. Participants also did not interact with the data tables, consisting of 51 columns and representing all underlying data for the models, but they were deemed to be important (see Trustworthy and rigorous data theme). “I did listen to what you described about the graphs on the right-hand side. But based on [the maps], you can predict what impact [MSUs] will have, I think. And that’s what [commissioners] need. So [commissioners] need something that’s visual. [Commissioners] need something that has some numbers associated with it, but also, because of the way that you’ve built this system, and I have to say, huge congratulations because it’s pretty intuitive, even with a very small amount of demonstration, you can get people to understand what their region looks like.” (Participant 14, stroke or critical care physician). Whilst the maps received most attention, participants often struggled to understand the concept of utility shift – a change in quality of life measured through health utility scores – which was used as the default outcome comparison between usual care. Even with explanation of utility shift by the researchers, participants found it difficult to translate into the actual benefit to patients when using the web application. While some stroke physicians were more familiar with the concept of utility shift this was not universal across all stroke physicians, restricting the usability of the web application to understand potential impact on patient outcomes. Concerns about utility shift, and how understandable this would be, were summed up by an ambulance service staff member: “I think…the biggest unanswered question as to what…the colours are lovely, and they’re dramatic and it makes you think, “Oh, wow, that’s a big difference.” But what is the difference?...I think I get my head around [utility shift]. I still think that’s- as someone lay, or whoever, or not even necessarily lay, that’s going to be their main question” (Participant 4, ambulance service staff). Despite the concerns regarding the understanding of utility shift, as represented by colours on a map, participants were very positive about the use of the coloured map to visually represent the data quickly as well as being more thought provoking as explained by a stroke physician, “it’s a really nice way of showing it…It makes it visibly more thought-provoking” (Participant 8, stroke or critical care physician). Participants were also able to provide a number of helpful suggestions about changes to the maps that could enhance the usability of the web app. Suggestions included mapping on road systems which would help users orientate more quickly as well as consider travel infrastructure in the area and including the locations of acute stroke units s on the MSU map, as this was currently only shown on the usual care map. Participants felt that, as MSUs would likely be managed by the ambulance service, having the locations of ambulance stations available would be beneficial, particularly when zooming into specific localities or regions. Lastly, participants felt that a number of selectable ‘overlays’ would be helpful which could show population density and demographics. These were then included in an update to the web application (figure 5). “…if you could integrate that with population density, that might be helpful… I suppose the older population, people over the age of 50 or 60, adding that as part of your population density map, accepting that stroke is commoner in later life…absolutely [include deprivation]…with deprivation, of course, you’re going to see that, in terms of scale and numbers, more in urban areas than in more rural areas…” (Participant 2, ambulance service staff). Informing commissioning and implementation decision-making Usability testing of the web application provided the opportunity to examine commissioning and implementation of MSUs, including how the web application could be utilised. As a result of the usability testing, we identified that commissioning of MSUs is unlikely to be a straightforward technical, data-informed process. Instead, it would be a complex socio-technical process based on multiple priorities. This is reflected by a participant with experience of working with commissioners, who felt that commissioners would also consider other healthcare conditions. “I, honestly, can’t see commissioners using this [web application]. I think it’s great for somebody like me, that’s overseeing huge projects across a regional footprint or, indeed, at system level. So I think it’s great for the providers, it’s great for the network, and you could say it would be great on an urgent and emergency care element as well… As far as our commissioners… certainly, at the moment, there is so little emphasis on stroke anyway that just keeping stroke on the radar at all is tricky” (Participant 5, ambulance service staff). Other participants disagreed as to whether commissioners would use the web application directly to inform commissioning decisions. In the cases where it was thought they would not use the web application, it would still support commissioning decisions by being able to provide the evidence for a business case put forward by services or to justify why MSUs should not be commissioned. This contention between participant views reflects the uncertainty that exists in commissioning a new innovation, including knowing who has responsibility for obtaining and examining evidence. This represents the social processes involved in commissioning, where a new innovation would likely then require a local champion. “…I think commissioning at a national level would use this and appreciate it. I don’t suspect they get this from many other things that we ask to be commissioned” (Participant 8, stroke or critical care physician). “…I think that, for an ambulance-based resource, it would probably be for us to make the case [for MSUs], but, you know, share the data, and give the Integrated Care Boards, the commissioning bodies, access to view this to support the conversation so they could see for themselves almost the benefits from it…” (Participant 6, ambulance service staff). Participants also discussed how the wider healthcare economy would contribute to MSU commissioning decisions. For instance, implementation of MSUs may result in some acute stroke units receiving fewer patients and thus make their own services financially unviable due to a loss of the tariff they currently receive for administering IVT. There were also conflicting views amongst participants regarding national or regional commissioning of MSUs. Some participants believed MSUs would need to be part of either a new or existing specialised commissioning pathway, such as the national commissioning of mechanical thrombectomy, whereas others thought MSUs would be commissioned as part of standard block commissioning contracts with Integrated Care Boards in partnership with local ambulance services. However, if regionally commissioned there were further concerns that ambulance services would have to find capital from within their existing budget to fund MSUs, but would not benefit from the gains, and implementation of MSUs. These concerns further highlight the social, particularly the political, challenges that were identified as a result of using the web application. “you’d get the primary stroke centres…they get a tariff don’t they for thrombolysing somebody in the primary stroke centre [ASU]. So they’d lose that and they’d lose some of their metrics for hyperacute stroke management, because in essence, you’re bypassing them to go straight to the CSC. So I think your primary stroke centres might not like [MSUs being commissioned]” (Participant 9, stroke or critical care physician) Alongside the potential for the web application to inform our understanding of commissioning decisions, either directly or indirectly, the web application also helped to develop an understanding of how it could be used operationally to support a commissioned MSU, particularly in the ambulance service. A critical care consultant and an ambulance service staff member both described how they felt the web application could be used as the basis for informing operational decisions once (and if) MSUs were commissioned. Participants recognised that this would likely require further modifications to the web application, particularly by integrating live ambulance service data on response times and predictive geographic models of stroke incidence. “…I suppose the beauty of having big data like this and a very visual representation is, when you pick a particular area…to put the origin of the call on this…and if the [origin of the call is] in the correct colour [showing benefit from MSU attendance]…you go or you don’t go.” (Participant 14, stroke or critical care physician) “…it probably would have…benefit…in terms of looking to plan the shift, plan the day, actually looking where we’ve historically seen higher instances of stroke and those that have subsequently turned out as a confirmed diagnosis. So for predicting around where to position the vehicle on a shift-by-shift basis…”(Participant 6, ambulance service staff) In discussions with participants who had lived experience of stroke, there were differing views on whether the web application would be useful, or even be used, by the general public. Of the participants who did feel the web application might be used, they explained it was because those people would likely have an interest in stroke care and developing treatments, as described by participant 12 who led a regional group for stroke survivors, “…Our group would be [interested in the web application]. They’re very open and want to learn more.” However, it was also highlighted that the data, and visual representation from the web application would still be of benefit to the public in educating, and gaining public support for, MSUs. This potential use was described by a person with lived experience of stroke and a critical care consultant: “…maybe if it was London and they were told this won’t- It might actually worsen outcomes, then people might not be for it, or as interested. But…those regions where it would help… I think, yeah, people would definitely be interested.” (Participant 11, patients and public involvement representative) “…if you’re going to commission anything, you have to get the public on board. You have to get people to understand what they’re going to get out of this, not in terms of clinical outcomes, not in terms of value for money, but actually, what does this mean for me tomorrow? And the patient public voice is really important…it’s a key aspect of stakeholder engagement.” (Participant 14, stroke or critical care physician) Informing views on intervention viability Once participants had used the web application and explored some of the early modelling data presented within it, they then provided their views on whether MSUs would be viable, and thus providing an additional use for the web application. In doing so, participants discussed how the maps showed relatively minimal gains to patient outcomes, with consideration to how many patients an MSU could realistically see in one day, and how many of those patients would be having an ischemic stroke and then be eligible for treatment. Alongside this, participants highlighted that the most gains appeared to be for patients with a large vessel occlusion (LVO), where the LVO would be recognised on the scan in the MSU, and patients could be directly transported to a comprehensive stroke centre (CSC) that is able to deliver mechanical thrombectomy, and thus avoiding secondary transfers. However, one member of the ambulance service explained this may not be an accepted approach by the CSCs. Specifically, it was unclear whether the CSCs would be able to review MSU scans obtained using the onboard Computer Tomography Angiography (CTA), and if they would be of sufficient technical quality for a direct admission decision to be made. “The issue…is…the… [CTA] scanner has got quite a narrow hole, it doesn’t do the neck vessels. So, the mechanical thrombectomy centres [CSCs] won’t accept those patients because they would have to repeat the CTA, and then they find they can’t get past the carotid [required to perform MT]” (Participant 4, ambulance service staff) Alongside this, participants also highlighted that the maps showed the most gains away from CSCs, which are based in large urban centres, and appeared to make the most improvement in rural areas where travel times to a hospital are the longest. This, again, raised the question of how efficient MSUs would be in these areas, given the relatively low population levels, but participants also recognised that improving access in these areas was important to reduce inequity in access to stroke care. Further, participants discussed the relative levels of poverty in some of these areas, such as some coastal communities, alongside higher age demographics in rural areas, both of which increase the risk of stroke. The difficulty in making this assessment was summed up by a critical care consultant, who using hypothetical numbers: “I don’t know the genuine numbers…but if you look at the tip of Cornwall for instance, so beyond Truro, should we place an MSU down there at the cost of £500,000 that will be used five times a week? Or should we place two MSUs in the middle of Plymouth that will cost £1m in total but be used 20 times a week? The former gives you equity of access to the MSU. The latter gives you volume of patients and the number needed to treat for mechanical thrombectomy” (Participant 14, stroke or critical care physician). Lastly, participants highlighted that MSUs were unlikely to provide the same level of benefit as investing money into MT services and widening the availability. However, they recognised that the main issue with increasing MT services was that there were not enough trained interventional neuroradiologists and therefore a like-for-like comparison on benefit was not possible and as expansion of MT services was not immediately available that MSUs could go some way to improving stroke care. This was explained by a stroke physician: “…I think if you’re realistic there’s no point commissioning more thrombectomy centres because we can’t staff the ones we’ve got. There’s no point pumping money into thrombectomy centres…We can’t train or recruit the specialists we need to deliver at thrombectomy centres. I don’t think throwing money at thrombectomy centres helps.” (Participant 8, stroke or critical care physician) Trustworthy and robust data Throughout all of the interviews there were reoccurring discussions about the need for health economics data. Participants felt strongly that incorporating the health economics into the web application would make it more useable in terms of informing their views on the viability of MSUs. They felt this would assist them in understanding not just the potential gains to individual patient outcomes, but also understanding the financial implications, which would make the web application even more useful for informing both commissioning and operational decisions regarding MSU implementation. The need for this was discussed by an ambulance service staff member: “I think the health economic bit would be important and an understanding of the numbers of patients that you could expect an MSU to attend in a 24-hour period or a 12-hour period, etc.” (Participant 2, ambulance service staff) The inclusion of health economics data was also thought to increase the chances of the web application being used directly by commissioners as this was seen to be a key element of their decision making. Without health economic data, stakeholders involved in commissioning decisions would not have all the information they needed. The emphasis on health economic data was explained by a patient and public involvement representative: “That is going to be really powerful, once that’s in [the health economics data], in terms of commissioning” (Participant 17, patient and public involvement representative) Lastly the health economics data was highlighted as essential to understanding how commissioning decisions would be informed by debates surrounding how best to improve equitable access to stroke care through implementing MSUs. For example, in terms of rural and urban inequity it was felt that MSUs would need to show they would be economically viable, by seeing enough patients, and that this would likely influence any decisions made about the extent to which MSUs could be used to reduce inequities in access to stroke care. “The health economic benefit of that, both the number of individuals who can help, but also the utility…Is the balance not that you just have to provide this to the most people you possibly can and accept there are going to be areas you cannot cover well?” (Participant 8, stroke or critical care physician) Discussion This study presents empirical evidence for how a digital web application can be used to both inform evidence-based commissioning of innovations in the context of acute stroke care and to explore the complexity of commissioning decisions amongst stakeholders in the context of usability testing. Some design elements were identified as requiring changes, the majority of which were specific to the implementation of MSUs in England, highlighting the importance of incorporating qualitative co-design into an agile development process for digital web applications to ensure that they are fit for purpose for all stakeholders involved in commissioning processes. More specifically, our findings emphasise that digital web applications which convey complicated information require, at minimum, brief training and tooltips to guide the user. The use of co-design in healthcare 30 , and agile processes in application development 36 , are now well established. The benefits of incorporating these approaches is now being recognised with examples of co-production and agile processes being used together in the development of health monitoring applications 45 , assistive technologies 46 and environmental monitoring 47 . It is acknowledged that building innovative technology is not possible without recognising the context within which they would deployed 48 , thus co-design, allowing for understanding of the social and professional contexts, alongside agile processes allowing for rapid change of the web application, ensure both its usability and that the developed application is attuned to the complex relational and transactional process that underpin it 26 , 27 , 49 . However, this approach also has challenges. There is a requirement for negotiation, at times, between the requests put forward by those involved with the co-production and what is achievable in development, which requires both parties to have clear discussions and compromise 50 , 51 . Lastly, it is also important that the process is properly facilitated to be of value to both the co-design participants and the researchers, which requires researchers to buy-in to the process and at times be outside of the comfort zone of traditionally controlled research processes 52 . The development and usability testing of the digital web application provided an additional benefit in that it allowed us to examine complex issues associated with commissioning that otherwise would not have been possible. We had previously identified several challenges to implementing MSUs in the English and Welsh NHS, including where to locate MSUs, how to staff them, and making dispatch decisions 53 . With a similar group of participants, we have now been able to identify that commissioning is an additional challenge to implementation. Stakeholders using the web application helped highlight the complexities of the commissioning process, 24 , 25 specifically uncertainty as to who would commission MSUs and whether this would be conducted locally, through centralised commissioning, or linked with specialist commissioning. This coincided with our experiences where, despite informal discussions with various types of commissioners, all declined to participate in the study as they did not feel they would ultimately be involved in the process of commissioning MSUs. The identified uncertainty around responsibility for commissioning is representative of a principal-agent problem resulting from a lack of role clarity. Principle-agent theory is drawn from political and economic science and provides a framework for understanding how work is delegated from principals to agents 54 . In the context of MSUs – where there currently is no policy for their implementation in the NHS – policymakers (principals) would delegate decisions to commissioners (agents), as they do in other complex pre-hospital care commissioning 55 . To address this problem, policy should be developed relating to commissioning decisions for MSUs in the NHS including the identification of who the agents would be. The web application also helps address a number of the previously identified barriers identified in the use of research in commissioning decisions, specifically that the web application would not be behind a paywall and was co-designed 25 , 31 with a focus on informing commissioning decisions 28 . The data presented within the web application can be used to both on a national, and local level, with the maps making the data easier to interpret quickly 28 , 29 . Consequently, the web application would most likely be used by people developing and presenting the case for MSUs to be commissioned rather than commissioners themselves, providing opportunity for the exploration and presentation of context of both existing and alternative services. In doing so, using a web application helps to reduce information asymmetry that may exist between commissioners and providers, which is important for making more optimal commissioning decisions 56 . However, this would only apply where the information is used in support of MSUs as opposed to using the web application to inform when MSUs should not be commissioned, and thus also represents a moral hazard that needs to be considered when commissioners review business cases. The use of the web application also allowed us to further explore some of the previously identified implementation challenges 53 , as well as identifying new challenges. One of the keys areas of debate was in relation to the base MSU location and particularly whether MSUs should be based in an urban area, where they might be of benefit to more of the population, or whether the focus should be on rural areas that have poorer access to acute stroke care. This challenge was further discussed following use of the web application which showed higher gains for those further from hospitals, such as rural areas, however participants remained concerned about how much the MSU would then be utilised. As a result of the agile process adopted the web application was able to be updated to include population density which assists in making a determination and was thought to be essential to support commissioners to balance value-based and cost-effective decisions 20 . The use of the web application also raised the concern about minimal gains to patients, as shown by the utility shift, particularly those without an LVO. This had not been a previous concern however participants then questioned whether MSUs should targeted towards those with an LVO. However, literature to date has shown conflicting results about whether MSUs speed up access to MT for those with LVO 7 , 8 , 14 . Lastly, the use of the web application highlighted the need for the health economics data to be incorporated into the web application, in line with the need for potential new services to be both of clinical benefit and financially viable 20 . Previous studies have shown that MSUs have the potential to be cost-effective 57 – 60 however this was very context dependent, as often seen in economic evaluations 61 . None of these studies were completed within a UK context, with MSUs trialled in urban areas, and reported that MSUs only became cost-effective once they reached a certain patient threshold. Therefore, the web application, with incorporated health economics data would assist with some of the uncertainties with the rural/urban debate and whether to focus the resource on LVO patients, by showing both the benefits to patients and whether MSUs would offer financial value. However, even with this data there remains an area of political consideration, likely to affect commissioning decisions, which is not addressed in relation to the loss of IVT tariffs for ASUs if MSUs were delivering IVT. This could have an impact on the acceptability of MSUs which is a key element to the successful implementation of new interventions 62 , 63 . This again highlights the complexities of commissioning and that even when something is evidenced, close relational involvement is still required 26 , 27 . Strengths and limitations A strength of the study was that it was conducted using real-world data for the implementation of MSUs in England, ensuring that the models underpinning the usability testing provided accurate representations of MSU implementation, subject to recognised implementation challenges identified elsewhere in the study. 53 , 64 However, this also contributes to a limitation of the study; the usability testing does not represent how commissioners would actually use the web application as part of the relational decision-making that occurs in commissioning. This requires further evaluation should MSUs be commissioned in the NHS. Whilst some participants would be involved in commissioning processes (we are unable to describe exact job roles to protect anonymity), the study would have benefited from further commissioner involvement. However, this was challenging due to the uncertainties identified in the findings as to who would have responsibility for commissioning MSUs. This represents a circular dependency borne from modelling a potential innovation in the NHS that has no current existing commissioning pathway or policy. Other methods would be required to address this circular dependency, such as the use of vignette-based interviews that could present concrete hypothetical situations to commissioners. These vignettes would need to include hypothetical policies to address the previously described principal-agent problem resulting from a lack of role clarity. Conclusion A combination of agile processes and co-production in the development of a web application allowed us to develop and present usable models on the effectiveness of MSUs within the English NHS. Usability testing of the web application identified areas for improvement and helped identify complexities in commissioning processes, including the lack of role clarity around who would be responsible for the commissioning of MSUs in the absence of policy. Further implementation challenges associated with MSUs were also identified, particularly in relation to potential minimal gains and that maximum gains are seen within rural areas that are unlikely to have the population density required to make an MSU viable. This study provides empirical evidence in support of developing innovative and accessible digital dissemination tools to inform commissioning processes. Future research should evaluate the implementation and use of tools to inform commissioning processes, including the relational aspects of commissioning. Declarations Author Contribution AL, MA, GAF, MJ, SM, GM, KP, DP, CP, LS, PW, DW, PM and JS conceived the study. AL, ML and KP developed the web application and underlying models of clinical benefit. LM, MA, SM, GM, LJP and JS collected the qualitative data. LM and JS analysed the qualitative data and wrote the main manuscript and prepared figure 1. AL and MA provided additional contributions and prepared figures 2-5. All authors provided comments on the manuscript and approved it for submission. Acknowledgement We would like to thank all participants for providing their time and views. MA and KP were part funded by the NIHR Applied Research Collaboration South West Peninsula. The views expressed in this publication are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care. Data availability The datasets generated and/or analysed during the current study are not publicly available due to not having ethical approval or participant consent for the sharing of transcripts beyond select quotations. Code availability The underlying code for this study is available on Github: https://github.com/stroke-modelling/muster_workshop_web_app_1 statement to the declarations: "Ethical approval was provided via Northumbria University ethics online system (reference: 4117). The study was deemed by the Health Research Authority (HRA) to not require HRA approval." References Stroke Association. Stroke Statistics. Stroke Association 2024. https://www.stroke.org.uk/stroke/statistics National Health Service. Stroke. https://www.nhs.uk/conditions/stroke/ Donkor ES. Stroke in the 21(st) Century: A Snapshot of the Burden, Epidemiology, and Quality of Life. Stroke Res Treat . 2018;2018:3238165. doi:10.1155/2018/3238165 King D, Wittenberg R, Patel A, Quayyum Z, Berdunov V, Knapp M. The future incidence, prevalence and costs of stroke in the UK. Age and Ageing . 2020;49(2):277-282. doi:10.1093/ageing/afz163 Saini V, Guada L, Yavagal DR. Global Epidemiology of Stroke and Access to Acute Ischemic Stroke Interventions. Neurology . 2021;97(20_Supplement_2):S6-S16. doi:doi:10.1212/WNL.0000000000012781 Walter S, Audebert HJ, Katsanos AH, et al. European Stroke Organisation (ESO) guidelines on mobile stroke units for prehospital stroke management. Eur Stroke J . Mar 2022;7(1):Xxvii-lix. doi:10.1177/23969873221079413 Bender MT, Mattingly TK, Rahmani R, et al. Mobile stroke care expedites intravenous thrombolysis and endovascular thrombectomy. Stroke and Vascular Neurology . 2022;7(3):209-214. doi:10.1136/svn-2021-001119 Bluhm S, Schramm P, Spreen-Ledebur Y, et al. Potential effects of a mobile stroke unit on time to treatment and outcome in patients treated with thrombectomy or thrombolysis: A Danish–German cross-border analysis. European Journal of Neurology . n/a(n/a):e16298. doi:https://doi.org/10.1111/ene.16298 Ebinger M, Siegerink B, Kunz A, et al. Association Between Dispatch of Mobile Stroke Units and Functional Outcomes Among Patients With Acute Ischemic Stroke in Berlin. Jama . Feb 2 2021;325(5):454-466. doi:10.1001/jama.2020.26345 Grotta JC, Yamal J-M, Parker SA, et al. Prospective, Multicenter, Controlled Trial of Mobile Stroke Units. New England Journal of Medicine . 2021;385(11):971-981. doi:10.1056/nejmoa2103879 Helwig SA, Ragoschke-Schumm A, Schwindling L, et al. Prehospital Stroke Management Optimized by Use of Clinical Scoring vs Mobile Stroke Unit for Triage of Patients With Stroke: A Randomized Clinical Trial. JAMA Neurol . Dec 1 2019;76(12):1484-1492. doi:10.1001/jamaneurol.2019.2829 Kunz A, Ebinger M, Geisler F, et al. Functional outcomes of pre-hospital thrombolysis in a mobile stroke treatment unit compared with conventional care: an observational registry study. Lancet Neurol . Sep 2016;15(10):1035-43. doi:10.1016/s1474-4422(16)30129-6 Larsen K, Jaeger HS, Tveit LH, et al. Ultraearly thrombolysis by an anesthesiologist in a mobile stroke unit: A prospective, controlled intervention study. Eur J Neurol . Aug 2021;28(8):2488-2496. doi:10.1111/ene.14877 Zhao H, Coote S, Easton D, et al. Melbourne Mobile Stroke Unit and Reperfusion Therapy. Stroke . 2020;51(3):922-930. doi:doi:10.1161/STROKEAHA.119.027843 Phipps MS, Cronin CA. Management of acute ischemic stroke. BMJ . 2020;368:l6983. doi:10.1136/bmj.l6983 NHS Commissioning before April 2013 (House of Commons Library) (2016). Allen P, Petsoulas C. Pricing in the English NHS quasi market: a national study of the allocation of financial risk through contracts. Public Money & Management . 2016/07/28 2016;36(5):341-348. doi:10.1080/09540962.2016.1194080 Checkland K, Harrison S, Snow S, McDermott I, Coleman A. Commissioning in the English National Health Service: What's the Problem? Journal of Social Policy . 2012;41(3):533-550. doi:10.1017/S0047279412000232 NHS England. Commissioning. 2024. https://www.england.nhs.uk/commissioning/ Tsevat J, Moriates C. Value-Based Health Care Meets Cost-Effectiveness Analysis. Ann Intern Med . Sep 4 2018;169(5):329-332. doi:10.7326/m18-0342 England NHS. NHS England Specialised Commissioning Service Development Policy . 2017. https://www.england.nhs.uk/wp-content/uploads/2017/09/B0026_NHS-England-Specialised-Commissioning-Service-Development-Policy.pdf Brownson RC, Fielding JE, Maylahn CM. Evidence-Based Public Health: A Fundamental Concept for Public Health Practice. Annual Review of Public Health . 2009;30(Volume 30, 2009):175-201. doi:https://doi.org/10.1146/annurev.publhealth.031308.100134 Kneale D, Rojas-García A, Raine R, Thomas J. The use of evidence in English local public health decision-making: a systematic scoping review. Implementation Science . 2017/04/20 2017;12(1):53. doi:10.1186/s13012-017-0577-9 Gongora-Salazar P, Glogowska M, Fitzpatrick R, Perera R, Tsiachristas A. Commissioning [Integrated] Care in England: An Analysis of the Current Decision Context. Int J Integr Care . Oct-Dec 2022;22(4):3. doi:10.5334/ijic.6693 Scott RJ, Mathie E, Newman HJH, Almack K, Brady L-M. Commissioning and co-production in health and care services in the United Kingdom and Ireland: An exploratory literature review. Health Expectations . 2024;27(3):e14053. doi:https://doi.org/10.1111/hex.14053 Porter A, Mays N, Shaw SE, Rosen R, Smith J. Commissioning healthcare for people with long term conditions: the persistence of relational contracting in England’s NHS quasi-market. BMC Health Services Research . 2013/05/24 2013;13(1):S2. doi:10.1186/1472-6963-13-S1-S2 Shaw SE, Smith JA, Porter A, Rosen R, Mays N. The work of commissioning: a multisite case study of healthcare commissioning in England's NHS. BMJ Open . 2013;3(9):e003341. doi:10.1136/bmjopen-2013-003341 Wye L, Brangan E, Cameron A, Gabbay J, Klein JH, Pope C. Evidence based policy making and the ‘art’ of commissioning – how English healthcare commissioners access and use information and academic research in ‘real life’ decision-making: an empirical qualitative study. BMC Health Services Research . 2015/09/29 2015;15(1):430. doi:10.1186/s12913-015-1091-x Jager A, Wong G, Papoutsi C, Roberts N. The usage of data in NHS primary care commissioning: a realist review. BMC Medicine . 2023/07/03 2023;21(1):236. doi:10.1186/s12916-023-02949-w National Health Service England. Working in partnership with people and communities: Statutory guidance . 2022. https://www.england.nhs.uk/long-read/working-in-partnership-with-people-and-communities-statutory-guidance/ Hart F. Is commissioning the enemy of co-production? Perspect Public Health . Jul 2022;142(4):191-192. doi:10.1177/17579139221103189 Marshall I, McKevitt C, Wang Y, et al. Stroke pathway ? An evidence base for commissioning ? An evidence review for NHS England and NHS Improvement [version 1; peer review: 2 approved with reservations]. NIHR Open Research . 2022;2(43)doi:10.3310/nihropenres.13257.1 Narayanan S, Balamurugan NM, M K, Palas PB. Leveraging Machine Learning Methods for Multiple Disease Prediction using Python ML Libraries and Flask API. 2022:694-701. Monks T, Harper A, Mustafee N. Towards sharing tools and artefacts for reusable simulations in healthcare. Journal of Simulation .1-20. doi:10.1080/17477778.2024.2347882 Laws A, Allen M, Scott J, et al. Modelling the potential clinical benefit of mobile stroke units in England. Zenodo . 2024;doi:https://doi.org/10.5281/zenodo.14132787 Al-Saqqa S, Sawalha S, Abdelnabi H. Agile Software Development: Methodologies and Trends. Int J Interact Mob Technol . 2020;14:246-270. Nickel GC, Wang S, Kwong JCC, Kvedar JC. The case for inclusive co-creation in digital health innovation. npj Digital Medicine . 2024/09/16 2024;7(1):251. doi:10.1038/s41746-024-01256-9 Chen J, Lin X, Cai Y, Huang R, Yang S, Zhang G. A Systematic Review of Mobile Stroke Unit Among Acute Stroke Patients: Time Metrics, Adverse Events, Functional Result and Cost-Effectiveness. Front Neurol . 2022;13:803162. doi:10.3389/fneur.2022.803162 Fatima N, Saqqur M, Hussain MS, Shuaib A. Mobile stroke unit versus standard medical care in the management of patients with acute stroke: A systematic review and meta-analysis. International Journal of Stroke . 2020;15(6):595-608. doi:10.1177/1747493020929964 Harvey N, Holmes CA. Nominal group technique: An effective method for obtaining group consensus. International Journal of Nursing Practice . 2012;18(2):188-194. doi:https://doi.org/10.1111/j.1440-172X.2012.02017.x Maramba I, Chatterjee A, Newman C. Methods of usability testing in the development of eHealth applications: A scoping review. Int J Med Inform . Jun 2019;126:95-104. doi:10.1016/j.ijmedinf.2019.03.018 National Institute for Health and Care Research. Modelling the resource requirements for implementation of mobile stroke units across the National Health Service, their cost-effectiveness, and their effect on equity of access to emergency stroke care. https://fundingawards.nihr.ac.uk/award/NIHR153982 Nvivo . Lumivero; 2020. Braun V, Clarke V. Thematic analysis : a practical guide . 1st edition.. ed. Thousand Oaks : SAGE Publications Ltd.; 2022. Fox S, Brown LJE, Antrobus S, et al. Co-design of a Smartphone App for People Living With Dementia by Applying Agile, Iterative Co-design Principles: Development and Usability Study. JMIR Mhealth Uhealth . 2022/1/14 2022;10(1):e24483. doi:10.2196/24483 Tessarolo F, Nollo G, Conotter V, et al. User-centered co-design and AGILE methodology for developing ambient assisting technologies: Study plan and methodological framework of the CAPTAIN project. 2019:283-286. LaScala-Gruenewald DE, Low NHN, Barry JP, et al. Building on a human-centred, iterative, and agile co-design strategy to facilitate the availability of deep ocean data. ICES Journal of Marine Science . 2022;80(2):347-351. doi:10.1093/icesjms/fsac145 Millard D, Howard Y, Gilbert L, Wills G. Co-Design and Co-Deployment Methodologies for Innovative m-Learning Systems. In: Goh TT, ed. Multiplatform E-Learning Systems and Technologies: Mobile Devices for Ubiquitous ICT-Based Education . IGI Global; 2010:147-163. Kokol P. Agile Software Development in Healthcare: A Synthetic Scoping Review. Applied Sciences . 2022;12(19). doi:10.3390/app12199462 Noorbergen TJ, Adam MTP, Teubner T, Collins CE. Using Co-design in Mobile Health System Development: A Qualitative Study With Experts in Co-design and Mobile Health System Development. JMIR Mhealth Uhealth . 2021/11/10 2021;9(11):e27896. doi:10.2196/27896 Tabeau K, de Mul M, Strating M, et al. The challenges of and solutions for combining cocreation and agile in the development of health information technologies. International Journal of Medical Informatics . 2024/11/01/ 2024;191:105557. doi:https://doi.org/10.1016/j.ijmedinf.2024.105557 Thabrew H, Fleming T, Hetrick S, Merry S. Co-design of eHealth Interventions With Children and Young People. Mini Review. Frontiers in Psychiatry . 2018-October-18 2018;9doi:10.3389/fpsyt.2018.00481 Moseley L, McMeekin P, Price C, et al. Practitioner, patient and public views on the acceptability of Mobile Stroke Units in England and Wales: a mixed methods study. medRxiv . 2024:2024.08. 26.24312612. Braun D, Guston DH. Principal-agent theory and research policy: an introduction. Science and public policy . 2003;30(5):302-308. Pettersen IJ, Nyland K, Robbins G. Public procurement performance and the challenge of service complexity – the case of pre-hospital healthcare. Journal of Public Procurement . 2020;20(4):403-421. doi:10.1108/JOPP-01-2020-0002 Popović M, Kuzmanović M, Gušavac BA. The agency dilemma: information asymmetry in the" principal-agent" problem. Management . 2012;62:15. Dietrich M, Walter S, Ragoschke-Schumm A, et al. Is Prehospital Treatment of Acute Stroke too Expensive An Economic Evaluation Based on the First Trial. Cerebrovascular Diseases . 2014;38(6):457-463. doi:10.1159/000371427 Gyrd-Hansen D, Olsen KR, Bollweg K, Kronborg C, Ebinger M, Audebert HJ. Cost-effectiveness estimate of prehospital thrombolysis: results of the PHANTOM-S study. Neurology . Mar 17 2015;84(11):1090-7. doi:10.1212/wnl.0000000000001366 Kim J, Easton D, Zhao H, et al. Economic evaluation of the Melbourne Mobile Stroke Unit. International Journal of Stroke . 2021;16(4):466-475. doi:10.1177/1747493020929944 Reimer AP, Zafar A, Hustey FM, et al. Cost-Consequence Analysis of Mobile Stroke Units vs. Standard Prehospital Care and Transport. Front Neurol . 2019;10:1422. doi:10.3389/fneur.2019.01422 Anderson R, Hardwick R. Realism and resources: Towards more explanatory economic evaluation. Evaluation (London, England 1995) . 2016;22(3):323-341. doi:10.1177/1356389016652742 Skivington K, Matthews L, Simpson SA, et al. A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. BMJ . 2021;374:n2061. doi:10.1136/bmj.n2061 Yardley L, Ainsworth B, Arden-Close E, Muller I. The person-based approach to enhancing the acceptability and feasibility of interventions. Pilot and Feasibility Studies . 2015/10/26 2015;1(1):37. doi:10.1186/s40814-015-0033-z Moseley L, McMeekin P, Allen M, et al. Co-design of a Mobile Stroke Unit pathway highlights uncertainties and trade-offs for viable system-wide implementation in the English and Welsh NHS. 2024; Additional Declarations No competing interests reported. Cite Share Download PDF Status: Published Journal Publication published 09 May, 2025 Read the published version in npj Digital Medicine → Version 1 posted You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5668150","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Article","associatedPublications":[],"authors":[{"id":407712822,"identity":"51e9be85-352f-43a3-8a88-af0d2051964e","order_by":0,"name":"Lisa Moseley","email":"","orcid":"","institution":"Northumbria University","correspondingAuthor":false,"prefix":"","firstName":"Lisa","middleName":"","lastName":"Moseley","suffix":""},{"id":407712823,"identity":"ffeb2104-f434-4d9b-9e91-5468dce28043","order_by":1,"name":"Anna Laws","email":"","orcid":"","institution":"University of Exeter","correspondingAuthor":false,"prefix":"","firstName":"Anna","middleName":"","lastName":"Laws","suffix":""},{"id":407712824,"identity":"49eb5559-d308-477d-9f30-1b509c84ae73","order_by":2,"name":"Michael Allen","email":"","orcid":"","institution":"University of Exeter","correspondingAuthor":false,"prefix":"","firstName":"Michael","middleName":"","lastName":"Allen","suffix":""},{"id":407712825,"identity":"cce30353-f452-4463-bbb8-60262e68323d","order_by":3,"name":"Gary A Ford","email":"","orcid":"","institution":"Oxford University Hospitals NHS Foundation Trust","correspondingAuthor":false,"prefix":"","firstName":"Gary","middleName":"A","lastName":"Ford","suffix":""},{"id":407712826,"identity":"69a5648d-2d31-4902-97c9-71fbed8143b7","order_by":4,"name":"Martin James","email":"","orcid":"","institution":"Royal Devon and Exeter Hospital","correspondingAuthor":false,"prefix":"","firstName":"Martin","middleName":"","lastName":"James","suffix":""},{"id":407712827,"identity":"0ff2d8d2-07d1-4116-a896-eeed3e3bd6d9","order_by":5,"name":"Stephen McCarthy","email":"","orcid":"","institution":"Northumbria University","correspondingAuthor":false,"prefix":"","firstName":"Stephen","middleName":"","lastName":"McCarthy","suffix":""},{"id":407712828,"identity":"de5f84ee-c1c6-49cd-b9d6-6c056b2c08c7","order_by":6,"name":"Graham McClelland","email":"","orcid":"","institution":"Northumbria University","correspondingAuthor":false,"prefix":"","firstName":"Graham","middleName":"","lastName":"McClelland","suffix":""},{"id":407712829,"identity":"c204a9c7-153c-4cce-a91f-426b36d53a54","order_by":7,"name":"Laura J Park","email":"","orcid":"","institution":"Northumbria University","correspondingAuthor":false,"prefix":"","firstName":"Laura","middleName":"J","lastName":"Park","suffix":""},{"id":407712830,"identity":"720d6590-de85-4b5c-8661-df885ffb62ce","order_by":8,"name":"Kerry Pearn","email":"","orcid":"","institution":"University of Exeter","correspondingAuthor":false,"prefix":"","firstName":"Kerry","middleName":"","lastName":"Pearn","suffix":""},{"id":407712831,"identity":"4c8bbc37-f6c8-4eb4-be4b-d7176fc145e1","order_by":9,"name":"Daniel Phillips","email":"","orcid":"","institution":"East of England Ambulance Service NHS Trust","correspondingAuthor":false,"prefix":"","firstName":"Daniel","middleName":"","lastName":"Phillips","suffix":""},{"id":407712832,"identity":"5a4ff14c-f14a-44c7-937a-bfe407cee1cd","order_by":10,"name":"Christopher Price","email":"","orcid":"","institution":"Newcastle University","correspondingAuthor":false,"prefix":"","firstName":"Christopher","middleName":"","lastName":"Price","suffix":""},{"id":407712833,"identity":"8970da6a-af35-4033-abaa-514e725d10e5","order_by":11,"name":"Lisa Shaw","email":"","orcid":"","institution":"Newcastle University","correspondingAuthor":false,"prefix":"","firstName":"Lisa","middleName":"","lastName":"Shaw","suffix":""},{"id":407712834,"identity":"dd96b69c-47a2-4d4c-8f22-24494ae0b4bb","order_by":12,"name":"Phil White","email":"","orcid":"","institution":"Newcastle University","correspondingAuthor":false,"prefix":"","firstName":"Phil","middleName":"","lastName":"White","suffix":""},{"id":407712835,"identity":"097beb4c-66a8-40fc-977c-c9218447c1a1","order_by":13,"name":"Dave Wilson","email":"","orcid":"","institution":"","correspondingAuthor":false,"prefix":"","firstName":"Dave","middleName":"","lastName":"Wilson","suffix":""},{"id":407712836,"identity":"b66425bd-152e-482e-9158-16f7391d3c87","order_by":14,"name":"Peter McMeekin","email":"","orcid":"","institution":"Northumbria University","correspondingAuthor":false,"prefix":"","firstName":"Peter","middleName":"","lastName":"McMeekin","suffix":""},{"id":407712837,"identity":"a57281f0-fd19-4847-a375-662979b1a0fe","order_by":15,"name":"Jason Scott","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABG0lEQVRIie2RsWrDMBCGrwg0qXiVCMSvIGNoyNJnUQm0Q2MwZDEEXEEgWQJe1bfIlKwJhnjRA6QUSoyhXdqpa4dYhhK3UejaQd/wI+70wZ0E4HD8Qy5k67w3yQFwk7/bpwpGyNyjfypHMML0qFiG+AbNirKKIfU9P9uO4+Ql7dH72yqJr8GbrTFTlsHmwzBUkAePU4SflR7RvopWgeYDoFpgtrDtMsQdAmvBt7VyORWU76IlkxwB7ACzvUXJ3l5rJW2UUUt5AP+cosRVraBGQS0lB24U22DqPQwJN7sMwg7Rgi30xyqQvCCBvpn0LesH2V1ZkaR+MbQpP0kiPF5Ey1J+jbvdIt88zS2KNMlPG0DOfaRvKzocDofjBwcRtV1nKspf3AAAAABJRU5ErkJggg==","orcid":"","institution":"Northumbria University","correspondingAuthor":true,"prefix":"","firstName":"Jason","middleName":"","lastName":"Scott","suffix":""}],"badges":[],"createdAt":"2024-12-18 09:38:26","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5668150/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5668150/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1038/s41746-025-01691-2","type":"published","date":"2025-05-09T15:57:42+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":75087638,"identity":"804c633e-5f2f-464e-8e92-c7653d596a8d","added_by":"auto","created_at":"2025-01-30 10:23:02","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":44455,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eThematic structure of the findings\u003c/em\u003e\u003c/p\u003e","description":"","filename":"1.png","url":"https://assets-eu.researchsquare.com/files/rs-5668150/v1/22577bf207018464ea89b94d.png"},{"id":75086754,"identity":"62767821-32b5-49cd-9e9a-4ce30559c006","added_by":"auto","created_at":"2025-01-30 10:15:02","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":39845,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eProcess for modifying pathway parameters within the web application for the standard pathway. Hovering the cursor over the question marks repeated the parameter headings. Additional parameters were included for MSU inputs (not pictured here). Not all model parameters were modifiable, such as distance and travel times. Travel times between all 32,843 Lower Super Output Areas in England and hospitals were calculated using Open Street Map data and Routino routing software, with travel times calibrated against Google Maps.\u003c/em\u003e\u003c/p\u003e","description":"","filename":"2.png","url":"https://assets-eu.researchsquare.com/files/rs-5668150/v1/b014b40177a681c1ad1ca0d1.png"},{"id":75087639,"identity":"0ee775ac-7bce-4396-8fcc-58ac58681fda","added_by":"auto","created_at":"2025-01-30 10:23:02","extension":"png","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":122712,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003ePresentation of potential locations for modelling MSU outcomes. Locations could be individually (de)selected by (un)ticking boxes in the right-hand side column.\u003c/em\u003e\u003c/p\u003e","description":"","filename":"3.png","url":"https://assets-eu.researchsquare.com/files/rs-5668150/v1/ddbc18366de1158e95c4e0a7.png"},{"id":75086761,"identity":"64fa408c-9b47-4b18-9564-8100128f60b6","added_by":"auto","created_at":"2025-01-30 10:15:02","extension":"png","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":319155,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eWeb application interface including maps and graphs demonstrating outcomes associated with implementing MSUs. All maps include zoom functionality.\u003c/em\u003e\u003c/p\u003e","description":"","filename":"4.png","url":"https://assets-eu.researchsquare.com/files/rs-5668150/v1/a83377eddbf3beeaeffb1047.png"},{"id":75087642,"identity":"f166e2e6-eebc-4f3e-9995-7df003d47636","added_by":"auto","created_at":"2025-01-30 10:23:02","extension":"png","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":368548,"visible":true,"origin":"","legend":"\u003cp\u003e\u003cem\u003eUpdated web application maps post-interviews. All maps include major arterial roads, with the left map presenting usual care, the middle map presenting clinical benefit over usual care using added utility, and the right map presenting population density. Population density per lower super output area (LSOA) is calculated by combining data from the Office for National Statistics on population per LSOA in 2018 and on the area of LSOAs in 2011. The default maximum population density of 100 is chosen to show the locations of large towns and city boundaries. All maps include zoom functionality.\u003c/em\u003e\u003c/p\u003e","description":"","filename":"5.png","url":"https://assets-eu.researchsquare.com/files/rs-5668150/v1/4e5760518c5167bae0a0a31e.png"},{"id":82537518,"identity":"0d452a8f-714f-40c7-bd8e-c9a94daaadda","added_by":"auto","created_at":"2025-05-12 16:07:54","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":1434605,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5668150/v1/353b1471-02b0-42b2-a77d-9f958a2cf32c.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"Usability testing a digital web application for evidence-based commissioning decisions in the context of implementing Mobile Stroke Units in England","fulltext":[{"header":"Introduction","content":"\u003cp\u003eStroke is one of the leading causes of death and disability worldwide\u003csup\u003e\u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e1\u003c/span\u003e\u003c/sup\u003e. Strokes are caused either by a clot, or a bleed, in the brain and are characterised by rapid onset of symptoms, which range in severity, and can impact all areas of function including mobility and speech\u003csup\u003e\u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2\u003c/span\u003e\u003c/sup\u003e. The long-term effects following a stroke can be severe, requiring ongoing health treatment as well as social care support\u003csup\u003e\u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e3\u003c/span\u003e\u003c/sup\u003e, and it is estimated that the number of strokes in the UK will rise by 59% by 2035, from the 2020 level of approximately 100,000 a year\u003csup\u003e\u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e4\u003c/span\u003e\u003c/sup\u003e. This makes stroke an important health concern due to the individual impact, alongside the economic burden. Treatment for ischaemic strokes, where the stroke is caused by a clot, can reduce, and even fully diminish, the impacts of a stroke. However, this treatment is time sensitive, making rapid diagnosis and intervention imperative\u003csup\u003e\u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e5\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eMobile Stroke Units (MSUs) have been growing in popularity as a means to diagnose and begin treatment of ischemic strokes, in pre-hospital settings, leading to the European Stroke Organisation (ESO) to recommend consideration of MSU deployment\u003csup\u003e\u003cspan citationid=\"CR6\" class=\"CitationRef\"\u003e6\u003c/span\u003e\u003c/sup\u003e. MSUs are specialist ambulances that have a computed tomography (CT) or computed tomography angiography (CT-A) scanner, capable of scanning the head with the latter using contrast to visualise blood vessels, and MSUs also have stroke specialist staffing and point of care testing\u003csup\u003e\u003cspan additionalcitationids=\"CR8 CR9 CR10 CR11 CR12 CR13\" citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e\u003c/sup\u003e. MSUs have focused on the treatment of ischaemic stroke, and have shown that MSUs can speed up to time to treatment with intravenous thrombolysis (IVT), a medication used in some cases to dissolve clots\u003csup\u003e\u003cspan additionalcitationids=\"CR10 CR11 CR12\" citationid=\"CR9\" class=\"CitationRef\"\u003e9\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e13\u003c/span\u003e\u003c/sup\u003e. An additional treatment, mechanical thrombectomy (MT), is used for some patients where a clot is occluding a large vessel, known as a large vessel occlusion (LVO). MT is performed by hospital-based interventional neuroradiologists who manually retrieve the clot to restore blood flow\u003csup\u003e\u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e15\u003c/span\u003e\u003c/sup\u003e. In the United Kingdom (UK), like most countries, MT is only available at certain specialist centres, comprehensive stroke centres (CSC), usually based in urban areas. However, patients with suspected stroke are currently often transported by the ambulance service to their nearest hospital with standard stroke services available, which can be an acute stroke unit (ASU) without MT capabilities, and thus requires an additional transfer of patients with LVO to CSCs. While initial trials of MSUs focused on the increased speed in which IVT was given, more recent evidence has found there is also the potential for MSUs to speed up access to MT, although this has not been consistently evidenced\u003csup\u003e\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e, \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e, \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e\u003c/sup\u003e. Despite the evidence favouring MSU in certain settings and healthcare systems, they are currently not commissioned in the NHS.\u003c/p\u003e \u003cp\u003eCommissioning has been a part of the National Health Service (NHS) since its inception in 1948, with services being provided by Local Authorities, hospitals and independent practitioners (i.e. General Practitioners), each having to submit estimated costs, but largely managing their own budgets locally\u003csup\u003e\u003cspan citationid=\"CR16\" class=\"CitationRef\"\u003e16\u003c/span\u003e\u003c/sup\u003e. In the 1990s the NHS was changed to a quasi-market to encourage competition between providers, aiming to reduce costs and improve efficiencies\u003csup\u003e\u003cspan citationid=\"CR17\" class=\"CitationRef\"\u003e17\u003c/span\u003e\u003c/sup\u003e. Arguments for the quasi-marketisation championed the purchasing of services being separated from those providing the service, with the view that this would improve equitable access to services universally, without influence from those providing services, particularly clinicians\u003csup\u003e\u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e18\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eCommissioning of services within the NHS remains complex, with decisions being made centrally and by local commissioners\u003csup\u003e\u003cspan citationid=\"CR19\" class=\"CitationRef\"\u003e19\u003c/span\u003e\u003c/sup\u003e. Increasingly there is also tension between cost-effectiveness commissioning compared to values-based commissioning which, while similar, differ in emphasis; the former focuses on societal benefits and the latter is intended to consider individual patient perspectives\u003csup\u003e\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e\u003c/sup\u003e. Evidence-based policy is key to all service planning within the NHS and is a key consideration in commissioning decisions\u003csup\u003e\u003cspan citationid=\"CR21\" class=\"CitationRef\"\u003e21\u003c/span\u003e\u003c/sup\u003e. The potential benefits of evidence-based commissioning decisions include a greater chance of a service being successfully implemented as well as more efficient use of public funds within the health service\u003csup\u003e\u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e22\u003c/span\u003e, \u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e23\u003c/span\u003e\u003c/sup\u003e. Despite the benefits of using data in commissioning decisions the reality of how commissioning decisions are made is much more complex.\u003csup\u003e\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e, \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e\u003c/sup\u003e Instead of being a transactional approach, commissioning involves close relational interaction between commissioners, providers of services and often patients and the public\u003csup\u003e\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eNumerous barriers exist to commissioners using evidence in their decision-making, including research articles being behind paywalls, research being clinically focused rather than commissioner focused, lack of localised knowledge or context in research, and the effort required to understand academic research\u003csup\u003e\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e\u003c/sup\u003e. A number of facilitators to data-informed commissioning have been identified including making data presented manipulatable, showing the data meaning at local levels, and making the data easy to interpret\u003csup\u003e\u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e\u003c/sup\u003e. Alongside these complexities in the commissioning of services there is now an emphasis on the co-production of health and care services, requiring services to be informed by relevant stakeholders\u003csup\u003e\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e\u003c/sup\u003e. However the use of co-production has posed further challenges to commissioners with difficulties involving stakeholders in an already complex commissioning process, the work required for co-production being unfeasible within commissioner time constraints, and the inflexibility of the commissioning processes to respond to additional input\u003csup\u003e\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e, \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eTo address these challenges, researchers are increasingly required to develop new approaches to sharing evidence created through research that can directly inform commissioning decisions, such as more accessible data summaries\u003csup\u003e\u003cspan citationid=\"CR32\" class=\"CitationRef\"\u003e32\u003c/span\u003e\u003c/sup\u003e. Web applications, as one such approach, are increasingly being used to provide end-user access to complex models (e.g. see Narayanan et al\u003csup\u003e\u003cspan citationid=\"CR33\" class=\"CitationRef\"\u003e33\u003c/span\u003e\u003c/sup\u003e) and have key advantages for making models available. These advantages include reduced barriers to end-user engagement, no specialist software or hardware needs for the end-user, rapid prototyping and development, the ability to work across multiple platforms (e.g. PC, tablet, phone), centralised updates/maintenance, immediate deployment of fixes or enhanced functionality, and scalable infrastructure which can adapt to demand. Within healthcare informatics, there is now an increased emphasis on sharing the underlying code of models\u003csup\u003e\u003cspan citationid=\"CR34\" class=\"CitationRef\"\u003e34\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eThis paper describes the approach taken to developing a web application for modelling the effectiveness of MSUs in England and examines the usability of the web application for informing commissioning decisions.\u003c/p\u003e"},{"header":"Methods","content":"\u003cp\u003e\u003cstrong\u003eDigital web application development\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe built models of clinical benefit of MSUs\u003csup\u003e35\u003c/sup\u003e, and we aimed to make those models available to potential end-users \u0026ndash; including stakeholders involved in commissioning processes \u0026ndash; as a digital web application. The chosen method of web application development was to follow an agile process\u003csup\u003e36\u003c/sup\u003e where iterative development, with user feedback, is used rather than a highly developed starting specification. User feedback was incorporated throughout the modelling and web application development via a process of qualitative co-design with stakeholders, recognised to be of importance for helping to ensure clinical relevance\u003csup\u003e37\u003c/sup\u003e. Model code development took place on GitHub (https://github.com/stroke-modelling/muster_workshop_web_app_1) and Streamlit (https://streamlit.io/) was chosen as the deployment platform, as this provides a platform designed for modellers and data scientists to easily implement web applications. Initial ranges and default values for the model parameters (which the end-user can change) were reached after examining published information on timings from MSU clinical trials (see Chen et al\u003csup\u003e38\u003c/sup\u003e and Fatima et al\u003csup\u003e39\u003c/sup\u003e for reviews). The choice of user-controlled input parameters and ranges for their values, and preferred model outputs, were then discussed with stakeholders using Nominal Group Technique\u003csup\u003e40\u003c/sup\u003e to examine their appropriateness and develop consensus. Some examples of the web application interface are provided in the findings.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eStudy design\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe study tested the usability of the developed web application using qualitative Think Aloud methodology\u003csup\u003e41\u003c/sup\u003e. Think Aloud is a common methodology in usability testing and asks participants to use the web application while talking out loud about what actions they are performing and why, and to provide thoughts on usability. We expanded the usability testing to examine how the web application could inform commissioning decisions in the context of deploying MSUs in England.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eParticipants, recruitment and sampling\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eParticipants were recruited via purposive, opportunistic and snowball sampling, as part of the wider research project in relation to MSUs\u003csup\u003e42\u003c/sup\u003e Stakeholders consisted of stroke clinicians (doctors and nurses), ambulance service staff, decision-makers involved in commissioning processes, and those with lived experience of stroke care (either as a stroke patient or supporting someone impacted by stroke). The research team used existing networks to recruit staff participants and the recruitment of patient and public involvement representatives (PPI) was supported by a national charity, the Stroke Association. PPI representatives who had expressed an interest to the Stroke Association in being part of research projects were contacted by the charity with an overview of the study and individuals responded if they wished to take part. The researchers then purposively sampled those who had responded, to ensure diverse representation, based on geographic location, age and type of stroke.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData collection\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eAll data were collected online via Microsoft Teams between September and October 2024 using interviews that consisted of a free-form Think-Aloud component, followed by semi-structured discussion. All discussions were audio recorded and transcribed verbatim. Prior to data collection the research team developed a topic guide (supplementary material) to cover key aspects of usability based on the major elements of the web application. These were used to facilitate reflection and discussion amongst participants during the Think Aloud component, particularly during periods of silence, and also to guide discussion during the semi-structured component of the interview. Prompts included: what they liked/disliked about the app; ease of use; any additional information they would like to see; how this might be used by commissioning and in wider practice and how the information influenced views on MSUs. Data collection was conducted by JS and LM.\u003c/p\u003e\n\u003cp\u003eParticipants were given a guided tutorial of the web application and the various functions by JS. Participants were then asked to access the web application themselves and share their screen with the researchers while using it. Both researchers observed how users engaged with the web application, whether certain parts of the application were used more than others, and areas that participants required additional guidance to utilise. To ensure participants were not digitally excluded from participating, they were also given the option for the interviewer to use the web application with the participant telling the interviewer what to do and giving their thoughts.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eData analysis\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eFollowing data collection all interviews were transcribed verbatim and analysed using Nvivo 12 (Lumivero)\u003csup\u003e43\u003c/sup\u003e by LM. To consider data from participants\u0026rsquo; actions that were seen during the usability rather than verbally recorded, LM and JS met after each interview to compare observations on the participants\u0026rsquo; use of the web application. These observations were incorporated into the analysis. Thematic analysis\u003csup\u003e44\u003c/sup\u003e was used by LM to create initial themes exploring commonalities in the data as well as contrasts, with additional input from JS. The themes were further scrutinised and refined with input from the wider research team resulting in the finalised themes presented.\u003c/p\u003e"},{"header":"Findings","content":"\u003cp\u003eSixteen people participated in usability testing, consisting of patient and public involvement representatives (n=6, 38%), stroke or critical care physicians (n=6, 38%), and ambulance service staff with front-line clinical and senior management expertise (n=4, 25%). Interviews lasted between 38 minutes and 89 minutes (mean = 58 minutes).\u003c/p\u003e\n\u003cp\u003eWe identified where users were able to successfully navigate and operate the web application, as well as areas that its usability could be improved. Together, these form the first theme: Usability of the web application. Central to this theme were the importance of clear user instructions and participants\u0026rsquo; preferences for data presentation. The second theme developed, Informing commissioning and implementation decision-making, examines how usability testing of the web application provided insight to the complexity of commissioning and implementation of MSUs, and allowed for a deeper exploration of how various stakeholders considered how the web application could be used. Related to this was a third theme, Informing views on intervention viability, that captures how usability testing allowed a further examination of whether MSUs could or should be commissioned. Finally, we developed a single cross-cutting theme that interacted with and was central to the second and third major themes; Trustworthy and robust data. This theme, particularly focusing on the need to include health economics data on cost-effectiveness of MSUs, reinforces the need for commissioning and implementation decisions to be data-driven and was seen as a requirement for participants to provide conclusive views on whether MSUs should be implemented. Figure 1 presents an overview of the thematic structure of the findings. Images of the web application are included to contextualise findings. Outcomes shown in images are for illustrative purposes only and not necessarily representative of the final models.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eUsability of the web application\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe overall view of the web application was positive, with most participants quickly able to use it themselves after a brief guided tutorial by researchers. However, participants highlighted that should the web application be distributed widely, it would not be feasible to give each individual a tutorial, and as such would benefit from a recorded video tutorial, as well as written instructions with screenshots. This concern was discussed by a stroke consultant:\u003c/p\u003e\n\u003cp\u003e\u0026ldquo;You are here, you explained it to me, and I found it very easy. So the question is, somebody who has not seen this and is seeing them for the first time, how do you make it user-friendly for them?...People differ, some people like that written instruction\u0026hellip;with some images attached\u0026hellip;Others would like the video\u0026hellip;I think you need to cater for those two groups\u0026hellip;\u0026rdquo; (Participant 13, stroke physician).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eWhen using the web application, participants particularly found the selection of model parameters intuitive, where they were able to easily modify some time parameters used in the modelling (see figure 2). This allowed participants to tailor the models, for instance by amending the ambulance response time using publicly available data on response times to stroke calls within their region. Participants felt that the modifiable parameters needed a brief explanation of how the parameter interacted with the pathway and evidence-based justification for default values, such as whether previous stroke data had been used to obtain average values. These observations were explained by a stroke physician and an ambulance service staff member:\u003c/p\u003e\n\u003cp\u003e\u0026ldquo;\u0026hellip;where there\u0026rsquo;s this little question mark [above each parameter]\u0026hellip;that\u0026rsquo;ll be nice if the question mark said, you know, the clarification of what it meant\u0026hellip;\u0026rdquo; (participant 9, stroke or critical physician)\u003c/p\u003e\n\u003cp\u003e\u0026ldquo;Is that just a random figure or is that based on data? Is that currently where we think we are [in relation to ambulance response times]?\u0026rdquo; (Participant 2, ambulance service staff)\u003c/p\u003e\n\u003cp\u003eObserving participant interactions with the web application revealed that they sometimes struggled to select the base location for an MSU, having to manually deselect default sites from a list of 128 locations. Despite being shown how to do this as part of the tutorial, some participants were unsure which boxes to (un)tick, and it was time-consuming for people to deselect multiple default options (figure 3).\u0026nbsp;\u003c/p\u003e\n\u003cp\u003eWithin the web application, data were presented in multiple formats including maps, graphs and tables (figure 4). Observing participant interactions with the web application, it was apparent that maps were the most accessible format, with participants paying little attention to the graphs. Participants also did not interact with the data tables, consisting of 51 columns and representing all underlying data for the models, but they were deemed to be important (see Trustworthy and rigorous data theme).\u003c/p\u003e\n\u003cp\u003e\u0026ldquo;I did listen to what you described about the graphs on the right-hand side. But based on [the maps], you can predict what impact [MSUs] will have, I think. And that\u0026rsquo;s what [commissioners] need. So [commissioners] need something that\u0026rsquo;s visual. [Commissioners] need something that has some numbers associated with it, but also, because of the way that you\u0026rsquo;ve built this system, and I have to say, huge congratulations because it\u0026rsquo;s pretty intuitive, even with a very small amount of demonstration, you can get people to understand what their region looks like.\u0026rdquo; (Participant 14, stroke or critical care physician).\u003c/p\u003e\n\u003cp\u003eWhilst the maps received most attention, participants often struggled to understand the concept of utility shift \u0026ndash; a change in quality of life measured through health utility scores \u0026ndash; which was used as the default outcome comparison between usual care. Even with explanation of utility shift by the researchers, participants found it difficult to translate into the actual benefit to patients when using the web application. While some stroke physicians were more familiar with the concept of utility shift this was not universal across all stroke physicians, restricting the usability of the web application to understand potential impact on patient outcomes. Concerns about utility shift, and how understandable this would be, were summed up by an ambulance service staff member:\u003c/p\u003e\n\u003cp\u003e\u0026ldquo;I think\u0026hellip;the biggest unanswered question as to what\u0026hellip;the colours are lovely, and they\u0026rsquo;re dramatic and it makes you think, \u0026ldquo;Oh, wow, that\u0026rsquo;s a big difference.\u0026rdquo; But what is the difference?...I think I get my head around [utility shift]. I still think that\u0026rsquo;s- as someone lay, or whoever, or not even necessarily lay, that\u0026rsquo;s going to be their main question\u0026rdquo; (Participant 4, ambulance service staff).\u003c/p\u003e\n\u003cp\u003eDespite the concerns regarding the understanding of utility shift, as represented by colours on a map, participants were very positive about the use of the coloured map to visually represent the data quickly as well as being more thought provoking as explained by a stroke physician, \u0026ldquo;it\u0026rsquo;s a really nice way of showing it\u0026hellip;It makes it visibly more thought-provoking\u0026rdquo; (Participant 8, stroke or critical care physician).\u003c/p\u003e\n\u003cp\u003eParticipants were also able to provide a number of helpful suggestions about changes to the maps that could enhance the usability of the web app. Suggestions included mapping on road systems which would help users orientate more quickly as well as consider travel infrastructure in the area and including the locations of acute stroke units s on the MSU map, as this was currently only shown on the usual care map.\u003c/p\u003e\n\u003cp\u003eParticipants felt that, as MSUs would likely be managed by the ambulance service, having the locations of ambulance stations available would be beneficial, particularly when zooming into specific localities or regions. Lastly, participants felt that a number of selectable \u0026lsquo;overlays\u0026rsquo; would be helpful which could show population density and demographics. These were then included in an update to the web application (figure 5).\u003c/p\u003e\n\u003cp\u003e\u0026ldquo;\u0026hellip;if you could integrate that with population density, that might be helpful\u0026hellip;\u0026nbsp;I suppose the older population, people over the age of 50 or 60, adding that as part of your population density map, accepting that stroke is commoner in later life\u0026hellip;absolutely [include deprivation]\u0026hellip;with deprivation, of course, you\u0026rsquo;re going to see that, in terms of scale and numbers, more in urban areas than in more rural areas\u0026hellip;\u0026rdquo; (Participant 2, ambulance service staff).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eInforming commissioning and implementation decision-making\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eUsability testing of the web application provided the opportunity to examine commissioning and implementation of MSUs, including how the web application could be utilised. As a result of the usability testing, we identified that commissioning of MSUs is unlikely to be a straightforward technical, data-informed process. Instead, it would be a complex socio-technical process based on multiple priorities. This is reflected by a participant with experience of working with commissioners, who felt that commissioners would also consider other healthcare conditions.\u003c/p\u003e\n\u003cp\u003e\u0026ldquo;I, honestly, can\u0026rsquo;t see commissioners using this [web application]. I think it\u0026rsquo;s great for somebody like me, that\u0026rsquo;s overseeing huge projects across a regional footprint or, indeed, at system level. So I think it\u0026rsquo;s great for the providers, it\u0026rsquo;s great for the network, and you could say it would be great on an urgent and emergency care element as well\u0026hellip;\u0026nbsp;As far as our commissioners\u0026hellip; certainly, at the moment, there is so little emphasis on stroke anyway that just keeping stroke on the radar at all is tricky\u0026rdquo; (Participant 5, ambulance service staff).\u003c/p\u003e\n\u003cp\u003eOther participants disagreed as to whether commissioners would use the web application directly to inform commissioning decisions. In the cases where it was thought they would not use the web application, it would still support commissioning decisions by being able to provide the evidence for a business case put forward by services or to justify why MSUs should not be commissioned. This contention between participant views reflects the uncertainty that exists in commissioning a new innovation, including knowing who has responsibility for obtaining and examining evidence. This represents the social processes involved in commissioning, where a new innovation would likely then require a local champion.\u003c/p\u003e\n\u003cp\u003e\u0026ldquo;\u0026hellip;I think commissioning at a national level would use this and appreciate it. I don\u0026rsquo;t suspect they get this from many other things that we ask to be commissioned\u0026rdquo; (Participant 8, stroke or critical care physician).\u003c/p\u003e\n\u003cp\u003e\u0026ldquo;\u0026hellip;I think that, for an ambulance-based resource, it would probably be for us to make the case [for MSUs], but, you know, share the data, and give the Integrated Care Boards, the commissioning bodies, access to view this to support the conversation so they could see for themselves almost the benefits from it\u0026hellip;\u0026rdquo; (Participant 6, ambulance service staff).\u003c/p\u003e\n\u003cp\u003eParticipants also discussed how the wider healthcare economy would contribute to MSU commissioning decisions. For instance, implementation of MSUs may result in some acute stroke units receiving fewer patients and thus make their own services financially unviable due to a loss of the tariff they currently receive for administering IVT. There were also conflicting views amongst participants regarding national or regional commissioning of MSUs. Some participants believed MSUs would need to be part of either a new or existing specialised commissioning pathway, such as the national commissioning of mechanical thrombectomy, whereas others thought MSUs would be commissioned as part of standard block commissioning contracts with Integrated Care Boards in partnership with local ambulance services. However, if regionally commissioned there were further concerns that ambulance services would have to find capital from within their existing budget to fund MSUs, but would not benefit from the gains, and implementation of MSUs. These concerns further highlight the social, particularly the political, challenges that were identified as a result of using the web application.\u003c/p\u003e\n\u003cp\u003e\u0026ldquo;you\u0026rsquo;d get the primary stroke centres\u0026hellip;they get a tariff don\u0026rsquo;t they for thrombolysing somebody in the primary stroke centre [ASU]. So they\u0026rsquo;d lose that and they\u0026rsquo;d lose some of their metrics for hyperacute stroke management, because in essence, you\u0026rsquo;re bypassing them to go straight to the CSC. So I think your primary stroke centres might not like [MSUs being commissioned]\u0026rdquo; (Participant 9, stroke or critical care physician)\u003c/p\u003e\n\u003cp\u003eAlongside the potential for the web application to inform our understanding of commissioning decisions, either directly or indirectly, the web application also helped to develop an understanding of how it could be used operationally to support a commissioned MSU, particularly in the ambulance service. A critical care consultant and an ambulance service staff member both described how they felt the web application could be used as the basis for informing operational decisions once (and if) MSUs were commissioned. Participants recognised that this would likely require further modifications to the web application, particularly by integrating live ambulance service data on response times and predictive geographic models of stroke incidence.\u003c/p\u003e\n\u003cp\u003e\u0026ldquo;\u0026hellip;I suppose the beauty of having big data like this and a very visual representation is, when you pick a particular area\u0026hellip;to put the origin of the call on this\u0026hellip;and if the [origin of the call is] in the correct colour [showing benefit from MSU attendance]\u0026hellip;you go or you don\u0026rsquo;t go.\u0026rdquo; (Participant 14, stroke or critical care physician)\u003c/p\u003e\n\u003cp\u003e\u0026ldquo;\u0026hellip;it probably would have\u0026hellip;benefit\u0026hellip;in terms of looking to plan the shift, plan the day, actually looking where we\u0026rsquo;ve historically seen higher instances of stroke and those that have subsequently turned out as a confirmed diagnosis. So for predicting around where to position the vehicle on a shift-by-shift basis\u0026hellip;\u0026rdquo;(Participant 6, ambulance service staff)\u003c/p\u003e\n\u003cp\u003eIn discussions with participants who had lived experience of stroke, there were differing views on whether the web application would be useful, or even be used, by the general public. Of the participants who did feel the web application might be used, they explained it was because those people would likely have an interest in stroke care and developing treatments, as described by participant 12 who led a regional group for stroke survivors, \u0026ldquo;\u0026hellip;Our group would be [interested in the web application]. They\u0026rsquo;re very open and want to learn more.\u0026rdquo;\u0026nbsp;However, it was also highlighted that the data, and visual representation from the web application would still be of benefit to the public in educating, and gaining public support for, MSUs. This potential use was described by a person with lived experience of stroke and a critical care consultant:\u003c/p\u003e\n\u003cp\u003e\u0026ldquo;\u0026hellip;maybe if it was London and they were told this won\u0026rsquo;t- It might actually worsen outcomes, then people might not be for it, or as interested. But\u0026hellip;those regions where it would help\u0026hellip; I think, yeah, people would definitely be interested.\u0026rdquo; (Participant 11, patients and public involvement representative)\u003c/p\u003e\n\u003cp\u003e\u0026ldquo;\u0026hellip;if you\u0026rsquo;re going to commission anything, you have to get the public on board. You have to get people to understand what they\u0026rsquo;re going to get out of this, not in terms of clinical outcomes, not in terms of value for money, but actually, what does this mean for me tomorrow? And the patient public voice is really important\u0026hellip;it\u0026rsquo;s a key aspect of stakeholder engagement.\u0026rdquo; (Participant 14, stroke or critical care physician)\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eInforming views on intervention viability\u0026nbsp;\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eOnce participants had used the web application and explored some of the early modelling data presented within it, they then provided their views on whether MSUs would be viable, and thus providing an additional use for the web application. In doing so, participants discussed how the maps showed relatively minimal gains to patient outcomes, with consideration to how many patients an MSU could realistically see in one day, and how many of those patients would be having an ischemic stroke and then be eligible for treatment. Alongside this, participants highlighted that the most gains appeared to be for patients with a large vessel occlusion (LVO), where the LVO would be recognised on the scan in the MSU, and patients could be directly transported to a comprehensive stroke centre (CSC) that is able to deliver mechanical thrombectomy, and thus avoiding secondary transfers. However, one member of the ambulance service explained this may not be an accepted approach by the CSCs. Specifically, it was unclear whether the CSCs would be able to review MSU scans obtained using the onboard Computer Tomography Angiography (CTA), and if they would be of sufficient technical quality for a direct admission decision to be made.\u003c/p\u003e\n\u003cp\u003e\u0026ldquo;The issue\u0026hellip;is\u0026hellip;the\u0026hellip; [CTA] scanner has got quite a narrow hole, it doesn\u0026rsquo;t do the neck vessels. So, the mechanical thrombectomy centres [CSCs] won\u0026rsquo;t accept those patients because they would have to repeat the CTA, and then they find they can\u0026rsquo;t get past the carotid [required to perform MT]\u0026rdquo; (Participant 4, ambulance service staff)\u003c/p\u003e\n\u003cp\u003eAlongside this, participants also highlighted that the maps showed the most gains away from CSCs, which are based in large urban centres, and appeared to make the most improvement in rural areas where travel times to a hospital are the longest. This, again, raised the question of how efficient MSUs would be in these areas, given the relatively low population levels, but participants also recognised that improving access in these areas was important to reduce inequity in access to stroke care. Further, participants discussed the relative levels of poverty in some of these areas, such as some coastal communities, alongside higher age demographics in rural areas, both of which increase the risk of stroke. The difficulty in making this assessment was summed up by a critical care consultant, who using hypothetical numbers:\u003c/p\u003e\n\u003cp\u003e\u0026ldquo;I don\u0026rsquo;t know the genuine numbers\u0026hellip;but if you look at the tip of Cornwall for instance, so beyond Truro, should we place an MSU down there at the cost of \u0026pound;500,000 that will be used five times a week? Or should we place two MSUs in the middle of Plymouth that will cost \u0026pound;1m in total but be used 20 times a week? The former gives you equity of access to the MSU. The latter gives you volume of patients and the number needed to treat for mechanical thrombectomy\u0026rdquo; (Participant 14, stroke or critical care physician).\u003c/p\u003e\n\u003cp\u003eLastly, participants highlighted that MSUs were unlikely to provide the same level of benefit as investing money into MT services and widening the availability. However, they recognised that the main issue with increasing MT services was that there were not enough trained interventional neuroradiologists and therefore a like-for-like comparison on benefit was not possible and as expansion of MT services was not immediately available that MSUs could go some way to improving stroke care. This was explained by a stroke physician:\u003c/p\u003e\n\u003cp\u003e\u0026ldquo;\u0026hellip;I think if you\u0026rsquo;re realistic there\u0026rsquo;s no point commissioning more thrombectomy centres because we can\u0026rsquo;t staff the ones we\u0026rsquo;ve got. There\u0026rsquo;s no point pumping money into thrombectomy centres\u0026hellip;We can\u0026rsquo;t train or recruit the specialists we need to deliver at thrombectomy centres. I don\u0026rsquo;t think throwing money at thrombectomy centres helps.\u0026rdquo; (Participant 8, stroke or critical care physician)\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eTrustworthy and robust data\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThroughout all of the interviews there were reoccurring discussions about the need for health economics data. Participants felt strongly that incorporating the health economics into the web application would make it more useable in terms of informing their views on the viability of MSUs. They felt this would assist them in understanding not just the potential gains to individual patient outcomes, but also understanding the financial implications, which would make the web application even more useful for informing both commissioning and operational decisions regarding MSU implementation. The need for this was discussed by an ambulance service staff member:\u003c/p\u003e\n\u003cp\u003e\u0026ldquo;I think the health economic bit would be important and an understanding of the numbers of patients that you could expect an MSU to attend in a 24-hour period or a 12-hour period, etc.\u0026rdquo; (Participant 2, ambulance service staff)\u003c/p\u003e\n\u003cp\u003eThe inclusion of health economics data was also thought to increase the chances of the web application being used directly by commissioners as this was seen to be a key element of their decision making. Without health economic data, stakeholders involved in commissioning decisions would not have all the information they needed. The emphasis on health economic data was explained by a patient and public involvement representative:\u003c/p\u003e\n\u003cp\u003e\u0026ldquo;That is going to be really powerful, once that\u0026rsquo;s in [the health economics data], in terms of commissioning\u0026rdquo; (Participant 17, patient and public involvement representative)\u003c/p\u003e\n\u003cp\u003eLastly the health economics data was highlighted as essential to understanding how commissioning decisions would be informed by debates surrounding how best to improve equitable access to stroke care through implementing MSUs. For example, in terms of rural and urban inequity it was felt that MSUs would need to show they would be economically viable, by seeing enough patients, and that this would likely influence any decisions made about the extent to which MSUs could be used to reduce inequities in access to stroke care.\u003c/p\u003e\n\u003cp\u003e\u0026ldquo;The health economic benefit of that, both the number of individuals who can help, but also the utility\u0026hellip;Is the balance not that you just have to provide this to the most people you possibly can and accept there are going to be areas you cannot cover well?\u0026rdquo; (Participant 8, stroke or critical care physician)\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eThis study presents empirical evidence for how a digital web application can be used to both inform evidence-based commissioning of innovations in the context of acute stroke care and to explore the complexity of commissioning decisions amongst stakeholders in the context of usability testing. Some design elements were identified as requiring changes, the majority of which were specific to the implementation of MSUs in England, highlighting the importance of incorporating qualitative co-design into an agile development process for digital web applications to ensure that they are fit for purpose for all stakeholders involved in commissioning processes.\u003c/p\u003e \u003cp\u003eMore specifically, our findings emphasise that digital web applications which convey complicated information require, at minimum, brief training and tooltips to guide the user. The use of co-design in healthcare\u003csup\u003e\u003cspan citationid=\"CR30\" class=\"CitationRef\"\u003e30\u003c/span\u003e\u003c/sup\u003e, and agile processes in application development\u003csup\u003e\u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e36\u003c/span\u003e\u003c/sup\u003e, are now well established. The benefits of incorporating these approaches is now being recognised with examples of co-production and agile processes being used together in the development of health monitoring applications\u003csup\u003e\u003cspan citationid=\"CR45\" class=\"CitationRef\"\u003e45\u003c/span\u003e\u003c/sup\u003e, assistive technologies\u003csup\u003e\u003cspan citationid=\"CR46\" class=\"CitationRef\"\u003e46\u003c/span\u003e\u003c/sup\u003e and environmental monitoring\u003csup\u003e\u003cspan citationid=\"CR47\" class=\"CitationRef\"\u003e47\u003c/span\u003e\u003c/sup\u003e. It is acknowledged that building innovative technology is not possible without recognising the context within which they would deployed\u003csup\u003e\u003cspan citationid=\"CR48\" class=\"CitationRef\"\u003e48\u003c/span\u003e\u003c/sup\u003e, thus co-design, allowing for understanding of the social and professional contexts, alongside agile processes allowing for rapid change of the web application, ensure both its usability and that the developed application is attuned to the complex relational and transactional process that underpin it\u003csup\u003e\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e, \u003cspan citationid=\"CR49\" class=\"CitationRef\"\u003e49\u003c/span\u003e\u003c/sup\u003e. However, this approach also has challenges. There is a requirement for negotiation, at times, between the requests put forward by those involved with the co-production and what is achievable in development, which requires both parties to have clear discussions and compromise\u003csup\u003e\u003cspan citationid=\"CR50\" class=\"CitationRef\"\u003e50\u003c/span\u003e, \u003cspan citationid=\"CR51\" class=\"CitationRef\"\u003e51\u003c/span\u003e\u003c/sup\u003e. Lastly, it is also important that the process is properly facilitated to be of value to both the co-design participants and the researchers, which requires researchers to buy-in to the process and at times be outside of the comfort zone of traditionally controlled research processes\u003csup\u003e\u003cspan citationid=\"CR52\" class=\"CitationRef\"\u003e52\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eThe development and usability testing of the digital web application provided an additional benefit in that it allowed us to examine complex issues associated with commissioning that otherwise would not have been possible. We had previously identified several challenges to implementing MSUs in the English and Welsh NHS, including where to locate MSUs, how to staff them, and making dispatch decisions\u003csup\u003e\u003cspan citationid=\"CR53\" class=\"CitationRef\"\u003e53\u003c/span\u003e\u003c/sup\u003e. With a similar group of participants, we have now been able to identify that commissioning is an additional challenge to implementation. Stakeholders using the web application helped highlight the complexities of the commissioning process,\u003csup\u003e\u003cspan citationid=\"CR24\" class=\"CitationRef\"\u003e24\u003c/span\u003e, \u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e\u003c/sup\u003e specifically uncertainty as to who would commission MSUs and whether this would be conducted locally, through centralised commissioning, or linked with specialist commissioning. This coincided with our experiences where, despite informal discussions with various types of commissioners, all declined to participate in the study as they did not feel they would ultimately be involved in the process of commissioning MSUs. The identified uncertainty around responsibility for commissioning is representative of a principal-agent problem resulting from a lack of role clarity. Principle-agent theory is drawn from political and economic science and provides a framework for understanding how work is delegated from principals to agents\u003csup\u003e\u003cspan citationid=\"CR54\" class=\"CitationRef\"\u003e54\u003c/span\u003e\u003c/sup\u003e. In the context of MSUs \u0026ndash; where there currently is no policy for their implementation in the NHS \u0026ndash; policymakers (principals) would delegate decisions to commissioners (agents), as they do in other complex pre-hospital care commissioning\u003csup\u003e\u003cspan citationid=\"CR55\" class=\"CitationRef\"\u003e55\u003c/span\u003e\u003c/sup\u003e. To address this problem, policy should be developed relating to commissioning decisions for MSUs in the NHS including the identification of who the agents would be.\u003c/p\u003e \u003cp\u003eThe web application also helps address a number of the previously identified barriers identified in the use of research in commissioning decisions, specifically that the web application would not be behind a paywall and was co-designed\u003csup\u003e\u003cspan citationid=\"CR25\" class=\"CitationRef\"\u003e25\u003c/span\u003e, \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e31\u003c/span\u003e\u003c/sup\u003e with a focus on informing commissioning decisions\u003csup\u003e\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e\u003c/sup\u003e. The data presented within the web application can be used to both on a national, and local level, with the maps making the data easier to interpret quickly\u003csup\u003e\u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e28\u003c/span\u003e, \u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e29\u003c/span\u003e\u003c/sup\u003e. Consequently, the web application would most likely be used by people developing and presenting the case for MSUs to be commissioned rather than commissioners themselves, providing opportunity for the exploration and presentation of context of both existing and alternative services. In doing so, using a web application helps to reduce information asymmetry that may exist between commissioners and providers, which is important for making more optimal commissioning decisions\u003csup\u003e\u003cspan citationid=\"CR56\" class=\"CitationRef\"\u003e56\u003c/span\u003e\u003c/sup\u003e. However, this would only apply where the information is used in support of MSUs as opposed to using the web application to inform when MSUs should not be commissioned, and thus also represents a moral hazard that needs to be considered when commissioners review business cases.\u003c/p\u003e \u003cp\u003eThe use of the web application also allowed us to further explore some of the previously identified implementation challenges\u003csup\u003e\u003cspan citationid=\"CR53\" class=\"CitationRef\"\u003e53\u003c/span\u003e\u003c/sup\u003e, as well as identifying new challenges. One of the keys areas of debate was in relation to the base MSU location and particularly whether MSUs should be based in an urban area, where they might be of benefit to more of the population, or whether the focus should be on rural areas that have poorer access to acute stroke care. This challenge was further discussed following use of the web application which showed higher gains for those further from hospitals, such as rural areas, however participants remained concerned about how much the MSU would then be utilised. As a result of the agile process adopted the web application was able to be updated to include population density which assists in making a determination and was thought to be essential to support commissioners to balance value-based and cost-effective decisions\u003csup\u003e\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e\u003c/sup\u003e. The use of the web application also raised the concern about minimal gains to patients, as shown by the utility shift, particularly those without an LVO. This had not been a previous concern however participants then questioned whether MSUs should targeted towards those with an LVO. However, literature to date has shown conflicting results about whether MSUs speed up access to MT for those with LVO\u003csup\u003e\u003cspan citationid=\"CR7\" class=\"CitationRef\"\u003e7\u003c/span\u003e, \u003cspan citationid=\"CR8\" class=\"CitationRef\"\u003e8\u003c/span\u003e, \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e14\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cp\u003eLastly, the use of the web application highlighted the need for the health economics data to be incorporated into the web application, in line with the need for potential new services to be both of clinical benefit and financially viable\u003csup\u003e\u003cspan citationid=\"CR20\" class=\"CitationRef\"\u003e20\u003c/span\u003e\u003c/sup\u003e. Previous studies have shown that MSUs have the potential to be cost-effective\u003csup\u003e\u003cspan additionalcitationids=\"CR58 CR59\" citationid=\"CR57\" class=\"CitationRef\"\u003e57\u003c/span\u003e\u0026ndash;\u003cspan citationid=\"CR60\" class=\"CitationRef\"\u003e60\u003c/span\u003e\u003c/sup\u003e however this was very context dependent, as often seen in economic evaluations\u003csup\u003e\u003cspan citationid=\"CR61\" class=\"CitationRef\"\u003e61\u003c/span\u003e\u003c/sup\u003e. None of these studies were completed within a UK context, with MSUs trialled in urban areas, and reported that MSUs only became cost-effective once they reached a certain patient threshold. Therefore, the web application, with incorporated health economics data would assist with some of the uncertainties with the rural/urban debate and whether to focus the resource on LVO patients, by showing both the benefits to patients and whether MSUs would offer financial value. However, even with this data there remains an area of political consideration, likely to affect commissioning decisions, which is not addressed in relation to the loss of IVT tariffs for ASUs if MSUs were delivering IVT. This could have an impact on the acceptability of MSUs which is a key element to the successful implementation of new interventions\u003csup\u003e\u003cspan citationid=\"CR62\" class=\"CitationRef\"\u003e62\u003c/span\u003e, \u003cspan citationid=\"CR63\" class=\"CitationRef\"\u003e63\u003c/span\u003e\u003c/sup\u003e. This again highlights the complexities of commissioning and that even when something is evidenced, close relational involvement is still required\u003csup\u003e\u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e26\u003c/span\u003e, \u003cspan citationid=\"CR27\" class=\"CitationRef\"\u003e27\u003c/span\u003e\u003c/sup\u003e.\u003c/p\u003e \u003cdiv id=\"Sec14\" class=\"Section2\"\u003e \u003ch2\u003eStrengths and limitations\u003c/h2\u003e \u003cp\u003eA strength of the study was that it was conducted using real-world data for the implementation of MSUs in England, ensuring that the models underpinning the usability testing provided accurate representations of MSU implementation, subject to recognised implementation challenges identified elsewhere in the study.\u003csup\u003e\u003cspan citationid=\"CR53\" class=\"CitationRef\"\u003e53\u003c/span\u003e, \u003cspan citationid=\"CR64\" class=\"CitationRef\"\u003e64\u003c/span\u003e\u003c/sup\u003e However, this also contributes to a limitation of the study; the usability testing does not represent how commissioners would actually use the web application as part of the relational decision-making that occurs in commissioning. This requires further evaluation should MSUs be commissioned in the NHS. Whilst some participants would be involved in commissioning processes (we are unable to describe exact job roles to protect anonymity), the study would have benefited from further commissioner involvement. However, this was challenging due to the uncertainties identified in the findings as to who would have responsibility for commissioning MSUs. This represents a circular dependency borne from modelling a potential innovation in the NHS that has no current existing commissioning pathway or policy. Other methods would be required to address this circular dependency, such as the use of vignette-based interviews that could present concrete hypothetical situations to commissioners. These vignettes would need to include hypothetical policies to address the previously described principal-agent problem resulting from a lack of role clarity.\u003c/p\u003e \u003c/div\u003e"},{"header":"Conclusion","content":"\u003cp\u003eA combination of agile processes and co-production in the development of a web application allowed us to develop and present usable models on the effectiveness of MSUs within the English NHS. Usability testing of the web application identified areas for improvement and helped identify complexities in commissioning processes, including the lack of role clarity around who would be responsible for the commissioning of MSUs in the absence of policy. Further implementation challenges associated with MSUs were also identified, particularly in relation to potential minimal gains and that maximum gains are seen within rural areas that are unlikely to have the population density required to make an MSU viable. This study provides empirical evidence in support of developing innovative and accessible digital dissemination tools to inform commissioning processes. Future research should evaluate the implementation and use of tools to inform commissioning processes, including the relational aspects of commissioning.\u003c/p\u003e"},{"header":"Declarations","content":"\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eAL, MA, GAF, MJ, SM, GM, KP, DP, CP, LS, PW, DW, PM and JS conceived the study. AL, ML and KP developed the web application and underlying models of clinical benefit. LM, MA, SM, GM, LJP and JS collected the qualitative data. LM and JS analysed the qualitative data and wrote the main manuscript and prepared figure 1. AL and MA provided additional contributions and prepared figures 2-5. All authors provided comments on the manuscript and approved it for submission.\u003c/p\u003e\u003ch2\u003eAcknowledgement\u003c/h2\u003e\u003cp\u003eWe would like to thank all participants for providing their time and views. MA and KP were part funded by the NIHR Applied Research Collaboration South West Peninsula. The views expressed in this publication are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care.\u003c/p\u003e\u003cp\u003e\u003cstrong\u003eData availability\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe datasets generated and/or analysed during the current study are not publicly available due to not having ethical approval or participant consent for the sharing of transcripts beyond select quotations. \u0026nbsp;\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eCode availability\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe underlying code for this study is available on Github: https://github.com/stroke-modelling/muster_workshop_web_app_1\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003estatement to the declarations:\u003c/strong\u003e \u0026quot;Ethical approval was provided via Northumbria University ethics online system (reference: 4117). The study was deemed by the Health Research Authority (HRA) to not require HRA approval.\u0026quot;\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eStroke Association. Stroke Statistics. Stroke Association 2024. https://www.stroke.org.uk/stroke/statistics\u003c/li\u003e\n\u003cli\u003eNational Health Service. Stroke. https://www.nhs.uk/conditions/stroke/\u003c/li\u003e\n\u003cli\u003eDonkor ES. Stroke in the 21(st) Century: A Snapshot of the Burden, Epidemiology, and Quality of Life. \u003cem\u003eStroke Res Treat\u003c/em\u003e. 2018;2018:3238165. doi:10.1155/2018/3238165\u003c/li\u003e\n\u003cli\u003eKing D, Wittenberg R, Patel A, Quayyum Z, Berdunov V, Knapp M. The future incidence, prevalence and costs of stroke in the UK. \u003cem\u003eAge and Ageing\u003c/em\u003e. 2020;49(2):277-282. doi:10.1093/ageing/afz163\u003c/li\u003e\n\u003cli\u003eSaini V, Guada L, Yavagal DR. Global Epidemiology of Stroke and Access to Acute Ischemic Stroke Interventions. \u003cem\u003eNeurology\u003c/em\u003e. 2021;97(20_Supplement_2):S6-S16. doi:doi:10.1212/WNL.0000000000012781\u003c/li\u003e\n\u003cli\u003eWalter S, Audebert HJ, Katsanos AH, et al. European Stroke Organisation (ESO) guidelines on mobile stroke units for prehospital stroke management. \u003cem\u003eEur Stroke J\u003c/em\u003e. Mar 2022;7(1):Xxvii-lix. doi:10.1177/23969873221079413\u003c/li\u003e\n\u003cli\u003eBender MT, Mattingly TK, Rahmani R, et al. Mobile stroke care expedites intravenous thrombolysis and endovascular thrombectomy. \u003cem\u003eStroke and Vascular Neurology\u003c/em\u003e. 2022;7(3):209-214. doi:10.1136/svn-2021-001119\u003c/li\u003e\n\u003cli\u003eBluhm S, Schramm P, Spreen-Ledebur Y, et al. Potential effects of a mobile stroke unit on time to treatment and outcome in patients treated with thrombectomy or thrombolysis: A Danish\u0026ndash;German cross-border analysis. \u003cem\u003eEuropean Journal of Neurology\u003c/em\u003e. n/a(n/a):e16298. doi:https://doi.org/10.1111/ene.16298\u003c/li\u003e\n\u003cli\u003eEbinger M, Siegerink B, Kunz A, et al. Association Between Dispatch of Mobile Stroke Units and Functional Outcomes Among Patients With Acute Ischemic Stroke in Berlin. \u003cem\u003eJama\u003c/em\u003e. Feb 2 2021;325(5):454-466. doi:10.1001/jama.2020.26345\u003c/li\u003e\n\u003cli\u003eGrotta JC, Yamal J-M, Parker SA, et al. Prospective, Multicenter, Controlled Trial of Mobile Stroke Units. \u003cem\u003eNew England Journal of Medicine\u003c/em\u003e. 2021;385(11):971-981. doi:10.1056/nejmoa2103879\u003c/li\u003e\n\u003cli\u003eHelwig SA, Ragoschke-Schumm A, Schwindling L, et al. Prehospital Stroke Management Optimized by Use of Clinical Scoring vs Mobile Stroke Unit for Triage of Patients With Stroke: A Randomized Clinical Trial. \u003cem\u003eJAMA Neurol\u003c/em\u003e. Dec 1 2019;76(12):1484-1492. doi:10.1001/jamaneurol.2019.2829\u003c/li\u003e\n\u003cli\u003eKunz A, Ebinger M, Geisler F, et al. Functional outcomes of pre-hospital thrombolysis in a mobile stroke treatment unit compared with conventional care: an observational registry study. \u003cem\u003eLancet Neurol\u003c/em\u003e. Sep 2016;15(10):1035-43. doi:10.1016/s1474-4422(16)30129-6\u003c/li\u003e\n\u003cli\u003eLarsen K, Jaeger HS, Tveit LH, et al. Ultraearly thrombolysis by an anesthesiologist in a mobile stroke unit: A prospective, controlled intervention study. \u003cem\u003eEur J Neurol\u003c/em\u003e. Aug 2021;28(8):2488-2496. doi:10.1111/ene.14877\u003c/li\u003e\n\u003cli\u003eZhao H, Coote S, Easton D, et al. Melbourne Mobile Stroke Unit and Reperfusion Therapy. \u003cem\u003eStroke\u003c/em\u003e. 2020;51(3):922-930. doi:doi:10.1161/STROKEAHA.119.027843\u003c/li\u003e\n\u003cli\u003ePhipps MS, Cronin CA. Management of acute ischemic stroke. \u003cem\u003eBMJ\u003c/em\u003e. 2020;368:l6983. doi:10.1136/bmj.l6983\u003c/li\u003e\n\u003cli\u003eNHS Commissioning before April 2013 (House of Commons Library) (2016).\u003c/li\u003e\n\u003cli\u003eAllen P, Petsoulas C. Pricing in the English NHS quasi market: a national study of the allocation of financial risk through contracts. \u003cem\u003ePublic Money \u0026amp; Management\u003c/em\u003e. 2016/07/28 2016;36(5):341-348. doi:10.1080/09540962.2016.1194080\u003c/li\u003e\n\u003cli\u003eCheckland K, Harrison S, Snow S, McDermott I, Coleman A. Commissioning in the English National Health Service: What\u0026apos;s the Problem? \u003cem\u003eJournal of Social Policy\u003c/em\u003e. 2012;41(3):533-550. doi:10.1017/S0047279412000232\u003c/li\u003e\n\u003cli\u003eNHS England. Commissioning. 2024. https://www.england.nhs.uk/commissioning/\u003c/li\u003e\n\u003cli\u003eTsevat J, Moriates C. Value-Based Health Care Meets Cost-Effectiveness Analysis. \u003cem\u003eAnn Intern Med\u003c/em\u003e. Sep 4 2018;169(5):329-332. doi:10.7326/m18-0342\u003c/li\u003e\n\u003cli\u003eEngland NHS. \u003cem\u003eNHS England Specialised Commissioning Service Development Policy\u003c/em\u003e. 2017. https://www.england.nhs.uk/wp-content/uploads/2017/09/B0026_NHS-England-Specialised-Commissioning-Service-Development-Policy.pdf\u003c/li\u003e\n\u003cli\u003eBrownson RC, Fielding JE, Maylahn CM. Evidence-Based Public Health: A Fundamental Concept for Public Health Practice. \u003cem\u003eAnnual Review of Public Health\u003c/em\u003e. 2009;30(Volume 30, 2009):175-201. doi:https://doi.org/10.1146/annurev.publhealth.031308.100134\u003c/li\u003e\n\u003cli\u003eKneale D, Rojas-Garc\u0026iacute;a A, Raine R, Thomas J. The use of evidence in English local public health decision-making: a systematic scoping review. \u003cem\u003eImplementation Science\u003c/em\u003e. 2017/04/20 2017;12(1):53. doi:10.1186/s13012-017-0577-9\u003c/li\u003e\n\u003cli\u003eGongora-Salazar P, Glogowska M, Fitzpatrick R, Perera R, Tsiachristas A. Commissioning [Integrated] Care in England: An Analysis of the Current Decision Context. \u003cem\u003eInt J Integr Care\u003c/em\u003e. Oct-Dec 2022;22(4):3. doi:10.5334/ijic.6693\u003c/li\u003e\n\u003cli\u003eScott RJ, Mathie E, Newman HJH, Almack K, Brady L-M. Commissioning and co-production in health and care services in the United Kingdom and Ireland: An exploratory literature review. \u003cem\u003eHealth Expectations\u003c/em\u003e. 2024;27(3):e14053. doi:https://doi.org/10.1111/hex.14053\u003c/li\u003e\n\u003cli\u003ePorter A, Mays N, Shaw SE, Rosen R, Smith J. Commissioning healthcare for people with long term conditions: the persistence of relational contracting in England\u0026rsquo;s NHS quasi-market. \u003cem\u003eBMC Health Services Research\u003c/em\u003e. 2013/05/24 2013;13(1):S2. doi:10.1186/1472-6963-13-S1-S2\u003c/li\u003e\n\u003cli\u003eShaw SE, Smith JA, Porter A, Rosen R, Mays N. The work of commissioning: a multisite case study of healthcare commissioning in England\u0026apos;s NHS. \u003cem\u003eBMJ Open\u003c/em\u003e. 2013;3(9):e003341. doi:10.1136/bmjopen-2013-003341\u003c/li\u003e\n\u003cli\u003eWye L, Brangan E, Cameron A, Gabbay J, Klein JH, Pope C. Evidence based policy making and the \u0026lsquo;art\u0026rsquo; of commissioning \u0026ndash; how English healthcare commissioners access and use information and academic research in \u0026lsquo;real life\u0026rsquo; decision-making: an empirical qualitative study. \u003cem\u003eBMC Health Services Research\u003c/em\u003e. 2015/09/29 2015;15(1):430. doi:10.1186/s12913-015-1091-x\u003c/li\u003e\n\u003cli\u003eJager A, Wong G, Papoutsi C, Roberts N. The usage of data in NHS primary care commissioning: a realist review. \u003cem\u003eBMC Medicine\u003c/em\u003e. 2023/07/03 2023;21(1):236. doi:10.1186/s12916-023-02949-w\u003c/li\u003e\n\u003cli\u003eNational Health Service England. \u003cem\u003eWorking in partnership with people and communities: Statutory guidance\u003c/em\u003e. 2022. https://www.england.nhs.uk/long-read/working-in-partnership-with-people-and-communities-statutory-guidance/\u003c/li\u003e\n\u003cli\u003eHart F. Is commissioning the enemy of co-production? \u003cem\u003ePerspect Public Health\u003c/em\u003e. Jul 2022;142(4):191-192. doi:10.1177/17579139221103189\u003c/li\u003e\n\u003cli\u003eMarshall I, McKevitt C, Wang Y, et al. Stroke pathway ? An evidence base for commissioning ? An evidence review for NHS England and NHS Improvement [version 1; peer review: 2 approved with reservations]. \u003cem\u003eNIHR Open Research\u003c/em\u003e. 2022;2(43)doi:10.3310/nihropenres.13257.1\u003c/li\u003e\n\u003cli\u003eNarayanan S, Balamurugan NM, M K, Palas PB. Leveraging Machine Learning Methods for Multiple Disease Prediction using Python ML Libraries and Flask API. 2022:694-701.\u003c/li\u003e\n\u003cli\u003eMonks T, Harper A, Mustafee N. Towards sharing tools and artefacts for reusable simulations in healthcare. \u003cem\u003eJournal of Simulation\u003c/em\u003e.1-20. doi:10.1080/17477778.2024.2347882\u003c/li\u003e\n\u003cli\u003eLaws A, Allen M, Scott J, et al. Modelling the potential clinical benefit of mobile stroke units in England. \u003cem\u003eZenodo\u003c/em\u003e. 2024;doi:https://doi.org/10.5281/zenodo.14132787\u003c/li\u003e\n\u003cli\u003eAl-Saqqa S, Sawalha S, Abdelnabi H. Agile Software Development: Methodologies and Trends. \u003cem\u003eInt J Interact Mob Technol\u003c/em\u003e. 2020;14:246-270.\u003c/li\u003e\n\u003cli\u003eNickel GC, Wang S, Kwong JCC, Kvedar JC. The case for inclusive co-creation in digital health innovation. \u003cem\u003enpj Digital Medicine\u003c/em\u003e. 2024/09/16 2024;7(1):251. doi:10.1038/s41746-024-01256-9\u003c/li\u003e\n\u003cli\u003eChen J, Lin X, Cai Y, Huang R, Yang S, Zhang G. A Systematic Review of Mobile Stroke Unit Among Acute Stroke Patients: Time Metrics, Adverse Events, Functional Result and Cost-Effectiveness. \u003cem\u003eFront Neurol\u003c/em\u003e. 2022;13:803162. doi:10.3389/fneur.2022.803162\u003c/li\u003e\n\u003cli\u003eFatima N, Saqqur M, Hussain MS, Shuaib A. Mobile stroke unit versus standard medical care in the management of patients with acute stroke: A systematic review and meta-analysis. \u003cem\u003eInternational Journal of Stroke\u003c/em\u003e. 2020;15(6):595-608. doi:10.1177/1747493020929964\u003c/li\u003e\n\u003cli\u003eHarvey N, Holmes CA. Nominal group technique: An effective method for obtaining group consensus. \u003cem\u003eInternational Journal of Nursing Practice\u003c/em\u003e. 2012;18(2):188-194. doi:https://doi.org/10.1111/j.1440-172X.2012.02017.x\u003c/li\u003e\n\u003cli\u003eMaramba I, Chatterjee A, Newman C. Methods of usability testing in the development of eHealth applications: A scoping review. \u003cem\u003eInt J Med Inform\u003c/em\u003e. Jun 2019;126:95-104. doi:10.1016/j.ijmedinf.2019.03.018\u003c/li\u003e\n\u003cli\u003eNational Institute for Health and Care Research. Modelling the resource requirements for implementation of mobile stroke units across the National Health Service, their cost-effectiveness, and their effect on equity of access to emergency stroke care. https://fundingawards.nihr.ac.uk/award/NIHR153982\u003c/li\u003e\n\u003cli\u003e\u003cem\u003eNvivo\u003c/em\u003e. Lumivero; 2020.\u003c/li\u003e\n\u003cli\u003eBraun V, Clarke V. \u003cem\u003eThematic analysis : a practical guide\u003c/em\u003e. 1st edition.. ed. Thousand Oaks : SAGE Publications Ltd.; 2022.\u003c/li\u003e\n\u003cli\u003eFox S, Brown LJE, Antrobus S, et al. Co-design of a Smartphone App for People Living With Dementia by Applying Agile, Iterative Co-design Principles: Development and Usability Study. \u003cem\u003eJMIR Mhealth Uhealth\u003c/em\u003e. 2022/1/14 2022;10(1):e24483. doi:10.2196/24483\u003c/li\u003e\n\u003cli\u003eTessarolo F, Nollo G, Conotter V, et al. User-centered co-design and AGILE methodology for developing ambient assisting technologies: Study plan and methodological framework of the CAPTAIN project. 2019:283-286.\u003c/li\u003e\n\u003cli\u003eLaScala-Gruenewald DE, Low NHN, Barry JP, et al. Building on a human-centred, iterative, and agile co-design strategy to facilitate the availability of deep ocean data. \u003cem\u003eICES Journal of Marine Science\u003c/em\u003e. 2022;80(2):347-351. doi:10.1093/icesjms/fsac145\u003c/li\u003e\n\u003cli\u003eMillard D, Howard Y, Gilbert L, Wills G. Co-Design and Co-Deployment Methodologies for Innovative m-Learning Systems. In: Goh TT, ed. \u003cem\u003eMultiplatform E-Learning Systems and Technologies: Mobile Devices for Ubiquitous ICT-Based Education\u003c/em\u003e. IGI Global; 2010:147-163.\u003c/li\u003e\n\u003cli\u003eKokol P. Agile Software Development in Healthcare: A Synthetic Scoping Review. \u003cem\u003eApplied Sciences\u003c/em\u003e. 2022;12(19). doi:10.3390/app12199462\u003c/li\u003e\n\u003cli\u003eNoorbergen TJ, Adam MTP, Teubner T, Collins CE. Using Co-design in Mobile Health System Development: A Qualitative Study With Experts in Co-design and Mobile Health System Development. \u003cem\u003eJMIR Mhealth Uhealth\u003c/em\u003e. 2021/11/10 2021;9(11):e27896. doi:10.2196/27896\u003c/li\u003e\n\u003cli\u003eTabeau K, de Mul M, Strating M, et al. The challenges of and solutions for combining cocreation and agile in the development of health information technologies. \u003cem\u003eInternational Journal of Medical Informatics\u003c/em\u003e. 2024/11/01/ 2024;191:105557. doi:https://doi.org/10.1016/j.ijmedinf.2024.105557\u003c/li\u003e\n\u003cli\u003eThabrew H, Fleming T, Hetrick S, Merry S. Co-design of eHealth Interventions With Children and Young People. Mini Review. \u003cem\u003eFrontiers in Psychiatry\u003c/em\u003e. 2018-October-18 2018;9doi:10.3389/fpsyt.2018.00481\u003c/li\u003e\n\u003cli\u003eMoseley L, McMeekin P, Price C, et al. Practitioner, patient and public views on the acceptability of Mobile Stroke Units in England and Wales: a mixed methods study. \u003cem\u003emedRxiv\u003c/em\u003e. 2024:2024.08. 26.24312612.\u003c/li\u003e\n\u003cli\u003eBraun D, Guston DH. Principal-agent theory and research policy: an introduction. \u003cem\u003eScience and public policy\u003c/em\u003e. 2003;30(5):302-308.\u003c/li\u003e\n\u003cli\u003ePettersen IJ, Nyland K, Robbins G. Public procurement performance and the challenge of service complexity \u0026ndash; the case of pre-hospital healthcare. \u003cem\u003eJournal of Public Procurement\u003c/em\u003e. 2020;20(4):403-421. doi:10.1108/JOPP-01-2020-0002\u003c/li\u003e\n\u003cli\u003ePopović M, Kuzmanović M, Gu\u0026scaron;avac BA. The agency dilemma: information asymmetry in the\u0026quot; principal-agent\u0026quot; problem. \u003cem\u003eManagement\u003c/em\u003e. 2012;62:15.\u003c/li\u003e\n\u003cli\u003eDietrich M, Walter S, Ragoschke-Schumm A, et al. Is Prehospital Treatment of Acute Stroke too Expensive An Economic Evaluation Based on the First Trial. \u003cem\u003eCerebrovascular Diseases\u003c/em\u003e. 2014;38(6):457-463. doi:10.1159/000371427\u003c/li\u003e\n\u003cli\u003eGyrd-Hansen D, Olsen KR, Bollweg K, Kronborg C, Ebinger M, Audebert HJ. Cost-effectiveness estimate of prehospital thrombolysis: results of the PHANTOM-S study. \u003cem\u003eNeurology\u003c/em\u003e. Mar 17 2015;84(11):1090-7. doi:10.1212/wnl.0000000000001366\u003c/li\u003e\n\u003cli\u003eKim J, Easton D, Zhao H, et al. Economic evaluation of the Melbourne Mobile Stroke Unit. \u003cem\u003eInternational Journal of Stroke\u003c/em\u003e. 2021;16(4):466-475. doi:10.1177/1747493020929944\u003c/li\u003e\n\u003cli\u003eReimer AP, Zafar A, Hustey FM, et al. Cost-Consequence Analysis of Mobile Stroke Units vs. Standard Prehospital Care and Transport. \u003cem\u003eFront Neurol\u003c/em\u003e. 2019;10:1422. doi:10.3389/fneur.2019.01422\u003c/li\u003e\n\u003cli\u003eAnderson R, Hardwick R. Realism and resources: Towards more explanatory economic evaluation. \u003cem\u003eEvaluation (London, England 1995)\u003c/em\u003e. 2016;22(3):323-341. doi:10.1177/1356389016652742\u003c/li\u003e\n\u003cli\u003eSkivington K, Matthews L, Simpson SA, et al. A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. \u003cem\u003eBMJ\u003c/em\u003e. 2021;374:n2061. doi:10.1136/bmj.n2061\u003c/li\u003e\n\u003cli\u003eYardley L, Ainsworth B, Arden-Close E, Muller I. The person-based approach to enhancing the acceptability and feasibility of interventions. \u003cem\u003ePilot and Feasibility Studies\u003c/em\u003e. 2015/10/26 2015;1(1):37. doi:10.1186/s40814-015-0033-z\u003c/li\u003e\n\u003cli\u003eMoseley L, McMeekin P, Allen M, et al. Co-design of a Mobile Stroke Unit pathway highlights uncertainties and trade-offs for viable system-wide implementation in the English and Welsh NHS. 2024;\u003cstrong\u003e\u003c/strong\u003e\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"npj-digital-medicine","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"npjdigitalmed","sideBox":"Learn more about [npj Digital Medicine](http://www.nature.com/npjdigitalmed/)","snPcode":"41746","submissionUrl":"https://submission.springernature.com/new-submission/41746/3","title":"npj Digital Medicine","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"","lastPublishedDoi":"10.21203/rs.3.rs-5668150/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5668150/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003eCommissioning of innovations in healthcare is a complex socio-technical process, ideally informed by high quality evidence. However, evidence is not always prepared and presented in a format usable for commissioning decisions. Agile methodology, combined with qualitative co-design, were used to develop a digital web application incorporating machine learning models of stroke outcomes to inform commissioning decisions for the implementation of Mobile Stroke Units (MSUs) in England, followed by usability testing using Think Aloud methodology. Sixteen stakeholders involved in developing consensus on model parameters and pathways participated with data thematically analysed. Required improvements to the web application were identified and novel insights into the complexity of context-specific commissioning decisions were generated, which also informed participants\u0026rsquo; views on the viability of MSUs. This study provides empirical evidence in support of developing innovative and accessible digital dissemination methods to engage with commissioning processes and prospectively understand commissioning challenges.\u003c/p\u003e","manuscriptTitle":"Usability testing a digital web application for evidence-based commissioning decisions in the context of implementing Mobile Stroke Units in England","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-01-30 10:14:57","doi":"10.21203/rs.3.rs-5668150/v1","editorialEvents":[{"type":"communityComments","content":0}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"npj-digital-medicine","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"npjdigitalmed","sideBox":"Learn more about [npj Digital Medicine](http://www.nature.com/npjdigitalmed/)","snPcode":"41746","submissionUrl":"https://submission.springernature.com/new-submission/41746/3","title":"npj Digital Medicine","twitterHandle":"","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"stoa","reportingPortfolio":"NPJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"10268b71-35c5-4017-850c-d62ce4a27f94","owner":[],"postedDate":"January 30th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[{"id":43470049,"name":"Health sciences/Health care/Health policy"},{"id":43470050,"name":"Health sciences/Health care/Health services"},{"id":43470051,"name":"Health sciences/Neurology/Neurological disorders/Stroke"}],"tags":[],"updatedAt":"2025-05-12T16:02:58+00:00","versionOfRecord":{"articleIdentity":"rs-5668150","link":"https://doi.org/10.1038/s41746-025-01691-2","journal":{"identity":"npj-digital-medicine","isVorOnly":false,"title":"npj Digital Medicine"},"publishedOn":"2025-05-09 15:57:42","publishedOnDateReadable":"May 9th, 2025"},"versionCreatedAt":"2025-01-30 10:14:57","video":"","vorDoi":"10.1038/s41746-025-01691-2","vorDoiUrl":"https://doi.org/10.1038/s41746-025-01691-2","workflowStages":[]},"version":"v1","identity":"rs-5668150","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5668150","identity":"rs-5668150","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}