Evaluating video and hybrid group consultations in general practice: mixed-methods, participatory study protocol (TOGETHER 2)

Keywords group consulting, remote care, general practice, invisible work, lived experience ALL Metrics - Views Downloads How to cite this article Papoutsi C, Abel G, Iglesias C et al. Evaluating video and hybrid group consultations in general practice: mixed-methods, participatory study protocol (TOGETHER 2) [version 2; peer review: 2 approved]. NIHR Open Res 2024, 4:30 (https://doi.org/10.3310/nihropenres.13584.2) NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article. Export Citation Sciwheel EndNote Ref. Manager Bibtex ProCite Sente Select a format first ▬ ✚ Study Protocol Revised Evaluating video and hybrid group consultations in general practice: mixed-methods, participatory study protocol (TOGETHER 2) [version 2; peer review: 2 approved] Chrysanthi Papoutsi https://orcid.org/0000-0003-1189-7100 1, Gary Abel2, Cynthia Iglesias3, [...] Jackie van Dael1, Claire Reidy1, Stuart D Faulkner1, Helene Raynsford4, Michele Siciliano https://orcid.org/0009-0000-7354-5368 3, Luis Beltran Galindo https://orcid.org/0000-0001-9199-8467 3, Vijay Gc https://orcid.org/0000-0003-0365-2605 5, John Campbell2, Trisha Greenhalgh https://orcid.org/0000-0003-2369-8088 1, Sara E Shaw https://orcid.org/0000-0002-7014-4793 1Chrysanthi Papoutsi https://orcid.org/0000-0003-1189-7100 1, Gary Abel2, [...] Cynthia Iglesias3, Jackie van Dael1, Claire Reidy1, Stuart D Faulkner1, Helene Raynsford4, Michele Siciliano https://orcid.org/0009-0000-7354-5368 3, Luis Beltran Galindo https://orcid.org/0000-0001-9199-8467 3, Vijay Gc https://orcid.org/0000-0003-0365-2605 5, John Campbell2, Trisha Greenhalgh https://orcid.org/0000-0003-2369-8088 1, Sara E Shaw https://orcid.org/0000-0002-7014-4793 1 PUBLISHED 18 Sep 2024 Author details Author details 1 Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, England, OX26GG, UK 2 Department of Health and Community Science, Smeall Building, St Luke’s Campus, University of Exeter, Exeter, England, EX1 2LU, UK 3 Department of Health Sciences, Seebohm Rowntree Building, University of York, York, England, YO10 5SQ, UK 4 Independent public contributor, UK, UK 5 School of Human & Health Sciences, University of Huddersfield, Huddersfield, England, UK 2 Department of Health and Community Science, Smeall Building, St Luke’s Campus, University of Exeter, Exeter, England, EX1 2LU, UK 3 Department of Health Sciences, Seebohm Rowntree Building, University of York, York, England, YO10 5SQ, UK 4 Independent public contributor, UK, UK 5 School of Human & Health Sciences, University of Huddersfield, Huddersfield, England, UK Chrysanthi Papoutsi Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing Gary Abel Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing Cynthia Iglesias Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing Jackie van Dael Roles: Investigation, Writing – Review & Editing Roles: Investigation, Writing – Review & Editing Claire Reidy Roles: Investigation, Writing – Review & Editing Roles: Investigation, Writing – Review & Editing Stuart D Faulkner Roles: Project Administration, Writing – Review & Editing Roles: Project Administration, Writing – Review & Editing Helene Raynsford Roles: Conceptualization, Writing – Review & Editing Roles: Conceptualization, Writing – Review & Editing Michele Siciliano Roles: Investigation, Writing – Review & Editing Roles: Investigation, Writing – Review & Editing Luis Beltran Galindo Roles: Investigation, Writing – Review & Editing Roles: Investigation, Writing – Review & Editing Vijay Gc Roles: Investigation, Writing – Review & Editing Roles: Investigation, Writing – Review & Editing John Campbell Roles: Conceptualization, Funding Acquisition, Methodology, Writing – Review & Editing Roles: Conceptualization, Funding Acquisition, Methodology, Writing – Review & Editing Trisha Greenhalgh Roles: Conceptualization, Funding Acquisition, Writing – Review & Editing Roles: Conceptualization, Funding Acquisition, Writing – Review & Editing Sara E Shaw Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Supervision, Writing – Review & Editing Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Supervision, Writing – Review & Editing OPEN PEER REVIEW REVIEWER STATUS General practice is facing an unprecedented challenge in managing the consequences of the pandemic. In the midst of a policy drive to balance remote and in-person service provision, substantial workload pressures remain, together with increasing prevalence of long-term conditions, and declining staff numbers and morale. To address these challenges, some practices in the UK have been delivering video and hybrid group consultations (VHGCs) providing clinical care to multiple patients at the same time. Despite positive initial findings and enthusiasm, there are still gaps in our understanding of the influence VHGCs have on patient experience, healthcare utilisation, quality, safety, equity and affordability. To generate an in-depth understanding of VHGCs for chronic conditions in general practice, surface assumptions and sociotechnical dynamics, inform practice and extend theorisation. Mixed-methods, multi-site research study using co-design and participatory methods, from qualitative, quantitative and cost-related perspectives. WP1 includes a national, cross-sectional survey on VHGC provision across the UK. In WP2 we will engage patients and general practice staff in co-design workshops to develop VHGC models with emphasis on digital inclusion and equity. In WP3 we will carry out a mixed-methods process evaluation in up to 10 GP practices across England (5 sites already running VHGCs and 5 comparison sites). Qualitative methods will include interviews, focus groups and ethnographic observation to examine the experiences of patients, carers, clinical and non-clinical NHS staff, commissioners and policy-makers. Quantitative methods will examine the impact of VHGCs on healthcare utilisation in primary and secondary care, patient satisfaction, engagement and activation. We will also assess value for money of group and individual care models from a health economics perspective. We aim to develop transferable learning on sociotechnical change in healthcare delivery, using VHGCs as an exemplar of technology-supported innovation. Findings will also inform the design of a future study. To better understand how group consultations for chronic conditions can benefit patients and the health service when delivered via video and/or in-person in general practice. Before the pandemic, group consultations were starting to gain ground in the UK as a new way of delivering clinical care to multiple patients at the same time, with potential benefits resulting from peer support and time efficiencies. When in-person care was restricted due to Covid-19, clinicians started delivering group consultations over video. Despite significant interest, we still know little about how group consultations delivered over video or hybrid models (combining video and in-person sessions) can be best implemented. In up to 5 GP practices across England, where group consultations are already being delivered, we will evaluate how these new approaches to clinical care are implemented, and how they may support an inclusive service that engages patients with different needs and preferences. The evaluation will include interviews with patients, carers, NHS staff, policy-makers and commissioners, as well as group discussions and observations, including research led by patients themselves. We will also collect numerical data on the number and type of patients attending, whether they are more satisfied or confident with their self-management, or less likely to need to go to hospital. We will explore costs associated with these new ways of delivering care and will develop comparisons to face-to-face individual appointments. We will also work with comparable clinical sites only delivering one-to-one appointments, to collect numerical data on patient attendance, satisfaction and use of health services. With the involvement of our patient and public involvement (PPI) group, we will bring together our data to develop practical knowledge. group consulting, remote care, general practice, invisible work, lived experience Corresponding Author(s) Chrysanthi Papoutsi ([email protected]) Grant information: The project is funded by the National Institute for Health and Care Research (NIHR) under its Health and Social Care Delivery Research (HSDR) Programme (Grant Reference Number NIHR133895). The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Copyright: © 2024 Papoutsi C et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. How to cite: Papoutsi C, Abel G, Iglesias C et al. Evaluating video and hybrid group consultations in general practice: mixed-methods, participatory study protocol (TOGETHER 2) [version 2; peer review: 2 approved]. NIHR Open Res 2024, 4:30 (https://doi.org/10.3310/nihropenres.13584.2) First published: 10 May 2024, 4:30 (https://doi.org/10.3310/nihropenres.13584.1) Latest published: 18 Sep 2024, 4:30 (https://doi.org/10.3310/nihropenres.13584.2) The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. In the revised version we have: a) added a definition of group consultations in the abstract, b) clarified that group consultations are delivered by different health professionals, c) explained the term sociotechnical change, d) provided references for the GP patient survey and QOF, d) clarified on proposed data collection methods and measurement, e) explained how the work-packages interlink, and f) provided specific details across different work-packages such as role of PPI, co-design workshop set-up, ability to draw conclusions from quantitative data etc. In the revised version we have: a) added a definition of group consultations in the abstract, b) clarified that group consultations are delivered by different health professionals, c) explained the term sociotechnical change, d) provided references for the GP patient survey and QOF, d) clarified on proposed data collection methods and measurement, e) explained how the work-packages interlink, and f) provided specific details across different work-packages such as role of PPI, co-design workshop set-up, ability to draw conclusions from quantitative data etc. See the authors' detailed response to the review by Rasmus Dahl Larsen See the authors' detailed response to the review by Jennifer Job Efforts are constantly underway to reconfigure healthcare provision, more recently driven by immense demand pressures and the need to provide good patient care with limited resources1,2. New service models often rely on digital health innovations in the hope that technology-supported care will contribute to streamlined workloads, better prioritisation of patient need, and time savings, among other benefits3. In the UK this became particularly pronounced in the context of Covid-19 when ‘remote-by-default’ care captured the imagination of policy-makers (until this policy was later reversed following public outcry)4–6. It is now becoming more widely understood that ‘remote services are difficult to set up, technically challenging, may increase workload at a stressful time, and could worsen health inequities’7. Alongside remote care, the role of ‘lived experience’ has also received significant attention in recent decades. Patient and public involvement (PPI), peer group and ‘expert patient’ programmes all aim to harness patient ‘experiences’ as a resource towards healthcare delivery and improvement8,9. Although a welcome shift from dominant paternalistic models viewing patients as passive recipients of care, an increased focus on ‘lived experience’ also involves a number of assumptions about whose experiences count, how these can be articulated and in what ways they become meaningful (or not) in the context of healthcare delivery10–12. The ‘diversity and plurality of forms and articulations of knowledge that characterise experiential knowledge, as well as the gradual, dynamic and entangled process that leads from experience to knowledge and expertise’ are often neglected10. In Together 2 we are studying video and hybrid group consultations, a relatively novel way of care provision combining the logics outlined above: a) an attempt to harness the potential of remote care while b) foregrounding experiential knowledge and sharing as a resource for service provision. In the UK, video and hybrid group consultations started to gain traction during the Covid-19 pandemic, where physical distancing invited imaginative solutions to care delivery. These built on existing models and approaches to in-person group consulting pre pandemic where small groups of patients came together to have their clinical consultations at the same time (e.g. diabetes annual review in general practice), rather than in separate individual appointments. Combining clinical consultation with education and peer support, group consultations are led by different health professionals (GPs, pharmacists, nurses) and have been delivered in a variety of formats for patients with different long-term conditions or shared health concerns13,14. A large part of the literature on group consultations has so far focused on assessing potential to improve clinical outcomes for specific conditions and to influence measures such as patient ‘satisfaction’ (e.g. see 15,16). Although some studies provide positive findings, especially in clinical areas where patient self-care is important (e.g. diabetes), these tend to focus on face-to-face delivery, report on group consultations conducted as part of randomised controlled trials rather than standard clinical care, or have been carried out in contexts other than the UK15,17–21. Only a small number of studies have focused (wholly or partly) on video group consultations in UK settings, including during the pandemic, and have provided insights in relation to facilitators/barriers or patient perceptions (e.g. see 16,22,23). Yet, more research is needed to understand whether and how group consultations, when provided remotely, can become embedded over time in different healthcare settings still recovering from the pandemic. Rich detail of the complexity of group consulting provision is only starting to emerge14,24–26; there is little theorisation of the role patients themselves (and other critical actors) play in group sessions and the ways good group-based care relies on the capacity to manage invisible work and mobilise lived experience. In Together 2 we seek to generate an in-depth understanding of different modes of remote group consulting (video and hybrid), surface assumptions (around what these models are meant to achieve) and sociotechnical dynamics (the interplay between technical and social elements) underpinning this relatively novel way of providing care, inform practice and extend theorisation. Informed by our previous work on in-person diabetes group consultations for socioeconomically disadvantaged young people in secondary care (NIHR HSDR-funded, 2016–21)25,26 and in-pandemic research on video group consultations in general practice (Health Foundation-funded, 2020–21)27, the following research questions act as our starting point: 1. What is the feasibility and acceptability of video and hybrid group consultations to different population groups? What are their (perceived) implications for access, efficiency and safety in general practice? 2. How do patients, carers and NHS staff experience these new models of care, compared to standard one-to-one service provision and in-person group consultations? 3. What are the costs of introducing video and hybrid group consultations? What is the organisational impact on general practice and other system-level stakeholders? 4. What would be the optimal design of a large-scale evaluation to assess clinical benefits and key outcome/activity parameters? This study follows from previous research (as explained above) where PPI contributors had been involved and helped shape the research questions and design set out in this protocol. PPI contributors supporting this study include people with a range of conditions, caring responsibilities (including for family members who do not speak English), ages, genders, and ethnicities. Members of the group meet formally every ~6 months with individual conversations organised between meetings as needed to discuss specific topics in more detail or where PPI contributors find it easier to engage separately. The evaluation will be underpinned by participatory principles, including working with PPI contributors on, e.g. setting the agenda for co-design, reviewing recruitment materials, helping with data interpretation and developing patient-facing guidance documents. This mixed methods project includes three overlapping and interlinked work packages: A) Cross-sectional, UK-wide survey on group consulting provision over time (WP1), B) Co-design with patients and general practice staff with emphasis on inclusion and equity (WP2), C) Mixed methods, multi-site process evaluation in up to 5 case and 5 comparison sites (primarily GP practices in England) to generate actionable learning and in-depth understanding of implementation, from qualitative, quantitative and cost-related perspectives (WP3). We plan a national scoping survey of existing and emerging group consulting service models across care levels in the UK. The scoping survey will provide a picture of how group consulting was deployed as a remote model of care in the context of the pandemic (2020–21), and how it has been used in the recovery context (2021–22), including the types of conditions and patient populations for which it has been used, the types of consultations carried out in groups (e.g. annual reviews, health checks), frequency and format of sessions (e.g. in-person or remote delivery mode, inclusion of underserved communities), staff training, and key implementation opportunities and challenges, with a particular focus on inclusion and equity. This online questionnaire will be developed using the JISC online surveys 2 platform (Google Forms is a potential open-access alternative) at the University of Oxford and will extend a survey instrument used in a previous study on video consultations across the UK in 202028. Data will be exported to Stata v17.0 for analysis (R is a potential open-access alternative). Specifically in relation to GP practices (subject to sufficient number of responses), we will examine associations between the historical deployment of group consulting, and patient experience as measured by the national GP Patient Survey (https://gp-patient.co.uk/), focusing on key conditions for which practices report carrying out group consultations (such as asthma, diabetes and cancer) and restricting data to those patients who self-report one of these conditions. Further comparisons between practices carrying out group consultations and those not doing so will be made on the basis of practice characteristics such as practice size, workforce, deprivation (using the index of multiple deprivation: https://data.cdrc.ac.uk/dataset/index-multiple-deprivation-imd), rurality and clinical quality as measured by the Quality Outcomes Framework (https://digital.nhs.uk/data-and-information/data-tools-and-services/data-services/general-practice-data-hub/quality-outcomes-framework-qof). Analysis of these differences will be made using mixed effects linear logistic models as appropriate to the outcome. A random effect model will be used to assess differences between practices unexplained by VHGCs or chance and will provide a context for the size of differences associated with VHGCs. Free text comments will be analysed using a combination of qualitative and quantitative content analysis. The survey will be distributed through multiple channels, including the Future NHS collaboration platform (e.g. community on remote and group consulting), NHS agencies and professional bodies (e.g. NHSE, NHS Scotland, NHS Wales, NHS Northern Ireland, Health Education England, RCGP, RCN), training programmes on group consulting, academic primary care networks and social media (e.g. Twitter). We will capitalise on existing professional networks with practitioners to ensure broad reach and expedite learning about existing and emerging group consulting models. Survey results will inform subsequent phases of the study e.g. in terms of site selection. The participatory co-design phase will focus on group consulting delivered remotely in general practice, i.e. video and hybrid formats (VHGCs). Drawing on principles of Experience-Based Co-Design (EBCD)29 we will use excerpts from de-identified interview data with patients and staff and recordings of group consultations (see next section) to trigger conversation and exchange in co-design workshops. To ensure co-design covers all areas where implementation and sustained use may encounter challenges, we will draw on the 7 domains of the NASSS (non-adoption, abandonment, scale-up, spread, and sustainability) framework, an evidence-informed approach to guide thinking on the implementation and sustainability of healthcare innovations30. This will help co-design participants focus discussions around opportunities for improvement and better integration of group consulting in the health service. Co-design will involve up to a total of 40 patient and public contributors, carers, clinical and non-clinical general practice staff in 4 virtual workshops (2 for patients and 2 for staff). These will be supplemented with 10 individual interviews (in-person/phone) with patient and public contributors to delve into more detail on sensitive topics and engage vulnerable or digitally excluded groups (e.g. those with sight loss, hearing loss and learning difficulties). The co-design process will be iterative, focusing on how VHGCs can be delivered to meet patient and staff needs in efficient and inclusive ways. We will draw on creative and visual techniques (e.g. persona development, journey mapping, storyboards) to facilitate discussion and collaborative design (with support from Design Science, an external design agency). Outputs will include practical guidance on setting up and delivering VHGCs which staff and patients can use flexibly (and adapt) depending on their needs. Co-design will iteratively inform the focus of data collection in WP3, especially qualitative and inclusion aspects, and emerging findings from WP3 will feed into the co-design workshops. The mixed methods process evaluation will use qualitative, quantitative and cost-related measures, based on a comparative case study design in up to 10 clinical sites in England (5 case and 5 comparison sites, primarily from general practice). Case sampling will focus on maximum variation in characteristics such as geographical location, size and digital maturity of the clinical site (digital readiness, capability and infrastructure), patient characteristics (ethnic background, digital literacy/confidence, deprivation) and VHGC delivery format and focus. i. Patient participants In up to 5 case sites we will carry out semi-structured, qualitative interviews with 40–50 patients and carers (~10 per case site), seeking to understand their views and experiences (also using interpreters where needed, to account for ethnic diversity). We will engage patients and carers who have taken part in VHGCs, as well as those who declined participation or were unable to attend sessions (e.g. due to digital exclusion). A subset of participants will be invited to follow-up interview 12 months later to reflect on longitudinal changes. Interviews will follow a flexible topic guide, will last 45–60mins, and will take place remotely (on the phone or video) or at participants’ homes, their GP practice or another preferred location. To allow participants to collectively debate their views on VHGCs, we will conduct 4–6 focus groups, each involving 6–8 patients and carers (n=24–48) who have not already participated in interviews. Each group will last 60–80 minutes and will be led by an experienced moderator using a flexible topic guide. We will aim to align some of these focus groups with existing VHGC sessions (especially when these are designed to be short) to capture immediate thoughts and feedback patients might have on what has gone well or less well. We will recruit patients across sociodemographic and ethnic backgrounds, health needs and conditions, and with different levels of self-assessed digital literacy/confidence. Groups with higher general practice consultation rates will be particularly targeted (asking each practice which groups tend to make most use of services locally), including those with long-term conditions, those with multi-morbidities, and people from underinvested communities. ii. Clinical and non-clinical NHS staff Recruitment will primarily focus on case sites, where we expect to engage at least 4–6 staff per site in semi-structured interviews (n=20–30 in total) to better understand VHGC delivery in the contact of broader organisational routines. This will include health professionals (GPs, trainees, practice nurses, healthcare assistants, pharmacists etc), practice managers, administrators, reception and IT staff. A subset of staff participants will be interviewed twice over 12 months, to be able to follow up on growing familiarity with VHGCs and any longer-term impacts on the service. Interviews will last 45 mins and will be carried out either in-person or on the phone/video. An indicative topic guide will be followed, tailored to different interviewees. We will also carry out ethnographic observation focusing on back-end operational and technical processes required to run VHGCs, to enrol patients in VHGCs and support their participation (at least 2 visits per case site over 12 months). The team will observe 25 VHGCs (virtually or in-person) across the 5 case sites (~5 each), using video or audio recordings and field notes. Any documentation, VHGC materials developed by the practice, significant event details or other relevant, de-identified records will be retained for analysis. iii. National and local policy-makers and commissioners To further our understanding of evolving policy and guidance, national-level incentives and regulation, we will conduct semi-structured, elite interviews with 5–10 key national decision-makers (e.g. NHSE). At a local level we will interview 5–7 commissioners across the 5 case site localities and observe relevant meetings, to gain further insights on VHGC impact on service planning and access. Interviews will last 45mins and will be carried out in-person or on phone/video. Relevant policy reports, public-facing content and other relevant materials will be retained for analysis. With informed consent (written or verbal), qualitative interviews, group discussions and a sample of VHGCs will be audio or video recorded, and professionally transcribed. Field notes will be taken during observation, and photos will be used to illustrate back-end operational processes. All data will be de-identified and uploaded to NVivo 12 to support data management (QualCoder is an open access alternative). Following data familiarisation and guided by our research questions, we will take an iterative approach to theory-informed, conceptual data coding in the first instance, following inductive, deductive and abductive approaches. We will compare and contrast data between cases, to develop an in-depth understanding of sociotechnical processes underpinning VHGC delivery and participation. Analysis will take place in parallel to data collection with a view to developing new theoretical insights. i. Description of VHGC delivery and resource use In each of the case sites we will collect descriptive data on the way VHGCs are delivered and the resources needed. These data will include the number and type of VHGC sessions, the number of patients participating in each VHGC and the number of patients invited to take part who choose not to participate. The data will be collected by the practice with support from the study team using structured proformas on a monthly basis. We will also collect key data, for those consultations observed by researchers, on timing and duration of VHGCs, resources needed (e.g. staff members) to deliver the sessions, and time and resources required to prepare for and follow up from each VHGC. The exact data required may depend on the way in which a practice chooses to deliver VHGCs but will likely include, the time at which the online session is started (e.g. by the ‘facilitator’), the time at which patients join the session, time spent facilitating technical issues, the time when the clinician joins the session (which in current VHGC models may be some time later than the patients), the time when the clinician leaves the session and the time when the session is ended. By examining this data we will gain an understanding of the time burden experienced by patients, GPs and other clinical and non-clinical staff, and how this changes over time. ii. Impact of VHGCs on patient experience, satisfaction and quality of life We aim to recruit up to 50 patients per practice who will consent to participate in a survey questionnaire comprising patient experience, satisfaction and healthcare-related outcome components, and to allow research access to their healthcare utilisation data. The survey will be administered online and offline, immediately following VHGC attendance (through a link shared on the same platform, e.g. Microsoft Teams, or in paper format for those attending in person), and will examine both patient experiences which relate directly to VHGCs (e.g. satisfaction with care received, inclusion and access implications, quality and safety of care, need for additional follow-up, travel time saved, time off work avoided etc.) as well as to wider experiences of healthcare in the practice (e.g. satisfaction with 1:1 care, perceptions on access, quality and safety). To develop the survey questionnaire, we will draw on questionnaires piloted in our previous research on patient experiences with video consultations and group clinics. We will also adopt or adapt items from existing surveys such as the GP Patient Survey (https://gp-patient.co.uk/) and the Primary Care Outcomes Questionnaire (PCOQ)31. We also expect new items to be developed. The findings will be used in descriptive analyses to provide a quantitative overview of patient experience of VHGCs. In addition to the case sites, we will recruit an equal number of comparison practices where VHGCs are not being employed (matched with case sites on location, setting, size, deprivation, QOF). As with case sites, we will aim to recruit 50 patients from each comparison site with recruitment targeted at similar patients as those taking part in VHGCs in terms of age, gender and chronic conditions treated. They will receive a shorter version of the patient survey used in case sites with questions relating only to wider experiences of healthcare at their practice and will be asked to consent to research access to their healthcare utilisation data. Our team will also provide support to any comparison or case sites who choose to administer the Patient Activation Measure (PAM) in the context of local service improvement. iii. Impact of VHCGs on healthcare utilisation We will work with practices to collect data (using Word proformas) on patients’ primary care utilisation including the number, type (staff members involved, e.g. GP, nurse or other healthcare professional, and mode of consultation e.g., in-person, telephone, online) and duration of appointments. We will also obtain secondary care utilisation data from NHS Digital using Hospital Episode Statistics (HES) data. Comparisons will be made between patients at case and comparison sites in terms of number and duration of primary care consultations, number of A&E attendances, outpatient appointments (including numbers or proportion of eligible patients waiting more than six months), hospital admissions, unplanned hospital admissions and hospital admissions for ambulatory care sensitive conditions (conditions for which admissions could, in principle, be avoided by good primary care32). Mixed effects models, with random intercepts for practice, will be used to analyse these data with the form of the model appropriate to the outcome (i.e. Poisson models for count of appointments/admissions and linear models for consultation durations). Adjustments will be made for patient factors including age, gender, deprivation, ethnicity and comorbidity status. Given the exploratory nature of this work we have not performed any formal power calculations and will interpret the results accordingly. Rather than focus on the presence or absence of statistically significant differences between groups, the analysis will instead examine confidence intervals as a guide to potential differences between the groups, to guide a larger, definitive study. To assess the value for money of GP models of healthcare provision with and without VHGC we will conduct an early health economic evaluation (HEE) – in the form of a model based cost-effectiveness analysis – from the perspective of the NHS and Personal Social Services. The cost-effectiveness analysis will complement the qualitative and quantitative analyses conducted in WP1-3. The health economics (HE) team will contribute to the development of quantitative and qualitative data collection tools to be used in WP1-3. Findings from WP1 (scoping survey) and WP2 (co-design) will inform the process of: i) defining the decision problem for the early economic evaluation ii) conceptualising a decision model structure that represents the decision problem; and iii) generating a set of initial model inputs for a preliminary evaluation of the decision model. Furthermore, in preparation for a large definitive trial, the early HEE study will ascertain the feasibility of (and best practice for) collecting data – at GP practice level – on healthcare resource use (e.g. venue, staffing, consultations’ duration, set-up time, technology costs) and patient reported outcomes (e.g., health related quality of life, satisfaction) associated with the delivery of VHGCs. Decision Model: conceptualisation and development of the decision model33 will be guided by the NASSS domains (i.e. clinical condition, technology, value proposition, adopter system, organisation, wider system and temporal change34 and their recent adaptation for remote consultations35). The process of conceptualising and developing the decision model will involve decomposing and characterising discrete elements of GP models of care. The decision model will enable estimation of the value for money of a GP model of healthcare provision that includes alternative types (i.e. one-to-one, group) and modes (i.e. one-to-one, group, in-person, home, video or hybrid) of GP consultations including VHGC, compared to a GP model of healthcare provision that does not include VHGC. Model inputs: the focus of the VHGC (e.g., diabetes annual reviews; peri and post menopause consultations) will be defined in consultation with case and comparison sites in WP3. A micro-costing analysis of VHGCs will be conducted in one of the five case sites in WP3 GP. Data on consultations (e.g. number and type of VHGCs) will be collected prospectively. The main source to estimate the unit cost of different types and modes of GP consultations will be the Unit Costs of Health and Social Care Manual (https://www.pssru.ac.uk/unitcostsreport/). Over an up to 6-month period, all patients attending a GP consultation (any type and mode, but with the same focus as VHGCs) in one case and one comparison site will be invited to complete an online Measures of Health Benefit survey. This will include questions on utility-based health-related quality of life (i.e. EQ-5D-5L)36, patient satisfaction (i.e. SOPS adaptation)37 and personal details (e.g. age, weight, height). Initial model inputs will be based on estimates of consultation costs, utility, and satisfaction. Given the limited data available to populate our decision model, and the need to make a number of expert-informed structural assumptions, we expect the results and conclusions derived from the early HEE to be subject to uncertainty. The impact of this on the decision uncertainty to consider VHGC a cost-effective addition to exiting models of GP healthcare provision will be explored using value of information analysis (VOI)38. This methodology will enable us to identify the model parameters (and structural assumptions) considered to be the key drivers of decision uncertainty. Results from the VOI analysis will be used to inform the design of a future definitive study of a GP model of healthcare provision with VHGCs. We are using the Non-adoption, Abandonment and Challenges to Spread, Scale-Up and Sustainability (NASSS) framework to inform study design and data collection on the multiple interacting influences affecting implementation and sustainment of VHGCs34. The framework will also contribute in crafting narrative case studies combining qualitative, quantitative and cost-related data for each site where VHGCs have been taking place. Integration of qualitative and quantitative data will inform the development of a theory of change, i.e. set of assumptions on successful delivery of VHGCs and un/anticipated implications in practice. Extending previous research on in-person and video group consulting26,27, we will draw on a number of theoretical threads from science and technology studies, medical sociology and organisational theory to underpin data analysis and synthesis. This includes theory on: a) burden of treatment and patient ‘work’ related to the responsibilities placed on patients in the context of group consultations38, b) digital exclusion and intersectionality around multiple dimensions that contribute to some people being served less well by remote means of group-based care39,40, c) health service complexity including a departure from linear cause-effect relationships to recognising interdependence and emergence in group-based care delivery41,42 and d) experiential knowledge mobilised to support clinical care10,12. Approval has been granted by the London Hampstead Research Ethics Committee and UK Health Research Authority, reference number 22/PR/0277 (19 May 2022), and subsequent amendments. The study is overseen by an independent external advisory group with a lay chair and diverse representation from policy, clinical care, the commercial sector, and people with experience on patient advocacy groups or regulatory bodies. We plan a number of key outputs including: a) empirical academic articles and conference presentations, b) co-designed guidance for patients and service providers on developing effective and inclusive video and hybrid group consulting, c) policy briefings around supporting equitable group-based digital services. We will facilitate knowledge sharing and capacity building across participating sites through joint co-design sessions, learning reports, and virtual learning events to share formative and summative evaluation findings and drive local improvement. We aim to develop transferable learning on sociotechnical change in healthcare, using VHGCs as an exemplar of technology-supported innovation. Findings will inform policy and practice, as well as the design of a future definitive study. Faculty Opinions recommendedReferences - 1. Cooksley T, Clarke S, Dean J, et al.: NHS crisis: rebuilding the NHS needs urgent action. BMJ. 2023; 380: 1. PubMed Abstract | Publisher Full Text - 2. UK Government: Health and care act 2022. 2022. Reference Source - 3. NHS England: NHS long term plan. London, 2019. Reference Source - 4. Greenhalgh T, Rosen R: Remote by default general practice: must we, should we, dare we? Br J Gen Pract. 2021; 71(705): 149–50. PubMed Abstract | Publisher Full Text | Free Full Text - 5. Mroz G, Papoutsi C, Greenhalgh T: From ‘a rapid and necessary revolution’ to ‘telemedicine killed the PE teacher’: changing representations of remote GP consultations in UK media during the COVID-19 pandemic. In: Lewis M, Govender E, Holland K, editors. Communicating COVID-19: Media, Trust, and Public Engagement. Cham: Springer International Publishing, 2024; 125–44. Publisher Full Text - 6. Shaw SE, Hughes G, Wherton J, et al.: Achieving spread, scale up and sustainability of video consulting services during the COVID-19 pandemic? findings from a comparative case study of policy implementation in England, Wales, Scotland and Northern Ireland. Front Digit Health. 2021; 3: 754319. PubMed Abstract | Publisher Full Text | Free Full Text - 7. Greenhalgh T, Shaw SE, Alvarez Nishio A, et al.: Protocol: remote care as the ‘new normal’? Multi-site case study in UK general practice [version 1; peer review: 3 approved]. NIHR Open Res. 2022; 2: 46. PubMed Abstract | Publisher Full Text | Free Full Text - 8. Veinot T: Power to the patient? a critical examination of patient empowerment discourses. In: Harris R, Wathen N, Wyatt S, editors. Configuring Health Consumers: Health Work and the Imperative of Personal Responsibility. London: Palgrave Macmillan UK, 2010; 30–41. Publisher Full Text - 9. Mazanderani F, Locock L, Powell J: Biographical value: towards a conceptualisation of the commodification of illness narratives in contemporary healthcare. Sociol Health Illn. 2013; 35(6): 891–905. PubMed Abstract | Publisher Full Text - 10. Halloy A, Simon E, Hejoaka F: Defining patient’s experiential knowledge: who, what and how patients know. A narrative critical review. Sociol Health Illn. 2023; 45(2): 405–22. PubMed Abstract | Publisher Full Text | Free Full Text - 11. Mazanderani F, Locock L, Powell J: Being differently the same: the mediation of identity tensions in the sharing of illness experiences. Soc Sci Med. 2012; 74(4): 546–53. PubMed Abstract | Publisher Full Text - 12. Mazanderani F, Noorani T, Dudhwala F, et al.: Knowledge, evidence, expertise? The epistemics of experience in contemporary healthcare. Evid Policy. 2020; 16(2): 267–84. Publisher Full Text - 13. Hayhoe B, Verma A, Kumar S: Shared medical appointments. BMJ. 2017; 358: j4034. PubMed Abstract | Publisher Full Text - 14. Papoutsi C, Colligan G, Hagell A, et al.: Promises and perils of group clinics for young people living with diabetes: a realist review. Diabetes Care. 2019; 42(5): 705–12. PubMed Abstract | Publisher Full Text - 15. Trento M, Gamba S, Gentile L, et al.: Rethink Organization to iMprove Education and Outcomes (ROMEO): a multicenter randomized trial of lifestyle intervention by group care to manage type 2 diabetes. Diabetes Care. 2010; 33(4): 745–7. PubMed Abstract | Publisher Full Text | Free Full Text - 16. Wong SH, Barrow N, Hall K, et al.: The Effective management of idiopathic intracranial hypertension delivered by in-person and virtual group consultations: results and reflections from a phase one service delivery. Neuroophthalmology. 2021; 45(4): 246–52. PubMed Abstract | Publisher Full Text | Free Full Text - 17. Graham F, Tang MY, Jackson K, et al.: Barriers and facilitators to implementation of shared medical appointments in primary care for the management of long-term conditions: a systematic review and synthesis of qualitative studies. BMJ Open. 2021; 11(8): e046842. PubMed Abstract | Publisher Full Text | Free Full Text - 18. Swaithes L, Paskins Z, Duffy H, et al.: Experience of implementing and delivering group consultations in UK general practice: a qualitative study. Br J Gen Pract. 2021; 71(707): e413–e422. PubMed Abstract | Publisher Full Text | Free Full Text - 19. Graham F, Martin H, Lecouturier J, et al.: Shared medical appointments in english primary care for long-term conditions: a qualitative study of the views and experiences of patients, primary care staff and other stakeholders. BMC Prim Care. 2022; 23(1): 180. PubMed Abstract | Publisher Full Text | Free Full Text - 20. Booth A, Cantrell A, Preston L, et al.: What is the evidence for the effectiveness, appropriateness and feasibility of group clinics for patients with chronic conditions? A systematic review. Health Serv Delivery Res. 2015; 3(46). PubMed Abstract | Publisher Full Text - 21. Nederveld A, Phimphasone-Brady P, Gurfinkel D, et al.: Delivering diabetes shared medical appointments in primary care: early and mid-program adaptations and implications for successful implementation. BMC Prim Care. 2023; 24(1): 52. PubMed Abstract | Publisher Full Text | Free Full Text - 22. Scott E, Swaithes L, Wynne-Jones G, et al.: Use of video group consultations by general practice staff during the COVID-19 pandemic. Prim Health Care. 2023; 33(3). Publisher Full Text - 23. Blatgé H, Shanmugam S, Slattery B, et al.: Patient perception of group consultations for knee osteoarthritis and influence of area-level deprivation: a service evaluation in a physiotherapy musculoskeletal setting. Musculoskeletal Care. 2024; 22(1): e1870. PubMed Abstract | Publisher Full Text - 24. Papoutsi C, Hargreaves D, Colligan G, et al.: Group clinics for young adults with diabetes in an ethnically diverse, socioeconomically deprived setting (TOGETHER study): protocol for a realist review, co-design and mixed methods, participatory evaluation of a new care model. BMJ Open. 2017; 7(6): e017363. PubMed Abstract | Publisher Full Text | Free Full Text - 25. Papoutsi C, Hargreaves D, Hagell A, et al.: Group clinics for young adults living with diabetes in an ethnically diverse, socioeconomically deprived population: mixed-methods evaluation. National Institute for Health and Care Research. Southampton (UK), 2022. PubMed Abstract | Publisher Full Text - 26. Papoutsi C, Hargreaves D, Hagell A, et al.: Implementation and delivery of group consultations for young people with diabetes in socioeconomically deprived, ethnically diverse settings. BMC Med. 2022; 20(1): 459. PubMed Abstract | Publisher Full Text | Free Full Text - 27. Papoutsi C, Shaw S, Greenhalgh T: Implementing video group consultations in general practice during COVID-19: a qualitative study. Br J Gen Pract. 2022; 72(720): e483–e91. PubMed Abstract | Publisher Full Text | Free Full Text - 28. Hughes G, Wood G, Shaw S: Video consultations in the NHS: summary findings from a UK-wide survey. 2020. Reference Source - 29. Donetto S, Tsianakas V, Robert G: Using Experience-Based Co-Design (EBCD) to improve the quality of healthcare: mapping where we are now and establishing future directions. London: King’s College London, 2014. Reference Source - 30. The King’s Fund: Experience-Based Co-Design toolkit 2013. Reference Source - 31. Murphy M, Hollinghurst S, Cowlishaw S, et al.: Primary Care Outcomes Questionnaire: psychometric testing of a new instrument. Br J Gen Pract. 2018; 68(671): e433–e40. PubMed Abstract | Publisher Full Text | Free Full Text - 32. Bardsley M, Blunt I, Davies S, et al.: Is secondary preventive care improving? Observational study of 10-year trends in emergency admissions for conditions amenable to ambulatory care. BMJ open. 2013; 3(1): e002007. PubMed Abstract | Publisher Full Text | Free Full Text - 33. Roberts M, Russell LB, Paltiel AD, et al.: Conceptualizing a model: a report of the ISPOR-SMDM modeling good research practices task force–2. Med Decis Making. 2012; 32(5): 678–89. PubMed Abstract | Publisher Full Text - 34. Greenhalgh T, Wherton J, Papoutsi C, et al.: Beyond adoption: a new framework for theorizing and evaluating Nonadoption, Abandonment, and challenges to the Scale-Up, Spread, and Sustainability of health and care technologies (NASSS framework). J Med Internet Res. 2017; 19(11): e367. PubMed Abstract | Publisher Full Text | Free Full Text - 35. Greenhalgh T, Shaw S, Byng R: Planning and Evaluating Remote Consultation Services (PERCS): a new conceptual framework incorporating complexity and practical ethics. Front Digit Health. under review. - 36. Herdman M, Gudex C, Lloyd A, et al.: Development and preliminary testing of the new five-level version of EQ-5D (EQ-5D-5L). Qual Life Res. 2011; 20(10): 1727–36. PubMed Abstract | Publisher Full Text | Free Full Text - 37. Salisbury C, Burgess A, Lattimer V, et al.: Developing a standard short questionnaire for the assessment of patient satisfaction with out-of-hours primary care. Fam Pract. 2005; 22(5): 560–9. PubMed Abstract | Publisher Full Text - 38. Veinot TC, Mitchell H, Ancker JS: Good intentions are not enough: how informatics interventions can worsen inequality. J Am Med Inform Assoc. 2018; 25(8): 1080–8. PubMed Abstract | Publisher Full Text | Free Full Text - 39. May CR, Eton DT, Boehmer K, et al.: Rethinking the patient: using Burden of Treatment Theory to understand the changing dynamics of illness. BMC Health Serv Res. 2014; 14(1): 281. PubMed Abstract | Publisher Full Text | Free Full Text - 40. Zheng Y, Walsham G: Inequality of what? An intersectional approach to digital inequality under COVID-19. Information and Organization. 2021; 31(1): 100341. Publisher Full Text | Free Full Text - 41. Greenhalgh T, Papoutsi C: Spreading and scaling up innovation and improvement. BMJ. 2019; 365: l2068. PubMed Abstract | Publisher Full Text | Free Full Text - 42. Greenhalgh T, Papoutsi C: Studying complexity in health services research: desperately seeking an overdue paradigm shift. BMC Med. 2018; 16(1): 95. PubMed Abstract | Publisher Full Text | Free Full Text Author details Author details 1 Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, England, OX26GG, UK 2 Department of Health and Community Science, Smeall Building, St Luke’s Campus, University of Exeter, Exeter, England, EX1 2LU, UK 3 Department of Health Sciences, Seebohm Rowntree Building, University of York, York, England, YO10 5SQ, UK 4 Independent public contributor, UK, UK 5 School of Human & Health Sciences, University of Huddersfield, Huddersfield, England, UK 2 Department of Health and Community Science, Smeall Building, St Luke’s Campus, University of Exeter, Exeter, England, EX1 2LU, UK 3 Department of Health Sciences, Seebohm Rowntree Building, University of York, York, England, YO10 5SQ, UK 4 Independent public contributor, UK, UK 5 School of Human & Health Sciences, University of Huddersfield, Huddersfield, England, UK Chrysanthi Papoutsi Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing Gary Abel Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing Cynthia Iglesias Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Supervision, Writing – Original Draft Preparation, Writing – Review & Editing Jackie van Dael Roles: Investigation, Writing – Review & Editing Roles: Investigation, Writing – Review & Editing Claire Reidy Roles: Investigation, Writing – Review & Editing Roles: Investigation, Writing – Review & Editing Stuart D Faulkner Roles: Project Administration, Writing – Review & Editing Roles: Project Administration, Writing – Review & Editing Helene Raynsford Roles: Conceptualization, Writing – Review & Editing Roles: Conceptualization, Writing – Review & Editing Michele Siciliano Roles: Investigation, Writing – Review & Editing Roles: Investigation, Writing – Review & Editing Luis Beltran Galindo Roles: Investigation, Writing – Review & Editing Roles: Investigation, Writing – Review & Editing Vijay Gc Roles: Investigation, Writing – Review & Editing Roles: Investigation, Writing – Review & Editing John Campbell Roles: Conceptualization, Funding Acquisition, Methodology, Writing – Review & Editing Roles: Conceptualization, Funding Acquisition, Methodology, Writing – Review & Editing Trisha Greenhalgh Roles: Conceptualization, Funding Acquisition, Writing – Review & Editing Roles: Conceptualization, Funding Acquisition, Writing – Review & Editing Sara E Shaw Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Supervision, Writing – Review & Editing Roles: Conceptualization, Funding Acquisition, Investigation, Methodology, Supervision, Writing – Review & Editing Competing interests No competing interests were disclosed. Grant information The project is funded by the National Institute for Health and Care Research (NIHR) under its Health and Social Care Delivery Research (HSDR) Programme (Grant Reference Number NIHR133895). The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Article Versions (2) Copyright © 2024 Papoutsi C et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. metrics VIEWS $counts.viewCount downloads Citations CITE how to cite this article Papoutsi C, Abel G, Iglesias C et al. Evaluating video and hybrid group consultations in general practice: mixed-methods, participatory study protocol (TOGETHER 2) [version 2; peer review: 2 approved]. NIHR Open Res 2024, 4:30 (https://doi.org/10.3310/nihropenres.13584.2) NOTE: If applicable, it is important to ensure the information in square brackets after the title is included in all citations of this article. track receive updates on this article Track an article to receive email alerts on any updates to this article. Current Reviewer Status: ? Key to Reviewer Statuses VIEW HIDE ApprovedThe paper is scientifically sound in its current form and only minor, if any, improvements are suggested Approved with reservations A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit. Not approvedFundamental flaws in the paper seriously undermine the findings and conclusions Version 2 VERSION 2 PUBLISHED 18 Sep 2024 Revised Views 0 How to cite this report: Job J. Reviewer Report For: Evaluating video and hybrid group consultations in general practice: mixed-methods, participatory study protocol (TOGETHER 2) [version 2; peer review: 2 approved]. NIHR Open Res 2024, 4:30 (https://doi.org/10.3310/nihropenres.14918.r32965) The direct URL for this report is: https://openresearch.nihr.ac.uk/articles/4-30/v2#referee-response-32965 https://openresearch.nihr.ac.uk/articles/4-30/v2#referee-response-32965 NOTE: it is important to ensure the information in square brackets after the title is included in this citation. Reviewer Report 20 Sep 2024 Approved VIEWS 0 Thank you. I approve the manuscript ... Continue reading I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Close Thank you. I approve the manuscript for indexing. Kind regards, Jenny Kind regards, Jenny Competing Interests: No competing interests were disclosed. Reviewer Expertise: Primary care research, implementation science CITE HOW TO CITE THIS REPORT Job J. Reviewer Report For: Evaluating video and hybrid group consultations in general practice: mixed-methods, participatory study protocol (TOGETHER 2) [version 2; peer review: 2 approved]. NIHR Open Res 2024, 4:30 (https://doi.org/10.3310/nihropenres.14918.r32965) The direct URL for this report is: https://openresearch.nihr.ac.uk/articles/4-30/v2#referee-response-32965 https://openresearch.nihr.ac.uk/articles/4-30/v2#referee-response-32965 NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article. Views 0 How to cite this report: Dahl Larsen R. Reviewer Report For: Evaluating video and hybrid group consultations in general practice: mixed-methods, participatory study protocol (TOGETHER 2) [version 2; peer review: 2 approved]. NIHR Open Res 2024, 4:30 (https://doi.org/10.3310/nihropenres.14918.r32964) The direct URL for this report is: https://openresearch.nihr.ac.uk/articles/4-30/v2#referee-response-32964 https://openresearch.nihr.ac.uk/articles/4-30/v2#referee-response-32964 NOTE: it is important to ensure the information in square brackets after the title is included in this citation. Reviewer Report 19 Sep 2024 Rasmus Dahl Larsen, Southern Denmark University, Odense, Denmark Approved VIEWS 0 ... Continue reading I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Close Approved Competing Interests: No competing interests were disclosed. Reviewer Expertise: Specialist in General Practice, MD, Phd-student in Video group consultation in GP and partycipatory design. CITE HOW TO CITE THIS REPORT Dahl Larsen R. Reviewer Report For: Evaluating video and hybrid group consultations in general practice: mixed-methods, participatory study protocol (TOGETHER 2) [version 2; peer review: 2 approved]. NIHR Open Res 2024, 4:30 (https://doi.org/10.3310/nihropenres.14918.r32964) The direct URL for this report is: https://openresearch.nihr.ac.uk/articles/4-30/v2#referee-response-32964 https://openresearch.nihr.ac.uk/articles/4-30/v2#referee-response-32964 NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article. Version 1 VERSION 1 PUBLISHED 10 May 2024 Views 0 How to cite this report: Dahl Larsen R. Reviewer Report For: Evaluating video and hybrid group consultations in general practice: mixed-methods, participatory study protocol (TOGETHER 2) [version 2; peer review: 2 approved]. NIHR Open Res 2024, 4:30 (https://doi.org/10.3310/nihropenres.14745.r31861) The direct URL for this report is: https://openresearch.nihr.ac.uk/articles/4-30/v1#referee-response-31861 https://openresearch.nihr.ac.uk/articles/4-30/v1#referee-response-31861 NOTE: it is important to ensure the information in square brackets after the title is included in this citation. Reviewer Report 13 Jun 2024 Rasmus Dahl Larsen, Southern Denmark University, Odense, Denmark Approved VIEWS 0 Thank you for the opportunity to review the study titled: Evaluating Video and Hybrid Group Consultations in General Practice: Mixed-Methods, Participatory Study Protocol. Overall, this PPI co-designed protocol is well-developed and appears very comprehensive, uncovering patients`, general practitioners', practice staffs ... Continue reading I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard. Close Thank you for the opportunity to review the study titled: Evaluating Video and Hybrid Group Consultations in General Practice: Mixed-Methods, Participatory Study Protocol. Overall, this PPI co-designed protocol is well-developed and appears very comprehensive, uncovering patients`, general practitioners', practice staffs and stakeholders’ thoughts and experiences with the use of video and hybrid group consultations in general practice and for which conditions group consultation has been used. Additionally, it addresses issues related to time resources and the economic aspects of using video and hybrid group consultations in general practice. The study is divided into three work packages, which is a logical approach. WP1 is exploratory, WP2 involves designing and testing, and WP3 focuses on further testing and evaluation. However, the connections (the common thread) between these packages could be made more visible, specifically if the work packages are interdependent and if yes, in what way. WP1: The questionnaire appears relevant for addressing clear questions regarding which chronic conditions and the overall setup of how video/hybrid consultations have been used and in which context. It also allows for comparisons with previous satisfaction surveys. Perhaps the authors could consider the following: - It might be interesting to involve PPI in the development and testing of this questionnaire. - Explain the purpose of the questionnaire and the specific information being gathered. - Describe how the data gathered in WP1 will be utilized and integrated into WP2. - Briefly explain what the GP survey is - As well as the OQF. - Consider using semi-structured interviews to further explore stakeholders’ answers from the questionnaire - what works and what doesn't, such as disease groups, practice and patient characteristics, etc. That could be a good preparation for WP2. WP2: Is very interesting and will reveal valuable insights about what works well in group settings for both patients and healthcare personnel. It is also beneficial to explore this for the vulnerable patient group, who can greatly benefit from peer-to-peer support. Perhaps the authors could consider the following: WP3: The evaluation of the project appears interesting, with important evaluation points such as health economics, sociotechnical change and dynamics, and an analysis of remote group consultations. - Is it the aim to develop a "golden standard of remote group consultations" in WP2, and then test it and compare it with the five control GPs in WP3? Its not clear for the reader. - is it possible to estimate the expected data volume for the quantitative part? Can robust and meaningful conclusions be made based on the quantitative data set? - The above reflections are also relevant for ensuring an accurate health economic analysis. - It could be elaborated upon what is alluded at with “surface assumptions”. Moreover, “sociotechnical change” as a concept and the various theories mentioned at the end of the review could be described in more detail, not least why these are found relevant. Overall: Furthermore, the authors could consider incorporating even more Patient and Public Involvement (PPI). For example, involving patients as co-creators in the development of the questionnaire in WP1, interview guide, and workshop, as well as in deciding which conditions and setups should be investigated, could provide valuable insights, and improve the study's relevance and impact. The study is divided into three work packages, which is a logical approach. WP1 is exploratory, WP2 involves designing and testing, and WP3 focuses on further testing and evaluation. However, the connections (the common thread) between these packages could be made more visible, specifically if the work packages are interdependent and if yes, in what way. WP1: The questionnaire appears relevant for addressing clear questions regarding which chronic conditions and the overall setup of how video/hybrid consultations have been used and in which context. It also allows for comparisons with previous satisfaction surveys. Perhaps the authors could consider the following: - It might be interesting to involve PPI in the development and testing of this questionnaire. - Explain the purpose of the questionnaire and the specific information being gathered. - Describe how the data gathered in WP1 will be utilized and integrated into WP2. - Briefly explain what the GP survey is - As well as the OQF. - Consider using semi-structured interviews to further explore stakeholders’ answers from the questionnaire - what works and what doesn't, such as disease groups, practice and patient characteristics, etc. That could be a good preparation for WP2. WP2: Is very interesting and will reveal valuable insights about what works well in group settings for both patients and healthcare personnel. It is also beneficial to explore this for the vulnerable patient group, who can greatly benefit from peer-to-peer support. Perhaps the authors could consider the following: - Provide a more detailed description of the workshop setup. Will it involve only patients? discussing with other patients, or will there be mixed groups of patients and GPs, staff members and stakeholders? - Is there a plan to conduct a mock-up remote group consultation and evaluation of it? - Once again, how will the results from WP1 be integrated into WP2. - How will the results from WP2 be used in WP3 – if at all? WP3: The evaluation of the project appears interesting, with important evaluation points such as health economics, sociotechnical change and dynamics, and an analysis of remote group consultations. - Is it the aim to develop a "golden standard of remote group consultations" in WP2, and then test it and compare it with the five control GPs in WP3? Its not clear for the reader. - is it possible to estimate the expected data volume for the quantitative part? Can robust and meaningful conclusions be made based on the quantitative data set? - The above reflections are also relevant for ensuring an accurate health economic analysis. - It could be elaborated upon what is alluded at with “surface assumptions”. Moreover, “sociotechnical change” as a concept and the various theories mentioned at the end of the review could be described in more detail, not least why these are found relevant. Overall: Furthermore, the authors could consider incorporating even more Patient and Public Involvement (PPI). For example, involving patients as co-creators in the development of the questionnaire in WP1, interview guide, and workshop, as well as in deciding which conditions and setups should be investigated, could provide valuable insights, and improve the study's relevance and impact. - Is the rationale for, and objectives of, the study clearly described? Yes - Is the study design appropriate for the research question? Yes - Are sufficient details of the methods provided to allow replication by others? Yes - Are the datasets clearly presented in a useable and accessible format? Yes Competing Interests: No competing interests were disclosed. Reviewer Expertise: Specialist in General Practice, MD, Phd-student in Video group consultation in GP and partycipatory design. CITE HOW TO CITE THIS REPORT Dahl Larsen R. Reviewer Report For: Evaluating video and hybrid group consultations in general practice: mixed-methods, participatory study protocol (TOGETHER 2) [version 2; peer review: 2 approved]. NIHR Open Res 2024, 4:30 (https://doi.org/10.3310/nihropenres.14745.r31861) The direct URL for this report is: https://openresearch.nihr.ac.uk/articles/4-30/v1#referee-response-31861 https://openresearch.nihr.ac.uk/articles/4-30/v1#referee-response-31861 NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article. - Author Response 18 Sep 2024Chrysanthi Papoutsi, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, OX26GG, UK18 Sep 2024Author ResponsePlease see below our point-by-point response to reviewer 2 comments: Thank you for the opportunity to review the study titled: Evaluating Video and Hybrid Group Consultations in General Practice: ... Continue reading Please see below our point-by-point response to reviewer 2 comments: Thank you for the opportunity to review the study titled: Evaluating Video and Hybrid Group Consultations in General Practice: Mixed-Methods, Participatory Study Protocol. Overall, this PPI co-designed protocol is well-developed and appears very comprehensive, uncovering patients`, general practitioners', practice staffs and stakeholders’ thoughts and experiences with the use of video and hybrid group consultations in general practice and for which conditions group consultation has been used. Additionally, it addresses issues related to time resources and the economic aspects of using video and hybrid group consultations in general practice. Thank you The study is divided into three work packages, which is a logical approach. WP1 is exploratory, WP2 involves designing and testing, and WP3 focuses on further testing and evaluation. However, the connections (the common thread) between these packages could be made more visible, specifically if the work packages are interdependent and if yes, in what way. Response: We have added further clarification: a) for WP1 ‘Survey results will inform subsequent phases of the study e.g. in terms of site selection’ in WP3, b) for WP2 ‘Co-design will iteratively inform the focus of data collection in WP3, especially qualitative and inclusion aspects, and emerging findings from WP3 will feed into the co-design workshops’, c) for WP3c the protocol already stated ‘Findings from WP1 (scoping survey) and WP2 (co-design) will inform the process of: i) defining the decision problem for the early economic evaluation ii) conceptualising a decision model structure that represents the decision problem; and iii) generating a set of initial model inputs for a preliminary evaluation of the decision model.’ Further details on interdependencies that are emerging as the project evolves will be included in the final findings paper. WP1: The questionnaire appears relevant for addressing clear questions regarding which chronic conditions and the overall setup of how video/hybrid consultations have been used and in which context. It also allows for comparisons with previous satisfaction surveys. Perhaps the authors could consider the following: - It might be interesting to involve PPI in the development and testing of this questionnaire. Response: As we explain the questionnaire extends a survey used in a previous study on video consultations in 2020 – both the previous study as well as the one described in this protocol have been informed by PPI contribution and this has included input to some of the survey questions, e.g. on inclusion. This is described in the findings paper for the study currently in development. - Explain the purpose of the questionnaire and the specific information being gathered. Response: The protocol explains as follows: ‘The scoping survey will provide a picture of how group consulting was deployed as a remote model of care in the context of the pandemic (2020-21), and how it has been used in the recovery context (2021-22), including the types of conditions and patient populations for which it has been used, the types of consultations carried out in groups (e.g. annual reviews, health checks), frequency and format of sessions (e.g. in-person or remote delivery mode, inclusion of underserved communities), staff training, and key implementation opportunities and challenges, with a particular focus on inclusion and equity.’ - Describe how the data gathered in WP1 will be utilized and integrated into WP2. Response: We have now explained: ‘Survey results will inform subsequent phases of the study e.g. in terms of site selection’ - Briefly explain what the GP survey is - As well as the OQF. Response: We have provided relevant reference. - Consider using semi-structured interviews to further explore stakeholders’ answers from the questionnaire - what works and what doesn't, such as disease groups, practice and patient characteristics, etc. That could be a good preparation for WP2. Response: The study includes relevant interviews in WP3. WP2: Is very interesting and will reveal valuable insights about what works well in group settings for both patients and healthcare personnel. It is also beneficial to explore this for the vulnerable patient group, who can greatly benefit from peer-to-peer support. Perhaps the authors could consider the following: - Provide a more detailed description of the workshop setup. Will it involve only patients? discussing with other patients, or will there be mixed groups of patients and GPs, staff members and stakeholders? Response: We have added this is 2 workshops for patients and 2 for staff. - Is there a plan to conduct a mock-up remote group consultation and evaluation of it? Response: This is an interesting idea – we used de-identified clips from WP3 to trigger discussion and will provide further explanation on this in the findings papers out of the study. The protocol states: ‘…we will use excerpts from de-identified interview data with patients and staff and recordings of group consultations (see next section) to trigger conversation and exchange in co-design workshops’ - Once again, how will the results from WP1 be integrated into WP2. Response: The results from WP1 mainly informed site selection and provided links to relevant stakeholders who were invited to the co-design workshops, as per the response in the relevant comment above. - How will the results from WP2 be used in WP3 – if at all? Response: We have provided relevant clarification (as explained above): ‘Co-design will iteratively inform the focus of data collection in WP3, especially qualitative and inclusion aspects, and emerging findings from WP3 will feed into the co-design workshops.’ WP3: The evaluation of the project appears interesting, with important evaluation points such as health economics, sociotechnical change and dynamics, and an analysis of remote group consultations. - Is it the aim to develop a "golden standard of remote group consultations" in WP2, and then test it and compare it with the five control GPs in WP3? Its not clear for the reader. Response: As the proposal suggests WP2 will develop practical guidance on setting up and delivering VHGCs – we do not believe there is a gold standard and only intend to develop guidance that staff and patients can use flexibly (and adapt) depending on their needs. We have now clarified this in the relevant section. - is it possible to estimate the expected data volume for the quantitative part? Can robust and meaningful conclusions be made based on the quantitative data set? Response: We have provided indicative numbers based on experience – the study is not looking to draw definitive conclusions based on quantitative data, rather we are looking to consolidate data across methods supported by theory to enhance the value of this work. Group consultations remain a relatively new model of care with remote delivery reaching relatively low volumes, therefore we are not expecting to be able to reach significant numbers at this stage and the study has been designed accordingly. - The above reflections are also relevant for ensuring an accurate health economic analysis. Please see response above. - It could be elaborated upon what is alluded at with “surface assumptions”. Moreover, “sociotechnical change” as a concept and the various theories mentioned at the end of the review could be described in more detail, not least why these are found relevant. Response: We have clarified what we mean by sociotechnical change also in response to reviewer 1 comments. We have also elaborated that assumptions relate to what these models are meant to achieve and provided details on the theories mentioned: ‘This includes theory on: a) burden of treatment and patient ‘work’ related to the responsibilities placed on patients in the context of group consultations (38), b) digital exclusion and intersectionality around multiple dimensions that contribute to some people being served less well by remote means of group-based care (39, 40), c) health service complexity including a departure from linear cause-effect relationships to recognising interdependence and emergence in group-based care delivery (41, 42) and d) experiential knowledge mobilised to support clinical care (10, 12)’. Overall: Furthermore, the authors could consider incorporating even more Patient and Public Involvement (PPI). For example, involving patients as co-creators in the development of the questionnaire in WP1, interview guide, and workshop, as well as in deciding which conditions and setups should be investigated, could provide valuable insights, and improve the study's relevance and impact. Response: Thank you for this helpful point – our evaluation is underpinned by participatory principles. In the protocol we have only provided some examples for PPI contribution. The findings papers will provide more detail on specific PPI activities, including the ones mentioned in the comment above.Please see below our point-by-point response to reviewer 2 comments:Competing Interests: n/a Close Thank you for the opportunity to review the study titled: Evaluating Video and Hybrid Group Consultations in General Practice: Mixed-Methods, Participatory Study Protocol. Overall, this PPI co-designed protocol is well-developed and appears very comprehensive, uncovering patients`, general practitioners', practice staffs and stakeholders’ thoughts and experiences with the use of video and hybrid group consultations in general practice and for which conditions group consultation has been used. Additionally, it addresses issues related to time resources and the economic aspects of using video and hybrid group consultations in general practice. Thank you The study is divided into three work packages, which is a logical approach. WP1 is exploratory, WP2 involves designing and testing, and WP3 focuses on further testing and evaluation. However, the connections (the common thread) between these packages could be made more visible, specifically if the work packages are interdependent and if yes, in what way. Response: We have added further clarification: a) for WP1 ‘Survey results will inform subsequent phases of the study e.g. in terms of site selection’ in WP3, b) for WP2 ‘Co-design will iteratively inform the focus of data collection in WP3, especially qualitative and inclusion aspects, and emerging findings from WP3 will feed into the co-design workshops’, c) for WP3c the protocol already stated ‘Findings from WP1 (scoping survey) and WP2 (co-design) will inform the process of: i) defining the decision problem for the early economic evaluation ii) conceptualising a decision model structure that represents the decision problem; and iii) generating a set of initial model inputs for a preliminary evaluation of the decision model.’ Further details on interdependencies that are emerging as the project evolves will be included in the final findings paper. WP1: The questionnaire appears relevant for addressing clear questions regarding which chronic conditions and the overall setup of how video/hybrid consultations have been used and in which context. It also allows for comparisons with previous satisfaction surveys. Perhaps the authors could consider the following: - It might be interesting to involve PPI in the development and testing of this questionnaire. Response: As we explain the questionnaire extends a survey used in a previous study on video consultations in 2020 – both the previous study as well as the one described in this protocol have been informed by PPI contribution and this has included input to some of the survey questions, e.g. on inclusion. This is described in the findings paper for the study currently in development. - Explain the purpose of the questionnaire and the specific information being gathered. Response: The protocol explains as follows: ‘The scoping survey will provide a picture of how group consulting was deployed as a remote model of care in the context of the pandemic (2020-21), and how it has been used in the recovery context (2021-22), including the types of conditions and patient populations for which it has been used, the types of consultations carried out in groups (e.g. annual reviews, health checks), frequency and format of sessions (e.g. in-person or remote delivery mode, inclusion of underserved communities), staff training, and key implementation opportunities and challenges, with a particular focus on inclusion and equity.’ - Describe how the data gathered in WP1 will be utilized and integrated into WP2. Response: We have now explained: ‘Survey results will inform subsequent phases of the study e.g. in terms of site selection’ - Briefly explain what the GP survey is - As well as the OQF. Response: We have provided relevant reference. - Consider using semi-structured interviews to further explore stakeholders’ answers from the questionnaire - what works and what doesn't, such as disease groups, practice and patient characteristics, etc. That could be a good preparation for WP2. Response: The study includes relevant interviews in WP3. WP2: Is very interesting and will reveal valuable insights about what works well in group settings for both patients and healthcare personnel. It is also beneficial to explore this for the vulnerable patient group, who can greatly benefit from peer-to-peer support. Perhaps the authors could consider the following: - Provide a more detailed description of the workshop setup. Will it involve only patients? discussing with other patients, or will there be mixed groups of patients and GPs, staff members and stakeholders? Response: We have added this is 2 workshops for patients and 2 for staff. - Is there a plan to conduct a mock-up remote group consultation and evaluation of it? Response: This is an interesting idea – we used de-identified clips from WP3 to trigger discussion and will provide further explanation on this in the findings papers out of the study. The protocol states: ‘…we will use excerpts from de-identified interview data with patients and staff and recordings of group consultations (see next section) to trigger conversation and exchange in co-design workshops’ - Once again, how will the results from WP1 be integrated into WP2. Response: The results from WP1 mainly informed site selection and provided links to relevant stakeholders who were invited to the co-design workshops, as per the response in the relevant comment above. - How will the results from WP2 be used in WP3 – if at all? Response: We have provided relevant clarification (as explained above): ‘Co-design will iteratively inform the focus of data collection in WP3, especially qualitative and inclusion aspects, and emerging findings from WP3 will feed into the co-design workshops.’ WP3: The evaluation of the project appears interesting, with important evaluation points such as health economics, sociotechnical change and dynamics, and an analysis of remote group consultations. - Is it the aim to develop a "golden standard of remote group consultations" in WP2, and then test it and compare it with the five control GPs in WP3? Its not clear for the reader. Response: As the proposal suggests WP2 will develop practical guidance on setting up and delivering VHGCs – we do not believe there is a gold standard and only intend to develop guidance that staff and patients can use flexibly (and adapt) depending on their needs. We have now clarified this in the relevant section. - is it possible to estimate the expected data volume for the quantitative part? Can robust and meaningful conclusions be made based on the quantitative data set? Response: We have provided indicative numbers based on experience – the study is not looking to draw definitive conclusions based on quantitative data, rather we are looking to consolidate data across methods supported by theory to enhance the value of this work. Group consultations remain a relatively new model of care with remote delivery reaching relatively low volumes, therefore we are not expecting to be able to reach significant numbers at this stage and the study has been designed accordingly. - The above reflections are also relevant for ensuring an accurate health economic analysis. Please see response above. - It could be elaborated upon what is alluded at with “surface assumptions”. Moreover, “sociotechnical change” as a concept and the various theories mentioned at the end of the review could be described in more detail, not least why these are found relevant. Response: We have clarified what we mean by sociotechnical change also in response to reviewer 1 comments. We have also elaborated that assumptions relate to what these models are meant to achieve and provided details on the theories mentioned: ‘This includes theory on: a) burden of treatment and patient ‘work’ related to the responsibilities placed on patients in the context of group consultations (38), b) digital exclusion and intersectionality around multiple dimensions that contribute to some people being served less well by remote means of group-based care (39, 40), c) health service complexity including a departure from linear cause-effect relationships to recognising interdependence and emergence in group-based care delivery (41, 42) and d) experiential knowledge mobilised to support clinical care (10, 12)’. Overall: Furthermore, the authors could consider incorporating even more Patient and Public Involvement (PPI). For example, involving patients as co-creators in the development of the questionnaire in WP1, interview guide, and workshop, as well as in deciding which conditions and setups should be investigated, could provide valuable insights, and improve the study's relevance and impact. Response: Thank you for this helpful point – our evaluation is underpinned by participatory principles. In the protocol we have only provided some examples for PPI contribution. The findings papers will provide more detail on specific PPI activities, including the ones mentioned in the comment above. COMMENTS ON THIS REPORT - Author Response 18 Sep 2024Chrysanthi Papoutsi, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, OX26GG, UK18 Sep 2024Author ResponsePlease see below our point-by-point response to reviewer 2 comments: Thank you for the opportunity to review the study titled: Evaluating Video and Hybrid Group Consultations in General Practice: ... Continue reading Please see below our point-by-point response to reviewer 2 comments: Thank you for the opportunity to review the study titled: Evaluating Video and Hybrid Group Consultations in General Practice: Mixed-Methods, Participatory Study Protocol. Overall, this PPI co-designed protocol is well-developed and appears very comprehensive, uncovering patients`, general practitioners', practice staffs and stakeholders’ thoughts and experiences with the use of video and hybrid group consultations in general practice and for which conditions group consultation has been used. Additionally, it addresses issues related to time resources and the economic aspects of using video and hybrid group consultations in general practice. Thank you The study is divided into three work packages, which is a logical approach. WP1 is exploratory, WP2 involves designing and testing, and WP3 focuses on further testing and evaluation. However, the connections (the common thread) between these packages could be made more visible, specifically if the work packages are interdependent and if yes, in what way. Response: We have added further clarification: a) for WP1 ‘Survey results will inform subsequent phases of the study e.g. in terms of site selection’ in WP3, b) for WP2 ‘Co-design will iteratively inform the focus of data collection in WP3, especially qualitative and inclusion aspects, and emerging findings from WP3 will feed into the co-design workshops’, c) for WP3c the protocol already stated ‘Findings from WP1 (scoping survey) and WP2 (co-design) will inform the process of: i) defining the decision problem for the early economic evaluation ii) conceptualising a decision model structure that represents the decision problem; and iii) generating a set of initial model inputs for a preliminary evaluation of the decision model.’ Further details on interdependencies that are emerging as the project evolves will be included in the final findings paper. WP1: The questionnaire appears relevant for addressing clear questions regarding which chronic conditions and the overall setup of how video/hybrid consultations have been used and in which context. It also allows for comparisons with previous satisfaction surveys. Perhaps the authors could consider the following: - It might be interesting to involve PPI in the development and testing of this questionnaire. Response: As we explain the questionnaire extends a survey used in a previous study on video consultations in 2020 – both the previous study as well as the one described in this protocol have been informed by PPI contribution and this has included input to some of the survey questions, e.g. on inclusion. This is described in the findings paper for the study currently in development. - Explain the purpose of the questionnaire and the specific information being gathered. Response: The protocol explains as follows: ‘The scoping survey will provide a picture of how group consulting was deployed as a remote model of care in the context of the pandemic (2020-21), and how it has been used in the recovery context (2021-22), including the types of conditions and patient populations for which it has been used, the types of consultations carried out in groups (e.g. annual reviews, health checks), frequency and format of sessions (e.g. in-person or remote delivery mode, inclusion of underserved communities), staff training, and key implementation opportunities and challenges, with a particular focus on inclusion and equity.’ - Describe how the data gathered in WP1 will be utilized and integrated into WP2. Response: We have now explained: ‘Survey results will inform subsequent phases of the study e.g. in terms of site selection’ - Briefly explain what the GP survey is - As well as the OQF. Response: We have provided relevant reference. - Consider using semi-structured interviews to further explore stakeholders’ answers from the questionnaire - what works and what doesn't, such as disease groups, practice and patient characteristics, etc. That could be a good preparation for WP2. Response: The study includes relevant interviews in WP3. WP2: Is very interesting and will reveal valuable insights about what works well in group settings for both patients and healthcare personnel. It is also beneficial to explore this for the vulnerable patient group, who can greatly benefit from peer-to-peer support. Perhaps the authors could consider the following: - Provide a more detailed description of the workshop setup. Will it involve only patients? discussing with other patients, or will there be mixed groups of patients and GPs, staff members and stakeholders? Response: We have added this is 2 workshops for patients and 2 for staff. - Is there a plan to conduct a mock-up remote group consultation and evaluation of it? Response: This is an interesting idea – we used de-identified clips from WP3 to trigger discussion and will provide further explanation on this in the findings papers out of the study. The protocol states: ‘…we will use excerpts from de-identified interview data with patients and staff and recordings of group consultations (see next section) to trigger conversation and exchange in co-design workshops’ - Once again, how will the results from WP1 be integrated into WP2. Response: The results from WP1 mainly informed site selection and provided links to relevant stakeholders who were invited to the co-design workshops, as per the response in the relevant comment above. - How will the results from WP2 be used in WP3 – if at all? Response: We have provided relevant clarification (as explained above): ‘Co-design will iteratively inform the focus of data collection in WP3, especially qualitative and inclusion aspects, and emerging findings from WP3 will feed into the co-design workshops.’ WP3: The evaluation of the project appears interesting, with important evaluation points such as health economics, sociotechnical change and dynamics, and an analysis of remote group consultations. - Is it the aim to develop a "golden standard of remote group consultations" in WP2, and then test it and compare it with the five control GPs in WP3? Its not clear for the reader. Response: As the proposal suggests WP2 will develop practical guidance on setting up and delivering VHGCs – we do not believe there is a gold standard and only intend to develop guidance that staff and patients can use flexibly (and adapt) depending on their needs. We have now clarified this in the relevant section. - is it possible to estimate the expected data volume for the quantitative part? Can robust and meaningful conclusions be made based on the quantitative data set? Response: We have provided indicative numbers based on experience – the study is not looking to draw definitive conclusions based on quantitative data, rather we are looking to consolidate data across methods supported by theory to enhance the value of this work. Group consultations remain a relatively new model of care with remote delivery reaching relatively low volumes, therefore we are not expecting to be able to reach significant numbers at this stage and the study has been designed accordingly. - The above reflections are also relevant for ensuring an accurate health economic analysis. Please see response above. - It could be elaborated upon what is alluded at with “surface assumptions”. Moreover, “sociotechnical change” as a concept and the various theories mentioned at the end of the review could be described in more detail, not least why these are found relevant. Response: We have clarified what we mean by sociotechnical change also in response to reviewer 1 comments. We have also elaborated that assumptions relate to what these models are meant to achieve and provided details on the theories mentioned: ‘This includes theory on: a) burden of treatment and patient ‘work’ related to the responsibilities placed on patients in the context of group consultations (38), b) digital exclusion and intersectionality around multiple dimensions that contribute to some people being served less well by remote means of group-based care (39, 40), c) health service complexity including a departure from linear cause-effect relationships to recognising interdependence and emergence in group-based care delivery (41, 42) and d) experiential knowledge mobilised to support clinical care (10, 12)’. Overall: Furthermore, the authors could consider incorporating even more Patient and Public Involvement (PPI). For example, involving patients as co-creators in the development of the questionnaire in WP1, interview guide, and workshop, as well as in deciding which conditions and setups should be investigated, could provide valuable insights, and improve the study's relevance and impact. Response: Thank you for this helpful point – our evaluation is underpinned by participatory principles. In the protocol we have only provided some examples for PPI contribution. The findings papers will provide more detail on specific PPI activities, including the ones mentioned in the comment above.Please see below our point-by-point response to reviewer 2 comments:Competing Interests: n/a Close Thank you for the opportunity to review the study titled: Evaluating Video and Hybrid Group Consultations in General Practice: Mixed-Methods, Participatory Study Protocol. Overall, this PPI co-designed protocol is well-developed and appears very comprehensive, uncovering patients`, general practitioners', practice staffs and stakeholders’ thoughts and experiences with the use of video and hybrid group consultations in general practice and for which conditions group consultation has been used. Additionally, it addresses issues related to time resources and the economic aspects of using video and hybrid group consultations in general practice. Thank you The study is divided into three work packages, which is a logical approach. WP1 is exploratory, WP2 involves designing and testing, and WP3 focuses on further testing and evaluation. However, the connections (the common thread) between these packages could be made more visible, specifically if the work packages are interdependent and if yes, in what way. Response: We have added further clarification: a) for WP1 ‘Survey results will inform subsequent phases of the study e.g. in terms of site selection’ in WP3, b) for WP2 ‘Co-design will iteratively inform the focus of data collection in WP3, especially qualitative and inclusion aspects, and emerging findings from WP3 will feed into the co-design workshops’, c) for WP3c the protocol already stated ‘Findings from WP1 (scoping survey) and WP2 (co-design) will inform the process of: i) defining the decision problem for the early economic evaluation ii) conceptualising a decision model structure that represents the decision problem; and iii) generating a set of initial model inputs for a preliminary evaluation of the decision model.’ Further details on interdependencies that are emerging as the project evolves will be included in the final findings paper. WP1: The questionnaire appears relevant for addressing clear questions regarding which chronic conditions and the overall setup of how video/hybrid consultations have been used and in which context. It also allows for comparisons with previous satisfaction surveys. Perhaps the authors could consider the following: - It might be interesting to involve PPI in the development and testing of this questionnaire. Response: As we explain the questionnaire extends a survey used in a previous study on video consultations in 2020 – both the previous study as well as the one described in this protocol have been informed by PPI contribution and this has included input to some of the survey questions, e.g. on inclusion. This is described in the findings paper for the study currently in development. - Explain the purpose of the questionnaire and the specific information being gathered. Response: The protocol explains as follows: ‘The scoping survey will provide a picture of how group consulting was deployed as a remote model of care in the context of the pandemic (2020-21), and how it has been used in the recovery context (2021-22), including the types of conditions and patient populations for which it has been used, the types of consultations carried out in groups (e.g. annual reviews, health checks), frequency and format of sessions (e.g. in-person or remote delivery mode, inclusion of underserved communities), staff training, and key implementation opportunities and challenges, with a particular focus on inclusion and equity.’ - Describe how the data gathered in WP1 will be utilized and integrated into WP2. Response: We have now explained: ‘Survey results will inform subsequent phases of the study e.g. in terms of site selection’ - Briefly explain what the GP survey is - As well as the OQF. Response: We have provided relevant reference. - Consider using semi-structured interviews to further explore stakeholders’ answers from the questionnaire - what works and what doesn't, such as disease groups, practice and patient characteristics, etc. That could be a good preparation for WP2. Response: The study includes relevant interviews in WP3. WP2: Is very interesting and will reveal valuable insights about what works well in group settings for both patients and healthcare personnel. It is also beneficial to explore this for the vulnerable patient group, who can greatly benefit from peer-to-peer support. Perhaps the authors could consider the following: - Provide a more detailed description of the workshop setup. Will it involve only patients? discussing with other patients, or will there be mixed groups of patients and GPs, staff members and stakeholders? Response: We have added this is 2 workshops for patients and 2 for staff. - Is there a plan to conduct a mock-up remote group consultation and evaluation of it? Response: This is an interesting idea – we used de-identified clips from WP3 to trigger discussion and will provide further explanation on this in the findings papers out of the study. The protocol states: ‘…we will use excerpts from de-identified interview data with patients and staff and recordings of group consultations (see next section) to trigger conversation and exchange in co-design workshops’ - Once again, how will the results from WP1 be integrated into WP2. Response: The results from WP1 mainly informed site selection and provided links to relevant stakeholders who were invited to the co-design workshops, as per the response in the relevant comment above. - How will the results from WP2 be used in WP3 – if at all? Response: We have provided relevant clarification (as explained above): ‘Co-design will iteratively inform the focus of data collection in WP3, especially qualitative and inclusion aspects, and emerging findings from WP3 will feed into the co-design workshops.’ WP3: The evaluation of the project appears interesting, with important evaluation points such as health economics, sociotechnical change and dynamics, and an analysis of remote group consultations. - Is it the aim to develop a "golden standard of remote group consultations" in WP2, and then test it and compare it with the five control GPs in WP3? Its not clear for the reader. Response: As the proposal suggests WP2 will develop practical guidance on setting up and delivering VHGCs – we do not believe there is a gold standard and only intend to develop guidance that staff and patients can use flexibly (and adapt) depending on their needs. We have now clarified this in the relevant section. - is it possible to estimate the expected data volume for the quantitative part? Can robust and meaningful conclusions be made based on the quantitative data set? Response: We have provided indicative numbers based on experience – the study is not looking to draw definitive conclusions based on quantitative data, rather we are looking to consolidate data across methods supported by theory to enhance the value of this work. Group consultations remain a relatively new model of care with remote delivery reaching relatively low volumes, therefore we are not expecting to be able to reach significant numbers at this stage and the study has been designed accordingly. - The above reflections are also relevant for ensuring an accurate health economic analysis. Please see response above. - It could be elaborated upon what is alluded at with “surface assumptions”. Moreover, “sociotechnical change” as a concept and the various theories mentioned at the end of the review could be described in more detail, not least why these are found relevant. Response: We have clarified what we mean by sociotechnical change also in response to reviewer 1 comments. We have also elaborated that assumptions relate to what these models are meant to achieve and provided details on the theories mentioned: ‘This includes theory on: a) burden of treatment and patient ‘work’ related to the responsibilities placed on patients in the context of group consultations (38), b) digital exclusion and intersectionality around multiple dimensions that contribute to some people being served less well by remote means of group-based care (39, 40), c) health service complexity including a departure from linear cause-effect relationships to recognising interdependence and emergence in group-based care delivery (41, 42) and d) experiential knowledge mobilised to support clinical care (10, 12)’. Overall: Furthermore, the authors could consider incorporating even more Patient and Public Involvement (PPI). For example, involving patients as co-creators in the development of the questionnaire in WP1, interview guide, and workshop, as well as in deciding which conditions and setups should be investigated, could provide valuable insights, and improve the study's relevance and impact. Response: Thank you for this helpful point – our evaluation is underpinned by participatory principles. In the protocol we have only provided some examples for PPI contribution. The findings papers will provide more detail on specific PPI activities, including the ones mentioned in the comment above. Views 0 How to cite this report: Job J. Reviewer Report For: Evaluating video and hybrid group consultations in general practice: mixed-methods, participatory study protocol (TOGETHER 2) [version 2; peer review: 2 approved]. NIHR Open Res 2024, 4:30 (https://doi.org/10.3310/nihropenres.14745.r31862) The direct URL for this report is: https://openresearch.nihr.ac.uk/articles/4-30/v1#referee-response-31862 https://openresearch.nihr.ac.uk/articles/4-30/v1#referee-response-31862 NOTE: it is important to ensure the information in square brackets after the title is included in this citation. Reviewer Report 27 May 2024 Approved with Reservations VIEWS 0 Thank you for the opportunity to review this well written manuscript outlining a protocol for the evaluation of the benefits to patients of group consultations when delivered via video and/or in-person in general practice for multiple patients, with the same ... Continue reading I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above. Close Thank you for the opportunity to review this well written manuscript outlining a protocol for the evaluation of the benefits to patients of group consultations when delivered via video and/or in-person in general practice for multiple patients, with the same chronic conditions. This manuscript proposes a valuable body of work which will provide important data on patient experience, utilisation, safety, equity, and affordability of this model. The rationale and research questions are clearly defined. The definition provided in the plain language summary was useful for the reader, who may be unfamiliar with the model, to clarify the authors meaning of VHGCs. Would including this in the abstract highlight the focus for potential readers, particularly those unfamiliar with the model? Are VHGCs delivered by a one practitioner? GPs? Other health professionals? The term “sociotechnical change” was unfamiliar – I wonder if it would be useful to expand on this. Page 5: National GP patient survey – could you provide the reference for this when it is first mentioned on page 5. Could you provide a reference for QOF? Page 7: It would be interesting to know the proposed data collection methods e.g. spreadsheets, or an online portal? How will ‘deprivation’ be measured? This manuscript proposes a valuable body of work which will provide important data on patient experience, utilisation, safety, equity, and affordability of this model. The rationale and research questions are clearly defined. The definition provided in the plain language summary was useful for the reader, who may be unfamiliar with the model, to clarify the authors meaning of VHGCs. Would including this in the abstract highlight the focus for potential readers, particularly those unfamiliar with the model? Are VHGCs delivered by a one practitioner? GPs? Other health professionals? The term “sociotechnical change” was unfamiliar – I wonder if it would be useful to expand on this. Page 5: National GP patient survey – could you provide the reference for this when it is first mentioned on page 5. Could you provide a reference for QOF? Page 7: It would be interesting to know the proposed data collection methods e.g. spreadsheets, or an online portal? How will ‘deprivation’ be measured? - Is the rationale for, and objectives of, the study clearly described? Yes - Is the study design appropriate for the research question? Yes - Are sufficient details of the methods provided to allow replication by others? Partly - Are the datasets clearly presented in a useable and accessible format? Not applicable Competing Interests: No competing interests were disclosed. Reviewer Expertise: Primary care research, implementation science CITE HOW TO CITE THIS REPORT Job J. Reviewer Report For: Evaluating video and hybrid group consultations in general practice: mixed-methods, participatory study protocol (TOGETHER 2) [version 2; peer review: 2 approved]. NIHR Open Res 2024, 4:30 (https://doi.org/10.3310/nihropenres.14745.r31862) The direct URL for this report is: https://openresearch.nihr.ac.uk/articles/4-30/v1#referee-response-31862 https://openresearch.nihr.ac.uk/articles/4-30/v1#referee-response-31862 NOTE: it is important to ensure the information in square brackets after the title is included in all citations of this article. - Author Response 18 Sep 2024Chrysanthi Papoutsi, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, OX26GG, UK18 Sep 2024Author ResponsePlease see below our response to Reviewer 1 comments - Reviewer Comments: Thank you for the opportunity to review this well written manuscript outlining a protocol for the evaluation ... Continue reading Please see below our response to Reviewer 1 comments - Reviewer Comments: Thank you for the opportunity to review this well written manuscript outlining a protocol for the evaluation of the benefits to patients of group consultations when delivered via video and/or in-person in general practice for multiple patients, with the same chronic conditions. This manuscript proposes a valuable body of work which will provide important data on patient experience, utilisation, safety, equity, and affordability of this model. The rationale and research questions are clearly defined. Author Response: Thank you Reviewer Comments: The definition provided in the plain language summary was useful for the reader, who may be unfamiliar with the model, to clarify the authors meaning of VHGCs. Would including this in the abstract highlight the focus for potential readers, particularly those unfamiliar with the model? Author Response: Helpful suggestion – we have now included the definition in the abstract Reviewer Comments: Are VHGCs delivered by a one practitioner? GPs? Other health professionals? Author Response: We have clarified in the introduction section: ‘Combining clinical consultation with education and peer support, group consultations are led by different health professionals (GPs, pharmacists, nurses) and have been delivered in a variety of formats for patients with different long-term conditions or shared health concerns (13, 14).’ Reviewer Comments: The term “sociotechnical change” was unfamiliar – I wonder if it would be useful to expand on this. Author Response: We have provided additional clarification: ‘In Together 2 we seek to generate an in-depth understanding of different modes of remote group consulting (video and hybrid), surface assumptions and sociotechnical dynamics (the interplay between technical and social elements) underpinning this relatively novel way of providing care, inform practice and extend theorisation.’ Reviewer Comments: Page 5: National GP patient survey – could you provide the reference for this when it is first mentioned on page 5. Author Response: We have provided relevant reference. Reviewer Comments: Could you provide a reference for QOF? Author Response: We have provided relevant reference. Reviewer Comments: Page 7: It would be interesting to know the proposed data collection methods e.g. spreadsheets, or an online portal? Author Response: It is not clear to us what part of page 7 this comment refers to but we have added clarification (where it seemed relevant) that we are using Word proformas. Reviewer Comments: How will ‘deprivation’ be measured? Author Response: We have clarified that we are using the index of multiple deprivation and provided reference: https://data.cdrc.ac.uk/dataset/index-multiple-deprivation-imdPlease see below our response to Reviewer 1 comments -Competing Interests: n/a Close Reviewer Comments: Thank you for the opportunity to review this well written manuscript outlining a protocol for the evaluation of the benefits to patients of group consultations when delivered via video and/or in-person in general practice for multiple patients, with the same chronic conditions. This manuscript proposes a valuable body of work which will provide important data on patient experience, utilisation, safety, equity, and affordability of this model. The rationale and research questions are clearly defined. Author Response: Thank you Reviewer Comments: The definition provided in the plain language summary was useful for the reader, who may be unfamiliar with the model, to clarify the authors meaning of VHGCs. Would including this in the abstract highlight the focus for potential readers, particularly those unfamiliar with the model? Author Response: Helpful suggestion – we have now included the definition in the abstract Reviewer Comments: Are VHGCs delivered by a one practitioner? GPs? Other health professionals? Author Response: We have clarified in the introduction section: ‘Combining clinical consultation with education and peer support, group consultations are led by different health professionals (GPs, pharmacists, nurses) and have been delivered in a variety of formats for patients with different long-term conditions or shared health concerns (13, 14).’ Reviewer Comments: The term “sociotechnical change” was unfamiliar – I wonder if it would be useful to expand on this. Author Response: We have provided additional clarification: ‘In Together 2 we seek to generate an in-depth understanding of different modes of remote group consulting (video and hybrid), surface assumptions and sociotechnical dynamics (the interplay between technical and social elements) underpinning this relatively novel way of providing care, inform practice and extend theorisation.’ Reviewer Comments: Page 5: National GP patient survey – could you provide the reference for this when it is first mentioned on page 5. Author Response: We have provided relevant reference. Reviewer Comments: Could you provide a reference for QOF? Author Response: We have provided relevant reference. Reviewer Comments: Page 7: It would be interesting to know the proposed data collection methods e.g. spreadsheets, or an online portal? Author Response: It is not clear to us what part of page 7 this comment refers to but we have added clarification (where it seemed relevant) that we are using Word proformas. Reviewer Comments: How will ‘deprivation’ be measured? Author Response: We have clarified that we are using the index of multiple deprivation and provided reference: https://data.cdrc.ac.uk/dataset/index-multiple-deprivation-imd COMMENTS ON THIS REPORT - Author Response 18 Sep 2024Chrysanthi Papoutsi, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, OX26GG, UK18 Sep 2024Author ResponsePlease see below our response to Reviewer 1 comments - Reviewer Comments: Thank you for the opportunity to review this well written manuscript outlining a protocol for the evaluation ... Continue reading Please see below our response to Reviewer 1 comments - Reviewer Comments: Thank you for the opportunity to review this well written manuscript outlining a protocol for the evaluation of the benefits to patients of group consultations when delivered via video and/or in-person in general practice for multiple patients, with the same chronic conditions. This manuscript proposes a valuable body of work which will provide important data on patient experience, utilisation, safety, equity, and affordability of this model. The rationale and research questions are clearly defined. Author Response: Thank you Reviewer Comments: The definition provided in the plain language summary was useful for the reader, who may be unfamiliar with the model, to clarify the authors meaning of VHGCs. Would including this in the abstract highlight the focus for potential readers, particularly those unfamiliar with the model? Author Response: Helpful suggestion – we have now included the definition in the abstract Reviewer Comments: Are VHGCs delivered by a one practitioner? GPs? Other health professionals? Author Response: We have clarified in the introduction section: ‘Combining clinical consultation with education and peer support, group consultations are led by different health professionals (GPs, pharmacists, nurses) and have been delivered in a variety of formats for patients with different long-term conditions or shared health concerns (13, 14).’ Reviewer Comments: The term “sociotechnical change” was unfamiliar – I wonder if it would be useful to expand on this. Author Response: We have provided additional clarification: ‘In Together 2 we seek to generate an in-depth understanding of different modes of remote group consulting (video and hybrid), surface assumptions and sociotechnical dynamics (the interplay between technical and social elements) underpinning this relatively novel way of providing care, inform practice and extend theorisation.’ Reviewer Comments: Page 5: National GP patient survey – could you provide the reference for this when it is first mentioned on page 5. Author Response: We have provided relevant reference. Reviewer Comments: Could you provide a reference for QOF? Author Response: We have provided relevant reference. Reviewer Comments: Page 7: It would be interesting to know the proposed data collection methods e.g. spreadsheets, or an online portal? Author Response: It is not clear to us what part of page 7 this comment refers to but we have added clarification (where it seemed relevant) that we are using Word proformas. Reviewer Comments: How will ‘deprivation’ be measured? Author Response: We have clarified that we are using the index of multiple deprivation and provided reference: https://data.cdrc.ac.uk/dataset/index-multiple-deprivation-imdPlease see below our response to Reviewer 1 comments -Competing Interests: n/a Close Reviewer Comments: Thank you for the opportunity to review this well written manuscript outlining a protocol for the evaluation of the benefits to patients of group consultations when delivered via video and/or in-person in general practice for multiple patients, with the same chronic conditions. This manuscript proposes a valuable body of work which will provide important data on patient experience, utilisation, safety, equity, and affordability of this model. The rationale and research questions are clearly defined. Author Response: Thank you Reviewer Comments: The definition provided in the plain language summary was useful for the reader, who may be unfamiliar with the model, to clarify the authors meaning of VHGCs. Would including this in the abstract highlight the focus for potential readers, particularly those unfamiliar with the model? Author Response: Helpful suggestion – we have now included the definition in the abstract Reviewer Comments: Are VHGCs delivered by a one practitioner? GPs? Other health professionals? Author Response: We have clarified in the introduction section: ‘Combining clinical consultation with education and peer support, group consultations are led by different health professionals (GPs, pharmacists, nurses) and have been delivered in a variety of formats for patients with different long-term conditions or shared health concerns (13, 14).’ Reviewer Comments: The term “sociotechnical change” was unfamiliar – I wonder if it would be useful to expand on this. Author Response: We have provided additional clarification: ‘In Together 2 we seek to generate an in-depth understanding of different modes of remote group consulting (video and hybrid), surface assumptions and sociotechnical dynamics (the interplay between technical and social elements) underpinning this relatively novel way of providing care, inform practice and extend theorisation.’ Reviewer Comments: Page 5: National GP patient survey – could you provide the reference for this when it is first mentioned on page 5. Author Response: We have provided relevant reference. Reviewer Comments: Could you provide a reference for QOF? Author Response: We have provided relevant reference. Reviewer Comments: Page 7: It would be interesting to know the proposed data collection methods e.g. spreadsheets, or an online portal? Author Response: It is not clear to us what part of page 7 this comment refers to but we have added clarification (where it seemed relevant) that we are using Word proformas. Reviewer Comments: How will ‘deprivation’ be measured? Author Response: We have clarified that we are using the index of multiple deprivation and provided reference: https://data.cdrc.ac.uk/dataset/index-multiple-deprivation-imd Alongside their report, reviewers assign a status to the article: - Approved - Approved with reservations - Not approved | Invited Reviewers | || |---|---|---| | 1 | 2 | | | Version 2 (revision) 18 Sep 24 | read | read | | Version 1 10 May 24 | read | read | Sign up for content alerts You are now signed up to receive this alert Alongside their report, reviewers assign a status to the article: Approved - the paper is scientifically sound in its current form and only minor, if any, improvements are suggested Approved with reservations - A number of small changes, sometimes more significant revisions are required to address specific details and improve the papers academic merit. Not approved - fundamental flaws in the paper seriously undermine the findings and conclusions Provide sufficient details of any financial or non-financial competing interests to enable users to assess whether your comments might lead a reasonable person to question your impartiality. Consider the following examples, but note that this is not an exhaustive list: Examples of 'Non-Financial Competing Interests' - Within the past 4 years, you have held joint grants, published or collaborated with any of the authors of the selected paper. - You have a close personal relationship (e.g. parent, spouse, sibling, or domestic partner) with any of the authors. - You are a close professional associate of any of the authors (e.g. scientific mentor, recent student). - You work at the same institute as any of the authors. - You hope/expect to benefit (e.g. favour or employment) as a result of your submission. - You are an Editor for the journal in which the article is published. Examples of 'Financial Competing Interests' - You expect to receive, or in the past 4 years have received, any of the following from any commercial organisation that may gain financially from your submission: a salary, fees, funding, reimbursements. - You expect to receive, or in the past 4 years have received, shared grant support or other funding with any of the authors. - You hold, or are currently applying for, any patents or significant stocks/shares relating to the subject matter of the paper you are commenting on. Sign up for content alerts and receive a weekly or monthly email with all newly published articles Register with NIHR Open Research Already registered? Sign in close Error If you are a previous or current NIHR award holder, sign up for information about developments, publishing and publications from NIHR Open Research. We'll keep you updated on any major new updates to NIHR Open Research Sign In If you've forgotten your password, please enter your email address below and we'll send you instructions on how to reset your password. Email us for further assistance. The email address should be the one you originally registered with F1000. Email address not valid, please try again You registered with F1000 via Google, so we cannot reset your password. To sign in, please click here. If you still need help with your Google account password, please click here. You registered with F1000 via Facebook, so we cannot reset your password. To sign in, please click here. If you still need help with your Facebook account password, please click here. Code not correct, please try again Server error, please try again. If your email address is registered with us, we will email you instructions to reset your password. If you think you should have received this email but it has not arrived, please check your spam filters and/or contact for further assistance. Please wait...