When scientific constraints and real-world implementation collide: Lessons from a hybrid type 3 study of a digital, direct-to-consumer HIV prevention program

When scientific constraints and real-world implementation collide: Lessons from a hybrid type 3 study of a digital, direct-to-consumer HIV prevention program | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article When scientific constraints and real-world implementation collide: Lessons from a hybrid type 3 study of a digital, direct-to-consumer HIV prevention program Kathryn R. Macapagal, Krystal L. Madkins, Ashley Knapp, Manuel Hurtado, Jr., and 6 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-9202620/v1 This work is licensed under a CC BY 4.0 License Status: Under Review Version 1 posted 5 You are reading this latest preprint version Abstract Background Digital HIV prevention interventions (DHIs) are efficacious in increasing prevention behaviors, and delivering them direct-to-consumer (DTC) expands their reach. However, few have successfully moved rigorously-studied DHIs from research to public health practice. Hybrid type 3 effectiveness-implementation studies promise to approximate more naturalistic settings and accelerate translation from research to practice, and few DTC DHIs have been tested this way. This study describes lessons learned from a hybrid type 3 study of a DTC DHI for young men who have sex with men (YMSM) ages 18–29 called Keep It Up! (KIU!) 3.0. Methods KIU! 3.0 was initially designed as a cluster-randomized hybrid type 3 implementation-effectiveness study across 44 United States counties with high HIV incidence among YMSM (22 DTC; 22 implemented in community-based organizations). The DTC strategy relied on online recruitment, at-home HIV/STI testing, and centralized intervention delivery. Over the course of the trial (October 2019-March 2023), we adapted our implementation four times in response to recruitment and retention challenges. Data sources included enrollment logs, advertising expenditures, participant communications, and internal documentation, which are used to characterize recruitment, costs, and recruitment/retention patterns. Results Due to challenges in recruitment and retention, four major changes were made over the course of the trial: 1) streamlining enrollment procedures, including shortening screening and verification steps, 2) modifying eligibility criteria, including expanding age limits, removing sexual risk requirements, and including gender-diverse participants, 3) introducing and increasing financial incentives for intervention completing and follow-up measures, and 4) shifting from county-level to nationwide recruitment. Following this fourth change in March 2021, enrollment increased by 301%, and cost per enrolled participant decreased. Ultimately 1,468 participants were enrolled nationwide. Nevertheless, retention and return of at-home STI test kits remained challenging, and guaranteed incentives increased the prevalence of imposter participants. Conclusion Implementing a DTC DHI within a hybrid type 3 study required balancing pragmatic implementation goals with effectiveness outcomes measurement. Future hybrid studies of DTC digital health interventions should consider eligibility criteria, incentive structures, outcomes measurement strategies, and the distinction between research and service components to better align research with real-world implementation contexts. Digital health interventions Direct-to-consumer (DTC) interventions hybrid effectiveness-implementation study HIV prevention Figures Figure 1 Figure 2 Figure 3 Contributions to the literature Hybrid type 3 studies intend to prioritize understanding of implementation over effectiveness. However, few hybrid type 3 studies of digital interventions that are delivered directly to the consumer have been reported in the literature. This study illuminated how various constraints and outer setting factors (e.g., pandemic, funder requirement to collect biomedical effectiveness data) created tensions between making the intervention delivery as “real world” as possible and engaging and retaining participants long enough to collect endpoint data. Findings can inform the design of future hybrid type 3 studies of digital interventions to increase their real-world applicability and reduce burden of effectiveness data collection. Introduction In the United States (U.S.), men who have sex with men (MSM) accounted for 67% of new HIV diagnoses in 2022 (CDC, 2025a ), most of which were within young MSM aged 13–34 (CDC, 2025a ). Digital HIV interventions (DHIs; e.g., interactive websites, smartphone apps, text messaging programs) tailored to their needs are efficacious in reducing HIV transmission risk behavior and sexually transmitted infection (STI) incidence and increasing health promotive behaviors (e.g., pre-exposure prophylaxis [PrEP] use, HIV testing). DHIs can reduce barriers to care if delivered “direct-to-consumer” (DTC) rather than through intermediaries, such as healthcare providers or community health centers (Li et al., 2019 ; Nguyen et al., 2019 ; Zhang et al., 2021 ). Currently, 29 DHIs are listed in a compendium of “best practices” in HIV risk reduction (CDC, 2025b ), but none are being implemented in public health practice. This is because despite significant federal investment in research to develop and test DHIs, relatively little research has attempted to implement them in real-world settings, and those who have reported significant obstacles (Hermes et al., 2019 ; Li et al., 2024 ). Researchers have studied DHI implementation in two main ways: via healthcare or community-based organizations (CBOs) trained to offer the intervention to clients and via a DTC model, where an organization hosts and/or markets the intervention to people who can self-enroll (Benbow et al., 2025 ; Mustanski, Benbow, et al., 2025b ; Mustanski, Macapagal, et al., 2025 ; Mustanski, Saber, Macapagal, et al., 2023 ). Most literature on DHI implementation has focused on healthcare settings with “captive audiences” of existing patients or clients. In contrast, a DTC model requires the implementer (e.g., digital health company, health department) to focus on creating significant demand for the intervention (via marketing, incentives, etc.) among people who may not perceive the need for one (Aref-Adib et al., 2019; Germain et al., 2021; Matson et al., 2025; Ross et al., 2018). Yet this may be challenging for preventive interventions as individuals are often less motivated to address health problems before they arise (Hardcastle et al., 2015 ; Hermes et al., 2019 ). In traditional effectiveness trials, demand creation activities are ancillary and often not detailed in publications, but in implementation studies they are essential in supporting DTC intervention delivery. How well these strategies operate in a “real-world” setting has not been well-documented. The Keep It Up! (KIU!) intervention We developed KIU!, a self-guided DHI for YMSM that is among the CDC’s best-evidence HIV prevention programs (Centers for Disease Control, 2024 ) and based on the Information-Motivation-Behavioral Skills theory of health behavior change (Fisher et al., 1994 ). KIU! was originally designed to be delivered in CBOs after YMSM tested HIV-negative to encourage them to “keep up” their preventive behaviors (KIU! 1.0; (Mustanski et al., 2013 ). A randomized controlled trial (KIU! 2.0) examined the effectiveness of CBO-based delivery in three U.S. cities and piloted a nationwide DTC strategy where staff recruited YMSM through online advertising, shipped HIV/STI test kits to participants, and granted access to KIU! when a participant tested HIV-negative (Mustanski et al., 2017 ). KIU! reduced self-reported condomless anal sex and gonorrhea and chlamydia incidence by 40% versus an HIV information-only control arm (Mustanski et al., 2018 ). From 2019 to 2023, we conducted a hybrid type 3 implementation–effectiveness study (KIU! 3.0) comparing the CBO and DTC strategies; the protocol and findings are described elsewhere (Mustanski, Benbow, et al., 2025b ; Mustanski, Saber, Jones, et al., 2023 ). We chose this design to approximate real-world implementation; however, the constraints of a research study necessarily limits flexibility. As few studies explore DTC delivery of DHIs, this trial offered a unique opportunity to examine KIU!’s effectiveness in a more naturalistic setting, but also examine what worked (and did not) regarding implementation. This paper describes the initial implementation plans for the KIU! 3.0 DTC strategy and how they evolved because of recruitment challenges and COVID-19 pandemic disruptions. We also discuss tensions between the research elements necessary to measure KIU!’s effectiveness and pragmatic elements more aligned with real-world delivery. Our goal is to offer a “behind the scenes” look at the study operations, decision making, and lessons learned in demand creation, and to share ideas about what can/cannot change during a hybrid 3 study of a DHI which can inform future pragmatic studies and reduce implementation barriers (Li et al., 2019 ). Methods Initial study design and context The KIU! 3.0 intervention consisted of five “episodes” (three reflecting the main intervention content and two boosters), each containing a webseries, didactics, and activities to augment learning and engagement that totaled ~ 1.5h. After episodes 1–3, participants waited 8h until the next episode. A feature of earlier KIU! versions, this break was based on guidelines to enhance information retention in multimedia learning (Clark & Mayer, 2008 ; Mayer, 2017 , 2016). Booster content in episodes 4 and 5 (at 3- and 6-months post-intervention, respectively) reinforced learning. Participants completed a final survey and STI testing at 12 months post-intervention. The study initially proposed a cluster-randomized design to compare the implementation and effectiveness of the KIU! CBO versus DTC strategies in 44 U.S. counties with high HIV incidence among YMSM (Mustanski, Saber, Jones, et al., 2023 ). Enrollment goals were 100 per county (N = 4400), and county-level randomization was selected to mirror how HIV public health funding often works and to compare implementation in high-incidence jurisdictions outlined in the federal Ending the HIV Epidemic initiative ( EHE Priority Jurisdictions , n.d.). In the CBO approach, CBOs in 22 counties received funding to deliver KIU! through their HIV prevention and testing programs. In the DTC approach, centrally-located staff recruited YMSM in 22 other counties using online advertising and shipped HIV self-tests and rectal and urethral chlamydia and gonorrhea sample collection kits (the latter two of which they were to ship to a lab for processing). In both approaches, after e-consenting to participation and providing documentation of an HIV-negative test result participants received access to KIU! Implementation assessments were based on RE-AIM and the Consolidated Framework for Implementation Research (CFIR) (Damschroder, Aron, et al., 2009a ; Glasgow et al., 1999 ). Effectiveness outcomes included STI diagnoses, PrEP uptake, or self-reported condomless anal sex at 12-month follow-up (later abbreviated to 3-months, described later). Due to the dual effectiveness and implementation foci, we needed to distinguish activities in the trial to properly attribute their effects on outcomes. Activities were either part of the service necessary for implementation (i.e., intervention content, delivery) or part of the research intended to assess intervention effectiveness (i.e., surveys, HIV/STI testing). All interested individuals were given access to the KIU! intervention but had to meet study eligibility criteria to complete the research components. Research procedures were approved by the Northwestern University IRB; service components were considered non-human subjects research (Mustanski, Benbow, et al., 2025a ; Mustanski, Saber, Jones, et al., 2023 ). All participants provided e-consent and/or verbal consent prior to enrollment in the research portions of the study. Assumptions and constraints As no DHIs were commercially available at the time of the study, we made assumptions about how DTC delivery would occur in the “real world” while also considering the research constraints of our trial (e.g., timeline, county-level recruitment, sampling quotas, budget constraints) (Benbow et al., 2025 ). One major assumption was that those voluntarily seeking HIV prevention or sexual health services would not expect to be paid, based on other digital health applications focused on prevention and behavior change (e.g., mindfulness, fitness) were available in the App Store or Google Play (i.e., DTC). As such, at the beginning of the study incentives were not guaranteed, but participants were entered into raffles for $ 50 gift cards or sexual health products for completing research surveys. Other updates were made to increase pragmatism based on assumptions about real-world DHIs. Table 1 compares DTC delivery of KIU! during the effectiveness trial (KIU! 2.0) with features of the KIU! 3.0 implementation study as originally planned, and the final design after all changes were made. Table 1. Eligibility and methods of KIU! effectiveness trial, vs. original design and final design of the hybrid study (DTC delivery only) Component KIU! 2.0 Effectiveness Trial KIU! 3.0 Implementation-Effectiveness Study (original design) KIU! 3.0 Implementation-Effectiveness Study (final design) Eligibility Age 18–29 18–29 18–34 HIV status Negative result from at-home HIV test kit Negative result from at-home HIV test kit Self-reported HIV negative or unknown status Relationship status Not in a monogamous/exclusive relationship lasting more than 6 months No requirement No requirement Sexual risk behavior Condomless anal sex with a male partner in prior 6 months Condomless anal sex with a male partner in prior 6 months No requirement PrEP use No requirement No PrEP use OR not adherent in prior 6 months No PrEP use OR not adherent in prior 6 months Screening Online screener Online screener Online screener Location Nationwide United States with counties (n = 22) with large diverse YMSM populations Nationwide Primary recruitment strategies Online ads Online ads; participant registries; friend and community partner referrals Online ads; participant registries; friend and community partner referrals Recruitment budget $ 7500 (total actual spend) $ 27,000 budgeted $ 47,626 (total actual spend) Sample size 144 (actual enrollment from DTC pilot) 2200 (goal) 1468 (actual enrollment) Delivery Online; 24-hour wait between intervention modules Online; 8-hour wait between intervention modules Online; no wait between intervention modules Compensation Up to $ 180 for completing surveys, intervention content, and at-home STI testing Entry into raffle for $ 50 for completing surveys Up to $ 100 for completing surveys, intervention content, and at-home STI testing Data sources Data came from weekly enrollment logs; budget spreadsheets and spending logs; secure participant text messages to our study; participant contact notes; and archived team meeting notes, e-mails, Data and Safety Monitoring Board (DSMB) reports, and memos to NIH program officers. Enrollment and recruitment data were analyzed descriptively. Below, we describe five phases of KIU! DTC delivery aligned with pivotal events and how implementation and study design evolved as we learned the limitations of what was feasible and achievable. Results DTC participants were on average 26.0 years old, with 57.6% identifying as Black, Latino, or another racial/ethnic minority. All U.S. states except Alaska, Wyoming, and Maine were represented in the DTC arm with 33.1% of participants living in the South, 26.2% in the West, 24.4% in the Midwest, and 16.3% in the Northeast Census Region. Other DTC participant characteristics are available elsewhere (Mustanski, Benbow, et al., 2025b ). To characterize the phases of DTC delivery, each section includes high-level enrollment data and advertising spending; quotes from participants’ text messages or phone calls with DTC staff; and information from investigator meetings and conversations with NIH program officers that guided study change decisions. The CBO and DTC teams each consisted of a co-investigator and two or three staff, respectively, who led arm-specific activities. The investigative team consisted of the PI, the two aforementioned co-investigators, and three implementation scientists, two health economists, a quantitative methodologist, and the software development team lead, all of whom operated across study arms. Staff analysts and software developers supported the investigative team and deployed the KIU! software and research surveys. During initial project meetings, details about CBO and DTC performance were not shared with the opposing teams to avoid contamination. Phase 1: Trial launch, October 2019-February 2020 . Enrollment began in October 2019. In contrast to the nationwide DTC pilot, the DTC team’s advertising strategy consisted of social media campaigns in ZIP codes within the 22 counties. Since financial incentives could not be guarantee, the advertisements focused on community benefits, opportunities to win sexual health products, and concerns about sexual health to create demand for research participation (Bassett et al., 2015 ; Macapagal et al., 2020 ). The advertising budget per DTC participant enrolled was smaller than in KIU! 2.0, where $ 7500 in advertising yielded 144 enrolled participants (~ $ 52/participant); in KIU! 3.0, our initial budget was $ 27,000 for a goal of 2200 participants (~ $ 12/participant). The DTC team anticipated that enrollment targets would not be achieved via paid online advertising alone given the budget constraints and limited geographic catchment areas. To address this concern, the DTC team identified health departments, LGBTQ- or HIV-focused advocacy organizations, social groups, or universities in each county. The intention was to find local champions for KIU! whose “personal touch” may lead to increased enrollment in these counties (Damschroder, Banaszak-Holl, et al., 2009 ; Schon, 1963 ; Valente & Pumpuang, 2007 ). Community members were encouraged to distribute information about KIU! at their organizations with flyers and pamphlets. The team also partnered with an LGBTQ-focused marketing company who placed paid advertisements on dating apps. A challenge for the DTC team was to reduce obstacles to accessing KIU! (a pragmatic implementation goal) while deterring bots and imposter participants (a research goal) (Teitcher et al., 2015 ). Initially, a multi-step enrollment process was planned to deter imposters (Fig. 1 ). After completing the eligibility survey, participant eligibility and mailing address was verified via a brief videochat or submission to a LexisNexis or WhitePages database, which would take 1–2 business days. Verified participants received an HIV/STI self-test kit in the mail. After participants submitted an HIV-negative test result to a secure online portal, they received the baseline survey, and then received the link to KIU! Participants were also asked to ship their STI samples to a lab. This yielded at minimum a 7–10 day lag between screener completion and receiving access to KIU! Given the relative ease of enrollment in KIU! 2.0 and a large ad buy on dating apps that had accelerated prior study enrollment, we did not expect to have spent $ 10,744 to only enroll 15 participants – a significantly higher per-person spend ( $ 716.27/participant) than in KIU! 2.0. We noticed significant attrition in the enrollment process. In February 2020, 55 people were preliminarily eligible, of which 38.2% (n = 21) were confirmed; 71.4% of these (n = 15) completed baseline and were enrolled (but only 27.3% of those preliminary eligible). Participants were required to answer comprehension questions about the consent form, and those who missed questions had to consent over the phone, which was a challenge to schedule: “I won’t be home until 11 tonight if that’s fine with you. Or you can call me before 9AM or after 7PM tomorrow, or anytime Saturday” (text message from participant, 11/07/19). Thus we brainstormed methods to streamline enrollment, such as shortening the screener and verifying respondents’ identity through social media and online directories rather than via LexisNexis. Phase 2: March 2020-June 2020 . After the COVID-19 pandemic restrictions began, the DTC team paused HIV/STI test shipments because staff were not permitted at the university. Requiring proof of an HIV-negative test result would prevent enrollment, so DTC participants were allowed to enroll in the intervention based on a self-reported negative or unknown HIV status. The changes brainstormed in Phase 1 were implemented to streamline the enrollment process (Fig. 2). To incentivize participants, in consultation with the investigative team, the DTC team increased the value and frequency of the raffle prizes from a $ 50 raffle for every 50 participants who completed a survey (which meant that there were long delays between raffles) to monthly raffles for $ 200 regardless of how many participants were enrolled. These changes were intended to increase demand for participation and ensure we collected sufficient data to power our effectiveness analyses. Figure 2. Comparison of Enrollment Processes Implemented during Study Phases 1 and 2 The investigative team had concerns about the impact of the pandemic on participants’ sexual behavior (and thus inclusion criteria) and related study outcomes (e.g., HIV transmission risk behavior). Participants shared that participating in KIU! 3.0 was no longer a priority as they dealt with health issues, employment and educational instability, and greater caretaking responsibilities. In response to a study reminder, one participant said, “Honestly with all the things going on [in] the world I just have so much else to focus [on]. I was laid off my job so looking for a job is my main priority” (04/03/2020). Another participant said: “It's been really difficult the last couple of months dealing with school, mental health, family and other aspects of life and I'm trying to get back to the online material as soon as possible.” (06/12/2020). In May 2020, staff could return to the office and a DTC team member resumed shipping of HIV/STI test kits every 2 weeks (vs. any time a participant enrolled) to balance team safety while continuing the study. We also contracted with a company who shipped HIV/STI test kits for us, which allowed greater flexibility for handling testing during COVID stay-at-home orders. Despite these adaptations, only 26 participants were enrolled at the end of this phase, and an additional $ 6,428 was spent on advertisements. Phase 3: July 2020-September 2020 . Several additional changes were implemented in this phase to increase demand for KIU! 3.0. We decided to guarantee incentives for completing episodes 1–3. To examine whether different incentives would result in varying engagement and retention, DTC counties were randomized to receive $ 10 or $ 25, modeled after prior research (Sharma et al., 2011 ). In August 2020 online recruitment on social media and dating apps’ self-service portals was supplemented by a marketing company experienced in online recruitment for health research studies. Snowball recruitment began as well, with enrolled participants allowed to refer up to five friends and receive $ 10 for each enrolled friend. Eligibility criteria were modified again in September 2020 to allow enrollment of gender-diverse participants assigned male at birth in recognition of the fact that sexual behavior, and not gender, drives HIV risk, and because in KIU! 3.0 and the team’s prior HIV prevention studies focused on cisgender men, nonbinary and gender-diverse participants comprised a significant proportion of ineligible individuals. Yet by the end of this phase, an additional $ 3,397 was spent on advertising, and DTC enrollment only increased to 64 participants. Individuals who had not finished the baseline survey were recontacted to increase enrollment, and feedback such as, “I didn't complete [the survey] because I am out of your age range” (10/07/2020), was common, suggesting that additional modifications to eligibility criteria could benefit the study. In addition, only 52.8% of participants had returned their baseline STI samples to the lab. Phase 4: October 2020-February 2021 . By this time it was evident that the several changes made to the study only modestly improved enrollment. The investigative team decided to inform the co-investigators leading the CBO and DTC arms that both groups were similarly struggling with enrollment and that more significant changes to the study design were necessary to avoid trial failure (i.e., the inability to enroll enough participants to assess either effectiveness or implementation). In October 2020, discussions began between the DTC team and implementation scientist co-investigators about expanding to nationwide recruitment. This change was deemed as most urgent because, out of all ineligible respondents for the DTC arm, nearly 25% failed screening due only to their county. This change also was perceived by the implementation scientist investigators to increase the DTC strategy pragmatism given that many commercially available DTC online health programs for other conditions are available more widely (e.g., via App Store, employer-sponsored insurance plans; Lau et al., 2020). However, this modification would fundamentally change the trial design (i.e., no longer cluster-randomized) and affect the ability to directly compare the effectiveness outcomes across arms due to the different scale of enrollment (i.e., county vs. national). Another change discussed was increasing incentives for completing the intervention content – not just the research surveys – to improve recruitment and retention. The investigative team believed this was justifiable as incentives were allowable in CDC funding opportunity announcements and found that it was common in public health practice to incentivize intervention completion (Zamantakis et al., 2025 ). In addition to DTC-strategy-specific changes, the investigative team brainstormed changes that could be applied irrespective of CBO/DTC arm to facilitate enrollment and between-strategy comparisons despite changes to the RCT design. This included: 1) increasing the upper age limit from 29 to 34 following age groupings in CDC epidemiological reports and other studies that included 34 year olds among “YMSM” (Gerke et al., 2022 ; Hill-Rorie et al., 2024 ); 2) no longer requiring participants to have had condomless anal sex in the past 3 months; 3) making breaks between episodes 1–3 optional such that participants could complete most of KIU! in one sitting; 4) abbreviating the breaks between episodes 1–3 and 5 and 6; 5) abbreviating the follow-up period from 12 to 3 months; and 6) eliminating STI testing from the final follow-up. Participant contact notes revealed that breaks were an obstacle to sustained attention, motivation, and engagement in KIU!: "I'm also stuck in the waiting period for the next episode unless that ran out and I missed it because what even is time” (1/23/2021). The DSMB and investigative team met in November 2020 to discuss the changes, which the DSMB supported and provided feedback regarding the impacts on aims, sample size, and safety. In December 2020, the proposed changes were submitted to the NIH program officers for approval. The program officers raised a concern about the request to drop STI testing at final timepoint as few HIV prevention studies at the time collected biomedical effectiveness outcomes; this resulted in retention of the STI testing endpoint. NIH and the university IRB approved all changes in February 2021. The study team spent the rest of this phase updating the KIU! materials, software, and advertising campaigns to reflect the changes. This phase ended with an additional $ 4,474 spent on advertising and DTC enrollment increased to 118 participants. Phase 5: March 2021 – March 2023 (end of trial) . When the changes proposed in Phase 4 went into effect, the DTC team launched national advertising campaigns on Facebook, Instagram, Snapchat, and Grindr and ran another campaign with the LGBTQ-focused marketing company. These were supplemented with offline strategies such as snowball recruitment, referrals from other studies, and participant registries. Those who were previously ineligible due to location were recontacted and informed about the new compensation structure: $ 50 for the first three episodes, $ 10 for the 6-week booster, and $ 15 for the 3-month booster. To facilitate adherence to STI testing at the 3-month endpoint, participants in both CBO and DTC strategies received an additional $ 10, for a total of up to $ 85 in incentives. These changes had a sizable and rapid impact on enrollment. One month after the changes, DTC enrollment increased from 118 to 355 participants (a 301% increase). The average cost of advertising per enrolled participant also dramatically decreased from $ 212 to $ 79. Many participants noted the incentives as an important motivator: “I was actually only interested in the incentives” (6/17/2021). Another sent the text message, “ill do it soon i want the money” (3/29/2021). However, the incentives were associated with a significant increase in imposter participants, resulting in additional measures to verify participants’ identity and their survey data. After reallocating a substantial amount of the budget to advertising in the prior phases, and after deciding to guarantee incentives for both surveys and intervention completion, it became clear that the goal of enrolling 2,200 participants with the remaining budget and timeframe was infeasible. Thus, in the final phase NIH gave permission to reduce the DTC enrollment goal from 2,200 to a minimum of 1,100, and to extend the trial end date from April to December 2022. The previous sample size needed to power the cluster-randomized trial analyses was no longer necessary, and the new sample size would still provide meaningful data. The last participant was recruited in August 2022 yielding 1,468 participants enrolled in KIU! DTC. Across the trial, a total of $ 47,626 (~ $ 32 per enrolled participant) was spent on DTC advertising, significantly more than the $ 27,000 budgeted, and over $ 100,000 was spent on incentives. Figure 3 depicts changes in DTC enrollment over the trial; Table 2 summarizes the barriers in each phase, solutions, and the impact on enrollment. Table 2 Challenges, solutions, and impact on sample size across the five phases of KIU! 3.0 Phase 1 Challenges Solutions Sample Size at Phase End Limitations of online recruitment Multistep eligibility screening and enrollment process Supplement online recruitment with outreach to community stakeholders who can refer participants into study Shortened eligibility screener 15 Phase 2 COVID-19 pandemic limits staff working in the office and impacts HIV/STI testing Lengthy eligibility screener survey Low enrollment At-home HIV and STI test kit preparation and shipments paused; third-party at-home testing option offered Participants allowed to self-report unknown or negative HIV status and enroll into study Registration form collecting basic contact and county info replaced the eligibility screener survey Raffle increased to $ 200 26 Phase 3 Low enrollment and intervention completion rates Restrictive eligibility criteria Randomized incentive of $ 10 or $ 25 offered to participants who complete first three KIU! episodes Online ad campaign in collaboration with LGBTQ-specific marketing and advertising company Incentivized snowball recruitment Eligibility criteria updated: Gender diverse participants assigned male at birth eligible for study 64 Phase 4 Restrictive eligibility criteria, low enrollment and retention rates, forced breaks between intervention content, lengthy study timeline Changes to eligibility criteria, study design, and incentives discussed and proposed to and approved by study DSMB, IRB, and NIH Updates made to KIU! app and databases to reflect these changes and to prepare for national recruitment campaigns 116 Phase 5 Restrictive eligibility criteria, low enrollment and retention rates, forced breaks between intervention content, lengthy study timeline Largest changes made: 1. Expanding DTC catchment area to national recruitment. 2. De-emphasizing effectiveness and focusing on implementation aims. 3. Increasing upper age limits of eligibility from 29 to 34 years. 4. Removing the condomless anal sex eligibility criteria. 5. Making breaks between the first three sessions of the intervention optional. 6. Shortening period of follow-up from 12 months to 3 months. 7. Collecting follow-up data at one time point only (3 months). 8. Increasing the incentive paid to DTC participants who complete the first three sessions of the intervention from $ 10 - $ 25 to $ 50. 9. Paying DTC participants who complete their 6-week and 3-month follow-ups $ 10 and $ 15, respectively. 1468 Retention period. Although enrollment in KIU! DTC improved, retention in the research study components remained a challenge even with incentives. First, retention at the 6-week and 3-month follow-up improved from 38% and 24%, respectively, to roughly 60% across both. In contrast, KIU! 2.0 had > 80% retention (Mustanski et al., 2018 ). The combination of intervention complexity, lower incentives, and the pandemic likely contributed to these lower rates. Also, participation in a study focused on preventing a health condition people did not have – or may not have felt the need for, given constrained opportunities for social and sexual contact in the early pandemic – was likely a lower priority based on communications between staff and participants. Yet final retention rates were still within ranges described in systematic reviews of digital health interventions, which demonstrate greater attrition in digital studies with larger samples, limited or no incentives, and limited or no human contact (Daniore et al., 2022 ; Torous et al., 2020 ). Second, the return rate for STI testing at 3-month follow-up (primary effectiveness outcome) was 48% (n = 699/1468). This lower-than-expected return rate may be explained by the initial decision to send test kits only to participants who reconfirmed their mailing address prior to the 3-month follow-up. The intent was to save resources and avoid mailing kits to participants who had moved or gave an incorrect address. Eventually these test kits were mailed to all participants regardless of address confirmation. This increased return rates from 19% at the end of Phase 4 to 48% at the end of Phase 5. After the last participant was enrolled, for the final 4 months of 2022, the DTC staff shifted into a retention-only phase, sending participants more frequent reminders to complete content and return STI testing kits via personalized calls, emails, and pre-recorded voice messages. Through this work, it became evident that participants thought that all retention reminders were automated. This led to an increase in receiving replies such as “STOP” or “UNSUBSCRIBE” via text. There was difficulty finding a middle ground between personalizing participant reminders for each participant’s situation and study behavior and automating them for efficiency in a large trial. Moreover, some participants seemed suspicious of study motivations when they perceived too many automated messages were sent, with one participant asking via a text: “How are you offering this [STI test] for free? Is my personal information being sold? Why do i need to complete a survey?” (12/01/2020) Discussion Overall, the execution of this project reflected a constant tension between the pragmatic implementation aim (reaching as many people as possible in as real-life of a way as possible) versus the effectiveness aim and the deliverables required by the funders (collecting as much effectiveness outcome data as possible). These tensions often influenced the ability to make decisions that would best facilitate uptake and use of KIU! Although in principle, hybrid type 3 designs prioritize implementation over demonstrating effectiveness, at times it seemed both outcomes were equally important. An exclusive focus on studying implementation of an established effective intervention may have made space for different decisions more aligned with enhancing implementation success. Hybrid effectiveness-implementation study designs in theory accelerate the translation of research to public health practice but in actuality may constrain what can be learned about implementation, at least in the context of an intervention with time-limited external funding that expected collection of biomedical outcome data. Given the challenges in collecting primary data in large implementation trials, it has been suggested that these studies may instead rely on other sources (e.g., administrative, EHR data) as a proxy (Curran et al., 2012 ). Nevertheless, we hope that this narrative of our experience attempting to implement a DTC digital health intervention within a large hybrid type 3 study may pull back the curtain on what one might experience when doing similar studies of other digital interventions. We offer several lessons learned for others planning to study implementation of a digital health intervention. First, we encourage investigators to consider what barriers to entry are critical (if any), including eligibility criteria, steps needed to access the intervention, and the time elapsed between screening eligible and entering the intervention, and we encourage investigators to do a real-time internal pilot of these steps if feasible. At the beginning of KIU! 3.0, study eligibility criteria were still relatively stringent, with multiple steps before participants could access the program. Although our cluster-randomized design was rigorous, ultimately it was not pragmatic for this intervention. In practice, one’s ability to access a self-guided, freely available digital intervention is rarely confined to the county they live in. These barriers, combined with our budget, constrained our ability to reach participants and led to significant attrition before people even accessed KIU! (Benbow et al., 2025 ). When research procedures (e.g., multistep screening, test kit verification, follow-up assessments) are the main barrier to entry, they can undermine implementation success, and may risk distorting what uptake might look like in real-world deployment. Second, we encourage investigators to consider whether their participant incentives are sufficient for their implementation study. Given our past experiences with grateful participants and the literature on altruism as a motivator for health research participation (Williams et al., 2008 ), we overestimated participants’ willingness to participate and underestimated their interest in financial incentives. Eventually the DTC team learned that the CBOs implementing KIU! offered monetary incentives to participants as high as $ 100. Given that the CBO and DTC teams initially operated independently of each other to prevent contamination (Mustanski, Saber, Jones, et al., 2023 ), the DTC team was not aware of these incentives until the trial design was changed. The existence of incentivized HIV prevention programs may have led some participants to expect similar payments for KIU!, and potential participants told us that they lost interest once learning that raffle prizes were the only incentive. In addition, guaranteed benefits likely play a role in reach and retention in implementation studies. In earlier KIU! evaluations, participants indicated that at-home STI testing was convenient as this preceded the rise of DTC health companies who now offer this service, and rectal STI testing was not yet widely available. Yet in this study, these same STI tests and the raffle prizes were insufficient to make study participation appealing. When asked about the reasons for not completing the STI tests, participants’ comments (e.g., “treatments are available”, “I only care about HIV”) suggested that STIs may be less of a concern within this population than HIV (Sarno et al., 2021 ) particularly as many STIs are treatable and preventable with antibiotics (Cannon & Celum, 2023 ). Moreover, since the KIU 2.0! trial, rectal STI testing has become more common in clinical settings, which may have made our provision of these tests less compelling. Providing incentives was valuable to study retention as well. Previous research has shown that user engagement with self-guided digital health apps is high initially, but few sustain engagement over time (Baumel et al., 2019 ). We addressed this issue by offering incentives for each survey and condensing the study timeline. Moreover, given that the study recruited during a time when many young people lost jobs, monetary incentives may have been more salient when making decisions about how they want to spend their time. Our economic analyses showed that even with these incentives, KIU! still represented a cost-saving intervention in the long term (Munroe et al., 2025 ; Mustanski, Benbow, et al., 2025b ). Together these experiences reflect the concept of “conditional altruism” (McCann et al., 2010 ) – although altruism itself may have been part of participants’ calculus, offering the right combination of incentives and/or benefits is likely necessary to yield enrollment. A third lesson learned is that dedicated staffing is a critical part of the implementation strategy for DTC self-guided digital interventions, particularly those that may require greater engagement to be effective. The adage “if you build it, they will come” did not hold true for our intervention, as simply advertising KIU! was not enough reach and retain people. Having a team who is skilled in reaching, engaging, and motivating participants from a distance can be the difference between successful and failed implementation of DTC digital interventions. Further, restrictions in audience targeting on self-service online marketing platforms (e.g., Meta), lower public trust in social media and online advertising (Park et al., 2020 ), and the increase in imposter participants in online research (Roehl & Harland, 2022 ) have made recruiting participants more challenging. Finally, although attrition in digital intervention studies (Daniore et al., 2022 ; Torous et al., 2020 ) is often high, if an implementation study requires effectiveness outcome data, retention is critical. Humanizing the intervention with personalized contacts and using techniques such as motivational interviewing to encourage study completion (Downs, 2019 ; Jake-Schoffman et al., 2021 ) enabled us to foster friendly relationships with study participants and troubleshoot barriers to study completion, but such frequent contacts were not always desired by participants. Finally, unanticipated or underestimated outer setting factors (e.g., the pandemic, attitudes about the necessity of HIV/STI prevention) were major determinants of how the DTC strategy performed (Damschroder, Aron, et al., 2009b ). However, researchers are not always equipped to deal with them, especially if they are newer to testing their interventions in implementation studies. One recommendation is to do a pre-mortem which may enable teams to better anticipate potential pitfalls and develop contingency plans in the event recruitment or retention does not go as planned (Wippold et al., 2025 ). Another is to test digital interventions in the context they were meant to be delivered from the very beginning (rather than beginning with traditional efficacy trials) (Beidas et al., 2023 ) which would enable teams to identify such roadblocks – and move interventions to public health practice – sooner. Conclusion Hybrid type 3 effectiveness-implementation studies of DTC digital health interventions remain relatively uncommon. Our experience in this study was illuminating and has since informed our approach to other hybrid studies of digital and multimedia interventions. This study attempted to approximate real-world conditions, yet research constraints precluded it from achieving naturalistic implementation conditions, and investigators seeking to do similar work may encounter frequent tensions between research and implementation priorities. As digital health services continue to expand to real-world settings, investigators and funders alike may need to reconsider how hybrid designs are structured, from barriers to entry, the extent to which incentives and sustained engagement are important to answer one’s research questions, which outcomes are essential and how they can be easily captured for real-world use, to how researchers who may also be implementers may consider the multiple “hats” they may wear in a study (Curran et al., 2012 ). Careful consideration of these issues can help ensure that the pursuit of rigor does not inadvertently constrain the very implementation processes these studies aim to advance. Declarations Competing interests: The authors declare that they have no competing interests. Ethics approval and consent to participate: Research procedures were approved by the Northwestern University IRB; service components were considered non-human subjects research. All participants provided e-consent and/or verbal consent prior to enrollment in the research portions of the study. Consent for publication: Not applicable. Funding: This project was funded by R01MH118213. Additional resources were provided by NIH grants P30AI117943 and UL1TR001422. AK’s time was supported by K01MH121854. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Author Contribution KRM and KLM contributed equally to the manuscript and are co-first authors. KRM contributed to conception and design of work, acquisition, analysis, and interpretation of data, and wrote the manuscript text. KLM contributed to acquisition, analysis, and interpretation of data, wrote the manuscript text, and prepared the figures and tables. AK contributed to interpretation of data and wrote and substantively revised the manuscript text. MH contributed to analysis of data and substantively revised the manuscript text. MP wrote and substantively revised the manuscript text. BN contributed to acquisition of data and substantively revised the manuscript text. JO contributed to acquisition of data and wrote the manuscript text. IG contributed to acquisition of data and wrote the manuscript text. DHL contributed to design of work, interpretation of data, and substantively revised the manuscript text. BM contributed to the conception and design of work and interpretation of data, and substantively revised the manuscript text. All authors reviewed the manuscript. Acknowledgement We would like to thank Reno Stephens, Chris Owens, Simon Abimosleh, Daniel Kellogg, Lauren Redman, Alex Saldivar, Emily Bettin, Val Wongsomboon, Rana Saber, and Greg Swann for their collaboration. We would also like to thank Maggi Price and Kelly Shaffer for their constructive feedback on an earlier draft of this manuscript, and to Susannah Allison and Rich Jenkins at NIH for their guidance throughout the changes to our study. We are grateful to all the participants in KIU, from whom we learned so much about implementation during a difficult time. Data Availability Limited data available upon request. References Bassett, I. V., Wilson, D., Taaffe, J., & Freedberg, K. A. (2015). Financial incentives to improve progression through the HIV treatment cascade. Current Opinion in HIV and AIDS , 10 (6), 451–463. https://doi.org/10.1097/COH.0000000000000196 Baumel, A., Muench, F., Edan, S., & Kane, J. M. (2019). Objective User Engagement With Mental Health Apps: Systematic Search and Panel-Based Usage Analysis. Journal of Medical Internet Research , 21 (9), e14567. https://doi.org/10.2196/14567 Beidas, R. S., Saldana, L., & Shelton, R. C. (2023). Testing psychosocial interventions in the contexts they are meant to be delivered. Journal of Consulting and Clinical Psychology , 91 (4), 189–191. https://doi.org/10.1037/ccp0000797 Benbow, N., Li, D. H., Macapagal, K., Madkins, K., Saber, R., zamantakis, alithia, Rudd, E., Smith, J. D., & Mustanski, B. (2025). Considerations for evaluating pragmatic design elements in digital health intervention trials: The case of Keep It Up! 3.0. Implementation Science Communications , 6 (1), 101. https://doi.org/10.1186/s43058-025-00777-y Cannon, C. A., & Celum, C. L. (2023). Doxycycline postexposure prophylaxis for prevention of sexually transmitted infections. Topics in Antiviral Medicine , 31 (5), 566–575. CDC. (2025a, February 7). Estimated HIV Incidence and Prevalence . HIV Data. https://www.cdc.gov/hiv-data/nhss/estimated-hiv-incidence-and-prevalence.html CDC. (2025b, May 27). HIV Compendium of Best Practices . HIV Public Health Partners. https://www.cdc.gov/hivpartners/php/compendium/index.html Centers for Disease Control. (2024). Compendium of Evidence-Based Interventions and Best Practices for HIV Prevention: Background, Methods, and Criteria . Division of HIV Prevention. https://www.cdc.gov/hiv/pdf/research/interventionresearch/compendium/PRS-Project-Compendium-Background-Methods-and-Criteria.pdf Clark, R. C., & Mayer, R. E. (2008). Learning by viewing versus learning by doing: Evidence-based guidelines for principled learning environments. Performance Improvement , 47 (9), 5–13. https://doi.org/10.1002/pfi.20028 Curran, G. M., Bauer, M., Mittman, B., Pyne, J. M., & Stetler, C. (2012). Effectiveness-implementation Hybrid Designs. Medical Care , 50 (3), 217–226. https://doi.org/10.1097/MLR.0b013e3182408812 Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009a). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science , 4 (1), 50. https://doi.org/10.1186/1748-5908-4-50 Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009b). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science: IS , 4 , 50. https://doi.org/10.1186/1748-5908-4-50 Damschroder, L. J., Banaszak-Holl, J., Kowalski, C. P., Forman, J., Saint, S., & Krein, S. L. (2009). The role of the champion in infection prevention: Results from a multisite qualitative study. Quality & Safety in Health Care , 18 (6), 434–440. https://doi.org/10.1136/qshc.2009.034199 Daniore, P., Nittas, V., & von Wyl, V. (2022). Enrollment and Retention of Participants in Remote Digital Health Studies: Scoping Review and Framework Proposal. Journal of Medical Internet Research , 24 (9), e39910. https://doi.org/10.2196/39910 Downs, H. (2019). Bridging the Gap: How the Generations Communicate. Concordia Journal of Communication Research , 6 (1). https://doi.org/https://doi.org/10.54416/SEZY7453 EHE Priority Jurisdictions . (n.d.). HIV.Gov. Retrieved February 6, 2026, from https://www.hiv.gov/federal-response/ending-the-hiv-epidemic/jurisdictions Fisher, J. D., Fisher, W. A., Williams, S. S., & Malloy, T. E. (1994). Empirical tests of an information-motivation-behavioral skills model of AIDS-preventive behavior with gay men and heterosexual university students. Health Psychology , 13 (3), 238–250. https://doi.org/10.1037/0278-6133.13.3.238 Gerke, D., Call, J., & Auslander, W. (2022). The Syndemic Factors of Violence Exposure, Substance Use, and Mental Health Problems: Relationships to Sexual Risk Behaviors in HIV-Negative Young Men Who Have Sex With Men. Journal of the Society for Social Work & Research , 13 (2). https://doi.org/10.1086/711612 Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health , 89 (9), 1322–1327. https://doi.org/10.2105/AJPH.89.9.1322 Hardcastle, S. J., Hancox, J., Hattar, A., Maxwell-Smith, C., Thøgersen-Ntoumani, C., & Hagger, M. S. (2015). Motivating the unmotivated: How can health behavior be changed in those unwilling to change? Frontiers in Psychology , 6 , 835. https://doi.org/10.3389/fpsyg.2015.00835 Hermes, E. D. A., Lyon, A. R., Schueller, S. M., & Glass, J. E. (2019). Measuring the Implementation of Behavioral Intervention Technologies: Recharacterization of Established Outcomes. Journal of Medical Internet Research , 21 (1). https://doi.org/10.2196/11752 Hill-Rorie, J., Biello, K. B., Quint, M., Johnson, B., Elopre, L., Johnson, K., Lillis, R., Burgan, K., Krakower, D., Whiteside, Y., & Mayer, K. H. (2024). Weighing the Options: Which PrEP (Pre-exposure Prophylaxis) Modality Attributes Influence Choice for Young Gay and Bisexual Men in the United States? AIDS and Behavior , 28 (9), 2970–2978. https://doi.org/10.1007/s10461-024-04384-1 Jake-Schoffman, D. E., Brown, S. D., Baiocchi, M., Bibeau, J. L., Daubenmier, J., Ferrara, A., Galarce, M. N., Hartogensis, W., Hecht, F. M., Hedderson, M. M., Moran, P. J., Pagoto, S. L., Tsai, A.-L., Waring, M. E., & Kiernan, M. (2021). Methods-Motivational Interviewing Approach for Enhanced Retention and Attendance. American Journal of Preventive Medicine , 61 (4), 606–617. https://doi.org/10.1016/j.amepre.2021.04.005 Li, D. H., Brown, C. H., Gallo, C., Morgan, E., Sullivan, P. S., Young, S. D., & Mustanski, B. (2019). Design Considerations for Implementing eHealth Behavioral Interventions for HIV Prevention in Evolving Sociotechnical Landscapes. Current HIV/AIDS Reports , 16 (4), 335–348. https://doi.org/10.1007/s11904-019-00455-4 Li, D. H., Macapagal, K., Mongrella, M., Saber, R., & Mustanski, B. (2024). “Your Package Could Not Be Delivered”: The State of Digital HIV Intervention Implementation in the US. Current HIV/AIDS Reports , 21 (3), 152–167. https://doi.org/10.1007/s11904-024-00693-1 Macapagal, K., Li, D. H., Clifford, A., Madkins, K., & Mustanski, B. (2020). The CAN-DO-IT Model: A Process for Developing and Refining Online Recruitment in HIV/AIDS and Sexual Health Research. Current HIV/AIDS Reports , 17 (3), 190–202. https://doi.org/10.1007/s11904-020-00491-5 Mayer, R. e. (2017). Using multimedia for e-learning. Journal of Computer Assisted Learning , 33 (5), 403–423. https://doi.org/10.1111/jcal.12197 McCann, S. K., Campbell, M. K., & Entwistle, V. A. (2010). Reasons for participating in randomised controlled trials: Conditional altruism and considerations for self. Trials , 11 (1), 31. https://doi.org/10.1186/1745-6215-11-31 Munroe, S., Schackman, B. R., Madkins, K., Saber, R., Macapagal, K., Vititow, J., Sweeney, N., Feder, N. M., Benbow, N., Mustanski, B., & Linas, B. P. (2025). Economic Evaluation of Delivering an Evidence-Based Online HIV Prevention Program to MSM via Direct-To-Consumer Marketing Versus Community-Based Organization Recruitment. JAIDS Journal of Acquired Immune Deficiency Syndromes , 98 (3), 257–264. https://doi.org/10.1097/QAI.0000000000003572 Mustanski, B., Benbow, N., Macapagal, K., Li, D., Madkins, K., Saber, R., Linas, B., Smith, J., Brown, C. H., Munroe, S., Reddy, S., Schackman, B. R., Swann, G., Janulis, P., Zamantakis, A., & Zapata, J. P. (2025a). Comparing Implementation and Effectiveness Outcomes for Two Implementation Strategies of the Keep It Up! Digital HIV Prevention Program: A Type 3 Hybrid Effectiveness-Implementation Trial. AIDS and Behavior , 29 (12), 4030–4042. https://doi.org/10.1007/s10461-025-04838-0 Mustanski, B., Benbow, N., Macapagal, K., Li, D., Madkins, K., Saber, R., Linas, B., Smith, J. D., Brown, C. H., Munroe, S., Reddy, S., Schackman, B. R., Swann, G., Janulis, P., Zamantakis, A., & Zapata, J. P. (2025b). Comparing Implementation and Effectiveness Outcomes for Two Implementation Strategies of the Keep It Up! Digital HIV Prevention Program: A Type 3 Hybrid Effectiveness-Implementation Trial. AIDS and Behavior , 29 (12), 4030–4042. https://doi.org/10.1007/s10461-025-04838-0 Mustanski, B., Garofalo, R., Monahan, C., Gratzer, B., & Andrews, R. (2013). Feasibility, Acceptability, and Preliminary Efficacy of an Online HIV Prevention Program for Diverse Young Men who have Sex with Men: The Keep It Up! Intervention. AIDS and Behavior , 17 (9), 2999–3012. https://doi.org/10.1007/s10461-013-0507-z Mustanski, B., Macapagal, K., Li, D. H., Rodriguez-Diaz, C. E., Saber, R., Matson, M., Moskowitz, D. A., Rendina, H. J., Laber, E., Ryan, D. T., & Newcomb, M. E. (2025). Effectiveness of the smart program: Stepped-care HIV prevention for gay and bisexual adolescent boys. Health Psychology , 44 (3), 321–331. https://doi.org/10.1037/hea0001471 Mustanski, B., Madkins, K., Greene, G. J., Parsons, J. T., Johnson, B. A., Sullivan, P., Bass, M., & Abel, R. (2017). Internet-Based HIV Prevention With At-Home Sexually Transmitted Infection Testing for Young Men Having Sex With Men: Study Protocol of a Randomized Controlled Trial of Keep It Up! 2.0. JMIR Research Protocols , 6 (1), e5740. https://doi.org/10.2196/resprot.5740 Mustanski, B., Parsons, J. T., Sullivan, P. S., Madkins, K., Rosenberg, E., & Swann, G. (2018). Biomedical and Behavioral Outcomes of Keep It Up!: An eHealth HIV Prevention Program RCT. American Journal of Preventive Medicine , 55 (2), 151–158. https://doi.org/10.1016/j.amepre.2018.04.026 Mustanski, B., Saber, R., Jones, J. P., Macapagal, K., Benbow, N., Li, D. H., Brown, C. H., Janulis, P., Smith, J. D., Marsh, E., Schackman, B. R., Linas, B. P., Madkins, K., Swann, G., Dean, A., Bettin, E., & Savinkina, A. (2023). Keep it up! 3.0: Study protocol for a type III hybrid implementation-effectiveness cluster-randomized trial. Contemporary Clinical Trials , 127 , 107134. https://doi.org/10.1016/j.cct.2023.107134 Mustanski, B., Saber, R., Macapagal, K., Matson, M., Laber, E., Rodrgiuez-Diaz, C., Moran, K. O., Carrion, A., Moskowitz, D. A., & Newcomb, M. E. (2023). Effectiveness of the SMART Sex Ed program among 13–18 year old English and Spanish speaking adolescent men who have sex with men. AIDS and Behavior , 27 (2), 733–744. https://doi.org/10.1007/s10461-022-03806-2 Nguyen, L. H., Tran, B. X., Rocha, L. E. C., Nguyen, H. L. T., Yang, C., Latkin, C. A., Thorson, A., & Strömdahl, S. (2019). A Systematic Review of eHealth Interventions Addressing HIV/STI Prevention Among Men Who Have Sex With Men. AIDS and Behavior , 23 (9), 2253–2272. https://doi.org/10.1007/s10461-019-02626-1 Park, S., Fisher, C., Flew, T., & Dulleck, U. (2020). Global Mistrust in News: The Impact of Social Media on Trust. International Journal on Media Management , 22 (2), 83–96. https://doi.org/10.1080/14241277.2020.1799794 Roehl, J. M., & Harland, D. J. (2022). “Imposter Participants: Overcoming Methodological Challenges Related to” by Jacqueline M. Roehl Ph.D. and Darci J. Harland Ph.D. https://nsuworks.nova.edu/tqr/vol27/iss11/5/ Ruth Colvin, & Mayer, R. E. (Eds.). (2016). E-Learning and the Science of Instruction (1st ed.). John Wiley & Sons, Ltd. https://doi.org/10.1002/9781119239086 Sarno, E. L., Macapagal, K., & Newcomb, M. E. (2021). “The Main Concern is HIV, Everything Else is Fixable”: Indifference Toward Sexually Transmitted Infections in the Era of Biomedical HIV Prevention. AIDS and Behavior , 25 (8), 2657–2660. https://doi.org/10.1007/s10461-021-03226-8 Schon, D. A. (1963). Champions for radical new inventions. Harvard Business Review : HBR, Harvard Business Review : HBR. - Boston, Mass. : Harvard Business School Publ. Corp., ISSN 0017-8012, ZDB-ID 2382-6. - Vol. 41.1963, 2, p. 77-86 , 41 (2), 77. Sharma, A., Sullivan, P. S., & Khosropour, C. M. (2011). Willingness to Take a Free Home HIV Test and Associated Factors among Internet-Using Men Who Have Sex with Men. Journal of the International Association of Physicians in AIDS Care , 10 (6), 357–364. https://doi.org/10.1177/1545109711404946 Teitcher, J. E. F., Bockting, W. O., Bauermeister, J. A., Hoefer, C. J., Miner, M. H., & Klitzman, R. L. (2015). Detecting, Preventing, and Responding to “Fraudsters” in Internet Research: Ethics and Tradeoffs. Journal of Law, Medicine & Ethics , 43 (1), 116–133. https://doi.org/10.1111/jlme.12200 Torous, J., Lipschitz, J., Ng, M., & Firth, J. (2020). Dropout rates in clinical trials of smartphone apps for depressive symptoms: A systematic review and meta-analysis. Journal of Affective Disorders , 263 , 413–419. https://doi.org/10.1016/j.jad.2019.11.167 Valente, T. W., & Pumpuang, P. (2007). Identifying Opinion Leaders to Promote Behavior Change. Health Education & Behavior , 34 (6), 881–896. https://doi.org/10.1177/1090198106297855 Williams, B., Entwistle, V., Haddow, G., & Wells, M. (2008). Promoting research participation: Why not advertise altruism? Social Science and Medicine , 66 (7), 1451–1456. https://doi.org/10.1016/j.socscimed.2007.12.013 Wippold, G. M., Garcia, K. A., Domlyn, A., Wong, D., & Fernandez, M. E. (2025). Proactive planning for contextual fit: The role of the implementation premortem. Implementation Science Communications , 6 (1), 84. https://doi.org/10.1186/s43058-025-00768-z Zamantakis, A., Zapata, J. P., Smith, J. D., Danielson, E. C., Macapagal, K., Saber, R., Li, D., Benbow, N., & Mustanski, B. (2025). Incentivizing Prevention: Community Based Organizations’ Perceptions on Financial Incentives for Recruitment and Retention of Participants in Keep it Up! (KIU!), an eHealth HIV Prevention Intervention. AIDS and Behavior . https://doi.org/10.1007/s10461-025-04963-w Zhang, Y., Guy, R. J., Smith, K. S., Jamil, M. S., Prestage, G., Applegate, T. L., Conway, D. P., Holt, M., Keen, P., Bavinton, B., McNulty, A. M., Batrouney, C., Russell, D., Vaughan, M., Chen, M., Fairley, C. K., Grulich, A. E., Kaldor, J. M., & Callander, D. (2021). Sustaining success: A qualitative study of gay and bisexual men’s experiences and perceptions of HIV self-testing in a randomized controlled trial. BMC Public Health , 21 , 2048. https://doi.org/10.1186/s12889-021-12011-0 Additional Declarations No competing interests reported. Cite Share Download PDF Status: Under Review Version 1 posted Reviewers agreed at journal 17 Apr, 2026 Reviewers invited by journal 14 Apr, 2026 Editor assigned by journal 07 Apr, 2026 Submission checks completed at journal 24 Mar, 2026 First submitted to journal 23 Mar, 2026 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-9202620","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":626323258,"identity":"6a3414c5-089d-4a1a-8356-50b711ff258a","order_by":0,"name":"Kathryn R. Macapagal","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAABT0lEQVRIie2RMUvDQBSAXwlclrNd7wi0v0DwCGSx4l9JOOgUxUko1DZdrptd03/R0c2Ug3RJzRqwgyWQSQcRRFHES1uhaUUcBfMN94733nfvuAMoKfmDEABUzFR1Lw/BRiooNBDQthSE8w47X7TfKsT+WaGDOEzbYg77A7l4OutcNBB9mKTtl7kzrMVqcwX1amJvKgbmOpuJDKyoZRp+OGXCOOUssjNn5HONzSIwaVGpA0e0LyRYgat8FFaE4VrUs+XJONHyEjjjLaWWrpX4XnvDH+GxoNFKuY6l/qqU3pZikK8piYuMPdFxBMHrKeoCFaXYB0WF+qlquJHYSjLrcHQZcIFdk3kt2fMTbqrTCBtFd4VHjp2MeueybsU8vT177h4N9YgtvKY0a8PJ4rEvmo3qtDBlSQUBhtU/yJ0i2ckseV8FpXS/bygpKSn513wCLb+AKBumg+oAAAAASUVORK5CYII=","orcid":"","institution":"Northwestern University","correspondingAuthor":true,"prefix":"","firstName":"Kathryn","middleName":"R.","lastName":"Macapagal","suffix":""},{"id":626323259,"identity":"4e8a29cf-6a86-4055-acf1-7ea286d67082","order_by":1,"name":"Krystal L. Madkins","email":"","orcid":"","institution":"Northwestern University","correspondingAuthor":false,"prefix":"","firstName":"Krystal","middleName":"L.","lastName":"Madkins","suffix":""},{"id":626323260,"identity":"a3a90683-cd56-482c-9a41-0ca6d851ac9f","order_by":2,"name":"Ashley Knapp","email":"","orcid":"","institution":"Northwestern University","correspondingAuthor":false,"prefix":"","firstName":"Ashley","middleName":"","lastName":"Knapp","suffix":""},{"id":626323261,"identity":"ae1607fc-3347-4c94-8531-7aed46088937","order_by":3,"name":"Manuel Hurtado, Jr.","email":"","orcid":"","institution":"Northwestern University","correspondingAuthor":false,"prefix":"","firstName":"Manuel","middleName":"","lastName":"Hurtado","suffix":"Jr."},{"id":626323262,"identity":"2d75fdf3-fcb1-453a-bf60-dd41e99e05ce","order_by":4,"name":"Mariajosé Paton","email":"","orcid":"","institution":"Northwestern University","correspondingAuthor":false,"prefix":"","firstName":"Mariajosé","middleName":"","lastName":"Paton","suffix":""},{"id":626323263,"identity":"46e69b0a-fcf9-4f85-a12b-1a2e2c9536a2","order_by":5,"name":"Bryant Norton","email":"","orcid":"","institution":"Northwestern University","correspondingAuthor":false,"prefix":"","firstName":"Bryant","middleName":"","lastName":"Norton","suffix":""},{"id":626323264,"identity":"90df1f1f-62c6-44ee-8aad-d2c1f6d27a85","order_by":6,"name":"Josephine Owusu","email":"","orcid":"","institution":"University of Illinois at Chicago","correspondingAuthor":false,"prefix":"","firstName":"Josephine","middleName":"","lastName":"Owusu","suffix":""},{"id":626323265,"identity":"45385d47-44ec-4313-abb0-1f8d8c9a2db8","order_by":7,"name":"Isaac Greenawalt","email":"","orcid":"","institution":"Long Island University","correspondingAuthor":false,"prefix":"","firstName":"Isaac","middleName":"","lastName":"Greenawalt","suffix":""},{"id":626323266,"identity":"417eee5b-9b8b-4142-a12b-57f97d96341c","order_by":8,"name":"Dennis H. Li","email":"","orcid":"","institution":"Northwestern University","correspondingAuthor":false,"prefix":"","firstName":"Dennis","middleName":"H.","lastName":"Li","suffix":""},{"id":626323267,"identity":"ce895b9b-7031-43d7-9805-0ca1ab486a27","order_by":9,"name":"Brian Mustanski","email":"","orcid":"","institution":"Northwestern University","correspondingAuthor":false,"prefix":"","firstName":"Brian","middleName":"","lastName":"Mustanski","suffix":""}],"badges":[],"createdAt":"2026-03-23 15:53:20","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-9202620/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-9202620/v1","draftVersion":[],"editorialEvents":[],"editorialNote":"","failedWorkflow":false,"files":[{"id":107615189,"identity":"092a1b22-a910-4b21-b487-838e9d4bde92","added_by":"auto","created_at":"2026-04-23 09:11:07","extension":"png","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":50403,"visible":true,"origin":"","legend":"\u003cp\u003eParticipant Workflow during Study Phase 1\u003c/p\u003e","description":"","filename":"floatimage1.png","url":"https://assets-eu.researchsquare.com/files/rs-9202620/v1/0768b20ac6fbaa833db13b61.png"},{"id":107615759,"identity":"ccdea5c7-a058-4feb-b3f7-f0cc80c84112","added_by":"auto","created_at":"2026-04-23 09:12:19","extension":"png","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":62695,"visible":true,"origin":"","legend":"\u003cp\u003eComparison of Enrollment Processes Implemented during Study Phases 1 and 2\u003c/p\u003e","description":"","filename":"floatimage2.png","url":"https://assets-eu.researchsquare.com/files/rs-9202620/v1/16ad3c8d2bc87656b6b44e02.png"},{"id":107615579,"identity":"11e06e84-b376-4f77-90bb-36528a375546","added_by":"auto","created_at":"2026-04-23 09:11:50","extension":"jpg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":75928,"visible":true,"origin":"","legend":"\u003cp\u003eParticipant enrollment through the five phases of the direct-to-consumer arm of the KIU! 3.0 hybrid effectiveness-implementation study, 2019-2022\u003c/p\u003e","description":"","filename":"3.jpg","url":"https://assets-eu.researchsquare.com/files/rs-9202620/v1/2e44470a7aaaca9598d9d1d5.jpg"},{"id":107616253,"identity":"253d7ff3-3c13-437f-a590-75ec74339936","added_by":"auto","created_at":"2026-04-23 09:13:00","extension":"pdf","order_by":0,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":863042,"visible":true,"origin":"","legend":"","description":"","filename":"manuscript.pdf","url":"https://assets-eu.researchsquare.com/files/rs-9202620/v1/b93ff74c-6e7b-4819-9db3-dedc1c8d5035.pdf"}],"financialInterests":"No competing interests reported.","formattedTitle":"When scientific constraints and real-world implementation collide: Lessons from a hybrid type 3 study of a digital, direct-to-consumer HIV prevention program","fulltext":[{"header":"Contributions to the literature","content":"\u003cul\u003e\n \u003cli\u003eHybrid type 3 studies intend to prioritize understanding of implementation over effectiveness. However, few hybrid type 3 studies of digital interventions that are delivered directly to the consumer have been reported in the literature.\u003c/li\u003e\n \u003cli\u003eThis study illuminated how various constraints and outer setting factors (e.g., pandemic, funder requirement to collect biomedical effectiveness data) created tensions between making the intervention delivery as \u0026ldquo;real world\u0026rdquo; as possible and engaging and retaining participants long enough to collect endpoint data.\u0026nbsp;\u003c/li\u003e\n \u003cli\u003eFindings can inform the design of future hybrid type 3 studies of digital interventions to increase their real-world applicability and reduce burden of effectiveness data collection.\u003c/li\u003e\n\u003c/ul\u003e"},{"header":"Introduction","content":"\u003cp\u003eIn the United States (U.S.), men who have sex with men (MSM) accounted for 67% of new HIV diagnoses in 2022 (CDC, \u003cspan class=\"CitationRef\"\u003e2025a\u003c/span\u003e), most of which were within young MSM aged 13–34 (CDC, \u003cspan class=\"CitationRef\"\u003e2025a\u003c/span\u003e). Digital HIV interventions (DHIs; e.g., interactive websites, smartphone apps, text messaging programs) tailored to their needs are efficacious in reducing HIV transmission risk behavior and sexually transmitted infection (STI) incidence and increasing health promotive behaviors (e.g., pre-exposure prophylaxis [PrEP] use, HIV testing). DHIs can reduce barriers to care if delivered “direct-to-consumer” (DTC) rather than through intermediaries, such as healthcare providers or community health centers (Li et al., \u003cspan class=\"CitationRef\"\u003e2019\u003c/span\u003e; Nguyen et al., \u003cspan class=\"CitationRef\"\u003e2019\u003c/span\u003e; Zhang et al., \u003cspan class=\"CitationRef\"\u003e2021\u003c/span\u003e). Currently, 29 DHIs are listed in a compendium of “best practices” in HIV risk reduction (CDC, \u003cspan class=\"CitationRef\"\u003e2025b\u003c/span\u003e), but none are being implemented in public health practice. This is because despite significant federal investment in research to \u003cem\u003edevelop and test\u003c/em\u003e DHIs, relatively little research has attempted to \u003cem\u003eimplement\u003c/em\u003e them in real-world settings, and those who have reported significant obstacles (Hermes et al., \u003cspan class=\"CitationRef\"\u003e2019\u003c/span\u003e; Li et al., \u003cspan class=\"CitationRef\"\u003e2024\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eResearchers have studied DHI implementation in two main ways: via healthcare or community-based organizations (CBOs) trained to offer the intervention to clients and via a DTC model, where an organization hosts and/or markets the intervention to people who can self-enroll (Benbow et al., \u003cspan class=\"CitationRef\"\u003e2025\u003c/span\u003e; Mustanski, Benbow, et al., \u003cspan class=\"CitationRef\"\u003e2025b\u003c/span\u003e; Mustanski, Macapagal, et al., \u003cspan class=\"CitationRef\"\u003e2025\u003c/span\u003e; Mustanski, Saber, Macapagal, et al., \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e). Most literature on DHI implementation has focused on healthcare settings with “captive audiences” of existing patients or clients. In contrast, a DTC model requires the implementer (e.g., digital health company, health department) to focus on creating significant demand for the intervention (via marketing, incentives, etc.) among people who may not perceive the need for one (Aref-Adib et al., 2019; Germain et al., 2021; Matson et al., 2025; Ross et al., 2018). Yet this may be challenging for preventive interventions as individuals are often less motivated to address health problems before they arise (Hardcastle et al., \u003cspan class=\"CitationRef\"\u003e2015\u003c/span\u003e; Hermes et al., \u003cspan class=\"CitationRef\"\u003e2019\u003c/span\u003e). In traditional effectiveness trials, demand creation activities are ancillary and often not detailed in publications, but in implementation studies they are essential in supporting DTC intervention delivery. How well these strategies operate in a “real-world” setting has not been well-documented.\u003c/p\u003e\n\u003ch3\u003eThe Keep It Up! (KIU!) intervention\u003c/h3\u003e\n\u003cp\u003eWe developed KIU!, a self-guided DHI for YMSM that is among the CDC’s best-evidence HIV prevention programs (Centers for Disease Control, \u003cspan class=\"CitationRef\"\u003e2024\u003c/span\u003e) and based on the Information-Motivation-Behavioral Skills theory of health behavior change (Fisher et al., \u003cspan class=\"CitationRef\"\u003e1994\u003c/span\u003e). KIU! was originally designed to be delivered in CBOs after YMSM tested HIV-negative to encourage them to “keep up” their preventive behaviors (KIU! 1.0; (Mustanski et al., \u003cspan class=\"CitationRef\"\u003e2013\u003c/span\u003e). A randomized controlled trial (KIU! 2.0) examined the effectiveness of CBO-based delivery in three U.S. cities and piloted a nationwide DTC strategy where staff recruited YMSM through online advertising, shipped HIV/STI test kits to participants, and granted access to KIU! when a participant tested HIV-negative (Mustanski et al., \u003cspan class=\"CitationRef\"\u003e2017\u003c/span\u003e). KIU! reduced self-reported condomless anal sex and gonorrhea and chlamydia incidence by 40% versus an HIV information-only control arm (Mustanski et al., \u003cspan class=\"CitationRef\"\u003e2018\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eFrom 2019 to 2023, we conducted a hybrid type 3 implementation–effectiveness study (KIU! 3.0) comparing the CBO and DTC strategies; the protocol and findings are described elsewhere (Mustanski, Benbow, et al., \u003cspan class=\"CitationRef\"\u003e2025b\u003c/span\u003e; Mustanski, Saber, Jones, et al., \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e). We chose this design to approximate real-world implementation; however, the constraints of a research study necessarily limits flexibility. As few studies explore DTC delivery of DHIs, this trial offered a unique opportunity to examine KIU!’s effectiveness in a more naturalistic setting, but also examine what worked (and did not) regarding implementation.\u003c/p\u003e \u003cp\u003eThis paper describes the initial implementation plans for the KIU! 3.0 DTC strategy and how they evolved because of recruitment challenges and COVID-19 pandemic disruptions. We also discuss tensions between the research elements necessary to measure KIU!’s effectiveness and pragmatic elements more aligned with real-world delivery. Our goal is to offer a “behind the scenes” look at the study operations, decision making, and lessons learned in demand creation, and to share ideas about what can/cannot change during a hybrid 3 study of a DHI which can inform future pragmatic studies and reduce implementation barriers (Li et al., \u003cspan class=\"CitationRef\"\u003e2019\u003c/span\u003e).\u003c/p\u003e "},{"header":"Methods","content":"\u003ch2\u003eInitial study design and context\u003c/h2\u003e\u003cp\u003eThe KIU! 3.0 intervention consisted of five “episodes” (three reflecting the main intervention content and two boosters), each containing a webseries, didactics, and activities to augment learning and engagement that totaled ~ 1.5h. After episodes 1–3, participants waited 8h until the next episode. A feature of earlier KIU! versions, this break was based on guidelines to enhance information retention in multimedia learning (Clark \u0026amp; Mayer, \u003cspan class=\"CitationRef\"\u003e2008\u003c/span\u003e; Mayer, \u003cspan class=\"CitationRef\"\u003e2017\u003c/span\u003e, 2016). Booster content in episodes 4 and 5 (at 3- and 6-months post-intervention, respectively) reinforced learning. Participants completed a final survey and STI testing at 12 months post-intervention.\u003c/p\u003e\u003cp\u003eThe study initially proposed a cluster-randomized design to compare the implementation and effectiveness of the KIU! CBO versus DTC strategies in 44 U.S. counties with high HIV incidence among YMSM (Mustanski, Saber, Jones, et al., \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e). Enrollment goals were 100 per county (N = 4400), and county-level randomization was selected to mirror how HIV public health funding often works and to compare implementation in high-incidence jurisdictions outlined in the federal Ending the HIV Epidemic initiative (\u003cem\u003eEHE Priority Jurisdictions\u003c/em\u003e, n.d.).\u003c/p\u003e\u003cp\u003eIn the CBO approach, CBOs in 22 counties received funding to deliver KIU! through their HIV prevention and testing programs. In the DTC approach, centrally-located staff recruited YMSM in 22 other counties using online advertising and shipped HIV self-tests and rectal and urethral chlamydia and gonorrhea sample collection kits (the latter two of which they were to ship to a lab for processing). In both approaches, after e-consenting to participation and providing documentation of an HIV-negative test result participants received access to KIU! Implementation assessments were based on RE-AIM and the Consolidated Framework for Implementation Research (CFIR) (Damschroder, Aron, et al., \u003cspan class=\"CitationRef\"\u003e2009a\u003c/span\u003e; Glasgow et al., \u003cspan class=\"CitationRef\"\u003e1999\u003c/span\u003e). Effectiveness outcomes included STI diagnoses, PrEP uptake, or self-reported condomless anal sex at 12-month follow-up (later abbreviated to 3-months, described later).\u003c/p\u003e\u003cp\u003eDue to the dual effectiveness and implementation foci, we needed to distinguish activities in the trial to properly attribute their effects on outcomes. Activities were either part of the \u003cem\u003eservice\u003c/em\u003e necessary for implementation (i.e., intervention content, delivery) or part of the \u003cem\u003eresearch\u003c/em\u003e intended to assess intervention effectiveness (i.e., surveys, HIV/STI testing). All interested individuals were given access to the KIU! intervention but had to meet study eligibility criteria to complete the research components.\u003c/p\u003e\u003cp\u003eResearch procedures were approved by the Northwestern University IRB; service components were considered non-human subjects research (Mustanski, Benbow, et al., \u003cspan class=\"CitationRef\"\u003e2025a\u003c/span\u003e; Mustanski, Saber, Jones, et al., \u003cspan class=\"CitationRef\"\u003e2023\u003c/span\u003e). All participants provided e-consent and/or verbal consent prior to enrollment in the research portions of the study.\u003c/p\u003e\u003ch3\u003eAssumptions and constraints\u003c/h3\u003e\u003cp\u003eAs no DHIs were commercially available at the time of the study, we made assumptions about how DTC delivery would occur in the “real world” while also considering the research constraints of our trial (e.g., timeline, county-level recruitment, sampling quotas, budget constraints) (Benbow et al., \u003cspan class=\"CitationRef\"\u003e2025\u003c/span\u003e). One major assumption was that those voluntarily seeking HIV prevention or sexual health services would not expect to be paid, based on other digital health applications focused on prevention and behavior change (e.g., mindfulness, fitness) were available in the App Store or Google Play (i.e., DTC). As such, at the beginning of the study incentives were not guaranteed, but participants were entered into raffles for \u003cspan\u003e$\u003c/span\u003e50 gift cards or sexual health products for completing research surveys. Other updates were made to increase pragmatism based on assumptions about real-world DHIs. Table\u0026nbsp;1 compares DTC delivery of KIU! during the effectiveness trial (KIU! 2.0) with features of the KIU! 3.0 implementation study as originally planned, and the final design after all changes were made.\u003c/p\u003e\u003cdiv class=\"gridtable\"\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003cdiv align=\"left\" class=\"colspec\"\u003e\u003c/div\u003e\u003ctable id=\"Taba\" border=\"1\"\u003e \u003ccolgroup cols=\"5\"\u003e \u003c/colgroup\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colspan=\"4\"\u003e \u003cp\u003eTable\u0026nbsp;1. Eligibility and methods of KIU! effectiveness trial, vs. original design and final design of the hybrid study (DTC delivery only)\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colspan=\"1\"\u003e\u0026nbsp;\u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eComponent\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eKIU! 2.0\u003c/p\u003e \u003cp\u003eEffectiveness Trial\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eKIU! 3.0\u003c/p\u003e \u003cp\u003eImplementation-Effectiveness Study\u003c/p\u003e \u003cp\u003e(original design)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\"\u003e \u003cp\u003eKIU! 3.0\u003c/p\u003e \u003cp\u003eImplementation-Effectiveness Study\u003c/p\u003e \u003cp\u003e(final design)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eEligibility\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\"\u003e\u0026nbsp;\u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\"\u003e\u0026nbsp;\u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eAge\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003e18–29\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003e18–29\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\"\u003e \u003cp\u003e18–34\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eHIV status\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eNegative result from\u003c/p\u003e \u003cp\u003eat-home HIV test kit\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eNegative result from at-home HIV test kit\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\"\u003e \u003cp\u003eSelf-reported HIV negative or unknown status\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eRelationship status\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eNot in a monogamous/exclusive relationship lasting more than 6 months\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eNo requirement\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\"\u003e \u003cp\u003eNo requirement\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eSexual risk behavior\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eCondomless anal sex with a male partner in prior\u003c/p\u003e \u003cp\u003e6 months\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eCondomless anal sex with a male partner in prior\u003c/p\u003e \u003cp\u003e6 months\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\"\u003e \u003cp\u003eNo requirement\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003ePrEP use\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eNo requirement\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eNo PrEP use OR not adherent in prior 6 months\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\"\u003e \u003cp\u003eNo PrEP use OR not adherent in prior 6 months\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eScreening\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eOnline screener\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eOnline screener\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\"\u003e \u003cp\u003eOnline screener\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eLocation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eNationwide\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eUnited States with counties (n = 22) with large diverse YMSM populations\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\"\u003e \u003cp\u003eNationwide\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003ePrimary recruitment strategies\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eOnline ads\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eOnline ads; participant registries; friend and community partner referrals\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\"\u003e \u003cp\u003eOnline ads; participant registries; friend and community partner referrals\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eRecruitment budget\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003e\u003cspan\u003e$\u003c/span\u003e7500 (total actual spend)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003e\u003cspan\u003e$\u003c/span\u003e27,000 budgeted\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\"\u003e \u003cp\u003e\u003cspan\u003e$\u003c/span\u003e47,626\u003c/p\u003e \u003cp\u003e(total actual spend)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eSample size\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003e144\u003c/p\u003e \u003cp\u003e(actual enrollment from DTC pilot)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003e2200 (goal)\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\"\u003e \u003cp\u003e1468 (actual enrollment)\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eDelivery\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eOnline; 24-hour wait between intervention modules\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eOnline; 8-hour wait between intervention modules\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\"\u003e \u003cp\u003eOnline; no wait between intervention modules\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eCompensation\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eUp to \u003cspan\u003e$\u003c/span\u003e180 for completing surveys, intervention content, and at-home STI testing\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\"\u003e \u003cp\u003eEntry into raffle for \u003cspan\u003e$\u003c/span\u003e50 for completing surveys\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colspan=\"2\"\u003e \u003cp\u003eUp to \u003cspan\u003e$\u003c/span\u003e100 for completing surveys, intervention content, and at-home STI testing\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/table\u003e\u003c/div\u003e\u003ch3\u003eData sources\u003c/h3\u003e\u003cp\u003eData came from weekly enrollment logs; budget spreadsheets and spending logs; secure participant text messages to our study; participant contact notes; and archived team meeting notes, e-mails, Data and Safety Monitoring Board (DSMB) reports, and memos to NIH program officers. Enrollment and recruitment data were analyzed descriptively. Below, we describe five phases of KIU! DTC delivery aligned with pivotal events and how implementation and study design evolved as we learned the limitations of what was feasible and achievable.\u003c/p\u003e"},{"header":"Results","content":"\u003cp\u003eDTC participants were on average 26.0 years old, with 57.6% identifying as Black, Latino, or another racial/ethnic minority. All U.S. states except Alaska, Wyoming, and Maine were represented in the DTC arm with 33.1% of participants living in the South, 26.2% in the West, 24.4% in the Midwest, and 16.3% in the Northeast Census Region. Other DTC participant characteristics are available elsewhere (Mustanski, Benbow, et al., \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e2025b\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eTo characterize the phases of DTC delivery, each section includes high-level enrollment data and advertising spending; quotes from participants\u0026rsquo; text messages or phone calls with DTC staff; and information from investigator meetings and conversations with NIH program officers that guided study change decisions.\u003c/p\u003e \u003cp\u003eThe CBO and DTC teams each consisted of a co-investigator and two or three staff, respectively, who led arm-specific activities. The investigative team consisted of the PI, the two aforementioned co-investigators, and three implementation scientists, two health economists, a quantitative methodologist, and the software development team lead, all of whom operated across study arms. Staff analysts and software developers supported the investigative team and deployed the KIU! software and research surveys. During initial project meetings, details about CBO and DTC performance were not shared with the opposing teams to avoid contamination.\u003c/p\u003e \u003cp\u003e\u003cb\u003ePhase 1: Trial launch, October 2019-February 2020\u003c/b\u003e. Enrollment began in October 2019. In contrast to the nationwide DTC pilot, the DTC team\u0026rsquo;s advertising strategy consisted of social media campaigns in ZIP codes within the 22 counties. Since financial incentives could not be guarantee, the advertisements focused on community benefits, opportunities to win sexual health products, and concerns about sexual health to create demand for research participation (Bassett et al., \u003cspan citationid=\"CR1\" class=\"CitationRef\"\u003e2015\u003c/span\u003e; Macapagal et al., \u003cspan citationid=\"CR26\" class=\"CitationRef\"\u003e2020\u003c/span\u003e). The advertising budget per DTC participant enrolled was smaller than in KIU! 2.0, where \u003cspan\u003e$\u003c/span\u003e7500 in advertising yielded 144 enrolled participants (~\u003cspan\u003e$\u003c/span\u003e52/participant); in KIU! 3.0, our initial budget was \u003cspan\u003e$\u003c/span\u003e27,000 for a goal of 2200 participants (~\u003cspan\u003e$\u003c/span\u003e12/participant).\u003c/p\u003e \u003cp\u003eThe DTC team anticipated that enrollment targets would not be achieved via paid online advertising alone given the budget constraints and limited geographic catchment areas. To address this concern, the DTC team identified health departments, LGBTQ- or HIV-focused advocacy organizations, social groups, or universities in each county. The intention was to find local champions for KIU! whose \u0026ldquo;personal touch\u0026rdquo; may lead to increased enrollment in these counties (Damschroder, Banaszak-Holl, et al., \u003cspan citationid=\"CR13\" class=\"CitationRef\"\u003e2009\u003c/span\u003e; Schon, \u003cspan citationid=\"CR43\" class=\"CitationRef\"\u003e1963\u003c/span\u003e; Valente \u0026amp; Pumpuang, \u003cspan citationid=\"CR47\" class=\"CitationRef\"\u003e2007\u003c/span\u003e). Community members were encouraged to distribute information about KIU! at their organizations with flyers and pamphlets. The team also partnered with an LGBTQ-focused marketing company who placed paid advertisements on dating apps.\u003c/p\u003e \u003cp\u003eA challenge for the DTC team was to reduce obstacles to accessing KIU! (a pragmatic implementation goal) while deterring bots and imposter participants (a research goal) (Teitcher et al., \u003cspan citationid=\"CR45\" class=\"CitationRef\"\u003e2015\u003c/span\u003e). Initially, a multi-step enrollment process was planned to deter imposters (Fig.\u0026nbsp;\u003cspan refid=\"Fig1\" class=\"InternalRef\"\u003e1\u003c/span\u003e).\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003eAfter completing the eligibility survey, participant eligibility and mailing address was verified via a brief videochat or submission to a LexisNexis or WhitePages database, which would take 1\u0026ndash;2 business days. Verified participants received an HIV/STI self-test kit in the mail. After participants submitted an HIV-negative test result to a secure online portal, they received the baseline survey, and then received the link to KIU! Participants were also asked to ship their STI samples to a lab. This yielded at minimum a 7\u0026ndash;10 day lag between screener completion and receiving access to KIU!\u003c/p\u003e \u003cp\u003eGiven the relative ease of enrollment in KIU! 2.0 and a large ad buy on dating apps that had accelerated prior study enrollment, we did not expect to have spent \u003cspan\u003e$\u003c/span\u003e10,744 to only enroll 15 participants \u0026ndash; a significantly higher per-person spend (\u003cspan\u003e$\u003c/span\u003e716.27/participant) than in KIU! 2.0. We noticed significant attrition in the enrollment process. In February 2020, 55 people were preliminarily eligible, of which 38.2% (n\u0026thinsp;=\u0026thinsp;21) were confirmed; 71.4% of these (n\u0026thinsp;=\u0026thinsp;15) completed baseline and were enrolled (but only 27.3% of those preliminary eligible). Participants were required to answer comprehension questions about the consent form, and those who missed questions had to consent over the phone, which was a challenge to schedule: \u0026ldquo;I won\u0026rsquo;t be home until 11 tonight if that\u0026rsquo;s fine with you. Or you can call me before 9AM or after 7PM tomorrow, or anytime Saturday\u0026rdquo; (text message from participant, 11/07/19). Thus we brainstormed methods to streamline enrollment, such as shortening the screener and verifying respondents\u0026rsquo; identity through social media and online directories rather than via LexisNexis.\u003c/p\u003e \u003cp\u003e\u003cb\u003ePhase 2: March 2020-June 2020\u003c/b\u003e. After the COVID-19 pandemic restrictions began, the DTC team paused HIV/STI test shipments because staff were not permitted at the university. Requiring proof of an HIV-negative test result would prevent enrollment, so DTC participants were allowed to enroll in the intervention based on a self-reported negative or unknown HIV status. The changes brainstormed in Phase 1 were implemented to streamline the enrollment process (Fig.\u0026nbsp;2). To incentivize participants, in consultation with the investigative team, the DTC team increased the value and frequency of the raffle prizes from a \u003cspan\u003e$\u003c/span\u003e50 raffle for every 50 participants who completed a survey (which meant that there were long delays between raffles) to monthly raffles for \u003cspan\u003e$\u003c/span\u003e200 regardless of how many participants were enrolled. These changes were intended to increase demand for participation and ensure we collected sufficient data to power our effectiveness analyses.\u003c/p\u003e \u003cp\u003eFigure 2. Comparison of Enrollment Processes Implemented during Study Phases 1 and 2\u003c/p\u003e \u003cp\u003e The investigative team had concerns about the impact of the pandemic on participants\u0026rsquo; sexual behavior (and thus inclusion criteria) and related study outcomes (e.g., HIV transmission risk behavior). Participants shared that participating in KIU! 3.0 was no longer a priority as they dealt with health issues, employment and educational instability, and greater caretaking responsibilities. In response to a study reminder, one participant said, \u0026ldquo;Honestly with all the things going on [in] the world I just have so much else to focus [on]. I was laid off my job so looking for a job is my main priority\u0026rdquo; (04/03/2020). Another participant said: \u0026ldquo;It's been really difficult the last couple of months dealing with school, mental health, family and other aspects of life and I'm trying to get back to the online material as soon as possible.\u0026rdquo; (06/12/2020).\u003c/p\u003e \u003cp\u003eIn May 2020, staff could return to the office and a DTC team member resumed shipping of HIV/STI test kits every 2 weeks (vs. any time a participant enrolled) to balance team safety while continuing the study. We also contracted with a company who shipped HIV/STI test kits for us, which allowed greater flexibility for handling testing during COVID stay-at-home orders. Despite these adaptations, only 26 participants were enrolled at the end of this phase, and an additional \u003cspan\u003e$\u003c/span\u003e6,428 was spent on advertisements.\u003c/p\u003e \u003cp\u003e \u003cb\u003ePhase 3: July 2020-September 2020\u003c/b\u003e. Several additional changes were implemented in this phase to increase demand for KIU! 3.0. We decided to guarantee incentives for completing episodes 1\u0026ndash;3. To examine whether different incentives would result in varying engagement and retention, DTC counties were randomized to receive \u003cspan\u003e$\u003c/span\u003e10 or \u003cspan\u003e$\u003c/span\u003e25, modeled after prior research (Sharma et al., \u003cspan citationid=\"CR44\" class=\"CitationRef\"\u003e2011\u003c/span\u003e). In August 2020 online recruitment on social media and dating apps\u0026rsquo; self-service portals was supplemented by a marketing company experienced in online recruitment for health research studies. Snowball recruitment began as well, with enrolled participants allowed to refer up to five friends and receive \u003cspan\u003e$\u003c/span\u003e10 for each enrolled friend. Eligibility criteria were modified again in September 2020 to allow enrollment of gender-diverse participants assigned male at birth in recognition of the fact that sexual behavior, and not gender, drives HIV risk, and because in KIU! 3.0 and the team\u0026rsquo;s prior HIV prevention studies focused on cisgender men, nonbinary and gender-diverse participants comprised a significant proportion of ineligible individuals. Yet by the end of this phase, an additional \u003cspan\u003e$\u003c/span\u003e3,397 was spent on advertising, and DTC enrollment only increased to 64 participants. Individuals who had not finished the baseline survey were recontacted to increase enrollment, and feedback such as, \u0026ldquo;I didn't complete [the survey] because I am out of your age range\u0026rdquo; (10/07/2020), was common, suggesting that additional modifications to eligibility criteria could benefit the study. In addition, only 52.8% of participants had returned their baseline STI samples to the lab.\u003c/p\u003e \u003cp\u003e \u003cb\u003ePhase 4: October 2020-February 2021\u003c/b\u003e. By this time it was evident that the several changes made to the study only modestly improved enrollment. The investigative team decided to inform the co-investigators leading the CBO and DTC arms that both groups were similarly struggling with enrollment and that more significant changes to the study design were necessary to avoid trial failure (i.e., the inability to enroll enough participants to assess either effectiveness or implementation).\u003c/p\u003e \u003cp\u003eIn October 2020, discussions began between the DTC team and implementation scientist co-investigators about expanding to nationwide recruitment. This change was deemed as most urgent because, out of all ineligible respondents for the DTC arm, nearly 25% failed screening due only to their county. This change also was perceived by the implementation scientist investigators to increase the DTC strategy pragmatism given that many commercially available DTC online health programs for other conditions are available more widely (e.g., via App Store, employer-sponsored insurance plans; Lau et al., 2020). However, this modification would fundamentally change the trial design (i.e., no longer cluster-randomized) and affect the ability to directly compare the effectiveness outcomes across arms due to the different scale of enrollment (i.e., county vs. national). Another change discussed was increasing incentives for completing the intervention content \u0026ndash; not just the research surveys \u0026ndash; to improve recruitment and retention. The investigative team believed this was justifiable as incentives were allowable in CDC funding opportunity announcements and found that it was common in public health practice to incentivize intervention completion (Zamantakis et al., \u003cspan citationid=\"CR50\" class=\"CitationRef\"\u003e2025\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eIn addition to DTC-strategy-specific changes, the investigative team brainstormed changes that could be applied irrespective of CBO/DTC arm to facilitate enrollment and between-strategy comparisons despite changes to the RCT design. This included: 1) increasing the upper age limit from 29 to 34 following age groupings in CDC epidemiological reports and other studies that included 34 year olds among \u0026ldquo;YMSM\u0026rdquo; (Gerke et al., \u003cspan citationid=\"CR18\" class=\"CitationRef\"\u003e2022\u003c/span\u003e; Hill-Rorie et al., \u003cspan citationid=\"CR22\" class=\"CitationRef\"\u003e2024\u003c/span\u003e); 2) no longer requiring participants to have had condomless anal sex in the past 3 months; 3) making breaks between episodes 1\u0026ndash;3 optional such that participants could complete most of KIU! in one sitting; 4) abbreviating the breaks between episodes 1\u0026ndash;3 and 5 and 6; 5) abbreviating the follow-up period from 12 to 3 months; and 6) eliminating STI testing from the final follow-up. Participant contact notes revealed that breaks were an obstacle to sustained attention, motivation, and engagement in KIU!: \"I'm also stuck in the waiting period for the next episode unless that ran out and I missed it because what even is time\u0026rdquo; (1/23/2021).\u003c/p\u003e \u003cp\u003eThe DSMB and investigative team met in November 2020 to discuss the changes, which the DSMB supported and provided feedback regarding the impacts on aims, sample size, and safety. In December 2020, the proposed changes were submitted to the NIH program officers for approval. The program officers raised a concern about the request to drop STI testing at final timepoint as few HIV prevention studies at the time collected biomedical effectiveness outcomes; this resulted in retention of the STI testing endpoint. NIH and the university IRB approved all changes in February 2021. The study team spent the rest of this phase updating the KIU! materials, software, and advertising campaigns to reflect the changes. This phase ended with an additional \u003cspan\u003e$\u003c/span\u003e4,474 spent on advertising and DTC enrollment increased to 118 participants.\u003c/p\u003e \u003cp\u003e \u003cb\u003ePhase 5: March 2021 \u0026ndash; March 2023 (end of trial)\u003c/b\u003e. When the changes proposed in Phase 4 went into effect, the DTC team launched national advertising campaigns on Facebook, Instagram, Snapchat, and Grindr and ran another campaign with the LGBTQ-focused marketing company. These were supplemented with offline strategies such as snowball recruitment, referrals from other studies, and participant registries. Those who were previously ineligible due to location were recontacted and informed about the new compensation structure: \u003cspan\u003e$\u003c/span\u003e50 for the first three episodes, \u003cspan\u003e$\u003c/span\u003e10 for the 6-week booster, and \u003cspan\u003e$\u003c/span\u003e15 for the 3-month booster. To facilitate adherence to STI testing at the 3-month endpoint, participants in both CBO and DTC strategies received an additional \u003cspan\u003e$\u003c/span\u003e10, for a total of up to \u003cspan\u003e$\u003c/span\u003e85 in incentives.\u003c/p\u003e \u003cp\u003eThese changes had a sizable and rapid impact on enrollment. One month after the changes, DTC enrollment increased from 118 to 355 participants (a 301% increase). The average cost of advertising per enrolled participant also dramatically decreased from \u003cspan\u003e$\u003c/span\u003e212 to \u003cspan\u003e$\u003c/span\u003e79. Many participants noted the incentives as an important motivator: \u0026ldquo;I was actually only interested in the incentives\u0026rdquo; (6/17/2021). Another sent the text message, \u0026ldquo;ill do it soon i want the money\u0026rdquo; (3/29/2021). However, the incentives were associated with a significant increase in imposter participants, resulting in additional measures to verify participants\u0026rsquo; identity and their survey data.\u003c/p\u003e \u003cp\u003eAfter reallocating a substantial amount of the budget to advertising in the prior phases, and after deciding to guarantee incentives for both surveys and intervention completion, it became clear that the goal of enrolling 2,200 participants with the remaining budget and timeframe was infeasible. Thus, in the final phase NIH gave permission to reduce the DTC enrollment goal from 2,200 to a minimum of 1,100, and to extend the trial end date from April to December 2022. The previous sample size needed to power the cluster-randomized trial analyses was no longer necessary, and the new sample size would still provide meaningful data. The last participant was recruited in August 2022 yielding 1,468 participants enrolled in KIU! DTC. Across the trial, a total of \u003cspan\u003e$\u003c/span\u003e47,626 (~\u003cspan\u003e$\u003c/span\u003e32 per enrolled participant) was spent on DTC advertising, significantly more than the \u003cspan\u003e$\u003c/span\u003e27,000 budgeted, and over \u003cspan\u003e$\u003c/span\u003e100,000 was spent on incentives. Figure\u0026nbsp;\u003cspan refid=\"Fig2\" class=\"InternalRef\"\u003e3\u003c/span\u003e depicts changes in DTC enrollment over the trial; Table\u0026nbsp;\u003cspan refid=\"Tab1\" class=\"InternalRef\"\u003e2\u003c/span\u003e summarizes the barriers in each phase, solutions, and the impact on enrollment.\u003c/p\u003e \u003cp\u003e \u003c/p\u003e \u003cp\u003e \u003cdiv class=\"gridtable\"\u003e\u003ctable float=\"Yes\" id=\"Tab1\" border=\"1\"\u003e \u003ccaption language=\"En\"\u003e \u003cdiv class=\"CaptionNumber\"\u003eTable 2\u003c/div\u003e \u003cdiv class=\"CaptionContent\"\u003e \u003cp\u003eChallenges, solutions, and impact on sample size across the five phases of KIU! 3.0\u003c/p\u003e \u003c/div\u003e \u003c/caption\u003e \u003ccolgroup cols=\"4\"\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c1\" colnum=\"1\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c2\" colnum=\"2\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c3\" colnum=\"3\"\u003e\u003c/div\u003e \u003cdiv align=\"left\" class=\"colspec\" colname=\"c4\" colnum=\"4\"\u003e\u003c/div\u003e \u003cthead\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c1\" morerows=\"1\" rowspan=\"2\"\u003e \u003cp\u003ePhase 1\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eChallenges\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSolutions\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003eSample Size at Phase End\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003ctr\u003e \u003cth align=\"left\" colname=\"c2\"\u003e \u003cp\u003eLimitations of online recruitment\u003c/p\u003e \u003cp\u003eMultistep eligibility screening and enrollment process\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c3\"\u003e \u003cp\u003eSupplement online recruitment with outreach to community stakeholders who can refer participants into study\u003c/p\u003e \u003cp\u003eShortened eligibility screener\u003c/p\u003e \u003c/th\u003e \u003cth align=\"left\" colname=\"c4\"\u003e \u003cp\u003e15\u003c/p\u003e \u003c/th\u003e \u003c/tr\u003e \u003c/thead\u003e \u003ctbody\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePhase 2\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eCOVID-19 pandemic limits staff working in the office and impacts HIV/STI testing\u003c/p\u003e \u003cp\u003eLengthy eligibility screener survey\u003c/p\u003e \u003cp\u003eLow enrollment\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eAt-home HIV and STI test kit preparation and shipments paused; third-party at-home testing option offered\u003c/p\u003e \u003cp\u003eParticipants allowed to self-report unknown or negative HIV status and enroll into study\u003c/p\u003e \u003cp\u003eRegistration form collecting basic contact and county info replaced the eligibility screener survey\u003c/p\u003e \u003cp\u003eRaffle increased to \u003cspan\u003e$\u003c/span\u003e200\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e26\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePhase 3\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eLow enrollment and intervention completion rates\u003c/p\u003e \u003cp\u003eRestrictive eligibility criteria\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eRandomized incentive of \u003cspan\u003e$\u003c/span\u003e10 or \u003cspan\u003e$\u003c/span\u003e25 offered to participants who complete first three KIU! episodes\u003c/p\u003e \u003cp\u003eOnline ad campaign in collaboration with LGBTQ-specific marketing and advertising company\u003c/p\u003e \u003cp\u003eIncentivized snowball recruitment\u003c/p\u003e \u003cp\u003eEligibility criteria updated: Gender diverse participants assigned male at birth eligible for study\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e64\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePhase 4\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eRestrictive eligibility criteria, low enrollment and retention rates, forced breaks between intervention content, lengthy study timeline\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eChanges to eligibility criteria, study design, and incentives discussed and proposed to and approved by study DSMB, IRB, and NIH\u003c/p\u003e \u003cp\u003eUpdates made to KIU! app and databases to reflect these changes and to prepare for national recruitment campaigns\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e116\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003ctr\u003e \u003ctd align=\"left\" colname=\"c1\"\u003e \u003cp\u003ePhase 5\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c2\"\u003e \u003cp\u003eRestrictive eligibility criteria, low enrollment and retention rates, forced breaks between intervention content, lengthy study timeline\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c3\"\u003e \u003cp\u003eLargest changes made:\u003c/p\u003e \u003cp\u003e1. Expanding DTC catchment area to national recruitment.\u003c/p\u003e \u003cp\u003e2. De-emphasizing effectiveness and focusing on implementation aims.\u003c/p\u003e \u003cp\u003e3. Increasing upper age limits of eligibility from 29 to 34 years.\u003c/p\u003e \u003cp\u003e4. Removing the condomless anal sex eligibility criteria.\u003c/p\u003e \u003cp\u003e5. Making breaks between the first three sessions of the intervention optional.\u003c/p\u003e \u003cp\u003e6. Shortening period of follow-up from 12 months to 3 months.\u003c/p\u003e \u003cp\u003e7. Collecting follow-up data at one time point only (3 months).\u003c/p\u003e \u003cp\u003e8. Increasing the incentive paid to DTC participants who complete the first three sessions of the intervention from \u003cspan\u003e$\u003c/span\u003e10 - \u003cspan\u003e$\u003c/span\u003e25 to \u003cspan\u003e$\u003c/span\u003e50.\u003c/p\u003e \u003cp\u003e9. Paying DTC participants who complete their 6-week and 3-month follow-ups \u003cspan\u003e$\u003c/span\u003e10 and \u003cspan\u003e$\u003c/span\u003e15, respectively.\u003c/p\u003e \u003c/td\u003e \u003ctd align=\"left\" colname=\"c4\"\u003e \u003cp\u003e1468\u003c/p\u003e \u003c/td\u003e \u003c/tr\u003e \u003c/tbody\u003e \u003c/colgroup\u003e \u003c/table\u003e\u003c/div\u003e \u003c/p\u003e \u003cp\u003e \u003cb\u003eRetention period.\u003c/b\u003e Although enrollment in KIU! DTC improved, retention in the research study components remained a challenge even with incentives. First, retention at the 6-week and 3-month follow-up improved from 38% and 24%, respectively, to roughly 60% across both. In contrast, KIU! 2.0 had\u0026thinsp;\u0026gt;\u0026thinsp;80% retention (Mustanski et al., \u003cspan citationid=\"CR35\" class=\"CitationRef\"\u003e2018\u003c/span\u003e). The combination of intervention complexity, lower incentives, and the pandemic likely contributed to these lower rates. Also, participation in a study focused on preventing a health condition people did not have \u0026ndash; or may not have felt the need for, given constrained opportunities for social and sexual contact in the early pandemic \u0026ndash; was likely a lower priority based on communications between staff and participants. Yet final retention rates were still within ranges described in systematic reviews of digital health interventions, which demonstrate greater attrition in digital studies with larger samples, limited or no incentives, and limited or no human contact (Daniore et al., \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e2022\u003c/span\u003e; Torous et al., \u003cspan citationid=\"CR46\" class=\"CitationRef\"\u003e2020\u003c/span\u003e).\u003c/p\u003e \u003cp\u003eSecond, the return rate for STI testing at 3-month follow-up (primary effectiveness outcome) was 48% (n\u0026thinsp;=\u0026thinsp;699/1468). This lower-than-expected return rate may be explained by the initial decision to send test kits only to participants who reconfirmed their mailing address prior to the 3-month follow-up. The intent was to save resources and avoid mailing kits to participants who had moved or gave an incorrect address. Eventually these test kits were mailed to all participants regardless of address confirmation. This increased return rates from 19% at the end of Phase 4 to 48% at the end of Phase 5.\u003c/p\u003e \u003cp\u003eAfter the last participant was enrolled, for the final 4 months of 2022, the DTC staff shifted into a retention-only phase, sending participants more frequent reminders to complete content and return STI testing kits via personalized calls, emails, and pre-recorded voice messages. Through this work, it became evident that participants thought that \u003cem\u003eall\u003c/em\u003e retention reminders were automated. This led to an increase in receiving replies such as \u0026ldquo;STOP\u0026rdquo; or \u0026ldquo;UNSUBSCRIBE\u0026rdquo; via text. There was difficulty finding a middle ground between personalizing participant reminders for each participant\u0026rsquo;s situation and study behavior and automating them for efficiency in a large trial. Moreover, some participants seemed suspicious of study motivations when they perceived too many automated messages were sent, with one participant asking via a text: \u0026ldquo;How are you offering this [STI test] for free? Is my personal information being sold? Why do i need to complete a survey?\u0026rdquo; (12/01/2020)\u003c/p\u003e"},{"header":"Discussion","content":"\u003cp\u003eOverall, the execution of this project reflected a constant tension between the pragmatic implementation aim (reaching as many people as possible in as real-life of a way as possible) versus the effectiveness aim and the deliverables required by the funders (collecting as much effectiveness outcome data as possible). These tensions often influenced the ability to make decisions that would best facilitate uptake and use of KIU! Although in principle, hybrid type 3 designs prioritize implementation over demonstrating effectiveness, at times it seemed both outcomes were equally important. An exclusive focus on studying implementation of an established effective intervention may have made space for different decisions more aligned with enhancing implementation success. Hybrid effectiveness-implementation study designs in theory accelerate the translation of research to public health practice but in actuality may constrain what can be learned about implementation, at least in the context of an intervention with time-limited external funding that expected collection of biomedical outcome data. Given the challenges in collecting primary data in large implementation trials, it has been suggested that these studies may instead rely on other sources (e.g., administrative, EHR data) as a proxy (Curran et al., \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2012\u003c/span\u003e). Nevertheless, we hope that this narrative of our experience attempting to implement a DTC digital health intervention within a large hybrid type 3 study may pull back the curtain on what one might experience when doing similar studies of other digital interventions.\u003c/p\u003e \u003cp\u003eWe offer several lessons learned for others planning to study implementation of a digital health intervention. First, we encourage investigators to consider what barriers to entry are critical (if any), including eligibility criteria, steps needed to access the intervention, and the time elapsed between screening eligible and entering the intervention, and we encourage investigators to do a real-time internal pilot of these steps if feasible. At the beginning of KIU! 3.0, study eligibility criteria were still relatively stringent, with multiple steps before participants could access the program. Although our cluster-randomized design was rigorous, ultimately it was not pragmatic for this intervention. In practice, one\u0026rsquo;s ability to access a self-guided, freely available digital intervention is rarely confined to the county they live in. These barriers, combined with our budget, constrained our ability to reach participants and led to significant attrition before people even accessed KIU! (Benbow et al., \u003cspan citationid=\"CR4\" class=\"CitationRef\"\u003e2025\u003c/span\u003e). When research procedures (e.g., multistep screening, test kit verification, follow-up assessments) are the main barrier to entry, they can undermine implementation success, and may risk distorting what uptake might look like in real-world deployment.\u003c/p\u003e \u003cp\u003eSecond, we encourage investigators to consider whether their participant incentives are sufficient for their implementation study. Given our past experiences with grateful participants and the literature on altruism as a motivator for health research participation (Williams et al., \u003cspan citationid=\"CR48\" class=\"CitationRef\"\u003e2008\u003c/span\u003e), we overestimated participants\u0026rsquo; willingness to participate and underestimated their interest in financial incentives. Eventually the DTC team learned that the CBOs implementing KIU! offered monetary incentives to participants as high as \u003cspan\u003e$\u003c/span\u003e100. Given that the CBO and DTC teams initially operated independently of each other to prevent contamination (Mustanski, Saber, Jones, et al., \u003cspan citationid=\"CR36\" class=\"CitationRef\"\u003e2023\u003c/span\u003e), the DTC team was not aware of these incentives until the trial design was changed. The existence of incentivized HIV prevention programs may have led some participants to expect similar payments for KIU!, and potential participants told us that they lost interest once learning that raffle prizes were the only incentive.\u003c/p\u003e \u003cp\u003eIn addition, guaranteed benefits likely play a role in reach and retention in implementation studies. In earlier KIU! evaluations, participants indicated that at-home STI testing was convenient as this preceded the rise of DTC health companies who now offer this service, and rectal STI testing was not yet widely available. Yet in this study, these same STI tests and the raffle prizes were insufficient to make study participation appealing. When asked about the reasons for not completing the STI tests, participants\u0026rsquo; comments (e.g., \u0026ldquo;treatments are available\u0026rdquo;, \u0026ldquo;I only care about HIV\u0026rdquo;) suggested that STIs may be less of a concern within this population than HIV (Sarno et al., \u003cspan citationid=\"CR42\" class=\"CitationRef\"\u003e2021\u003c/span\u003e) particularly as many STIs are treatable and preventable with antibiotics (Cannon \u0026amp; Celum, \u003cspan citationid=\"CR5\" class=\"CitationRef\"\u003e2023\u003c/span\u003e). Moreover, since the KIU 2.0! trial, rectal STI testing has become more common in clinical settings, which may have made our provision of these tests less compelling. Providing incentives was valuable to study retention as well. Previous research has shown that user engagement with self-guided digital health apps is high initially, but few sustain engagement over time (Baumel et al., \u003cspan citationid=\"CR2\" class=\"CitationRef\"\u003e2019\u003c/span\u003e). We addressed this issue by offering incentives for each survey and condensing the study timeline. Moreover, given that the study recruited during a time when many young people lost jobs, monetary incentives may have been more salient when making decisions about how they want to spend their time. Our economic analyses showed that even with these incentives, KIU! still represented a cost-saving intervention in the long term (Munroe et al., \u003cspan citationid=\"CR29\" class=\"CitationRef\"\u003e2025\u003c/span\u003e; Mustanski, Benbow, et al., \u003cspan citationid=\"CR31\" class=\"CitationRef\"\u003e2025b\u003c/span\u003e). Together these experiences reflect the concept of \u0026ldquo;conditional altruism\u0026rdquo; (McCann et al., \u003cspan citationid=\"CR28\" class=\"CitationRef\"\u003e2010\u003c/span\u003e) \u0026ndash; although altruism itself may have been part of participants\u0026rsquo; calculus, offering the right combination of incentives and/or benefits is likely necessary to yield enrollment.\u003c/p\u003e \u003cp\u003eA third lesson learned is that dedicated staffing is a critical part of the implementation strategy for DTC self-guided digital interventions, particularly those that may require greater engagement to be effective. The adage \u0026ldquo;if you build it, they will come\u0026rdquo; did not hold true for our intervention, as simply advertising KIU! was not enough reach and retain people. Having a team who is skilled in reaching, engaging, and motivating participants from a distance can be the difference between successful and failed implementation of DTC digital interventions. Further, restrictions in audience targeting on self-service online marketing platforms (e.g., Meta), lower public trust in social media and online advertising (Park et al., \u003cspan citationid=\"CR39\" class=\"CitationRef\"\u003e2020\u003c/span\u003e), and the increase in imposter participants in online research (Roehl \u0026amp; Harland, \u003cspan citationid=\"CR40\" class=\"CitationRef\"\u003e2022\u003c/span\u003e) have made recruiting participants more challenging. Finally, although attrition in digital intervention studies (Daniore et al., \u003cspan citationid=\"CR14\" class=\"CitationRef\"\u003e2022\u003c/span\u003e; Torous et al., \u003cspan citationid=\"CR46\" class=\"CitationRef\"\u003e2020\u003c/span\u003e) is often high, if an implementation study requires effectiveness outcome data, retention is critical. Humanizing the intervention with personalized contacts and using techniques such as motivational interviewing to encourage study completion (Downs, \u003cspan citationid=\"CR15\" class=\"CitationRef\"\u003e2019\u003c/span\u003e; Jake-Schoffman et al., \u003cspan citationid=\"CR23\" class=\"CitationRef\"\u003e2021\u003c/span\u003e) enabled us to foster friendly relationships with study participants and troubleshoot barriers to study completion, but such frequent contacts were not always desired by participants.\u003c/p\u003e \u003cp\u003eFinally, unanticipated or underestimated outer setting factors (e.g., the pandemic, attitudes about the necessity of HIV/STI prevention) were major determinants of how the DTC strategy performed (Damschroder, Aron, et al., \u003cspan citationid=\"CR12\" class=\"CitationRef\"\u003e2009b\u003c/span\u003e). However, researchers are not always equipped to deal with them, especially if they are newer to testing their interventions in implementation studies. One recommendation is to do a pre-mortem which may enable teams to better anticipate potential pitfalls and develop contingency plans in the event recruitment or retention does not go as planned (Wippold et al., \u003cspan citationid=\"CR49\" class=\"CitationRef\"\u003e2025\u003c/span\u003e). Another is to test digital interventions in the context they were meant to be delivered from the very beginning (rather than beginning with traditional efficacy trials) (Beidas et al., \u003cspan citationid=\"CR3\" class=\"CitationRef\"\u003e2023\u003c/span\u003e) which would enable teams to identify such roadblocks \u0026ndash; and move interventions to public health practice \u0026ndash; sooner.\u003c/p\u003e"},{"header":"Conclusion","content":"\u003cp\u003eHybrid type 3 effectiveness-implementation studies of DTC digital health interventions remain relatively uncommon. Our experience in this study was illuminating and has since informed our approach to other hybrid studies of digital and multimedia interventions. This study attempted to approximate real-world conditions, yet research constraints precluded it from achieving naturalistic implementation conditions, and investigators seeking to do similar work may encounter frequent tensions between research and implementation priorities. As digital health services continue to expand to real-world settings, investigators and funders alike may need to reconsider how hybrid designs are structured, from barriers to entry, the extent to which incentives and sustained engagement are important to answer one\u0026rsquo;s research questions, which outcomes are essential and how they can be easily captured for real-world use, to how researchers who may also be implementers may consider the multiple \u0026ldquo;hats\u0026rdquo; they may wear in a study (Curran et al., \u003cspan citationid=\"CR10\" class=\"CitationRef\"\u003e2012\u003c/span\u003e). Careful consideration of these issues can help ensure that the pursuit of rigor does not inadvertently constrain the very implementation processes these studies aim to advance.\u003c/p\u003e"},{"header":"Declarations","content":"\u003cp\u003eCompeting interests: The authors declare that they have no competing interests.\u003c/p\u003e\u003cp\u003e \u003cstrong\u003eEthics approval and consent to participate:\u003c/strong\u003e \u003cp\u003e Research procedures were approved by the Northwestern University IRB; service components were considered non-human subjects research. All participants provided e-consent and/or verbal consent prior to enrollment in the research portions of the study.\u003c/p\u003e \u003c/p\u003e \u003cp\u003e \u003cstrong\u003eConsent for publication:\u003c/strong\u003e \u003cp\u003eNot applicable.\u003c/p\u003e \u003c/p\u003e\u003ch2\u003eFunding:\u003c/h2\u003e \u003cp\u003eThis project was funded by R01MH118213. Additional resources were provided by NIH grants P30AI117943 and UL1TR001422. AK\u0026rsquo;s time was supported by K01MH121854. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.\u003c/p\u003e\u003ch2\u003eAuthor Contribution\u003c/h2\u003e\u003cp\u003eKRM and KLM contributed equally to the manuscript and are co-first authors. KRM contributed to conception and design of work, acquisition, analysis, and interpretation of data, and wrote the manuscript text. KLM contributed to acquisition, analysis, and interpretation of data, wrote the manuscript text, and prepared the figures and tables. AK contributed to interpretation of data and wrote and substantively revised the manuscript text. MH contributed to analysis of data and substantively revised the manuscript text. MP wrote and substantively revised the manuscript text. BN contributed to acquisition of data and substantively revised the manuscript text. JO contributed to acquisition of data and wrote the manuscript text. IG contributed to acquisition of data and wrote the manuscript text. DHL contributed to design of work, interpretation of data, and substantively revised the manuscript text. BM contributed to the conception and design of work and interpretation of data, and substantively revised the manuscript text. All authors reviewed the manuscript.\u003c/p\u003e\u003ch2\u003eAcknowledgement\u003c/h2\u003e\u003cp\u003eWe would like to thank Reno Stephens, Chris Owens, Simon Abimosleh, Daniel Kellogg, Lauren Redman, Alex Saldivar, Emily Bettin, Val Wongsomboon, Rana Saber, and Greg Swann for their collaboration. We would also like to thank Maggi Price and Kelly Shaffer for their constructive feedback on an earlier draft of this manuscript, and to Susannah Allison and Rich Jenkins at NIH for their guidance throughout the changes to our study. We are grateful to all the participants in KIU, from whom we learned so much about implementation during a difficult time.\u003c/p\u003e\u003ch2\u003eData Availability\u003c/h2\u003e\u003cp\u003eLimited data available upon request.\u003c/p\u003e"},{"header":"References","content":"\u003col\u003e\n\u003cli\u003eBassett, I. V., Wilson, D., Taaffe, J., \u0026amp; Freedberg, K. A. (2015). Financial incentives to improve progression through the HIV treatment cascade. \u003cem\u003eCurrent Opinion in HIV and AIDS\u003c/em\u003e, \u003cem\u003e10\u003c/em\u003e(6), 451\u0026ndash;463. https://doi.org/10.1097/COH.0000000000000196\u003c/li\u003e\n\u003cli\u003eBaumel, A., Muench, F., Edan, S., \u0026amp; Kane, J. M. (2019). Objective User Engagement With Mental Health Apps: Systematic Search and Panel-Based Usage Analysis. \u003cem\u003eJournal of Medical Internet Research\u003c/em\u003e, \u003cem\u003e21\u003c/em\u003e(9), e14567. https://doi.org/10.2196/14567\u003c/li\u003e\n\u003cli\u003eBeidas, R. S., Saldana, L., \u0026amp; Shelton, R. C. (2023). Testing psychosocial interventions in the contexts they are meant to be delivered. \u003cem\u003eJournal of Consulting and Clinical Psychology\u003c/em\u003e, \u003cem\u003e91\u003c/em\u003e(4), 189\u0026ndash;191. https://doi.org/10.1037/ccp0000797\u003c/li\u003e\n\u003cli\u003eBenbow, N., Li, D. H., Macapagal, K., Madkins, K., Saber, R., zamantakis, alithia, Rudd, E., Smith, J. D., \u0026amp; Mustanski, B. (2025). Considerations for evaluating pragmatic design elements in digital health intervention trials: The case of Keep It Up! 3.0. \u003cem\u003eImplementation Science Communications\u003c/em\u003e, \u003cem\u003e6\u003c/em\u003e(1), 101. https://doi.org/10.1186/s43058-025-00777-y\u003c/li\u003e\n\u003cli\u003eCannon, C. A., \u0026amp; Celum, C. L. (2023). Doxycycline postexposure prophylaxis for prevention of sexually transmitted infections. \u003cem\u003eTopics in Antiviral Medicine\u003c/em\u003e, \u003cem\u003e31\u003c/em\u003e(5), 566\u0026ndash;575.\u003c/li\u003e\n\u003cli\u003eCDC. (2025a, February 7). \u003cem\u003eEstimated HIV Incidence and Prevalence\u003c/em\u003e. HIV Data. https://www.cdc.gov/hiv-data/nhss/estimated-hiv-incidence-and-prevalence.html\u003c/li\u003e\n\u003cli\u003eCDC. (2025b, May 27). \u003cem\u003eHIV Compendium of Best Practices\u003c/em\u003e. HIV Public Health Partners. https://www.cdc.gov/hivpartners/php/compendium/index.html\u003c/li\u003e\n\u003cli\u003eCenters for Disease Control. (2024). \u003cem\u003eCompendium of Evidence-Based Interventions and Best Practices for HIV Prevention: Background, Methods, and Criteria\u003c/em\u003e. Division of HIV Prevention. https://www.cdc.gov/hiv/pdf/research/interventionresearch/compendium/PRS-Project-Compendium-Background-Methods-and-Criteria.pdf\u003c/li\u003e\n\u003cli\u003eClark, R. C., \u0026amp; Mayer, R. E. (2008). Learning by viewing versus learning by doing: Evidence-based guidelines for principled learning environments. \u003cem\u003ePerformance Improvement\u003c/em\u003e, \u003cem\u003e47\u003c/em\u003e(9), 5\u0026ndash;13. https://doi.org/10.1002/pfi.20028\u003c/li\u003e\n\u003cli\u003eCurran, G. M., Bauer, M., Mittman, B., Pyne, J. M., \u0026amp; Stetler, C. (2012). Effectiveness-implementation Hybrid Designs. \u003cem\u003eMedical Care\u003c/em\u003e, \u003cem\u003e50\u003c/em\u003e(3), 217\u0026ndash;226. https://doi.org/10.1097/MLR.0b013e3182408812\u003c/li\u003e\n\u003cli\u003eDamschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., \u0026amp; Lowery, J. C. (2009a). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. \u003cem\u003eImplementation Science\u003c/em\u003e, \u003cem\u003e4\u003c/em\u003e(1), 50. https://doi.org/10.1186/1748-5908-4-50\u003c/li\u003e\n\u003cli\u003eDamschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., \u0026amp; Lowery, J. C. (2009b). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. \u003cem\u003eImplementation Science: IS\u003c/em\u003e, \u003cem\u003e4\u003c/em\u003e, 50. https://doi.org/10.1186/1748-5908-4-50\u003c/li\u003e\n\u003cli\u003eDamschroder, L. J., Banaszak-Holl, J., Kowalski, C. P., Forman, J., Saint, S., \u0026amp; Krein, S. L. (2009). The role of the champion in infection prevention: Results from a multisite qualitative study. \u003cem\u003eQuality \u0026amp; Safety in Health Care\u003c/em\u003e, \u003cem\u003e18\u003c/em\u003e(6), 434\u0026ndash;440. https://doi.org/10.1136/qshc.2009.034199\u003c/li\u003e\n\u003cli\u003eDaniore, P., Nittas, V., \u0026amp; von Wyl, V. (2022). Enrollment and Retention of Participants in Remote Digital Health Studies: Scoping Review and Framework Proposal. \u003cem\u003eJournal of Medical Internet Research\u003c/em\u003e, \u003cem\u003e24\u003c/em\u003e(9), e39910. https://doi.org/10.2196/39910\u003c/li\u003e\n\u003cli\u003eDowns, H. (2019). Bridging the Gap: How the Generations Communicate. \u003cem\u003eConcordia Journal of Communication Research\u003c/em\u003e, \u003cem\u003e6\u003c/em\u003e(1). https://doi.org/https://doi.org/10.54416/SEZY7453\u003c/li\u003e\n\u003cli\u003e\u003cem\u003eEHE Priority Jurisdictions\u003c/em\u003e. (n.d.). HIV.Gov. Retrieved February 6, 2026, from https://www.hiv.gov/federal-response/ending-the-hiv-epidemic/jurisdictions\u003c/li\u003e\n\u003cli\u003eFisher, J. D., Fisher, W. A., Williams, S. S., \u0026amp; Malloy, T. E. (1994). Empirical tests of an information-motivation-behavioral skills model of AIDS-preventive behavior with gay men and heterosexual university students. \u003cem\u003eHealth Psychology\u003c/em\u003e, \u003cem\u003e13\u003c/em\u003e(3), 238\u0026ndash;250. https://doi.org/10.1037/0278-6133.13.3.238\u003c/li\u003e\n\u003cli\u003eGerke, D., Call, J., \u0026amp; Auslander, W. (2022). The Syndemic Factors of Violence Exposure, Substance Use, and Mental Health Problems: Relationships to Sexual Risk Behaviors in HIV-Negative Young Men Who Have Sex With Men. \u003cem\u003eJournal of the Society for Social Work \u0026amp; Research\u003c/em\u003e, \u003cem\u003e13\u003c/em\u003e(2). https://doi.org/10.1086/711612\u003c/li\u003e\n\u003cli\u003eGlasgow, R. E., Vogt, T. M., \u0026amp; Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. \u003cem\u003eAmerican Journal of Public Health\u003c/em\u003e, \u003cem\u003e89\u003c/em\u003e(9), 1322\u0026ndash;1327. https://doi.org/10.2105/AJPH.89.9.1322\u003c/li\u003e\n\u003cli\u003eHardcastle, S. J., Hancox, J., Hattar, A., Maxwell-Smith, C., Th\u0026oslash;gersen-Ntoumani, C., \u0026amp; Hagger, M. S. (2015). Motivating the unmotivated: How can health behavior be changed in those unwilling to change? \u003cem\u003eFrontiers in Psychology\u003c/em\u003e, \u003cem\u003e6\u003c/em\u003e, 835. https://doi.org/10.3389/fpsyg.2015.00835\u003c/li\u003e\n\u003cli\u003eHermes, E. D. A., Lyon, A. R., Schueller, S. M., \u0026amp; Glass, J. E. (2019). Measuring the Implementation of Behavioral Intervention Technologies: Recharacterization of Established Outcomes. \u003cem\u003eJournal of Medical Internet Research\u003c/em\u003e, \u003cem\u003e21\u003c/em\u003e(1). https://doi.org/10.2196/11752\u003c/li\u003e\n\u003cli\u003eHill-Rorie, J., Biello, K. B., Quint, M., Johnson, B., Elopre, L., Johnson, K., Lillis, R., Burgan, K., Krakower, D., Whiteside, Y., \u0026amp; Mayer, K. H. (2024). Weighing the Options: Which PrEP (Pre-exposure Prophylaxis) Modality Attributes Influence Choice for Young Gay and Bisexual Men in the United States? \u003cem\u003eAIDS and Behavior\u003c/em\u003e, \u003cem\u003e28\u003c/em\u003e(9), 2970\u0026ndash;2978. https://doi.org/10.1007/s10461-024-04384-1\u003c/li\u003e\n\u003cli\u003eJake-Schoffman, D. E., Brown, S. D., Baiocchi, M., Bibeau, J. L., Daubenmier, J., Ferrara, A., Galarce, M. N., Hartogensis, W., Hecht, F. M., Hedderson, M. M., Moran, P. J., Pagoto, S. L., Tsai, A.-L., Waring, M. E., \u0026amp; Kiernan, M. (2021). Methods-Motivational Interviewing Approach for Enhanced Retention and Attendance. \u003cem\u003eAmerican Journal of Preventive Medicine\u003c/em\u003e, \u003cem\u003e61\u003c/em\u003e(4), 606\u0026ndash;617. https://doi.org/10.1016/j.amepre.2021.04.005\u003c/li\u003e\n\u003cli\u003eLi, D. H., Brown, C. H., Gallo, C., Morgan, E., Sullivan, P. S., Young, S. D., \u0026amp; Mustanski, B. (2019). Design Considerations for Implementing eHealth Behavioral Interventions for HIV Prevention in Evolving Sociotechnical Landscapes. \u003cem\u003eCurrent HIV/AIDS Reports\u003c/em\u003e, \u003cem\u003e16\u003c/em\u003e(4), 335\u0026ndash;348. https://doi.org/10.1007/s11904-019-00455-4\u003c/li\u003e\n\u003cli\u003eLi, D. H., Macapagal, K., Mongrella, M., Saber, R., \u0026amp; Mustanski, B. (2024). \u0026ldquo;Your Package Could Not Be Delivered\u0026rdquo;: The State of Digital HIV Intervention Implementation in the US. \u003cem\u003eCurrent HIV/AIDS Reports\u003c/em\u003e, \u003cem\u003e21\u003c/em\u003e(3), 152\u0026ndash;167. https://doi.org/10.1007/s11904-024-00693-1\u003c/li\u003e\n\u003cli\u003eMacapagal, K., Li, D. H., Clifford, A., Madkins, K., \u0026amp; Mustanski, B. (2020). The CAN-DO-IT Model: A Process for Developing and Refining Online Recruitment in HIV/AIDS and Sexual Health Research. \u003cem\u003eCurrent HIV/AIDS Reports\u003c/em\u003e, \u003cem\u003e17\u003c/em\u003e(3), 190\u0026ndash;202. https://doi.org/10.1007/s11904-020-00491-5\u003c/li\u003e\n\u003cli\u003eMayer, R. e. (2017). Using multimedia for e-learning. \u003cem\u003eJournal of Computer Assisted Learning\u003c/em\u003e, \u003cem\u003e33\u003c/em\u003e(5), 403\u0026ndash;423. https://doi.org/10.1111/jcal.12197\u003c/li\u003e\n\u003cli\u003eMcCann, S. K., Campbell, M. K., \u0026amp; Entwistle, V. A. (2010). Reasons for participating in randomised controlled trials: Conditional altruism and considerations for self. \u003cem\u003eTrials\u003c/em\u003e, \u003cem\u003e11\u003c/em\u003e(1), 31. https://doi.org/10.1186/1745-6215-11-31\u003c/li\u003e\n\u003cli\u003eMunroe, S., Schackman, B. R., Madkins, K., Saber, R., Macapagal, K., Vititow, J., Sweeney, N., Feder, N. M., Benbow, N., Mustanski, B., \u0026amp; Linas, B. P. (2025). Economic Evaluation of Delivering an Evidence-Based Online HIV Prevention Program to MSM via Direct-To-Consumer Marketing Versus Community-Based Organization Recruitment. \u003cem\u003eJAIDS Journal of Acquired Immune Deficiency Syndromes\u003c/em\u003e, \u003cem\u003e98\u003c/em\u003e(3), 257\u0026ndash;264. https://doi.org/10.1097/QAI.0000000000003572\u003c/li\u003e\n\u003cli\u003eMustanski, B., Benbow, N., Macapagal, K., Li, D., Madkins, K., Saber, R., Linas, B., Smith, J., Brown, C. H., Munroe, S., Reddy, S., Schackman, B. R., Swann, G., Janulis, P., Zamantakis, A., \u0026amp; Zapata, J. P. (2025a). Comparing Implementation and Effectiveness Outcomes for Two Implementation Strategies of the Keep It Up! Digital HIV Prevention Program: A Type 3 Hybrid Effectiveness-Implementation Trial. \u003cem\u003eAIDS and Behavior\u003c/em\u003e, \u003cem\u003e29\u003c/em\u003e(12), 4030\u0026ndash;4042. https://doi.org/10.1007/s10461-025-04838-0\u003c/li\u003e\n\u003cli\u003eMustanski, B., Benbow, N., Macapagal, K., Li, D., Madkins, K., Saber, R., Linas, B., Smith, J. D., Brown, C. H., Munroe, S., Reddy, S., Schackman, B. R., Swann, G., Janulis, P., Zamantakis, A., \u0026amp; Zapata, J. P. (2025b). Comparing Implementation and Effectiveness Outcomes for Two Implementation Strategies of the Keep It Up! Digital HIV Prevention Program: A Type 3 Hybrid Effectiveness-Implementation Trial. \u003cem\u003eAIDS and Behavior\u003c/em\u003e, \u003cem\u003e29\u003c/em\u003e(12), 4030\u0026ndash;4042. https://doi.org/10.1007/s10461-025-04838-0\u003c/li\u003e\n\u003cli\u003eMustanski, B., Garofalo, R., Monahan, C., Gratzer, B., \u0026amp; Andrews, R. (2013). Feasibility, Acceptability, and Preliminary Efficacy of an Online HIV Prevention Program for Diverse Young Men who have Sex with Men: The Keep It Up! Intervention. \u003cem\u003eAIDS and Behavior\u003c/em\u003e, \u003cem\u003e17\u003c/em\u003e(9), 2999\u0026ndash;3012. https://doi.org/10.1007/s10461-013-0507-z\u003c/li\u003e\n\u003cli\u003eMustanski, B., Macapagal, K., Li, D. H., Rodriguez-Diaz, C. E., Saber, R., Matson, M., Moskowitz, D. A., Rendina, H. J., Laber, E., Ryan, D. T., \u0026amp; Newcomb, M. E. (2025). Effectiveness of the smart program: Stepped-care HIV prevention for gay and bisexual adolescent boys. \u003cem\u003eHealth Psychology\u003c/em\u003e, \u003cem\u003e44\u003c/em\u003e(3), 321\u0026ndash;331. https://doi.org/10.1037/hea0001471\u003c/li\u003e\n\u003cli\u003eMustanski, B., Madkins, K., Greene, G. J., Parsons, J. T., Johnson, B. A., Sullivan, P., Bass, M., \u0026amp; Abel, R. (2017). Internet-Based HIV Prevention With At-Home Sexually Transmitted Infection Testing for Young Men Having Sex With Men: Study Protocol of a Randomized Controlled Trial of Keep It Up! 2.0. \u003cem\u003eJMIR Research Protocols\u003c/em\u003e, \u003cem\u003e6\u003c/em\u003e(1), e5740. https://doi.org/10.2196/resprot.5740\u003c/li\u003e\n\u003cli\u003eMustanski, B., Parsons, J. T., Sullivan, P. S., Madkins, K., Rosenberg, E., \u0026amp; Swann, G. (2018). Biomedical and Behavioral Outcomes of Keep It Up!: An eHealth HIV Prevention Program RCT. \u003cem\u003eAmerican Journal of Preventive Medicine\u003c/em\u003e, \u003cem\u003e55\u003c/em\u003e(2), 151\u0026ndash;158. https://doi.org/10.1016/j.amepre.2018.04.026\u003c/li\u003e\n\u003cli\u003eMustanski, B., Saber, R., Jones, J. P., Macapagal, K., Benbow, N., Li, D. H., Brown, C. H., Janulis, P., Smith, J. D., Marsh, E., Schackman, B. R., Linas, B. P., Madkins, K., Swann, G., Dean, A., Bettin, E., \u0026amp; Savinkina, A. (2023). Keep it up! 3.0: Study protocol for a type III hybrid implementation-effectiveness cluster-randomized trial. \u003cem\u003eContemporary Clinical Trials\u003c/em\u003e, \u003cem\u003e127\u003c/em\u003e, 107134. https://doi.org/10.1016/j.cct.2023.107134\u003c/li\u003e\n\u003cli\u003eMustanski, B., Saber, R., Macapagal, K., Matson, M., Laber, E., Rodrgiuez-Diaz, C., Moran, K. O., Carrion, A., Moskowitz, D. A., \u0026amp; Newcomb, M. E. (2023). Effectiveness of the SMART Sex Ed program among 13\u0026ndash;18 year old English and Spanish speaking adolescent men who have sex with men. \u003cem\u003eAIDS and Behavior\u003c/em\u003e, \u003cem\u003e27\u003c/em\u003e(2), 733\u0026ndash;744. https://doi.org/10.1007/s10461-022-03806-2\u003c/li\u003e\n\u003cli\u003eNguyen, L. H., Tran, B. X., Rocha, L. E. C., Nguyen, H. L. T., Yang, C., Latkin, C. A., Thorson, A., \u0026amp; Str\u0026ouml;mdahl, S. (2019). A Systematic Review of eHealth Interventions Addressing HIV/STI Prevention Among Men Who Have Sex With Men. \u003cem\u003eAIDS and Behavior\u003c/em\u003e, \u003cem\u003e23\u003c/em\u003e(9), 2253\u0026ndash;2272. https://doi.org/10.1007/s10461-019-02626-1\u003c/li\u003e\n\u003cli\u003ePark, S., Fisher, C., Flew, T., \u0026amp; Dulleck, U. (2020). Global Mistrust in News: The Impact of Social Media on Trust. \u003cem\u003eInternational Journal on Media Management\u003c/em\u003e, \u003cem\u003e22\u003c/em\u003e(2), 83\u0026ndash;96. https://doi.org/10.1080/14241277.2020.1799794\u003c/li\u003e\n\u003cli\u003eRoehl, J. M., \u0026amp; Harland, D. J. (2022). \u003cem\u003e\u0026ldquo;Imposter Participants: Overcoming Methodological Challenges Related to\u0026rdquo; by Jacqueline M. Roehl Ph.D. and Darci J. Harland Ph.D.\u003c/em\u003e https://nsuworks.nova.edu/tqr/vol27/iss11/5/\u003c/li\u003e\n\u003cli\u003eRuth Colvin, \u0026amp; Mayer, R. E. (Eds.). (2016). \u003cem\u003eE-Learning and the Science of Instruction\u003c/em\u003e (1st ed.). John Wiley \u0026amp; Sons, Ltd. https://doi.org/10.1002/9781119239086\u003c/li\u003e\n\u003cli\u003eSarno, E. L., Macapagal, K., \u0026amp; Newcomb, M. E. (2021). \u0026ldquo;The Main Concern is HIV, Everything Else is Fixable\u0026rdquo;: Indifference Toward Sexually Transmitted Infections in the Era of Biomedical HIV Prevention. \u003cem\u003eAIDS and Behavior\u003c/em\u003e, \u003cem\u003e25\u003c/em\u003e(8), 2657\u0026ndash;2660. https://doi.org/10.1007/s10461-021-03226-8\u003c/li\u003e\n\u003cli\u003eSchon, D. A. (1963). Champions for radical new inventions. \u003cem\u003eHarvard Business Review : HBR, Harvard Business Review : HBR. - Boston, Mass. : Harvard Business School Publ. Corp., ISSN 0017-8012, ZDB-ID 2382-6. - Vol. 41.1963, 2, p. 77-86\u003c/em\u003e, \u003cem\u003e41\u003c/em\u003e(2), 77.\u003c/li\u003e\n\u003cli\u003eSharma, A., Sullivan, P. S., \u0026amp; Khosropour, C. M. (2011). Willingness to Take a Free Home HIV Test and Associated Factors among Internet-Using Men Who Have Sex with Men. \u003cem\u003eJournal of the International Association of Physicians in AIDS Care\u003c/em\u003e, \u003cem\u003e10\u003c/em\u003e(6), 357\u0026ndash;364. https://doi.org/10.1177/1545109711404946\u003c/li\u003e\n\u003cli\u003eTeitcher, J. E. F., Bockting, W. O., Bauermeister, J. A., Hoefer, C. J., Miner, M. H., \u0026amp; Klitzman, R. L. (2015). Detecting, Preventing, and Responding to \u0026ldquo;Fraudsters\u0026rdquo; in Internet Research: Ethics and Tradeoffs. \u003cem\u003eJournal of Law, Medicine \u0026amp; Ethics\u003c/em\u003e, \u003cem\u003e43\u003c/em\u003e(1), 116\u0026ndash;133. https://doi.org/10.1111/jlme.12200\u003c/li\u003e\n\u003cli\u003eTorous, J., Lipschitz, J., Ng, M., \u0026amp; Firth, J. (2020). Dropout rates in clinical trials of smartphone apps for depressive symptoms: A systematic review and meta-analysis. \u003cem\u003eJournal of Affective Disorders\u003c/em\u003e, \u003cem\u003e263\u003c/em\u003e, 413\u0026ndash;419. https://doi.org/10.1016/j.jad.2019.11.167\u003c/li\u003e\n\u003cli\u003eValente, T. W., \u0026amp; Pumpuang, P. (2007). Identifying Opinion Leaders to Promote Behavior Change. \u003cem\u003eHealth Education \u0026amp; Behavior\u003c/em\u003e, \u003cem\u003e34\u003c/em\u003e(6), 881\u0026ndash;896. https://doi.org/10.1177/1090198106297855\u003c/li\u003e\n\u003cli\u003eWilliams, B., Entwistle, V., Haddow, G., \u0026amp; Wells, M. (2008). Promoting research participation: Why not advertise altruism? \u003cem\u003eSocial Science and Medicine\u003c/em\u003e, \u003cem\u003e66\u003c/em\u003e(7), 1451\u0026ndash;1456. https://doi.org/10.1016/j.socscimed.2007.12.013\u003c/li\u003e\n\u003cli\u003eWippold, G. M., Garcia, K. A., Domlyn, A., Wong, D., \u0026amp; Fernandez, M. E. (2025). Proactive planning for contextual fit: The role of the implementation premortem. \u003cem\u003eImplementation Science Communications\u003c/em\u003e, \u003cem\u003e6\u003c/em\u003e(1), 84. https://doi.org/10.1186/s43058-025-00768-z\u003c/li\u003e\n\u003cli\u003eZamantakis, A., Zapata, J. P., Smith, J. D., Danielson, E. C., Macapagal, K., Saber, R., Li, D., Benbow, N., \u0026amp; Mustanski, B. (2025). Incentivizing Prevention: Community Based Organizations\u0026rsquo; Perceptions on Financial Incentives for Recruitment and Retention of Participants in Keep it Up! (KIU!), an eHealth HIV Prevention Intervention. \u003cem\u003eAIDS and Behavior\u003c/em\u003e. https://doi.org/10.1007/s10461-025-04963-w\u003c/li\u003e\n\u003cli\u003eZhang, Y., Guy, R. J., Smith, K. S., Jamil, M. S., Prestage, G., Applegate, T. L., Conway, D. P., Holt, M., Keen, P., Bavinton, B., McNulty, A. M., Batrouney, C., Russell, D., Vaughan, M., Chen, M., Fairley, C. K., Grulich, A. E., Kaldor, J. M., \u0026amp; Callander, D. (2021). Sustaining success: A qualitative study of gay and bisexual men\u0026rsquo;s experiences and perceptions of HIV self-testing in a randomized controlled trial. \u003cem\u003eBMC Public Health\u003c/em\u003e, \u003cem\u003e21\u003c/em\u003e, 2048. https://doi.org/10.1186/s12889-021-12011-0\u003c/li\u003e\n\u003c/ol\u003e"}],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":true,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":false,"isAuthorSuppliedPdf":false,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":false,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"implementation-science-communications","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"iscm","sideBox":"Learn more about [Implementation Science Communications](https://implementationsciencecomms.biomedcentral.com)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/ISCM/default.aspx","title":"Implementation Science Communications","twitterHandle":"@ImplementSci","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"Digital health interventions, Direct-to-consumer (DTC) interventions, hybrid effectiveness-implementation study, HIV prevention","lastPublishedDoi":"10.21203/rs.3.rs-9202620/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-9202620/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003ch2\u003eBackground\u003c/h2\u003e \u003cp\u003eDigital HIV prevention interventions (DHIs) are efficacious in increasing prevention behaviors, and delivering them direct-to-consumer (DTC) expands their reach. However, few have successfully moved rigorously-studied DHIs from research to public health practice. Hybrid type 3 effectiveness-implementation studies promise to approximate more naturalistic settings and accelerate translation from research to practice, and few DTC DHIs have been tested this way. This study describes lessons learned from a hybrid type 3 study of a DTC DHI for young men who have sex with men (YMSM) ages 18\u0026ndash;29 called Keep It Up! (KIU!) 3.0.\u003c/p\u003e\u003ch2\u003eMethods\u003c/h2\u003e \u003cp\u003eKIU! 3.0 was initially designed as a cluster-randomized hybrid type 3 implementation-effectiveness study across 44 United States counties with high HIV incidence among YMSM (22 DTC; 22 implemented in community-based organizations). The DTC strategy relied on online recruitment, at-home HIV/STI testing, and centralized intervention delivery. Over the course of the trial (October 2019-March 2023), we adapted our implementation four times in response to recruitment and retention challenges. Data sources included enrollment logs, advertising expenditures, participant communications, and internal documentation, which are used to characterize recruitment, costs, and recruitment/retention patterns.\u003c/p\u003e\u003ch2\u003eResults\u003c/h2\u003e \u003cp\u003eDue to challenges in recruitment and retention, four major changes were made over the course of the trial: 1) streamlining enrollment procedures, including shortening screening and verification steps, 2) modifying eligibility criteria, including expanding age limits, removing sexual risk requirements, and including gender-diverse participants, 3) introducing and increasing financial incentives for intervention completing and follow-up measures, and 4) shifting from county-level to nationwide recruitment. Following this fourth change in March 2021, enrollment increased by 301%, and cost per enrolled participant decreased. Ultimately 1,468 participants were enrolled nationwide. Nevertheless, retention and return of at-home STI test kits remained challenging, and guaranteed incentives increased the prevalence of imposter participants.\u003c/p\u003e\u003ch2\u003eConclusion\u003c/h2\u003e \u003cp\u003eImplementing a DTC DHI within a hybrid type 3 study required balancing pragmatic implementation goals with effectiveness outcomes measurement. Future hybrid studies of DTC digital health interventions should consider eligibility criteria, incentive structures, outcomes measurement strategies, and the distinction between research and service components to better align research with real-world implementation contexts.\u003c/p\u003e","manuscriptTitle":"When scientific constraints and real-world implementation collide: Lessons from a hybrid type 3 study of a digital, direct-to-consumer HIV prevention program","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2026-04-23 09:08:21","doi":"10.21203/rs.3.rs-9202620/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"reviewerAgreed","content":"77904015001567138350943778999417866793","date":"2026-04-17T12:41:59+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2026-04-14T13:14:34+00:00","index":"","fulltext":""},{"type":"editorAssigned","content":"","date":"2026-04-08T03:00:12+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2026-03-24T23:46:55+00:00","index":"","fulltext":""},{"type":"submitted","content":"Implementation Science Communications","date":"2026-03-23T15:38:05+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"implementation-science-communications","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"iscm","sideBox":"Learn more about [Implementation Science Communications](https://implementationsciencecomms.biomedcentral.com)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/ISCM/default.aspx","title":"Implementation Science Communications","twitterHandle":"@ImplementSci","acdcEnabled":true,"dfaEnabled":true,"editorialSystem":"em","reportingPortfolio":"BMC/SO AJ","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"9e45cda8-5e34-4bb1-92aa-670db0e370b8","owner":[],"postedDate":"April 23rd, 2026","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"under-review","subjectAreas":[],"tags":[],"updatedAt":"2026-04-23T09:08:23+00:00","versionOfRecord":[],"versionCreatedAt":"2026-04-23 09:08:21","video":"","vorDoi":"","vorDoiUrl":"","workflowStages":[]},"version":"v1","identity":"rs-9202620","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-9202620","identity":"rs-9202620","version":["v1"]},"buildId":"XKTyCvWXoU3ODBz1xrDgd","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}