Daily angina documentation versus subsequent recall: development of a symptom smartphone app.

The emphasis on symptom relief in stable angina has increased since the value of coronary revascularization for reducing cardiovascular outcomes like myocardial infarction and death was questioned. 1 , 2 . It is important that symptom relief is measured in an accurate, reliable, and patient-centred manner. Modes of assessment include exercise time, angina frequency, e.g. Seattle Angina Questionnaire (SAQ), 3 and quality of life, e.g. EuroQOL-5D (EQ-5D). Symptoms can be self-reported or physician-assessed, but both rely on patients recalling their symptoms over a period of time. ORBITA was a randomized placebo-controlled trial of percutaneous coronary intervention (PCI) for patients with single-vessel coronary artery disease. 4 Participants were randomized to PCI or placebo in a 1 to 1 ratio. The primary endpoint of ORBITA was exercise time. ORBITA-2 5 was designed in line with feedback from ORBITA participants and the clinical and scientific cardiology community. ORBITA-2 is recruiting people with symptoms of stable angina, evidence of ischaemia, and significant coronary stenosis in at least one vessel. In particular, the endpoint is an ordinal clinical outcome scale for angina which contains daily angina frequency. 6 This paper describes the development of the smartphone application used for documenting daily symptoms, including a feasibility and acceptability assessment, and a comparison with subsequent recall.

The design focus group only contains patients who participated in ORBITA. Thus, it does not include people who may not agree to participate in a trial or undergo a randomized blinded procedure. It may therefore not be representative of the general population of patients with angina. However, we were doing this work with the intention of applying to the ORBITA-2 trial and therefore we believe that the focus group is well qualified to advise. The Recall Substudy did not include all participants in ORBITA-2 but rather all recruited within a particular time window. We have no reason to believe that these participants were any different from the generality of ORBITA-2 participants. For the reference measurement, we used the daily reports from participants via the symptom app. We did not make daily phone calls nor ask them to keep a separate paper diary. Therefore, we cannot exclude the possibility that they entered incorrect information on the app. On the other hand, telephone calls would be unnecessarily irritating for participants, jeopardizing retention in the trial. With paper diaries, there is no way to tell that entries are made contemporaneously rather than just before a subsequent visit. Therefore, we consider the daily entries on the app to be a suitably contemporaneous reference data set. We did not try multiple different protocols for reminding participants, for example, starting when they are only 1 day late. We settled with our focus group on a pattern that they considered acceptable. While it is possible that patients may report more symptoms including potentially trivial symptoms via the app, contemporaneous reporting is the gold standard for symptoms and limits post hoc interpretation of the symptoms.

The app was designed in partnership with patients who participated in the ORBITA trial which was completed in 2017. The aim of the app is to collect data for the blinded ORBITA-2 trial, i.e. to measure the clinical response to PCI or placebo. The trial methods have been previously described. 5 The primary outcome of ORBITA was exercise time and angina was primarily measured with the SAQ. The primary outcome of the ORBITA-2 trial is an ordinal clinical outcome scale which includes daily angina frequency reported on the smartphone app and the SAQ is a secondary outcome. The app was developed using a user-centred iterative design and testing cycle. This involved a focus group of people with lived experience of stable angina and participation in clinical trials. They were asked what elements would adequately cover their symptoms for reporting to researchers. Their feedback was recorded as notes during the meetings. They were each given a prototype of the app to try recording symptoms, and again their feedback was recorded. The app was iteratively improved and retested by members of the focus group. The first prototype of the app was based on feedback from ORBITA participants and investigators regarding the limitations of other angina assessments such as exercise testing and the Seattle Angina Questionnaire. The next iteration incorporated the results of a competitive analysis. The competitive analysis involved an appraisal of existing angina apps [Angina Control (Google Playstore), Angina Recorder (Apple App store), and Heart Keeper (Google Playstore)] and apps for other chronic pain conditions (Sora, Migraine Buddy, and RheumaBuddy for endometriosis, migraines, and rheumatoid arthritis, respectively) to inform the design of the smartphone app. Participants record the number of episodes experienced each day. The app also records the severity of the episodes. Participants use a slider to grade their worst episode that day from mild to severe on a scale (which they can interpret as the intensity and/or duration depending on their personal preference). However, in the ORBITA-2 trial, only angina frequency is included in the primary endpoint. Feasibility was assessed in two ways: first by focus group members using the prototype app and second with pilot data from actual ORBITA-2 participants. The pilot data included completion rates of daily symptom app entries, the proportion of participants requiring reminders to complete the app, and how many reminders were required. The app data dashboard indicated when a participant had not recorded their symptoms for 3 days and this triggered a reminder to be sent. Acceptability was also assessed by focus group members using the prototype app and with pilot data from actual ORBITA-2 participants. Focus group member opinions were collected through verbal feedback and a written survey. ORBITA-2 participant’s feedback was collected verbally at the end of their involvement in the study. The design focus group participants were participants from the previously completed ORBITA trial. All participants from ORBITA who agreed to subsequent contact were invited to focus group meetings. The recall assessment group was a substudy of ORBITA-2, composed of ORBITA-2 participants who had their final visit between 1 August 2020 and 30 April 2021 inclusive. The completion assessment group were all participants in ORBITA-2 who had their final visit before 30 April 2021. The ability to use the app is not an inclusion criterion for ORBITA-2. All participants receive one-to-one onboarding with a researcher to teach them to use the app and answer any queries. All participants who consented to participate have been able to complete the app. It is possible that the use of an app is a deterrent to some potential participants, but almost all of those approached already had devices compatible with apps. The aims of the focus groups were to gather feedback for the design of the ORBITA-2 trial which included the primary endpoint, smartphone app, patient-facing literature, and the patient experience. The app design team created a progressive web app, designed to be available on any smartphone, tablet, or computer. Usability was tested across these platforms. The user interface was tested and iteratively improved by focus group members who could try it on their phones and iPads available during focus group meetings. Maintenance of data integrity was tested by the app design team, including ensuring adequate logging of dates, times, users, and changes. In this phase, we optimized features such as font, readability, the onboarding process, error messages, e.g. for lack of Internet connection, and the frequently asked questions section, with input from the focus group members. A data protection impact assessment was carried out and a GDPR statement was prepared. Users are required to agree to a privacy policy during onboarding. According to Medicines and Healthcare Products Regulatory Agency (MHRA) Guidance: Medical device stand-alone software including apps (including IVDMDs) v1.08 7 the app is not a device. This is contingent on the app not being used to make decisions about clinical care. It is merely a tool to collect data for research purposes. Each day participants record the number of angina episodes. These data are used for angina frequency and are not transformed or adjusted. Data are summarized as n (%) and median (interquartile range). Trends in proportions were assessed using logistic regression. Analyses were performed using the open-source statistical environment ‘R’ Version 4.1.0. 8

We developed a working prototype web-based app tested across various Android smartphones, iPhones, tablets, and computer platforms. The design focus group was composed of 10 participants of the previous ORBITA trial, of whom one was female. The recall assessment group consisted of 29 ORBITA-2 participants who had their final study visit between 1 August 2020 and 30 April 2021. The completion assessment group consisted of the 142 ORBITA-2 participants who had their final study visit by 30 April 2021. The design focus group had meetings during October 2018. They reviewed prototype apps on their phones and/or iPad tablets which were available for their use during the meetings. They tried out the onboarding process. Table 1 shows their comments and the resulting changes made to the app and the participant onboarding process. Recommendations from the design focus group Discussion with the design focus group reminded us that people may modify their activity to avoid episodes of angina. We therefore wanted ORBITA-2 to include questions about symptom responses to reference activities that were stable during trial participation but were individually tailored to the participant at enrolment. The design focus group completed a written survey of activities that triggered their angina episodes. Their responses are shown in Table 2 . The design focus group advised us that changes in symptom response to these particular stimuli would be the best indicator of whether the treatment had worked, since it was relevant to their life and was the reason they originally sought medical help. Focus group member responses to survey of angina triggers We used their responses to provide in the ORBITA-2 app a list of activities. This is shown in Figure 1 . During onboarding, participants select two activities that currently cause angina. The list also contains an option for participants to enter an unlisted activity in free text form. This is shown in Figure 1 . Subsequently, every week during the trial, participants are asked whether they had angina when conducting the two pre-specified personalized activities. Options and examples of weekly provocation questions on app. The design focus group completed a written survey of activities that triggered their angina episodes. Their responses are shown in Table 2 . The design focus group advised us that changes in symptom response to these particular stimuli would be the best indicator of whether the treatment had worked, since it was relevant to their life and was the reason they originally sought medical help. We used their responses to provide in the ORBITA-2 app a list of activities. This is shown in Figure 1 . During onboarding, participants select two activities that currently cause angina. The list also contains an option for participants to enter an unlisted activity in free text form. This is shown in Figure 1 . Subsequently, every week during the trial, participants are asked whether they had angina when conducting the two pre-specified personalized activities. Reminders required over the duration of participation were not specific to the time period within the trial ( Figure 2B ). ( A ) Distribution of the number of reminders needed in the completion assessment group. ( B ) Reminders by week of trial in the completion assessment group. The recall assessment group were asked at the end of their participation to recall their number of episodes in each of the seven previous days individually. All said they could recall the previous day’s symptoms, but the proportion declined for each day before that, reaching 10 of 18 (55%) for 7 days previously ( Figure 3A , P  < 0.001 for trend). ( A ) Stated ability and actual ability to recall numbers of angina episodes. ( B ) Stated ability and actual ability to recall numbers of angina episodes when at least one episode was recorded. Recollections did not always match what they had entered in the app. For the previous day, 14 of 17 (82%) were able to give a correct recollection. This proportion declined for each day before that, reaching 55% for 7 days previously ( Figure 3A , P  = 0.04 for trend). Rates of correct recall were lower in those who had had at least one episode ( Figure 3B ). The recall assessment group were also asked to recall the total number of episodes in the past 4 weeks. Six of 29 (21%) said they could not recall. The remaining 23 participants reported a total of 157 episodes in the past 4 weeks, median 2 (IQR 0–12). Ten of 23 (43%) recalled correctly, of whom 8 of 10 (80%) had had no episodes.

Clinical trials of angina often use questionnaires at the end of a follow-up period to assess angina outcomes. These rely on participant recall of symptoms. The ability to recall symptoms correctly declines dramatically with time. Daily symptom reporting on a smartphone application was acceptable to clinical trial participants. Two-thirds of participants required reminders when they were behind schedule on data entry. Patients who have participated in randomized trials assessing angina are an ideal expert group to advise on how angina can be assessed without being unreasonably intrusive. We recommend this partnership model for the development of future symptom reporting apps for clinical trials of symptom relief. Daily symptom reporting and a protocol of reminders when behind schedule are acceptable to real-life clinical trial participants. ORBITA-2 is funded by Imperial College NIHR BRC. A.N.N. is supported by the NIHR Academy. Conflict of interest: R.K.A.-L. receives speaker’s honoraria from Phillips Volcano and Menarini Pharmaceuticals. A.N.N. is supported by the NIHR Academy. M.F. and C.R. are supported by the MRC. T.R.K. has research grants from Abbott Vascular, Zoll Corporation, and speaker fee honoria from Abbott Vascular, Astra Zeneca, and BD. We acknowledge support from the National Institute for Health Research Imperial Biomedical Research Centre and the Imperial British Heart Foundation Centre of Research Excellence. F.E.H. receives consulting fees from Regeneron, Johnson & Johnson/Jenssen, Annexon, Provention Bio, and AstraZeneca for unrelated projects. Data will be made available on request.

As time passes, it becomes increasingly difficult to recall episodes of angina accurately. When patients have had angina, accurate recall falls to 25% even within 1 week. For clinical trials focusing on angina endpoints, daily documentation is therefore advisable. All participants found a smartphone app easy to use. One-third of patients entered all their data without needing reminders, and the other two-thirds required a mean of 0.4 reminders per week of participation. Overall, data collection was 99% complete.

In this study, we found that recalling numbers of episodes of angina is difficult or impossible after a few days unless that number is zero. For patients who experienced angina, episodes that occurred more than 1 day ago were only 14% likely to be correctly recalled, while they were 29% likely to be incorrectly recalled, with the remainder stating that they could not recall. The ORBITA-2 symptom app was developed with patients who had completed the first ORBITA trial, because they recommended symptoms to be the primary focus of ORBITA-2. In the completion assessment group, 99% completed data entry in full. The design focus group advised that simply counting episodes would understate the impact of the condition because they reported adapting their activity patterns to avoid experiencing angina. ORBITA-2, therefore, asked each participant at enrolment to identify two reference activities which characteristically caused them angina and which they would continue to carry out at least weekly throughout their participation. Separate from the main assessment which was daily episode count, the app asked them weekly whether they experienced angina on each of the two personalized reference activities. Members of the design focus group, during their discussions, noticed that their individual experiences of angina were different although generally stereotyped within an individual. They wanted this to be formally documented somehow in ORBITA-2. We were mindful that repeatedly asking large numbers of categorical questions could be irritating and still leave the participant with the feeling that their individual symptom has not been adequately described. Certainly repeating the process regularly during the study would impair the participant goodwill on which the trial depends. ORBITA-2 therefore asks participants to describe their symptoms in their own words. This personal definition of angina is available on the app for them to refer back to. This allows a single daily question to cover the participant’s own angina without the excessive wordiness of listing all possible variants. The ability to recall (correctly or incorrectly) episodes of angina declines progressively within a few days to reach only 55% at 7 days. Even this limited recall is bolstered by those who actually had zero episodes. For those with one or more episodes to recall, the ability to recall (correctly or incorrectly) reached only 25% at 7 days. We were surprised by the rapidity of decline in these proportions. The incorrect recall was only a problem for symptoms in the very recent past. For symptoms that were even further back, participants would simply say they could not recall at all. ORBITA-2 collects daily data via the symptom app so that its angina episode counts can be as representative as possible. The episode counts reported at the end of the trial occur only because these are part of conventional questionnaires. The inaccurate recall is a renowned issue in clinical research for chronic pain conditions and a daily smartphone approach known as Ecological Momentary Assessment has been used for conditions such as breakthrough cancer pain 9 and fibromyalgia 10 for this reason. The app is being used for daily angina reporting in other studies assessing symptom relief for stable angina. 11 , 12 The use of the app for monitoring in a clinical setting warrants further investigation in future, e.g. for patients within cardiac rehabilitation programmes or monitoring response to therapy.