Informing the Design of Individualized Self-Management Regimens from the Human, Data, and Machine Learning Perspectives

Intelligent systems — powered by AI and large volumes of patient-contributed data — have the potential to support humans in managing chronic illness [ 13 ]. However, their real potential has not yet been fully realized [ 10 , 38 ]. Arguably, one barrier is limitations in existing design approaches: user-centered design, while well-established in the context of more traditional software applications, does not account for the unique constrains and inherent structures of AI models and, thus, may lead to requirements not possible to meet with existing AI algorithms [ 25 , 127 ]. On the other hand, technology-driven approaches to the development of AI may lead to interactive systems inconsistent with user needs and result in limited adoption and unintended harms [ 18 ]. To move towards integrating these systems into the management of illness, we propose and implement an HAI framework — which we have termed Multi-Perspective Directed Analysis. We use MPDA to conduct a mixed-methods study to map and synthesize human, data, and ML requirements and constraints to generate design recommendations for an AI-enabled solution in the context of uncertainty in chronic illness. This approach also enables us to identify and elaborate on several sociotechnical gaps [ 2 ], where we document a mismatch between the complex, nuanced demands of real-world self-management and the rigid limitations of existing technologies. Here, we discuss implications from synthesizing needs identified by users, “in the wild” self-tracked data, and constraints of an RL approach for management of a complex, poorly understood disease. Developing AI-enabled systems presents unique challenges compared to traditional software development, since AI models operate within fixed conceptual spaces with specific constraints. Unlike conventional methods, where user needs can directly inform system features, designing AI systems requires adapting to the capabilities of particular AI methods, such as carefully configuring input and output features for neural networks. This highlights the need for approaches that attend to both user needs and technical capabilities of ML. Sociotechnical approaches, including participatory design, have long emphasized the importance of aligning human and technical factors in system design [ 58 , 109 , 130 , 146 ]. However, the complexities introduced by AI require a broader lens to address its more complex sociotechnical dynamics [ 51 ], which traditional methods may not fully accommodate [ 25 ]. Some human-centered design approaches offer valuable practices — multidisciplinary collaboration, iterative prototyping, and continuous evaluation [ 9 , 79 , 112 , 113 ] — to design tools that are both technically feasible while also aligning with user needs. Yet, there is significant value in a framework that explicitly accounts for human, machine, and data perspectives in the design of intelligent systems. We proposed an HAI framework to synthesize human, data, and ML perspectives — we use high-level concepts of a particular ML approach, RL, as guiding principles to synthesize and triangulate findings across quantitative and qualitative data sources. Each “perspective” informs and complements the others, serving the analysis in important ways. First, the ML perspective allows us to account for the specific requirements and constraints of a particular ML approach, which is a novel aspect of this framework. Second, the human perspective ensures the system is grounded in real-world needs, perspectives, and practices, with qualitative data guiding the development of the final results categories. Third, the data perspective allows us to validate user-reported insights, enhance qualitative findings with a larger sample, and identify gaps in existing data. Self-tracked, “in-the-wild” data offer a unique view of actual user behaviors and practices, informing additional data requirements. This integrated approach provides a comprehensive understanding of both technical and human factors to inform the design of a real-world system. We demonstrate that the MPDA framework provides a structured approach to the design process, offering a unique method for integrating insights from three distinct perspectives: human, machine, and data. MPDA enables a comprehensive understanding of complex tasks and contexts, such as those related to specific illnesses and care tasks. By organizing and synthesizing these perspectives, MPDA helps to reduce the complexity of the problem space, yielding nuanced insights into human experiences and translating them into actionable design recommendations. While MPDA cannot guarantee the success of an RL algorithm, it offers a systematic way to evaluate its potential by addressing key questions around system requirements, user acceptance, and data feasibility. Additionally, the framework has broader implications for HAI, re-affirming some established principles while also uncovering less explored areas. Below, we describe how MPDA connects to and advances the principles of HAI. The framework facilitates human-centeredness in two key ways. First, the framework itself explicitly accounts for the perspectives of people, data, and a specific algorithm — relying on human users as experts helps to avoid dehumanizing them or creating harmful technologies [ 33 ]. Second, using this framework has enabled us to generate recommendations that foreground the human-centerdness required for a successful real-world intelligent system, rather than allowing the algorithmic perspective to dominate. MPDA introduces a principled way to integrate both human and technological perspectives systematically, ensuring that human-centered principles drive our design recommendations. Designing for control and user autonomy. Facilitating human intuition integrated with computational support is a fundamental goal of HAI [ 136 ]. Thus, enabling user control and autonomy are key features for designing human-centered AI systems. The MPDA framework helped identify areas where incorporating user input and human-in-the-loop aspects of RL could enhance control and offset computational challenges. In particular, in defining and constraining the action space, users wish to input preferences and limits upfront, as well as provide feedback to the system as recommendations are given (e.g., if they want to engage in a strategy or how they might want to modify it so that it fits their current needs). Existing applications provide examples for how features of interactive RL could accomplish these goals (e.g., guided exploration of the action space, or action advice where the user suggests the action they believe is optimal at a given state) [ 12 ], which can work to simultaneously meet the needs of users and the requirements of RL, while advancing principles of HAI. Enabling explainability. Explainability can enable personalization [ 141 ] and offer insights into managing chronic conditions [ 23 , 157 ], and is itself central to HAI [ 59 ]. Explainability in this context is particularly important, due to illness uncertainty and the high demands of user involvement with an RL, so users must constantly decide when to follow system recommendations. For an explainable RL-based system, designers should avoid black-box algorithms [ 11 , 106 , 137 ], opting instead for model-based techniques like MDP, bandit algorithms [ 7 ], or Bayesian methods, e.g., via Thompson sampling based flexible solutions [ 149 ] or via hierarchical models that learn both within and across individuals [ 145 ]. A small but quickly growing body of literature highlights explainable RL approaches specifically — largely applied in dissimilar contexts like gaming, computer vision, and robotics, but which could be adapted to this personal health context [ 49 , 69 , 85 , 101 , 123 , 155 ]. The importance of safety, privacy, and trust. Although safety, privacy, and trust are critical principles of HAI systems [ 116 ], participants in our study did not identify these as important features. Rather than not being important to users, this likely reflects a dangerous assumption that these aspects are already addressed. The absence of these concerns in our study highlights a limitation of our approach, emphasizing the need for researchers to proactively consider critical HAI elements that may not be identified by users. Additionally, we identify several key insights into less commonly emphasized aspects of HAI that arose in the context of endometriosis. Taken together, the user needs and design recommendations generated in this study are consequences of the complexity and unpredictability of managing endometriosis. This is not a given — other aspects of the disease emerge as most prominent in different settings. When seeking a diagnosis, the invisible nature of the disease is prominent [ 77 ]. In the context of shared decision making, research documents needs related to the stigma of the disease, privacy around disclosures, and managing emotions associated with illness [ 120 ]. This study’s findings related to illness complexity and uncertainty resulted in specific requirements and recommendations. We identify opportunities and challenges of using ML in a design space with nuanced, holistic representations of patient data. Personalization for individualized, internal context. We document wide variations in illness experiences and identify diverse approaches to self-management. While the need for personalization in chronic disease self-management is well understood [ 55 , 142 ], it presents a considerable challenge for ML and AI. This study documented needs and opportunities for personalization from the perspective of RL, specifically, the need to individualize the action space , state space , reward , and the agent/policy . Translating complex human experiences into representations that can be used by RL remains challenging, requiring reconciliation between human and algorithmic inputs. However, personalized user modeling has been explored in both HCI [ 65 ] and ML research [ 154 ]. There are also new ML approaches to guide the development of personalized reward functions [ 97 ]. Furthermore, considering the prospect of designing a personalized ML-enabled system from the data perspective suggests that individuals are willing to engage in self-monitoring, which could support the application of RL. At the same time, the study also identified the need to constrain the action space, providing several actionable strategies to address this challenge. Embodied health experiences. Individuals’ perceptions of their illness experiences are multi-faceted, complex, unique, and sometimes subjective, often relating to nuanced, embodied sensations. At the same time, computational tools require tractable representations of health status to evaluate if self-management strategies offer improvements. In an RL-enabled system, the state space must account for and represent these complex and subjective health states, while the reward mechanism needs to assess the effectiveness of strategies based on individuals’ unique experiences and evaluations of their health. Thus, interactive systems will need to translate these embodied health experiences into something that could be quantified, measured, and compared across time. Feminist scholars [ 122 ] and information theorists [ 24 ] have discussed how these properties of data require simplifying reality and imposing classifications that may not align with an individual’s experience. The gap between the messy, embodied tracking needs of menstruators and existing technologies has been previously documented [ 119 ]. In Data Feminism , D’Ignazio and Klein [ 54 ] explain that data serve to render the health experiences of individuals visible, from the patient perspective, and that to be useful, it is necessary to create machine-readable representations. However, they also warn that care must be taken to avoid flattening experiences or dehumanizing users. They recommend representations move beyond simplistic binaries and categories to capture the rich nuance and complexity of lived health experiences, and to contextualize information, rather than relying on the notion that data can “speak for themselves” [ 54 ]. Addressing challenges of creating nuanced representations of complex, embodied data requires creative, human-centered technical innovations, which might involve capturing multi-modal data (e.g., voice, video, or artistic creations) [ 74 , 89 ] and applying digital phenotyping methods to generate rich health representations [ 114 ]. These approaches should actively engage users in categorizing and labeling their data to ensure alignment with their lived experiences. Real-time, real-world context. An intelligent interactive system for self-management must adapt to individuals’ needs and preferences, respond to life circumstances and uncertainty, and provide dynamic recommendations that account for real-time context. Thus, representations that account for real-time, real-word context are important for defining the state space and for learning individualized policies — i.e., tailoring recommendations, recognizing triggers or patterns, and enabling explainability about why a recommendation was made [ 3 ]. Disregarding context results in fragmented, shallow representations of users, detached from their non-clinical experiences and broader environments [ 138 ], which can negatively impact patient outcomes [ 19 ]. The importance of incorporating context is well-established. Context-aware solutions that use situational information that are relevant to a user for a particular task [ 53 ] can enable personalized recommendations [ 151 ] and facilitate behavior change [ 83 ]. Context-sensitive recommender systems and personal informatics tools further highlight how integrating real-world circumstances supports tailored interventions [ 3 , 125 , 151 ]. Nevertheless, in the context of RL for a complex chronic illness, it will be challenging to capture, represent, and use complex, real-world, real-time context in ways that are not at odds with an RL’s requirement for a constrained state and action space. These key challenges represent examples of Ackerman’s sociotechnical gap [ 2 ], where the shortcomings of technical systems cannot fully support the flexible, nuanced, and contextualized human experiences and activities required in the task of self-management. People with endometriosis describe complex and unpredictable illness experiences, which are difficult to translate into solutions compatible with existing technical capabilities. In this work, we not only articulate where these gaps arise but also present recommendations for solutions that partially solve these problems with known tradeoffs and provide work-arounds, in particular that enhance human-centered elements of control and autonomy. Unlike conditions with clearly established and well-defined treatment guidelines, people with endometriosis lack reliable, evidence-based treatments, leaving them to navigate their care through self-management [ 132 , 133 , 158 , 159 ]. Yet despite this uncertainty, self-management helps empower individuals with endometriosis in their own care, mitigate their symptoms, and cope with the burden of their illness [ 120 ]. Even though recommendations provided by intelligent systems may be limited or ineffective, the trial-and-error process of finding strategies that work for an individual remains an essential aspect of care. As the standard of care advances, patients would benefit from systems that support this exploratory process. Given the documented complexity and uncertainty surrounding endometriosis, technology that can help patients identify patterns and discover new management strategies has significant implications for improving the state of care for endometriosis. We document that individuals already engage in a complex, personal trial-and-error process and are interested in computational tools to support their self-management. Even in the absence of an AI, individuals already log sufficient self-tracked data to suggest that an RL-enabled system could be effective in assisting endometriosis care. We advocate for solutions that offer computational support for both self-experimentation with strategies already identified (i.e., evaluating if strategies are effective) and for discovery (i.e., recommending new strategies). Such tools could empower individuals in their care, support them in developing effective individualized management regimens, and improve quality of life. And while HAI scholars warn against falling into the “solutionism trap” that frames ML as a simple solution to a difficult problem [ 107 , 134 ], our research affirms that ML may be an appropriate approach to this complex, unresolved problem. We also situate our work within a feminist research perspective [ 16 , 17 , 54 ], particularly recognizing that individuals coping with a burdensome women’s health disease like endometriosis must contend with issues of bodily autonomy, health equity, and society’s disregard for women’s health concerns [ 43 , 118 , 132 , 156 ]. The deeply personalized and embodied experiences of endometriosis are often minimized and neglected, underscoring the need for solutions that center the highly contextualized and varied symptomatic manifestations of the illness. We call for human-centered technologies that leverage self-tracking data to elevate the perspective of users, enabling complex, contextualized representations of illness that can facilitate solutions to support users in their care tasks. We also emphasize the importance of innovating methods to translate individual, nuanced experiences of illness to tractable representations that can be leveraged in intelligent systems to advance personal health goals. We provide an illustrative case of eliciting the needs of users within the constraints of a specific ML approach and its computational requirements and translating these into actionable recommendations for the design of an intelligent system. While this framework has been successful in answering the research question within the context of self-management for a particular complex chronic illness, the same approach could be adapted for other contexts that need to attend to human, data, and ML perspectives at the same time. We expect that MPDA will help to integrate human needs, challenges, and aspirations with ML and computational feasibility across a range of scenarios (i.e., other illness contexts, tasks, and ML paradigms), leading to design requirements that are not only desirable by end-users but also achievable. For example, if we were to follow the same process in a different disease where the set of actions was very well known, then individuals would not need to define their own strategies. Thus, aspects of the RL, including the action space and policy learning, could be population-based. For a different task, the high-level concepts would be operationalized differently. In a different ML model, the high-level concepts used as organizing principles would be different. Our work has illustrated that the framework can also lead to the foregrounding of human-centered principles in the design of AI-enabled systems, which is also generalizable to other contexts. In a different disease, we might find that control and autonomy would need to be augmented in very different ways than is the case in endometriosis, e.g., individual-level learning may not be needed but cultural factors may impact the options that users want to control. MPDA also proved valuable in documenting and elaborating the sociotechnical gaps [ 2 ] that arise in the context of designing intelligent systems for a complex human task like self-management, which could be extended to other tasks, diseases, and algorithms. At the same time, we also recognize limitations in this approach, for example the important aspects of safety, privacy, and trust were not elicited from end-users as design requirements.

In order to align design requirements of an intelligent system for supporting experimentation with self-management across human, data, and machine learning perspectives, we propose and implement a novel HAI framework to conduct a mixed-methods study. From the ML perspective (described in Section 4.1 ), we use high-level concepts from RL to conduct directed coding. From the human perspective (described in Section 4.2 ), we rely on the input of individuals with endometriosis to understand their needs and values when conducting self-management and receiving recommendations. The original high-level RL concepts form the basis for the directed content analysis of the qualitative data. From the data perspective (described in Section 4.3 ), we conduct analysis of “in the wild” self-tracking data from current users of the Phendo app, who log their experiences in their day-to-day lives (i.e., in the absence of any ML algorithm) to understand patterns of illness experiences and self-management behavior from real-world data. The organizing principles from RL, which are revised according to the empirical data from the qualitative analysis, provide structure to the quantitative analysis of usage logs. Through iterative and ongoing discussions amongst the authors, we triangulate findings from across these three different perspectives to arrive at a set of conclusions and design recommendations. An overview of the organizing principles and data sources is presented in Figure 2 . Our institutional review board approved all study procedures. This current study is situated within our broader research initiative focused on characterizing the enigmatic disease endometriosis and developing tools to support individuals in managing this complex chronic illness. This study leverages the Phendo app, as described in Section 2.2 — we analyze data that have been self-tracked with the app, re-analyze transcripts from prior focus groups, and conduct interviews with active Phendo users. We select high-level RL concepts (i.e., action space, state space, reward, agent/policy — identified by Sutton and Barto [ 139 ]) as the basis for this analysis. These concepts are presented in the first column of Figure 2 . In the context of this study, the action space represents the items available for the intelligent interactive system to recommend — self-management strategies. The state space represents the illness, contextual, and broader environmental variables that the agent will take into account for its recommendation. The reward is the function that will be used to evaluate the success of each recommended strategy. The agent/policy refers to the actual individualized self-management recommendations and behavior of the intelligent interactive system. A mapping between the RL concepts and the corresponding components of an intelligent system for self-management is presented in Figure 3 . A wide body of literature for RL policy learning exists [ 5 , 87 ], with recent work in the context of mobile health data [ 76 , 90 ]. Since the aim in the current study is to support individuals in developing a personalized self-management routine that might evolve over time, we focus our attention on learning individualized sequences of strategies that maximize users’ well-being as they interact with them (i.e., online learning). At the same time, we can leverage available historical data (i.e., offline data) for the design and modeling of the algorithm, inform its value and policy functions, and to avoid the cold-start problem. An intelligent system that will be used and useful to individuals with endometriosis will have to meet their needs, operate acceptably and within expectations, and fit into their lives and self-management routines. To ensure these requirements are met, a key step undertaken in this study is to map the RL concepts and intervention components to the real-world aspects of human end-users. The design items to consider are: the scope of the action space; how to represent the state space and context; how to model the reward signal; how to balance exploitation vs exploration; what are the decision points (e.g., every hour vs daily opportunities) and decision rules; how much time can users dedicate for each recommendation; how often (dose schedule) and for how long (study duration) are users willing to experiment with the automated recommendation system. We answer these questions based on input from Phendo users and already available Phendo data. Below, we illustrate how our framework can use these RL-specific constructs to structure both qualitative analysis of user interviews and quantitative analysis of Phendo usage logs. For the qualitative analysis, we analyze transcripts from discussions with people with endometriosis. We use directed content analysis (as described by Hsieh [ 75 ]) based around the high-level RL concepts described in the previous section to code, understand, and organize the data into meaningful findings. The way that the RL concepts are applied in this part of the study is depicted in the second column of Figure 2 . Note that data from the quantitative analysis also represent the real-world experiences of users, but are discussed separately within the data perspective in the next section. We re-analyze focus group transcripts from prior studies and conduct additional interviews with Phendo users with follow-up questions specifically relating to a tool for self-management. In total, we re-analyze 10 transcripts from focus groups with 48 participants (all women with a formal diagnosis of endometriosis) from prior work and data from three follow-up interviews. Focus groups from 2016 (5 groups, n = 27) originally informed the design of the Phendo app. Focus groups from 2019 (5 groups, n = 21) were initially used to elicit design needs for tools to support care and self-management of endometriosis. This secondary analysis of focus group transcripts provides a valuable source of information about Phendo users’ management practices, how people with endometriosis select and evaluate strategies, and where participants have unmet care needs that technology could support. But while these focus group transcripts do directly address self-management practices and technology needs, the discussions do not specifically address an interactive system for experimenting with self-management strategies. To fill this gap and strengthen the evidence collected from the secondary analysis, we conduct in-depth follow-up interviews (n = 3) to ask potential users directly about an interactive system for developing individualized regimens. Interviews probe about participants’ experiences of self-management, the process by which participants have developed their own regimens, and what opportunities and constraints participants foresee in an automated system that provides intelligent recommendations that could support their management. From these data sources, we are able to directly and indirectly elicit constraints and boundaries of an interactive system for self-management. Additional details on these methods, including a description of the study sample and topics covered in the interviews and focus groups, can be found in Appendix A.1 . The qualitative analysis focuses on predetermined high-level RL concepts in order to guide the analysis towards evidence that answers our specific research question. Our inquiry seeks to identify technical requirements and parameters for an intelligent interactive system for self-management, to figure out a range of what users want and are willing to do when using the system, to outline constraints, and to assess potential feasibility of such a system. We use directed content analysis to code the data, starting from the four high-level RL concepts as predetermined codes (i.e., action space, state space, reward, agent/policy ), then refine the codes during analysis based on empirical data — the initial codes and final coding categories used as the framework for the study are presented in Table 1 . During the analysis, the authors worked together to review, refine, and name the categories and extract key examples to present. Given the reflexive and collaborative nature of the analysis methods, we focused more on exploration of themes and consensus building rather than consistency in coding [ 29 ]. The codes generated through the analysis are used solely for conceptual purposes, serving to establish foundations for an RL model rather than for direct use in RL training. Future work developing the RL agent may employ an MPDA-based approach, where coding consistency will be critical. For the quantitative analysis, we leverage data collected from the Phendo app to capture the illness experiences and self-management behaviors of users “in the wild,” i.e., without any intervention or recommendation, how individuals track their illness and what strategies they rely on for self-management. The Phendo data and analyses used for each of the RL concepts can be found in the final column of Figure 2 . Phendo users self-track a variety of details about their illness (see Table 2 ). We identify two broad domains of questions from the Phendo app that are relevant for this analysis: those related to the state space, or the personal experience of illness (colored in taupe in Table 2 ), and those related to the action space, or self-management (colored pink in Table 2 ). For each of the customizable self-management strategies (e.g., exercise, food), we manually map and normalize all responses to a smaller, coherent set of pre-determined strategies in order to reduce the number of strategies in the dataset and to facilitate a population- vs individual-level analysis. E.g., the user-entered exercises ‘walk,’ ‘hiking,’ and ‘walking dog’ are all harmonized to a ‘walking’ entry, and the curated ‘strength’ strategy includes responses from customized answers like ‘crunches,’ ‘sit ups,’ ‘core exercises,’ ‘squats,’ ‘planks,’ ‘push ups,’ and ‘weightlifting.’ Since this study focuses on self-management, we extract all data for users who have self-tracked at least one of the Phendo self-management questions marked with an asterisk (*) in Table 2 . For each participant, we collect their longitudinal self-tracking data as entered in the Phendo app, so that we can quantify both their self-management habits and information on their health status through time. The dataset contains information from 10,463 users, with n = 399,786 self-tracked responses, of which n = 290,503 are self-management instances. The quantitative analysis was conducted in alignment with the pre-selected coding categories, and iteratively revised alongside the qualitative analysis, as the categories were revised inductively based on empirical data. For the action space , we analyze the answers to the self-management Phendo questions detailed in Table 2 to characterize the breadth of strategies users experiment with, as well as for how long and how often do they engage with each strategy . Similarly, for the state space , we analyze the answers to the questions related to the personal experience of illness, detailed in Table 2 . For the analysis related to the reward , we define simple goals related to pain, GI symptoms, and other symptoms. We quantify how frequently users track this information before and after engaging in self-management activities. An example is highlighted in green in Table 2 . For the analysis related to the agent/policy concepts of RL, we quantify the effects of strategies through an analysis of self-management trials, and assess engagement based on how many trials users log “in the wild.” We define a self-management trial as an event where a user tracks a self-management strategy, as well as goal information a day before and after the tracked strategy. That is, a trial consists of a triad of pre-management goal information, a self-management instance, and post-management goal information, all contained within a day of the self-tracked strategy. We then quantify effect estimates of each strategy on symptom severity: i.e., we compute the difference between the post- and pre-strategy goal response for each trial. E.g., a self-management pre-post negative pain-severity effect estimate indicates the strategy reduces pain, while a positive pain-severity effect showcases worsening of the experienced outcomes of the self-management strategy. For a given set of per-strategy trial effects, we compute effect estimates both at the population level (by averaging per-strategy effects) and for individuals (by averaging per-individual longitudinal per-strategy effect). An example is shown in blue in Table 2 . In investigating if an RL policy is feasible for this task, this component of the analysis can help us understand if data tracking patterns “in the wild” will align with the ML and human requirements and constraints identified in the qualitative analysis.

The Human-Computer Interaction (HCI) community has established a rich body of literature on developing intelligent systems and tools to use personal health data to support individuals in their health-related goals. Personal informatics examines solutions that focus on facilitating individuals’ engagement with personal data. Li et al [ 88 ] proposed a stage-based model of personal informatics that outlines various stages of user engagement with their personal data. Many personal informatics solutions have focused on the presentation of personal data to support reflection and improve self-knowledge [ 28 , 37 , 82 , 103 ]. These systems have also been augmented with ML to better support reflection and action [ 94 , 104 ]. Other research has explored the use of ML in sociotechnical systems, for example in clinical medicine [ 124 ] and mental health [ 33 , 144 ]. One common approach to supporting increased self-knowledge using personal health data is through self-experimentation. Traditional personal informatics tools, even without the use of AI or statistical analysis, can help individuals identify trends and patterns in their health data to support personal discovery and behavior change [ 15 , 86 , 128 ]. Personal informatics interventions have also been developed specifically for this purpose. These systems are frequently designed to facilitate conducting n-of-1 trials (also called single case designs), where users act as their own controls, to highlight an individual’s response to a treatment rather than a group’s [ 44 , 81 ]. While many of these systems have shown promise in supporting experimentation, they have also been constrained by rigid options and limited personalization [ 45 , 46 , 80 , 86 , 143 ]. But users with complex health conditions are still in need of personalized, customizable solutions to support self-experimentation towards developing effective long-term regimens. Some research has explored the use of advanced computational frameworks that use ML to develop more flexible, person-centered systems for self-experimentation [ 47 , 129 ]. Other research has investigated automated, individualized intervention approaches such as adaptive treatment strategies [ 8 , 104 ], micro-randomized trials (MRTs) [ 83 ], and just-in-time adaptive interventions (JITAIs) [ 110 ], for a variety of health-related outcomes [ 41 ]. These tools exist in domains that have more well-defined parameters and outcomes, but could provide guidance in the context of complex chronic illness self-management. Some of these existing approaches to adaptive intervention fit into the RL paradigm [ 139 , 140 ]: a computational approach to understanding, automating, and optimizing a sequenced set of actions that maximize an outcome of interest. An RL agent learns which intervention is best to suggest from a pre-determined (continuous or discrete) set of actions , given a state , towards maximizing a total reward . JITAIs adjust the type, timing, and framing of support provided based on a user’s dynamic internal and contextual state, aimed at maximizing the positive impact of an intervention. JITAIs, which sometimes use RL approaches to learn the best intervention for users, result in (expert or learned) decision rules that map an individual’s current state to a particular intervention or treatment at each decision time point. Therefore, an RL approach to an intelligent system for supporting the trial-and-error process in self-management of a complex chronic condition is a promising solution to investigate. RL agents make sequential decisions as they interact with the environment, i.e., as the world changes, toward achievement of a specified goal. An RL agent learns how to update its recommendation policy from previous actions and evaluated rewards, based on the variables it observes. The policy dictates the behavior of the agent (system) and uses observations to map from the state space to the action space when in particular observed states. The state, i.e., the information used for individualization, helps decide when and/or how to intervene with particular actions. A reward signal defines the goal of the RL problem, which the agent maximizes, while the value function (which the RL agent updates as it interacts with the environment) quantifies the long-term desirability of states (after taking into account likely subsequent states and corresponding rewards). RL offers potential benefits that make it a promising candidate for providing automated, adaptive recommendations for individualized self-management in an illness context with a lot of complexity and uncertainty. Since it uses the results of each iteration to update the policy and inform future algorithmic decisions, it is particularly well-suited to help users self-experiment with self-management strategies, when it is not known beforehand what will be helpful, when, or for whom. In short, RL engages algorithmically in a trial-and-error process similar to the goal-directed, sequential learning that individuals follow when working to develop their own individualized self-management regimens [ 92 ]. RL can handle the dynamics of sequential interactions between the user and a system by utilizing feedback from the environment to adjust its actions; RL can account for long-term user engagement with a system; and RL can optimize a policy by sequentially interacting with the environment without requiring explicit user input [ 4 ]. In the context of endometriosis, RL-based recommendations offer several advantages. The goal of an RL agent is not only to optimize interactions with the environment, but also to learn insights that allow individualized interventions. Since RL allows for individual-level optimization, rather than focusing on population-level estimates, given that endometriosis shows strong person-to-person variation in symptoms and treatment responses, person-level personalized self-management recommendation is needed. RL is a sequential decision-making algorithm that can leverage multiple correlated time points during learning, making it suitable for learning with correlated longitudinal data. Since RL is able to provide sequential, adaptable, and individualized recommendations, it could be set up to conduct n-of-1 trials to facilitate self-experimentation at the individual-level — and helpful for other chronic diseases beyond endometriosis. RL has shown exceptional performance in multiple scenarios [ 106 , 137 ]. However, despite the previous successes of this approach and numerous benefits, there are also critical unresolved impracticalities in the context of this study: ( i ) successful examples have been limited to highly structured, controlled, and well-defined environments, i.e., they do not translate easily to practical, yet more complex situations; ( ii ) the state-of-the-art, deep-learning based RL techniques are data-hungry, i.e., they require inordinately large, repeated interactions with the environment to learn effective policies; and ( iii ) they are black-box models, i.e., it is inherently difficult to interpret the complex features learned by these models and why recommendations are made. The real-world task of figuring out what self-management strategies might work for an individual with an illness like endometriosis is an inherently interactive process that calls for a sequential decision-making process. Thus, it is possible to formulate the problem as a Markov decision process (MDP) and solve it with RL [ 135 , 163 ]. The fundamental trade-off between exploitation (optimizing recommendations) and exploration (learning insights about the environment it is interacting with) is critical for efficiently updating the RL policies that decide which action to recommend next. To support individuals in developing a personalized management routine, an RL agent must learn a policy that can prescribe, based on observed context (i.e., user state), appropriate self-management strategies (i.e., actions at each interaction) that best improve an individual’s health status (i.e., the reward signal). In real-world situations, especially when the RL directly interacts with and learns from humans’ actions, there are multiple challenges to consider in designing an RL agent [ 70 ]. On one hand, the operational success of an RL policy is constrained by the size of the data it can learn from, the type of information it has access to, the number of interactions it can leverage, and the type of actions it can recommend. On the other, users of the system have their own preferences (e.g., the framing and tone of a recommendation) and values (e.g., preserving agency in their self-management activities). Misalignment between human and machine can create unhelpful recommendations (e.g., a 20-minute jump rope session for an individual with chronic pelvic pain) and as a consequence, at best break trust and at worst result in harm. Hence, understanding user-centered requirements for RL and its different components is critical to designing human-centered RL systems. In addition, an intelligent system that uses RL may ease the cognitive burden put on individuals to find their optimal self-management regimen. At the same time, this type of system requires delivery as a mobile intervention, which must incorporate human feedback. Delivery via a mobile self-tracking app requires the RL system to be able to determine when and how to best deliver the intervention components at different decision points. An interactive solution that incorporates human feedback in the learning process is necessary [ 12 ].

We integrate results from the qualitative and quantitative analysis, with the RL concepts as the guiding framework. In synthesizing these findings, we identify promising circumstances where an intelligent system for self-experimentation with self-management could be successfully deployed; but we also find barriers and challenges that will need to be resolved. We present eight findings across the four RL categories ( Figure 4 ): 1. Action Space — (1a) The space of actions/self-management activities individuals experiment with is large; (1b) Individuals are willing to explore the action space as long as they can set some boundaries; 2. State Space — (2a) The state space, that determines which actions to recommend, is complex and can vary across individuals; (2b) Individuals favor recommendations that are responsive to the user’s current context; 3. Reward — (3) Individuals assess the success of a strategy in the short term; 4 . Agent/Policy — (4a) Success of specific strategies varies across the population, rendering the need for tailored recommendations from individualized policies; (4b) Individuals want to understand why they received recommendations and why they should try recommended strategies; and (4c) “In the wild” user engagement with self-management and self-tracking is frequent enough to satisfy RL data requirements. Illustrative quotes are included with each of the results (where the interviews are labeled “Int” and the focus groups are labeled “FG”), and Recommendations for design are included in Table 3 at the end of the section. In the context of chronic disease self-management, the action space corresponds to the space of possible self-management actions that an individual can take. As such, we code all participant descriptions of self-management approaches under this category and analyze self-tracking data related to self-management. Participants detail a multitude of different self-management strategies and self-care approaches for coping with symptoms and for promoting health more broadly. Their regimens incorporate strategies that are included in the Phendo self-tracking vocabulary (see Table 2 ), both pre-set (e.g., “In the middle of the pain, just give me my heating pad and my drugs.” [FG 2]) and user-customized, such as exercise and diet (e.g., “For me — no dairy, no gluten, very low sugar. Minimize raw food intake — mostly cooked vegetables, dried fruits. Fried foods. Preservatives. Processed foods. So, an endo-friendly diet.” [Int 3]). Participants also describe strategies and self-care tactics beyond what is included in Phendo, for example: “I’ve found hobbies that I can do with two hands help me feel better. I crochet, I am kind of a plant lady, I have a garden.” [FG 7]. Strategies like these could be added into Phendo as user-customized vocabulary. Participants report using various strategies in combination and as needed, for example: “I like to take a holistic approach — I put marijuana in tea and I drink it, sometimes I use the CBD pen. I have ginger and turmeric tea on a regular basis. And I try to exercise at least three times a week.” [FG 7]. From existing “in the wild” data, we corroborate these findings: current Phendo users track a wide range of strategies across the population and individual users often track several in combination. First, we showcase in Figure 5 the count of users that have self-tracked at least 10 instances of each strategy within the self-management Phendo questions. These results align with the participant reports of the wide variety of strategies they use (and are willing to use). Second, we illustrate in Figure 6 the count of users that self-track strategies in combination. We notice that, although a lot of Phendo users focus on one or two distinct self-management strategies, there are many Phendo users already tracking regimens in the app that combine several strategies together. From the ML perspective, it is beneficial that users engage with a range of strategies, yet an RL algorithm will benefit from a constrained action space to effectively learn meaningful policies. Design decisions will need to be made in order to satisfy both human user requirements and the RL requirements. Taken together, the data from users and self-tracking data suggests that there is a broad range of strategies used at the population level, but individuals are likely to only experiment with a few strategies at a time. The second code within the category of the action space relates to users’ willingness to explore different strategy recommendations. Here, we code discussions about what participants are willing to try and limits and barriers that constrain what recommendations they are open to. Participants frequently discuss turning to exploration and trial- and-error self-experiments, since individuals managing endometriosis face a lot of uncertainty in care and are often left without effective treatments. Participants tell us they are interested in trying a broad range of self-management options and are willing to try strategies that don’t necessarily make sense to them. When one interview participant was asked about trying strategies the intelligent system might recommend, she reports: “I’ll always give the new thing a try and put the old thing on hold.” [Int 3]. Further, while users are willing to explore a wide variety of strategies and try suggestions that they may not otherwise, they do want an intelligent system that becomes more tailored to them and personalized over time (shifting from explore to exploit in the RL framework). As this same participant explains: “In the beginning I’d be open to explore all options and see what there is to offer. And then, throughout usage it becomes more fine-tailored to me, I think that would be ideal.” [Int 3]. Participants warn that adding too many strategies to someone’s care routine may become overwhelming, so generally limit experiments to a single strategy. At the same time, they describe benefits to engaging in multiple strategies that target different symptoms (e.g., one food strategy, one physical activity strategy, and one pain management strategy for flares). As one participant explains: One thing, because that’s all I can handle. When I’m trying a new thing, I try to just ‘suss’ that out for a while just to see if it’s working. But at the same time, not like two things to cut out or add to my diet, but maybe something with diet and something with activity or movement — a combination of things. While participants are open to exploring different self-management strategies, which is beneficial to the learning process of RL, they are not willing to attempt very exhausting or taxing activities (e.g., “I do think that there’s a healthy limit to pushing yourself a little bit. But anything that causes me more pain is never going to be helpful.” [Int 1]). While participants describe a range of barriers that limit the options that are feasible for them, cost, logistics, and emotional challenges are the most often mentioned. Participants see value in setting hard limits ahead of time about what strategies are recommended. At the same time, they also imagine advantages to receiving recommendations with a variety of novel strategies that they otherwise might not try and to give feedback to the system in real-time as recommendations are given. For example, one interview participant explains: Based on my previous experience with that type of activity, I already know it doesn’t work for me. […] I would want to tell the app, ‘Please don’t ever recommend that to me again.’ But, I would say not setting all of the limits at the beginning, because there is something about ‘pushing yourself’ a little bit and trying something a little bit new. […] So I actually think it would be helpful to get the things, but there are going to be some things for certain women, that are just going to be a no-go. [Int 1] Participants imagine giving input or feedback to the system about what they want to try (or not try), for example, interview participants asked about the intelligent system propose features that could provide options alongside recommendations to see if users are willing to try a strategy, such as: “yes,” “not now,” “not ever.” They also suggest offering different options of strategies across difficulty level (from gentle to rigorous) to choose from when presented with self-management recommendations. These suggestions from participants could enhance user control and integrate human intuition into the decision-making process, while enabling efficient RL learning with constrained action spaces. Based on my previous experience with that type of activity, I already know it doesn’t work for me. […] I would want to tell the app, ‘Please don’t ever recommend that to me again.’ But, I would say not setting all of the limits at the beginning, because there is something about ‘pushing yourself’ a little bit and trying something a little bit new. […] So I actually think it would be helpful to get the things, but there are going to be some things for certain women, that are just going to be a no-go. In the case of chronic disease self-management, the state space corresponds to the ways that an individual’s illness state, current context, and the broader environment are represented. In this category, we code all descriptions of how individuals assess their health status and the context and environmental factors that impact their engagement in self-management and analyze self-tracking data related to illness experience and context. Participants describe rich, nuanced accounts of illness and discuss highly personalized and often embodied ways that they characterize and assess changes in their health status. In detailing the ways that they understand their own disease, participants emphasize the importance of holistic and individualized representations of their illness experiences and talk about factors that are not always deemed clinically relevant. Pain and fatigue are the most frequently discussed indicators, but GI symptoms (e.g., cramping, diarrhea, nausea, bloating), breakthrough bleeding or spotting, skin issues (pimples/acne, rashes, hives), migraines, emotions (e.g., “mood swings,” depression, anxiety), manifestations of pain (pain-somnia, pain-aggression), and other personal signs of inflammation (e.g., in the face or belly) are seen as necessary to provide a holistic and individualized picture of a person’s health status. Participants also distinguish between dealing with typical, day-to-day endometriosis symptoms and the flare-ups that some individuals experience, as one focus group participant explains: “When it comes to endo, you have the pain aspect but then you also have the flare-up aspect and it kind of goes hand in hand and sometimes it doesn’t go hand in hand.” [FG 2]. Some participants report on indicators that suggest they are in a flare or a flare is coming. Participants recount their unique manifestations of “bad days” when they “feel terrible” or “horrible” (even though they often “don’t look sick”). They talk about being “incapacitated” in bed because of pain (sometimes tied to their menstrual cycle, ovulation, or menses) and feeling so ill they can’t move, are exhausted, or “collapse” when they get home. They talk about knowing their bodies, so they can know when something is not right ( “You just have to know your body, and you’d be like, ‘Okay, well, something’s off. How can I help it?’ ” [FG 5]). The analysis of Phendo data corroborates, as shown in Figure 7 , that current users self-track the experience of illness via a multitude of dimensions that are available in the app — the list of available Phendo questions directly related to the individual’s disease experience is found in Table 2 . These variables align with the description of what users consider important to represent their illness experiences to facilitate holistic state space representations and that can be used to modulate the recommendations from an intelligent system. The second code within the category of the state space relates to the relevant context features where users are open to receiving different recommendations. As such, in this category, we code participant discussions about what details they consider in deciding on self-management strategies to use and analyze contextual data tracked immediately before self-management strategies are logged. Users describe wanting an intelligent system to consider contextual details related to their illness, their environment, and their day-to-day lives when tailoring strategies to recommend — many of which may be subjective, esoteric, and/or highly personalized. Users request for recommendations to be responsive to their current context, as one interview participant explains: I track emotional moments when things arise for me. So if the Phendo app could know, ‘OK she’s been logging these things, she’s been anxious, or there’s fatigue.’ Then now is not a good time for the app to recommend training for the 5k. But if I log those things and the app suggests meditating today for 30 minutes, or ‘Why don’t you journal?’ — it would be useful if the recommendations coincide with whatever symptoms I’m tracking. [Int 3] On top of this request for context-awareness, users also want strategies to tailor for particular symptoms and for daily management compared to a debilitating flare. I track emotional moments when things arise for me. So if the Phendo app could know, ‘OK she’s been logging these things, she’s been anxious, or there’s fatigue.’ Then now is not a good time for the app to recommend training for the 5k. But if I log those things and the app suggests meditating today for 30 minutes, or ‘Why don’t you journal?’ — it would be useful if the recommendations coincide with whatever symptoms I’m tracking. Participants discuss fitting strategies into their daily lives, and mention wanting a system to consider their schedules and daily routines and fit strategies into them in a way that makes sense. Participants also see value in a system that could push recommendations for strategies that would be helpful in the moment. Some users are open to pre-scheduled time set aside to do the self-management strategies offered by the system, but users desire a system that can tailor strategies to the particular context of situations as they arise. As one interview participant describes: There could be benefits to both. I like the push of, ‘Hey stretch right now.’ However I typically implement those types of things when my threshold for pain has gotten so high to the point where I can’t think, and that’s where I would step away and do that, as opposed to just getting home later tonight and stretching. [Int 1] Participants imagine personalized recommendations would be responsive to a user’s state, including health status and more broadly how are they feeling. But context about the day-to-day activities of users and their environments is currently infeasible, difficult, or impossible to capture (e.g., “What headspace am I in when that is suggested?” [Int 3]). Nonetheless, potential users want a system to take these aspects of their day-to-day lives into account. There could be benefits to both. I like the push of, ‘Hey stretch right now.’ However I typically implement those types of things when my threshold for pain has gotten so high to the point where I can’t think, and that’s where I would step away and do that, as opposed to just getting home later tonight and stretching. We corroborate in Figure 8 that there are many existing instances of the self-tracked illness experiences discussed as important in Section 5.3 , which are tracked by users within a day before they track a self-management strategy, suggesting that the contextual data currently tracked by users in the app can be useful for tailoring recommended actions. We observe that Phendo users are very likely to report information about their experiences with activities of daily living, as well as with information on symptoms they are experiencing the day before and the day after they self-manage. Therefore, there are already existing Phendo answers that can be used to compute a meaningful state space for an intelligent system and facilitate contextually-tailored recommendations, although we would still miss things that are not currently tracked or cannot be tracked (e.g., “current headspace,” “timing is not working out today,” or “pain so bad can’t think”). In addition, there are sparsity and heterogeneity challenges that are raised by the “in the wild” data analysis with current Phendo data: i.e., not all users track the same set of experiences, and they do not track them consistently for every self-management strategy they try. In an intelligent system for chronic disease self-management, the reward corresponds to the signal that will be optimized when suggesting actions and how the success of actions tried will be evaluated. As such, we code participant descriptions of how individuals evaluate if self-management strategies are working under this category and use simple self-tracked data to evaluate the effect of strategies tracked “in the wild.” In endometriosis, symptoms often persist and sometimes progress, since there is no cure, which means that getting rid of symptoms altogether is often impossible. So, people with endometriosis report having to rely on assessing how they are feeling in the moment and trends or deviations from their individualized baseline to figure out if strategies are effective or not. When asked about assessing the success of self-management strategies and evaluating the progress of their health goals, individuals largely describe short-term symptom or health-related quality of life indicators. While patients often have longer-term goals for their care and management, these are not what participants report relying on for developing their care regimens. Participants talk about having difficulty figuring out if strategies are working and tell us that they generally rely on checking in with how they feel in the moment ( “How do I know something is working? I’m not great at figuring that out all the time. […] For me, if the pain goes away in the moment, then I will stick with that.” [FG 6]). They report relying on mind and body pain and sensations, immediately or within a short time-frame. Even still, users are not looking to eliminate pain and symptoms (which is often not possible), and instead focus on evaluating if strategies are helping to improve their symptoms ( “That’s the key word, ‘Not as bad.’ So, better.” [FG 1]). When evaluating if self-management strategies are working, participants all talk about assessing their pain symptoms; GI symptoms are also frequently used as an indicator. Participants also describe clues or indicators that are specific to them, which they use to help them determine if they are feeling better or worse (e.g., skin rashes, acne breakouts, and the ability to sit in a chair without high pain). In order to measure the success of self-management strategies in a data-driven fashion, we need to determine the effect signal for each individual: i.e., we need to define the reward function that an intelligent system will use as feedback. When looking at existing Phendo data, we observe that pain, GI, and other symptomatic experiences are frequently self-tracked by current Phendo users, both before and after self-tracked self-management strategies. More specifically, in Figure 9 we showcase the abundance of Phendo self-management trials for which there is pain-related pre-post information. These results suggest that Phendo users track sufficient information related to their self-management goals (pain results are shown here, but similar stories hold for GI and other symptoms) for the design of a meaningful reward function. We have used simple, human-driven functions in this analysis, illustrating that an RL algorithm could have sufficient reward signal to detect an effect, based only on “in the wild” data. The reward function is critical for any RL agent, as it provides the signal from which to learn how to find the right exploration-exploitation tradeoff. Although the simple functions suggest sufficient data and effects exist to learn RL policies, the user perspective emphasizes that we will need to expand on these reward functions to represent individualized metrics. In the context of chronic disease self-management, the agent/policy dictates the behavior of the intelligent system. As such, we code participant discussions about developing their own self-management regimens under this category and analyze engagement and existing self-management trials with self-tracking data. This category has been separated into three codes during analysis; conversations discussing the personalized nature of self-management regimens are coded here. Users are enthusiastic about an intelligent system that could customize personalized recommendations based on their prior experiences, rather than suggesting strategies because it worked for someone else who is “similar” to them. They describe their desire for recommendations to be tailored to them based on their own data ( “Based on my stuff I logged — the experiences that I’ve had so far.” [FG 2]), since each person is different. Phendo users emphasize that a personalized approach to self-management will be required to meet their needs, particularly since they are heterogeneous in which strategies individuals use for different symptoms and health states. Across the wide array of self-management strategies, individuals turn to different activities to address particular symptoms. Some examples of strategies users enact based on various symptoms include: You can pretty much always do deep breathing. I don’t care how much pain you are in, as a default. I’ll read and write a lot when I’m having a bad day, because it’s a way to escape and be in a space that’s more enjoyable. When I’m having a good day, I’m more of a physical person. I’ll be like, ‘Oh, I’d love to go kayaking,’ or ‘Oh, let me go work in my garden.’ I went to acupuncture when I was having really severe pain before my surgery. Cooking has been helpful to me. I have a task in front of me, I can get into a flow, and it is also good because it makes me feel like I’m in control of what I’m eating, what I’m creating. I can just close my eyes, deep breath and be distracted by something else. When I’m ovulating and menstruating, soups and broths help for that. Aligned with participant perspectives, we find that current Phendo users’ responses to the same management strategy is heterogeneous, both at the population and individual levels. We showcase in Figure 10 walking to pain severity self-management pre-post effects for the population of Phendo users and specific individuals: i.e., a negatively skewed pre-post pain severity effect histogram implies that walking mostly reduces pain, while a positively skewed histogram showcases worsening of the experienced pain after walking. In Figure 10a , we observe how, although the effect of walking is null for a vast majority of Phendo users (notice the spike near the histogram origin), many users report both positive and negative effects in their pain severity within a day window. When looking at the individual (i.e., n-of-1) effects, we observe that the effect of the same self-management strategy (e.g., walking) is wildly heterogeneous for different users in the Phendo cohort. We illustrate the wide range of pain severity reported effects within a day of walking in Figure 10b – Figure 10e , where we observe how walking clearly helps reduce pain for the individual depicted in Figure 10e , but hurts the individual in Figure 10b ; the same strategy does not have any impact for the individual in Figure 10c , while it is unclear on the effect for the individual in Figure 10d . We also corroborate the heterogeneity in population and individual effects for different strategies, as well as different goals (pain, GI, other symptoms) at the population level. This finding justifies the need for fully personalized self-management recommendation policies: i.e., one strategy does not suit all individuals with endometriosis. The second code within the agent/policy category relates to the desire for explainability of recommendations. We code discussions about information participants want alongside recommendations in this category. Participants are interested in the reasons for trying a particular strategy and want to set their expectations about how long it may take to see an improvement in symptoms. All three participants interviewed about the intelligent system specifically tell us that they want explanations for recommendations provided by an intelligent system. They explain that they want to know why a strategy is being recommended, especially when the agent suggests strategies that have not yet been effective for an individual. Explanations would ideally include information about why the recommendation was chosen or tailored for the symptom or context. Participants suggest that explanations for what is offered and why would be helpful to users, especially when exploring options that may not make sense to the user or if the suggestions conflict with existing strategies that work. One person suggests that explanations might help motivate them to try the suggested strategies: “I would be more prone to try it if there was something behind it.” [Int 2]. These participant perspectives indicate that explanations could also help users build trust with the system. Participants are interested in implementing strategies that are responsive to their contextual environment, and report that insight from an intelligent system could help their understanding and exploration. One interview participant suggests that she be able to inspect and explore self-management options that are personally tailored for her, so that she may use the system’s insights along with her own to decide on an appropriate action that is “connected to how [I’m] actually feeling” [Int 1]. The final code within the category of the agent/policy relates to users’ willingness to engage with self-management and an intelligent system to support them. We code conversations about engaging in self-management and interacting with an intelligent system in this category. Users tell us that they are willing to extensively self-track their illness experiences and understand that an intelligent system for self-management would require active engagement. Participants are not concerned about the burden of using such a system (e.g., “Logging observations is not cumbersome to me.” [Int 1]). The analysis of current Phendo users’ self-tracking engagement confirms that the frequency at which they track different self-management strategies is high — we find that many users track self-management strategies with associated goals many times within their timeline. In Figure 11 , we observe that there are many Phendo users who have tracked up to 10 trials of pain to various strategies: walking ( Figure 11a ), carbs, grains or gluten based foods ( Figure 11b ), or talk-therapy ( Figure 11c ). In Figure 12 , we also observe regular engagement: self-management trials occur as often as every day or every other day (see Figure 12a and Figure 12b ), but also periodically, as in weekly or bi-weekly for talk-therapy to pain trials (observe the spikes at 8 and 15 days in Figure 12c ). The observed “in the wild” number of interactions and frequency provides evidence for the feasibility of an RL-enabled system and the acceptability to users, but the specifics for programming an RL algorithm in practice will need to be determined. Participants tell us that self-management is already part of their day-to-day routines and that they already dedicate significant time to these tasks, so committing to the self-experiments recommended by the system is seen to fit into their existing care regimens. Interview participants tell us they could easily incorporate 10–30 minutes into their daily routines, but also mention they would be open to recommendations that take one, two, or even three hours if those strategies might help their symptoms. However, participants explain that some strategies could be carried out more frequently than others. Participants generally agree that they try self-management strategies for about a month before deciding if they work or not, although sometimes they can tell sooner, even immediately. At the longer end of the scale, participants report willingness to experiment with strategies for several months or even up to a year to develop a regimen that works (e.g., “Sometimes it was 60 days, sometimes it was a whole year.” [FG 9]). Further, participants explain that self-management will likely remain part of their ongoing care plan. While some users may use an intelligent system for a short time or on-and-off (e.g., during a flare-up) and then set it aside until the next episode where they need support (e.g., “I would probably just move on until I have another acute pain episode, then I would be likely to use it again. I think, you try a couple of things. Maybe you learned something helpful.” [Int 1]), others envision using a system to support individualized self-management long-term (e.g., “That’s something I’m going to do the rest of my life, I’ve committed to that. […] So, ongoing for sure. I don’t think I’ll ever reach my peak where I don’t need this anymore, no.” [Int 3]). These levels of engagement, while not guaranteed to result in a feasible RL algorithm, are promising from both the human and data perspectives. Here, we lay out guidance for designing an intelligent interactive system that aligns with the human-data-machine insights and requirements elicited in this study. These recommendations address tradeoffs with the capabilities and constraints elaborated in this study and provide a concrete mechanism to convert the ideals articulated in the findings into real-world design decisions. The recommendations presented here provide a workable starting point for the design of an RL-based intelligent system for self-management of endometriosis. They also help to identify places where the sociotechnical gap [ 2 ] might be exacerbated by the use of ML or by the complex illness context, which provides guidance for future work and needs for innovation. These recommendations are presented in Table 3 .

We investigate the specific research question of this study in the context of endometriosis, where the need for individualized recommendations is particularly acute. Endometriosis is a chronic condition defined by the presence of a tissue similar to uterine endometrium located in physiologically inappropriate body locations, leading to chronic, cyclic, and persistent or progressive symptoms [ 6 , 164 ]. These symptoms (e.g., pain, gastro-urinary symptoms, and fatigue) can be debilitating and impact activities of daily living [ 126 ]. Diagnosis and treatment remain problematic since the disease is poorly understood scientifically, and while surgery to remove diseased tissue may alleviate symptoms, no cure exists and patients actively engage in ongoing care and management [ 1 , 26 , 68 , 117 ]. Endometriosis is an illness that has many uncertainties, and wide heterogeneity in disease presentation and in response to treatment and self-management [ 148 ]. The Phendo app [ 60 , 61 ] is a mobile research app that was developed in partnership with endometriosis patients to capture the real-world experience of the disease by allowing users to catalog the day-to-day signs and symptoms, self-management activities, and other lived experiences of endometriosis outside of the clinic. A broader goal is to leverage citizen science to facilitate ML from the collected data to discover new insights about the disease and support individuals in the care and management of their illness. A series of prior studies explored users’ motivations for tracking and important dimensions to track, via interviews and focus groups [ 99 ], and further elicited variables (e.g., symptoms, self-management strategies) that people with endometriosis find important via online surveys and content analysis of an online endometriosis community [ 98 ]. This self-tracking app was designed alongside end-users — individuals with endometriosis — so that the collected data reflect the illness experience of individuals from the patient perspective. Users of the Phendo app self-track a wide range of information about the different dimensions of their illness — screenshots from the Phendo app are shown in Figure 1 . Table 2 provides the full list of Phendo questions, along with the vocabulary type (pre-set, customized, or free-text) of the available answers. The community of Phendo users is active and engaged with self-tracking activities that provide a rich day-to-day picture of the disease [ 64 ]. Our analysis of Phendo app usage patterns showed that long-term users of the app are more likely than short-term users to self-track their self-management activities [ 62 ]. Prior work with Phendo data has focused on characterizing the enigmatic illness — an analysis of self-tracked Phendo data has helped to augment what is known about the disease and fill gaps in the medical literature [ 148 ]. Other analyses have detailed specific experiences of endometriosis self-management, in particular, the impacts of physical activity [ 63 ]. More recent work has sought to understand the needs of individuals and their providers in supporting care and management with personal informatics tools. In a study that we conducted to elicit the needs of patients and providers in caring for endometriosis, all stakeholders conveyed a dire need for support in self-management of this complex condition [ 120 ]. Because of the enigmatic and chronic nature of endometriosis, treatments are often ineffective and management regimens are highly personalized, commonly requiring lengthy and involved trial-and-error with various strategies. While patients emphasize that they want low-impact, restorative self-management strategies to help them live with and ameliorate symptoms, they are not well-supported in their experimentation over time and have difficulty identifying strategies, sticking with them, and evaluating if they are working. Participants imagined features of intelligent systems that could help them scaffold this trial-and-error self-experimentation process — helping them figure out what to try, determine if it is working, and structure the data collection process. The current study is situated within this broader research initiative and leverages our previous work. In this study, the data that have been collected serve as the basis of the quantitative analysis while users of the Phendo app are consulted in the qualitative part.

This study puts forth a novel HAI framework, MPDA, to design an intelligent system for providing adaptive recommendations to support experimenting with self-management and developing personalized regimens. This framework simultaneously attends to both human and technical requirements and constraints. By mapping the needs of human users, a specific ML approach, and data that have been tracked “in the wild” independently of any algorithm, we uncover important insights and design recommendations that are likely to be feasible in real-world settings. There is a need to help individuals use their health data to create personalized self-management regimens, especially for chronic conditions with substantial complexity and uncertainty, like endometriosis. Intelligent systems could help fill this gap by providing adaptive recommendations. Our investigation suggests that an RL-powered system is a promising solution for real-world self-management. Findings provide useful guidance for the development and deployment of an RL system for experimenting with endometriosis self-management strategies. Beyond providing interesting insights and design guidance, this study importantly contributes a novel framework that could be adapted by other researchers who need to attend to specific human and technical requirements simultaneously. Future work will address how to operationalize the key parts of the RL agent in the context of an intelligent system, such as data-driven learning of user states and composite reward functions, for developing individualized endometriosis self-management regimens.

Care for chronic conditions is a major health priority globally [ 100 ]. Self-management — the day-to-day activities individuals undertake outside of the clinic to cope with their chronic illness — plays a critical role in managing and preventing the progression of disease [ 21 , 22 , 39 , 40 , 73 , 152 ]. However, establishing a self-management care regimen can be a major hurdle. Faced with often generic guidance, individuals are left with the burden of translating this information into their day-to-day lives [ 30 , 57 , 102 , 105 , 111 ]. For instance, the recommendation to “engage in regular exercise” is left up to individuals to determine how to implement (e.g., which exercises would make sense for them, how often, and how intense). Furthermore, it is not known a-priori which strategies will be successful for a given person. Thus, individuals have to experiment through trial-and-error, which can be a lengthy process. This approach becomes even more complex when individuals have to choose among multiple strategies that can be combined into a regimen. Finally, in conditions with limited scientific knowledge and no established self-management guidelines, there is an additional burden on the individual to identify candidate strategies [ 36 , 158 , 161 ] and effective personalized regimens [ 36 , 48 ]. Personal health informatics solutions have been proposed to support self-management for individuals [ 35 , 42 , 52 , 71 , 78 , 93 , 121 , 131 ], scaffold for problem solving [ 95 , 96 ], and promote experimentation to identify triggers of disease flares [ 44 , 45 , 81 ]. Tools for self-tracking and reflection have helped in multiple contexts like migraine [ 128 ], IBS [ 37 , 80 ], autism spectrum disorder [ 82 ], HIV [ 28 ], diabetes [ 27 , 115 ], and for those with multiple chronic conditions [ 20 , 91 ]. However, most solutions still leave a lot of the analytical work to individuals, e.g., which strategies to experiment with (either alone or in concert of each other), how to go about experimenting with them, and determining if they work. As such, designing intelligent systems to support management in the context of complex chronic illness represents an example of Ackerman’s sociotechnical gap [ 2 ] — i.e., there is a known discrepancy between the nuanced, flexible, and contextual real-world task of self-management, and the rigid and brittle capabilities of technology. In prior research with endometriosis patients [ 120 ], we documented a need for solutions that can provide individualized self-management recommendations and help to identify strategies with a positive health impact. These recommendations should take individual factors and context into consideration, help discover and select strategies to try out, and learn which ones are effective for each person. Given the structure of the problem identified in this work, a promising research direction for individualized, adaptive recommendations for self-management is the use of artificial intelligence (AI) methods in general, and reinforcement learning (RL), in particular. RL is unique in its ability to make sequential recommendations that adapt to changes in a complex environment. With RL, an agent learns a policy, i.e., a mapping from states to recommended actions (e.g., for a person living with diabetes, blood glucose level measurements mapped to recommended insulin doses), to maximize a pre-defined reward (e.g., decrease in HbA1c levels in a person living with diabetes) while simultaneously adapting to changes in the state, i.e., the environment resulting from the actions (e.g., the health condition of the person with diabetes). This ability to adapt in decision-making under uncertainty makes RL a plausible candidate to power an intelligent personal informatics system to support the trial-and-error process of self-management, by providing individualized recommendations of strategies for users to try and evaluating their success — over time and through interactions with the system — for learning individualized self-management regimens that work for each person [ 67 ]. However, RL also has a somewhat rigid conceptual model and requires structuring of a problem space into its framework of action space , state space , reward , and agent/policy . Furthermore, RL is notoriously data hungry and requires large training datasets to enable learning. As a result, while RL has many unique benefits, the introduction of RL may widen the sociotechnical gap [ 2 ] that already exists in intelligent solutions for personal health. In order to design intelligent, personalized systems that meet the real-world needs of users, a human-centered approach is necessary [ 18 ]. User-centered and participatory design have been established as critical pathways towards developing technological solutions, which have been increasingly applied in AI [ 72 , 108 , 153 ]. However, traditional user-centered design methods often prioritize user needs over technical capabilities and limitations [ 14 ]. While this approach has been shown to be widely successful in traditional software development [ 34 , 50 , 56 , 84 , 147 ], it may have limitations when applied to machine learning (ML) and AI. Given the need for human-centered design and an uptick in interactive systems that incorporate ML/AI [ 31 , 33 , 150 ], there is a need for new design methods that balance human needs with the technical capabilities of these systems. Various human-centered AI (HAI) principles have been put forward to address this gap [ 66 ]. Chancellor [ 32 ] argues that human-centered machine learning practices must be applied throughout the whole ML pipeline of problem brainstorming, development, and deployment. Human-Centered Algorithm Design proposed by Baumer [ 18 ] outlines design strategies across theoretical, speculative, and participatory processes, with the focus on incorporating social interpretations into the design approach. Value-Sensitive Algorithm Design [ 162 ] focuses on the early elicitation of human insights to guide the abstract and analytical creation of algorithms, aiming to mitigate bias and avoid compromising user values; however, there is no way to explicitly account for the demands and specifications of specific ML techniques, and it does not account for the data perspective. By contrast, Stakeholder-Centered AI Design [ 160 ] accounts for both human and data perspectives, using co-design with stakeholders’ own data to prototype with AI, but only addresses generic algorithm considerations. While these approaches include user, data, and/or ML perspectives, they do not account for all of these perspectives simultaneously and have no way to design for a specific ML algorithm. In fact, to a large degree, these new efforts continue to prioritize user needs and values and use them as a blueprint for designing new technologies. Since AI-enabled technologies often have unique capabilities and hard to change restrictions that must be considered when designing user-facing systems [ 25 ], neither the traditional user-centered design, nor the newer user-centered methods for AI account for these constraints. While these approaches offer some directions, a gap remains in practices to design HAI technologies, particularly for a given ML solution [ 127 ]. This requires new design approaches that place both users and AI on the same level and enables negotiation between them. In this paper, we propose and implement an HAI framework for designing intelligent personal informatics systems — Multi-Perspective Directed Analysis (MPDA) — that accounts for human, data, and machine learning requirements and constraints concurrently. MPDA uses constructs extracted from an ML approach, RL, to elicit both user needs, through directed content analysis of user interviews, and practical data constrains, critical for ML and AI-driven systems, through analysis of user engagement logs with an app for collecting self-monitoring data. We apply the proposed framework to gathering and triangulating human-machine-data requirements for a self-management tool for individuals with endometriosis — a poorly understood, complex chronic condition with no cure or reliable treatment. In this study, we ask the following research question: What insights at the intersection of human needs and values, human self-tracking behaviors as evidenced by “in the wild” self-tracking data, and capabilities and constraints of RL, can inform the design of RL-based intelligent systems for self-management of endometriosis? The work presented here makes these specific contributions: First, we propose (and provide an example of) a novel human-centered AI framework — Multi-Perspective Directed Analysis — that maps the human, data, and machine learning requirements and constraints within a complex real-world scenario for the design of an intelligent interactive system. We identify high-level concepts from RL to use as organizing principles for analyzing and synthesizing both qualitative and quantitative data. We use the MPDA framework to answer the research question asked in this study, which allows us to synthesize the human, data, and ML perspectives, evaluate trade-offs, and generate recommendations for design. Second, we identify promising directions for an RL-based intelligent interactive system for management of complex chronic illness, and elaborate several sociotechnical gaps [ 2 ]. In simultaneously attending to human, data, and ML requirements, we are able to surface recommendations to guide the design of an RL-enabled system to support users in leveraging their personal health data to experiment with self-management regimens in the context of endometriosis. Such a system for developing individualized self-management regimens in the case of endometriosis holds significant promise in improving care and management of this burdensome condition.