Tele-patient-reported outcome measures (telePROM) in follow-up of endometriosis: a validity and test-retest reliability study of an endometriosis-specific questionnaire (EQ)

Objective: Patient-Reported Outcome (PRO) measures supported by a severity algorithm may serve as a decision aid for triage and consultation in follow-up of patients with endometriosis. In a new follow- up regime, patients filled out an endometriosis-specific questionnaire (EQ) at home before outpatient consultation (tele-Patient-Reported Outcome Measures; telePROM). A severity algorithm was assigned patients’ answers using a color code thereby reflecting the need of clinical attention. Our study aimed to assess the test-retest reliability of the severity algorithm and of the single items as well as to evalu - ate the face- and content validity of the EQ.

The study was carried out in a referral endometriosis clinic at a Danish University Hospital. The validation was based on an initial version of the EQ, which was adjusted simultaneously with its severity algorithm, to meet the purpose of this study. Reliability was assessed by a test-retest setting of the questionnaire including patients with endometriosis, � 18 years and Danish speaking. Kappa statistics and interclass correlation analyses were applied to assess test-retest reliability. Face- and con - tent validity was explored by focus group interviewing of patients.

In total, 14 patients answered the questionnaire twice. Results indicate that the EQ demon - strated substantial reliability in three out of five domain indicators in the severity algorithm and 65% of items with kappa values above 0.60. Further, focus-group interview of five patients resulted in add - ing an open-ended question regarding important issues to discuss at the consultation.

TelePROM in outpatient follow-up of endometriosis is feasible as patients viewed the questionnaire relevant for their clinical follow-up. Yet, due to the small sample size results should be interpreted with caution. Further validation of the EQ is recommended. ARTICLE HISTORY Received 12 July 2024 Revised 12 February 2025 Accepted 17 February 2025

Patient reported outcome (PRO); endometriosis; outpatient; reliability; validity

Endometriosis is a painful, chronic disease that affects up till 10% of women in their reproductive age 1 . Endometriosis is defined as the presence of endometrial tissue outside the uterus 2 , no cure exists and treatment options are both med - ical and surgical 3 ; surgery depending on severity and/or symptom burden. Due to the chronic aspect the disease requires repetitive standardized consultations and so far, sparse is available regarding the optimal follow-up schedule 4 . Moreover, there is a growing awareness that follow-up should be more influenced by the patient’s own perception of symptoms thereby focusing on symptom relief rather than surgery, for which recurrence rates are high (40–50% in 5 years) 5 , 6 . The European Society of Human Reproduction and Embryology’s Guideline on Endometriosis underlines that follow-up should be further individualized based on pre - vious and current treatments, and severity of the disease and symptoms 4 . Patient-reported outcome (PRO) measures is a well-known tool to collect data about the patient’s health status reported directly from the patient 7 , 8 , and thereby a potential valuable instrument for individualized follow-up. When filled in remotely, it is defined as telePRO 9 . PRO measures are widely used in several settings, e.g. epilepsy, oncology and CONTACT Maria Monberg Feenstra [email protected] Department of Gynecology and Obstetrics, Odense University Hospital; Research Unit of Gynaecology and Obstetrics, Department of Clinical Research, Faculty of Health Sciences, University of Southern Denmark, Odense, Denmark Supplemental data for this article can be accessed online at https://doi.org/10.1080/03007995.2025.2470749. � 2025 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited, and is not altered, transformed, or built upon in any way. The terms on which this article has been published allow the posting of the Accepted Manuscript in a repository by the author(s) or with their consent. www.cmrojournal.com CURRENT MEDICAL RESEARCH AND OPINION 2025, VOL. 41, NO. 2, 307–316 https://doi.org/10.1080/03007995.2025.2470749 Article ST-0706/2470749 pregnancy 10 , 11 . Potential advantages of PRO measures are better monitoring of treatment on patients’ health 12 and more tailored dialogues 12 , 13 . PRO may increase symptom management actions in patients and healthcare professionals (HCPs) 14 , early diagnosis 15 , and potentially better treatment of symptoms and unmet needs through, e.g. medication, self-management strategies and insight in the current state of the disease 16 , 17 . In addition, PRO measures may support HCPs as a decision support-tool, since patients can assess their need of contact with the healthcare system and vice versa through real-time monitoring of symptoms by use of PRO 10 , 18 . Currently, very limited data exist regarding the use of PRO measures in follow-up of patients with endometriosis in an outpatient setting. In a systematic literature review by Nicolas-Boluda et al. 19 , they evaluated PRO measures for use in endometriosis care, however, not identifying any current use of PRO measures in routine clinical care. Yet patients find patient-initiated follow-up in endometriosis valuable in contrast to fixed controls 20 . As seen before, PRO measures supported by a severity algorithm could aid in clinical deci - sion-making during triage and consultations 9 . PRO measures including a severity algorithm can be use - ful and incorporated into general clinical practice and func - tion as a tool for HCPs during patient visits, but it is crucial that the collected data are precise in order to assess the accurate disease or treatment burden 21 , as invalid outcomes could pose a risk to patient safety and treatment effective - ness. Validity and reliability constitute crucial psychometric attributes of an instrument to ensure measurements are accurate and dependable 22 , 23 . The endometriosis-specific questionnaire (EQ) presented in this study including a severity algorithm was originally developed prior to this study to monitor patients’ health sta - tus and need of outpatient follow-up. The algorithm was applied to determine whether the contact should be with a physician or a nurse and with a focus on strengthening per - sonalized dialogue between patients and HCPs 24 . The pre - sent algorithm employs three colors: Red (signifying a significant problem encountered by the patient regarding the item); Yellow (indicating a mild to moderate problem experienced by the patient); and Green (denoting the absence of any problems within the item as reported by the patient). However, the EQ and algorithm was never validated.

The objective of this study was to evaluate the test-retest reliability of the severity algorithm used for triage, along with single items of patients’ symptoms and general well- being, collectively forming the endometriosis-specific ques - tionnaire (EQ). Further, we aimed to evaluate the face- and content validity of the endometriosis-specific question - naire (EQ).

Development and preparation phase The development of a first-version of the EQ and its severity algorithm was done in close collaboration between patients and HCPs representing six different Danish hospitals from both regional and University Hospitals, representing four dif - ferent Danish regions. The initial development of the EQ was facilitated by AmbuFlex, Center for Patient Reported Outcomes, in the Region of Central Denmark and consisted of both observations of consultations, interviews with patients, consensus meetings, and, finally, pilot testing in nine patients with endometriosis. Observation of patients’ consultations and interviews were carried out to identify symptoms and problems discussed during consultations. Initial questions were discussed in terms of relevance and understandability. The first-version of the EQ was then tested in a small group of patients to evaluate patients understand - ing of each question, the overall questionnaire, and its intended purpose. Variation in the target group was sought by age and location of endometriosis. The methods of “Think aloud” and “Verbal probing” 25 was used to conduct cognitive semi-structured interviews throughout the development pro - cess and analysis of interviews was inspired from thematic analysis 26 . Prior to this study, AmbuFlex reported in-depth descriptions of the initial development process elsewhere in Danish 26 , 27 , but their methods are reported internationally 9 . However, the EQ was never validated. In our validation study, the EQ and its severity algorithm was reviewed and further adjusted with input from four gynecologists specialized in endometriosis, along with feed - back from one patient. All were affiliated with the single study site; a tertiary referral endometriosis outpatient clinic at a Danish University Hospital. In addition, the app named “My Hospital” (in Danish: “Mit Sygehus”) was chosen for dis - tribution of the EQ. A web-version (www.mitsygehus.dk) was also available. The patient pilot tested the final and digital version of the EQ before validation of the EQ. Content of the endometriosis-specific questionnaire (EQ) Relevant items were either adopted from previous validated questionnaires, or developed when no relevant instruments or items were identified. Questions about pain during peri - ods, pain in between periods, and abdominal pain were developed by AmbuFlex. Pain during defecation, when uri - nating or breathing were adopted from Riiskjær et.al. 28 All were measured by a numeric rating scale (NRS) from 0 to 10, where 0 indicates no pain and 10 reflects worst pain ever 29 . Symptoms reflecting feeling bloated, obstipated, having diar - rhea, passing blood in stool, having nausea, vomiting, prob - lems with urinating, frequent urination, and limitations in regard to work or daily activities were inspired from the European Organization for Research and Treatment (EORTC) quality of life questionnaires 30 , since the questions are con - sistent with symptoms reported among patients with endo - metriosis. Questions about infertility and sexual function (including dyspareunia) were adapted from the disease- 308 M. M. FEENSTRA ET AL. specific questionnaire Endometriosis Health Profile (EHP- 30) 31 . WHO well-being index (WHO-5) was used to measure mental well-being 32 , and one question from the question - naire RAND Short-Form 36 (SF-36) was included in regard to patients’ perceptions of their health in general 33 . Further, questions added to the EQ were related to medi - cation use (hormones, painkillers, and laxatives) and blood in the urine. Likewise, symptoms related to pain in lower back or back, shoulder, and legs, as well as two items regarding the number of days required to stay at home or in bed were created. Finally, two items were developed regarding part - ner’s concern for their relationship with the patient and understanding of the patient’s challenges. Overall, the EQ consisted of 54 questions in total (Supplementary Material A). Not all needed to be filled out, only those relevant for the patient. Five domain indicators were created, consolidating all items in the EQ, and presenting an organized overview of the patient’s problems by the following categories: Pain con - cerning menstrual bleeding (“Pain bleeding”), Pain concern - ing breathing, urinating or defecation (“Pain other”), Other Symptoms including sexuality and infertility (“Symptoms”), Well-being and depression (“Depression”), and Need of Contact (“Contact”). Adjustment of the severity algorithm The purpose of the EQ was to support triage and dialogue in outpatient follow-up care of patients with endometriosis. The severity algorithm needed to be further developed for this purpose. Answers to new questions added to the original EQ were assigned a color by the team of endometriosis special - ists at the study site. Each domain indicator was colored by the item within the domain representing the worst color code (e.g. red). Text summaries to support triage of patients were developed by creating a flowchart visualizing the domain indicator (worst color code of any symptom) com - bined with the domain indicator “Contact” reflecting the patient’s need of contact. A heat-map of an anonymized dataset guided this process. The severity algorithm could both be displayed per item, per domain indicator, and as a text summary for the total PRO measures; generating a sug - gestion as to whether the patient should see a physician or a nurse (or displays no need to be seen at all) and selecting the appropriate method (attendance at the clinic, video- or phone consultation). Visualization of the EQ in the system “My Hospital” can be seen in Supplementary Material B. Clinical purpose of the endometriosis-specific questionnaire (EQ) Introducing the EQ for use in general routine outpatient follow- up of all patients with endometriosis implied that every patient booked for a follow-up consultation received the EQ automatic - ally through the app “My Hospital”. The EQ was available to be answered 14 days prior to their consultation. All responses were stored in the medical record and a graphical PRO-overview of the patient responses was generated. Further, responses could be displayed over time (See Supplementary Material B). In add - ition, patients could contact the clinic through a message-func - tion in the app, requesting consultation. This was followed by a distribution of the EQ to be completed by the patient through the app. After reviewing the patient’s answers, the suggestions generated by the severity algorithm, and reviewing the latest notes in the electronic patient journal; the patient would be booked for an outpatient consultation based on her current symptom status to be consulted by either a physician or nurse, and by relevant method, e.g. attendance at the clinic, video- or phone. Nurses specialized in endometriosis were responsible for reviewing patients’ answers ensuring relevant consultation for the patient. The new PRO-based follow-up regime was named tele-patient-reported outcome measures (telePROM). Study population Patients diagnosed with endometriosis from the study site were invited to practice video consultations with the HCPs as a new initiative. In addition, patients should fill in the EQ for the HCPs to practice using the EQ for dialogue support. Patients were screened for eligibility by a gynecologist specialized in endometriosis. Inclusion criteria were diagnosis of endometri - osis registered in the electronic patient journal, age � 18 years and, in a stable period of endometriosis as they should not need a physical examination. Exclusion criteria was patients, who were not able to speak or read Danish. Patients for this study were recruited after their video consultation, which all were conducted in September 2019 as part of testing the new initiative of video consultations. All patients that fulfilled inclusion criteria were provided with written information about the study before enrolling. Additional verbal information was provided when needed. The invitation was sent electronically through “e-Boks” 34 which is a secure electronic mailbox for all citizens in Denmark. Evaluation of test-retest reliability All patients received the questionnaire (test 1) through an app or web-version of “My Hospital” to be completed prior to the video consultation with the physician and nurse. Patients accepting the re-test assessment (test 2) were asked to answer the same questionnaire again as to assess test- retest reliability 35 . To avoid patients seeing previous answers, the second questionnaire (re-test) was distributed through SurveyXact. Non-responders were contacted two times; first through e-Boks and secondly by telephone. No sample size was calculated, as we approached all patients, striving to meet the recommended number of 50 participants 36 . Test-retest reliability of domain indicators The EQ severity algorithm applied to each item a color code (green, yellow, red) and summarized answers according to the items in the five domain indicators. The primary analysis concerned the five domain indicators. Test and re-test classi - fications were tabulated, and the reliability quantified using the intraclass (IC) kappa coefficient 37 , 38 . Confidence intervals CURRENT MEDICAL RESEARCH AND OPINION 309 95% for the IC kappa were calculated using non-parametric bootstrapping based on 1000 replications. Kappa values were interpreted as follows �0.2 (slight), 0.21–0.4 (fair), 0.41– 0.6 (moderate), 0.61–0.8 (substantial), 0.81–1.0 (almost per - fect) 39 . The analysis was performed by implementing the intra-class kappa formula from Kraemer et al. 37 (section 2.1). The code was tested by comparing to the SAS-procedure given in Fisher et al. 38 and the two implementations were found to coincide in a large, simulated data set. As a sensitiv - ity analysis, the primary analyses were repeated after exclud - ing participants, who changed their hormonal treatment between the test and re-test. Test-retest reliability of items Secondary analyses were performed on the EQ item level. Reliability of binary items was quantified using the IC kappa as described above for the overall indicators. For continuous and ordinal items, reliability is quantified as the intra-class correlation coefficient (ICC), which was estimated from a mixed model. The model included no fixed effects and a ran - dom effect on the patient level (its variance component sig - nifying the between-patient variation). The ICC was calculated as the ratio of between-patient variance to the total variance (between- plus within-patient variance), and a confidence interval was constructed on logistic scale by the delta method and then backtransformed. Model validation was performed by inspection of residuals, fitted values and BLUP estimates of random effects. Normality assumptions were found to be violated for some items. Note that for ordinal items, the ICC is equivalent to a weighted kappa using quadratic weights 40 . Evaluation of face- and content validity We evaluated the face- and content validity of the EQ as well as intended use and relevance by performing a focus group interview (FGI) with a sub-sample of patients, who had answered both questionnaires. A semi-structured inter - view guide was developed to explore the relevance of the questions in the EQ, the comprehensiveness as to no impor - tant aspects of the construct should be missing, and finally the comprehensibility as to the questions should be under - stood by patients as intended 23 . Additional questions regard - ing intended use of the questionnaire in clinical practice and distribution options were added. The interview guide is pro - vided in Supplementary Material C. Before beginning the interview, patients read the ques - tionnaire again (paper version). The interview was carried out by the first author in November 2019, it lasted 58 min, was audio-recorded and verbatim transcribed. First-author had some experience in interviewing patients, and a nurse col - league observed the participants during the interview as to non-verbal communication and if everyone seemed to express their opinion freely. Afterwards the first-author and the nurse evaluated the overall quality of the interview based on the observations. Final pilot testing on patients from the endometriosis outpatient clinic was planned to reassess face- and content validity regarding any adjustments made based on the results of the FGI. Other analyses Descriptive summaries of baseline variables are presented as median with interquartile range (IQR) for continuous variables and as numbers (%) with percentage of total for categorical variables. The Consensus-based Standards for the Selection of health Measurement Instruments (COSMIN) Risk of bias check - list was used for content validity and reliability 41–43 . All analysis were performed by Stata 17 (StataCorp. 2021. Stata Statistical Software: Release 17. College Station, TX: StataCorp LLC).

Reliability A total of 48 out of 291 screened patients were found eli - gible and invited to fill in the EQ and participate in a video consultation ( Figure 1 ). Overall, 26 patients accepted and filled in the first questionnaire (test 1). Two patients were unable to attend their video consultation, resulting in 24 patients invited for this study. A total of 14 patients answered both the test and re-test questionnaire and were included for data analysis. Questionnaires were answered with a median duration of 34 days (IQR 13 days) and with a range between patients’ answers of 27 days to 50 days. Patients’ characteristics at baseline are displayed in Table 1 . Kappa values estimates for the domain indicators in the EQ ranged from slight to almost perfect 39 ( Figure 2 ). Except for the domain indicators “Contact” with a kappa value of −0.01, 95% CI (-0.35–0.69), all disclosed kappa value esti - mates above 0.41, 95% CI (-0.05 to 1.0) categorized as mod - erate to almost perfect. When excluding patients, who reported a change in hormonal treatment (n ¼ 5), lower kappa value estimates between 0.30–0.60, 95% CI (-0.17–1) were observed in three domain indicators (“Depression”, “Symptoms”, “Pain bleeding”). Domain indicators “Pain other” and “Contact” showed higher kappa value estimates of 0.30 and 1, 95% CI (-0.17–1) ( Figure 3 ). Kappa value estimates of single items in the EQ are illus - trated in Figure 4 . Low kappa value estimates were observed in relation to questions about reporting blood in the urine (0.00) or in the feces (0.143) and to having experienced pain while bleeding during the last four weeks (-0.12). In total, 65% of items showed substantial reliability as kappa values were 0.61 or above. Face- and content validity A total of six patients accepted to participate in a FGI. One patient was unable to participate at the day of the FGI, leav - ing five participants, aged 25–47. Time of diagnosis varied from within the last year to 9–10 years ago. Patients were generally satisfied with the questionnaire and its content. It was easy to fill out digitally, and notifications were a relevant

for distribution of reminders to fill out the EQ. A 310 M. M. FEENSTRA ET AL. total of seven answer categories were modified. A single open-ended question was added asking patients “Is there anything else that is important for you to talk about at the consultation?” . Patients stressed the importance of HCPs using the EQ actively during consultation and having a clear purpose of using the tool. The full results of the FGI are dis - played in Supplementary Material D. Afterwards, the EQ was pilot tested in two rounds at the study site with, respectively, six and eight patients with endometriosis. Minor editorial adjustments were made, and face- and content validity was found acceptable for implementation in the endometriosis outpatient clinic.

Studies of PRO measures for routine follow-up of endometri - osis are sparse. This study shows that the EQ had acceptable face- and content validity amongst Danish patients with Figure 1. Inclusion of study participants. CURRENT MEDICAL RESEARCH AND OPINION 311 endometriosis, and that the EQ demonstrated substantial reli - ability estimates in three out of five domain indicators as well as 65% of items with kappa value estimates above 0.60. Yet, the wide confidence intervals (CI) indicate that results should be interpreted with caution due to the small sample size and that no confirmatory conclusions as to the test-retest reliability of the EQ can be made. PRO measures during endometriosis follow-up Face- and content validity is the most important property of an instrument 23 . Our study pointed out that patients found the questions relevant as well as appropriate for use during outpatient follow-up. Several items are inspired from already validated instruments; e.g. the NRS is relevant for pain meas - urement in endometriosis 29 , Rand SF-36 and its psychometric properties are well documented 44 and EHP-30 has been vali - dated in Danish 45 . Patients addressed the need of the possi - bility to add further information unique for the individual patient, e.g. that bleeding persists even after a hysterectomy; reflecting that some items needed a possibility for further elaboration with a focus on individual considerations. Collectively, these findings contribute to the groundwork, serving as the basis for future adaptation and integration of the EQ in outpatient follow-up of endometriosis. Test-retest of a hormone-dependent disease As endometriosis is a hormone-dependent disease, symptoms might vary during a short period of time, making it complex to ensure a stable study population in the test-retest period. In a test-retest study among patients with endometriosis 45 , an inter - val of more than four weeks (i.e. more than a menstrual period) was chosen with our study having a range between patients’ answers of 27 days to 50 days. Our results might reflect patients had a benefit of receiving a video-consultation during the test-retest period as not all domain indicators and items demonstrated fair kappa values. Negative kappa values could indicate that the study population had been too homogeneous as to all having symptoms from endometriosis in the same degree, and the study population not reflecting much variance in disease severity. This would affect the kappa value. In respect to patients’ need of contact with the outpatient clinic, seven out of ten patients did not report having a need for contact in test 2. This result might reflect the fact that patients had a video consultation with an HCP between test 1 and test 2, and the study design could Table 1. Participants’ characteristics at time of test 1, N ¼ 14. Age, median (IQR) 40.6 (18.5) Hormonal treatment a , n (%) 11 (78.57) Hormone spiral (Mirena, Kyleena or Jaydess) 5 (35.71) Mini-pills 4 (28.57) Birth control pills 3 (21.43) Anti-hormones (e.g. Zoladex) 3 (21.43) Estrogen (for example together with hormones) 2 (14.29) Algorithm domain indicators in test 1 Pain bleeding, n (%) Green b 6 (42.86) Yellow b 4 (28.57) Red b 4 (28.57) Pain other, n (%) Green b 4 (28.57) Yellow b 4 (28.57) Red b 6 (42.86) Symptoms, n (%) Green b 0 (0) Yellow b 4 (28.57) Red b 10 (71.43) Depression, WHO-5, n (%) Green b 4 (28.57) Yellow b 10 (71.43) Red b 0 (0) Contact, n (%) Green b 4 (28.57) Yellow b 0 (0) Red b 10 (71.43) Depression, WHO-5, median (IQR) 48 (44) Patient perceived health (SF-36), n (%) Excellent/very good 4 (28.57) Good 3 (21.43) Fair/poor 7 (50) a Patients could report more than one treatment option. b Green indicating no problem, yellow indicating mild to moderate problem and red indicating significant problem. Figure 2. Test-retest reliability of domain indicators. 312 M. M. FEENSTRA ET AL. have caused a bias in patients feeling better supported at test 2 due to the video consultation with a HCP. Use of PRO measures in clinical practice In this study, patients underlined the importance of knowing the actual purpose of the questionnaire. Like other studies 46 , 47 , patients advocated for HCPs to use their answers actively dur - ing follow-up consultations. To do so, special attention must be given to both patients and HCPs when implementing telePROM in outpatient follow-up. A special focus should be on information and training in using the EQ. When the EQ sup - ported by the severity algorithm is used as a decision aid for patients’ actual need and choice of consultation method, it should not stand alone but include additional data from the electronic patient journal in evaluating the patient’s actual dis - ease burden 48 . It adds to the safety of the initiative, as each patient is thoroughly evaluated by combining relevant data from different sources, which should be described in local guidelines to support HCPs during implementation and in daily clinical practice. Further, if PRO measures are to be used for clinical research it is important to acknowledge that PRO measures represent a snapshot of patients’ experiences, which can fluctuate rapidly due to the hormonal influence upon endometriosis as a disease. Moreover, awareness of the data collection timing should be considered as it may influence the data, whether it is collected, e.g. before, on the day of consult - ation or in the following days. Figure 3. Test-retest reliability of domain indicators sens hormones. Figure 4. Test-retest reliability of single items. CURRENT MEDICAL RESEARCH AND OPINION 313

There are limitations to consider in this study. The study population represents only Danish-speaking patients as the questionnaire was in Danish, and thus; for use in other lan - guages the questionnaire needs to be translated and validated according to relevant COSMIN guidelines. Today, non-Danish speaking patients have the possibility to bring a relative to translate and/or fill in the questionnaire in the app but this study cannot be valid nor feasible regarding such patients. The participant rate was below COSMIN rec - ommendations of a minimum of 50 patients as sample size 36 , wherefore there could be a risk of selection bias and thus a reduced generalization of the results. Reasons for dropouts were not recorded, but may be influenced by the fact that video consultation took place on a Saturday. More research is needed to further validate and test the reliability of the instrument in patients with endometriosis. To determine change over time, each construct or item could be provided with a filter question as to whether there has been any changes since the last questionnaire was filled out. In telePROM, additional information about the patient were available in the follow-up procedure; e.g. the patient’s electronic journal, messages in-between consultations with the endometriosis team sent through the app as well as patient’s own assessment of the need of a consultation self- reported in the EQ. Therefore, patients’ PRO measures become an integrated part of different data sources used for triage of the patient. Finally, the EQ has been further revised since data collecting in 2019, why more tests of validity and reliability are recommended, which are also of great impor - tance of improving data quality 49 .

Our study is one of the first to test psychometric properties of a Danish EQ for use in routine outpatient follow-up. Results demonstrated an acceptable face- and content validity of the EQ, and patients reported the instrument as relevant during outpatient follow-up. Implementation is feasible as the EQ would be supplemented by several data sources, when decid - ing upon patients’ need and type of consultation. Due to the small sample size further psychometric validation of the EQ and in a broader sample of patients is recommended. Transparency Declaration of funding This study is funded by the Department of Clinical Research, University of Southern Denmark and Department of Gynecology and Obstetrics, Odense University Hospital. The funding body had no role in the design of the study, choice of methods, data collection, analysis or reporting of the results. Declaration of financial/other relationships The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or roy - alties. Peer reviewers on this manuscript have no relevant financial or other relationships to disclose. Author contributions The first- and last author contributed to the study conception, design,

preparation and data collection. Data analysis were performed by Simon Bang Kristensen, Maria Monberg Feenstra and Martin Rudnicki. The first draft of the manuscript was written by Maria Monberg Feenstra and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

We would like to send a special thank you to the patients participating in this study and providing us their unique perspectives and valuable data on symptoms and general well-being. An acknowledgement to Dr Poul Bak Thorsen, MD, PhD for skillful assistance and discussion during the preparation phase and data collection of this study. Lastly, thank you to all funding organizations contributing to the realization of this study. Data availability statement The data supporting the findings of this study are not publicly available due to restrictions, e.g. patients have not consented hereto, and data contain personal information. Ethical statement Approval from the Data Protection Agency of the Region of Southern Denmark was granted (jrn.nr 20/35407) for this study. Approval from the local Ethics Committee was not required according to Danish law. All patients in this study gave informed consent to participate. ORCID Maria Monberg Feenstra http://orcid.org/0000-0002-3645-0543 Anne Sidenius http://orcid.org/0000-0002-8680-8769 Charlotte Nielsen http://orcid.org/0000-0002-8920-7758 Simon Bang Kristensen http://orcid.org/0000-0002-5533-6922 Martin Rudnicki http://orcid.org/0000-0002-5115-1445

0[1] Horne AW, Saunders PTK. SnapShot: endometriosis. Cell. 2019; 179(7):1677–1677.e1. doi: 10.1016/j.cell.2019.11.033. 0[2] Kennedy S, Bergqvist A, Chapron C, et al. ESHRE guideline for the diagnosis and treatment of endometriosis. Hum Reprod. 2005; 20(10):2698–2704. doi: 10.1093/humrep/dei135. 0[3] Lemaire GS. More than just menstrual cramps: symptoms and uncertainty among women with endometriosis. J Obstet Gynecol Neonatal Nurs. 2004;33(1):71–79. doi: 10.1177/0884217503261085. 0[4] European Society of Human Reproduction and Embryology Guideline Development Group. Endometriosis. Guideline of European Society of Human Reproduction and Embryology; 2022 [cited 2024 Jul 4]. Available from: https://www.eshre.eu/ Guidelines-and-Legal/Guidelines/Endometriosis-guideline. 0[5] Horne AW, Missmer SA. Pathophysiology, diagnosis, and manage - ment of endometriosis. BMJ. 2022;379:e070750. doi: 10.1136/bmj- 2022-070750. 314 M. M. FEENSTRA ET AL. 0[6] Saunders PTK, Horne AW. Endometriosis: etiology, pathobiology, and therapeutic prospects. Cell. 2021;184(11):2807–2824. doi: 10. 1016/j.cell.2021.04.041. 0[7] US Department of Health and Human Services Food and Drug Administration. Guidance for industry: patient-reported outcome measures: use in medical product development to support label - ing claims; 2009 [cited 2023 Sep 9]. Available from: https://www. fda.gov/media/77832/download. 0[8] de Vet Ht CB, Mokkink BL, Knol DL. Measurement in medicine. 1st ed. New York Cambridge University Press; 2011. 0[9] Hjollund NHI, Larsen LP, de Thurah AL, et al. Patient-reported outcome (PRO) measurements in chronic and malignant diseases: ten years’ experience with PRO-algorithm-based patient-clinician interaction (telePRO) in AmbuFlex. Qual Life Res. 2023;32(4): 1053–1067. doi: 10.1007/s11136-022-03322-9. [10] Videns-og Kompetencecenter for Brugerinddragelse i Sundhedsvæsenet (ViBIS). Program PRO. Anvendelse af PRO-data i kvalitetsudviklingen af det danske sundhedsvæsen – anbefalin - ger og vidensgrundlag. [Program PRO. Use of PRO data in the quality development of the Danish healthcare system – recom - mendations and knowledge base]; 2016 [cited 2024 July 4]. Available from: https://danskepatienter.dk/sites/danskepatienter. dk/files/media/Publikationer%20-%20Egne/B_ViBIS/A_Rapporter% 20og%20unders%C3%B8gelser/program_pro-rapport.pdf. [11] Chen A, V €ayrynen K, Schmidt A, et al. The impact of implement - ing patient-reported measures in routine maternity care: a sys - tematic review. Acta Obstet Gynecol Scand. 2022;101(11):1184– 1196. doi: 10.1111/aogs.14446. [12] Valderas JM, Kotzeva A, Espallargues M, et al. The impact of measuring patient-reported outcomes in clinical practice: a systematic review of the literature. Qual Life Res. 2008;17(2):179– 193. doi: 10.1007/s11136-007-9295-0. [13] Ishaque S, Karnon J, Chen G, et al. A systematic review of rando - mised controlled trials evaluating the use of patient-reported out - come measures (PROMs). Qual Life Res. 2019;28(3):567–592. doi: 10.1007/s11136-018-2016-z. [14] Howell D, Molloy S, Wilkinson K, et al. Patient-reported outcomes in routine cancer clinical practice: a scoping review of use, impact on health outcomes, and implementation factors. Ann Oncol. 2015;26(9):1846–1858. doi: 10.1093/annonc/mdv181. [15] Ballard KD, Seaman HE, de Vries CS, et al. Can symptomatology help in the diagnosis of endometriosis? Findings from a national case-control study–Part 1. BJOG. 2008;115(11):1382–1391. doi: 10. 1111/j.1471-0528.2008.01878.x. [16] Lavallee DC, Chenok KE, Love RM, et al. Incorporating patient- reported outcomes into health care to engage patients and enhance care. Health Aff (Millwood). 2016;35(4):575–582. doi: 10. 1377/hlthaff.2015.1362. [17] Lancaster K, Abuzour A, Khaira M, et al. The use and effects of electronic health tools for patient self-monitoring and reporting of outcomes following medication use: systematic review. J Med Internet Res. 2018;20(12):e294. doi: 10.2196/jmir.9284. [18] Aiyegbusi OL, Hughes SE, Peipert JD, et al. Reducing the pres - sures of outpatient care: the potential role of patient-reported outcomes. J R Soc Med. 2023;116(2):44–64. doi: 10.1177/ 01410768231152222. [19] Nicolas-Boluda A, Oppenheimer A, Bouaziz J, et al. Patient- reported outcome measures in endometriosis. J Clin Med. 2021; 10(21):5106. doi: 10.3390/jcm10215106. [20] Chainrai M, Kershaw VF, Gray TG, et al. Patient-initiated follow-up in gynaecology: patient and clinician views. Eur J Obstet Gynecol Reprod Biol. 2024;298:18–22. doi: 10.1016/j.ejogrb.2024.04.032. [21] Kottner J, Audig �e L, Brorson S, et al. Guidelines for reporting reli - ability and agreement studies (GRRAS) were proposed. J Clin Epidemiol. 2011;64(1):96–106. doi: 10.1016/j.jclinepi.2010.03.002. [22] Mokkink LB, Boers M, van der Vleuten CPM, et al. COSMIN risk of bias tool to assess the quality of studies on reliability or measure - ment error of outcome measurement instruments: a Delphi study. BMC Med Res Methodol. 2020;20(1):293. doi: 10.1186/s12874-020- 01179-5. [23] Terwee CB, Prinsen C, Chiarotto A, et al. COSMIN methodology for assessing the content validity of PROMs–user manual. Amsterdam: VU University Medical Center; 2018. [24] Region of Central Denmark. Endometriose [Endometriosis]. 2024. https://www.fagperson.sundhed.rm.dk/til-ansatte-og-samarbejd - spartnere/ambuflex/pro-losninger/kvindesygdomme-og-fodsler/ endometriose/. [25] Beatty PC, Willis GB. Research synthesis: the practice of cognitive interviewing. Public Opinion Quarterly. 2007;71(2):287–311. doi: 10.1093/poq/nfm006. [26] Ambuflex. Test af spørgeskema om endometriose [Test of endo - metriosis questionnaire]; 2019. p. 1–45. [27] Ambuflex. Spørgeskema om endometriose. Patientinddragelse [Questionnaire about endometriosis. Patientinvolvement]; 2018. p. 1–17. [28] Riiskjær M, Egekvist AG, Hartwell D, et al. Bowel endometriosis syndrome: a new scoring system for pelvic organ dysfunction and quality of life. Hum Reprod. 2017;32(9):1812–1818. doi: 10. 1093/humrep/dex248. [29] Bourdel N, Alves J, Pickering G, et al. Systematic review of endo - metriosis pain assessment: how to choose a scale? Hum Reprod Update. 2015;21(1):136–152. doi: 10.1093/humupd/dmu046. [30] Aaronson NK, Ahmedzai S, Bergman B, et al. The European Organization for Research and Treatment of Cancer QLQ-C30: a quality-of-life instrument for use in international clinical trials in oncology. J Natl Cancer Inst. 1993;85(5):365–376. doi: 10.1093/ jnci/85.5.365. [31] Jones G, Kennedy S, Barnard A, et al. Development of an endo - metriosis quality-of-life instrument: the Endometriosis Health Profile-30. Obstet Gynecol. 2001;98(2):258–264. doi: 10.1016/ s0029-7844(01)01433-8. [32] Topp CW, Østergaard SD, Søndergaard S, et al. The WHO-5 Well- Being Index: a systematic review of the literature. Psychother Psychosom. 2015;84(3):167–176. doi: 10.1159/000376585. [33] Bjorner JB, Thunedborg K, Kristensen TS, et al. The Danish SF-36 Health Survey: translation and preliminary validity studies. J Clin Epidemiol. 1998;51(11):991–999. doi: 10.1016/s0895-4356(98) 00091-2. [34] Postnord Na. eBoks. 2023 [cited 2023 Nov 28]. Available from: https://private.e-boks.com/danmark/en. [35] Mokkink LB, Terwee CB, Patrick DL, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient- reported outcomes. J Clin Epidemiol. 2010;63(7):737–745. doi: 10. 1016/j.jclinepi.2010.02.006. [36] Mokkink LB, Prinsen CA, Bouter LM, et al. The consensus-based standards for the selection of health measurement instruments (COSMIN) and how to select an outcome measurement instru - ment. Braz J Phys Ther. 2016;20(2):105–113. doi: 10.1590/bjpt-rbf. 2014.0143. [37] Chmura Kraemer H, Periyakoil VS, Noda A. Kappa coefficients in medical research. Stat Med. 2002;21(14):2109–2129. doi: 10.1002/ sim.1180. [38] Fisher Glr DG, Neri E, Art N, et al. Measuring test-retest reliability: the Intraclass Kappa. SAS Conf Proc. 2019:1–9. [39] Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–174. doi: 10.2307/ 2529310. [40] Fleiss JL, Cohen J. The equivalence of weighted Kappa and the intraclass correlation coefficient as measures of reliability. Educ Psychol Measure. 1973;33(3):613–619. doi: 10.1177/0013164 47303300309. [41] Mokkink LB, de Vet HCW, Prinsen CAC, et al. COSMIN risk of bias checklist for systematic reviews of patient-reported outcome measures. Qual Life Res. 2018;27(5):1171–1179. doi: 10.1007/ s11136-017-1765-4. [42] Prinsen CAC, Mokkink LB, Bouter LM, et al. COSMIN guideline for systematic reviews of patient-reported outcome measures. Qual Life Res. 2018;27(5):1147–1157. doi: 10.1007/s11136-018-1798-3. CURRENT MEDICAL RESEARCH AND OPINION 315 [43] Terwee CB, Prinsen CAC, Chiarotto A, et al. COSMIN methodology for evaluating the content validity of patient-reported outcome measures: a Delphi study. Qual Life Res. 2018;27(5):1159–1170. doi: 10.1007/s11136-018-1829-0. [44] Bjorner JB, Damsgaard MT, Watt T, et al. Tests of data quality, scaling assumptions, and reliability of the Danish SF-36. J Clin Epidemiol. 1998;51(11):1001–1011. doi: 10.1016/s0895- 4356(98)00092-4. [45] Hansen KE, Lambek R, Røssaak K, et al. Health-related quality of life in women with endometriosis: psychometric validation of the Endometriosis Health Profile 30 questionnaire using confirmatory factor analysis. Hum Reprod Open. 2022;2022(1):hoab042. doi: 10. 1093/hropen/hoab042. [46] Grove BE, Valen Schougaard LM, Ivarsen P, et al. Remote follow- up based on patient-reported outcomes in patients with chronic kidney disease: a qualitative study of patient perspectives. PLoS One. 2023;18(2):e0281393. doi: 10.1371/journal.pone.0281393. [47] Mejdahl CT, Schougaard LMV, Hjollund NH, et al. Patient-reported outcome measures in the interaction between patient and clin - ician – a multi-perspective qualitative study. J Patient Rep Outcomes. 2020;4(1):3. doi: 10.1186/s41687-019-0170-x. [48] Schougaard LMV, Friis RB, Grytnes R, et al. Exploring the nurses’ perspective on using remote electronic symptom monitoring in clinical decision-making among patients with metastatic lung cancer. Semin Oncol Nurs. 2023;39(6):151517. doi: 10.1016/j. soncn.2023.151517. [49] Schougaard LM, Larsen LP, Jessen A, et al. AmbuFlex: tele- patient-reported outcomes (telePRO) as the basis for follow-up in chronic and malignant diseases. Qual Life Res. 2016;25(3):525– 534. doi: 10.1007/s11136-015-1207-0. 316 M. M. FEENSTRA ET AL.