{"paper_id":"4bb1a72d-379e-4b5e-8fa4-43c6446e6a8d","body_text":"Current Medical Research and Opinion\nISSN: 0300-7995 (Print) 1473-4877 (Online) Journal homepage: www.tandfonline.com/journals/icmo20\nTele-patient-reported outcome measures\n(telePROM) in follow-up of endometriosis: a\nvalidity and test-retest reliability study of an\nendometriosis-specific questionnaire (EQ)\nMaria Monberg Feenstra, Anne Sidenius, Charlotte Nielsen, Simon Bang\nKristensen & Martin Rudnicki\nTo cite this article: Maria Monberg Feenstra, Anne Sidenius, Charlotte Nielsen, Simon Bang\nKristensen & Martin Rudnicki (2025) Tele-patient-reported outcome measures (telePROM)\nin follow-up of endometriosis: a validity and test-retest reliability study of an endometriosis-\nspecific questionnaire (EQ), Current Medical Research and Opinion, 41:2, 307-316, DOI:\n10.1080/03007995.2025.2470749\nTo link to this article:  https://doi.org/10.1080/03007995.2025.2470749\n© 2025 The Author(s). Published by Informa\nUK Limited, trading as Taylor & Francis\nGroup\nView supplementary material \nPublished online: 03 Mar 2025.\n Submit your article to this journal \nArticle views: 1312\n View related articles \nView Crossmark data\n Citing articles: 2 View citing articles \nFull Terms & Conditions of access and use can be found at\nhttps://www.tandfonline.com/action/journalInformation?journalCode=icmo20\n\nRESEARCH ARTICLE                                            \nTele-patient-reported outcome measures (telePROM) in follow-up of \nendometriosis: a validity and test-retest reliability study of an endometriosis- \nspecific questionnaire (EQ)\nMaria Monberg Feenstra\na ,b \n, Anne Sidenius\nc \n, Charlotte Nielsen\nb, d,e \n, Simon Bang Kristensen\nf ,g \nand \nMartin Rudnicki\na ,b \na\nDepartment of Gynecology and Obstetrics, Odense University Hospital, Odense, Denmark; \nb\nResearch Unit of Gynaecology and Obstetrics, \nDepartment of Clinical Research, Faculty of Health Sciences, University of Southern Denmark, Odense, Denmark; \nc\nDepartment of Prevention, \nHealth Promotion & Community Care, Copenhagen University Hospital – Steno Diabetes Center Copenhagen, Herlev, Denmark; \nd\nResearch \nUnit for Plastic Surgery, Odense University Hospital, Odense, Denmark; \ne\nDepartment of Oral and Maxillofacial Surgery, Odense University \nHospital, Odense, Denmark; \nf\nOPEN – Open Patient data Explorative Network, Odense University Hospital, Odense, Denmark; \ng\nDepartment of \nPublic Health, Aarhus University, Aarhus, Denmark \nABSTRACT \nObjective: Patient-Reported Outcome (PRO) measures supported by a severity algorithm may serve as \na decision aid for triage and consultation in follow-up of patients with endometriosis. In a new follow- \nup regime, patients filled out an endometriosis-specific questionnaire (EQ) at home before outpatient \nconsultation (tele-Patient-Reported Outcome Measures; telePROM). A severity algorithm was assigned \npatients’ answers using a color code thereby reflecting the need of clinical attention. Our study aimed \nto assess the test-retest reliability of the severity algorithm and of the single items as well as to evalu -\nate the face- and content validity of the EQ.\nMethods: The study was carried out in a referral endometriosis clinic at a Danish University Hospital. \nThe validation was based on an initial version of the EQ, which was adjusted simultaneously with its \nseverity algorithm, to meet the purpose of this study. Reliability was assessed by a test-retest setting \nof the questionnaire including patients with endometriosis, � 18 years and Danish speaking. Kappa \nstatistics and interclass correlation analyses were applied to assess test-retest reliability. Face- and con -\ntent validity was explored by focus group interviewing of patients.\nResults: In total, 14 patients answered the questionnaire twice. Results indicate that the EQ demon -\nstrated substantial reliability in three out of five domain indicators in the severity algorithm and 65% \nof items with kappa values above 0.60. Further, focus-group interview of five patients resulted in add -\ning an open-ended question regarding important issues to discuss at the consultation.\nConclusion: TelePROM in outpatient follow-up of endometriosis is feasible as patients viewed the \nquestionnaire relevant for their clinical follow-up. Yet, due to the small sample size results should be \ninterpreted with caution. Further validation of the EQ is recommended.\nARTICLE HISTORY \nReceived 12 July 2024 \nRevised 12 February 2025 \nAccepted 17 February 2025 \nKEYWORDS \nPatient reported outcome \n(PRO); endometriosis; \noutpatient; reliability; \nvalidity   \nIntroduction\nEndometriosis is a painful, chronic disease that affects up till \n10% of women in their reproductive age\n1\n. Endometriosis is \ndefined as the presence of endometrial tissue outside the \nuterus\n2\n, no cure exists and treatment options are both med -\nical and surgical\n3\n; surgery depending on severity and/or \nsymptom burden. Due to the chronic aspect the disease \nrequires repetitive standardized consultations and so far, \nsparse is available regarding the optimal follow-up schedule\n4\n. \nMoreover, there is a growing awareness that follow-up \nshould be more influenced by the patient’s own perception \nof symptoms thereby focusing on symptom relief rather than \nsurgery, for which recurrence rates are high (40–50% in \n5 years)\n5 , 6\n. The European Society of Human Reproduction \nand Embryology’s Guideline on Endometriosis underlines \nthat follow-up should be further individualized based on pre -\nvious and current treatments, and severity of the disease and \nsymptoms\n4\n.\nPatient-reported outcome (PRO) measures is a well-known \ntool to collect data about the patient’s health status reported \ndirectly from the patient\n7 , 8\n, and thereby a potential valuable \ninstrument for individualized follow-up. When filled in \nremotely, it is defined as telePRO\n9\n. PRO measures are widely \nused in several settings, e.g. epilepsy, oncology and \nCONTACT Maria Monberg Feenstra \n maria.feenstra@rsyd.dk \n Department of Gynecology and Obstetrics, Odense University Hospital; Research Unit of \nGynaecology and Obstetrics, Department of Clinical Research, Faculty of Health Sciences, University of Southern Denmark, Odense, Denmark \nSupplemental data for this article can be accessed online at https://doi.org/10.1080/03007995.2025.2470749. \n� 2025 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group  \nThis is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivatives License (http://creativecommons.org/licenses/by-nc-nd/4.0/), \nwhich permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited, and is not altered, transformed, or built upon in any way. \nThe terms on which this article has been published allow the posting of the Accepted Manuscript in a repository by the author(s) or with their consent. \nwww.cmrojournal.com\nCURRENT MEDICAL RESEARCH AND OPINION \n2025, VOL. 41, NO. 2, 307–316 \nhttps://doi.org/10.1080/03007995.2025.2470749 \nArticle ST-0706/2470749 \n\npregnancy\n10 , 11\n. Potential advantages of PRO measures are \nbetter monitoring of treatment on patients’ health\n12\nand \nmore tailored dialogues\n12 , 13\n. PRO may increase symptom \nmanagement actions in patients and healthcare professionals \n(HCPs)\n14\n, early diagnosis\n15\n, and potentially better treatment \nof symptoms and unmet needs through, e.g. medication, \nself-management strategies and insight in the current state \nof the disease\n16 , 17\n. In addition, PRO measures may support \nHCPs as a decision support-tool, since patients can assess \ntheir need of contact with the healthcare system and vice \nversa through real-time monitoring of symptoms by use of \nPRO\n10 , 18\n.\nCurrently, very limited data exist regarding the use of PRO \nmeasures in follow-up of patients with endometriosis in an \noutpatient setting. In a systematic literature review by \nNicolas-Boluda et al.\n19\n, they evaluated PRO measures for use \nin endometriosis care, however, not identifying any current \nuse of PRO measures in routine clinical care. Yet patients \nfind patient-initiated follow-up in endometriosis valuable in \ncontrast to fixed controls\n20\n. As seen before, PRO measures \nsupported by a severity algorithm could aid in clinical deci -\nsion-making during triage and consultations\n9\n.\nPRO measures including a severity algorithm can be use -\nful and incorporated into general clinical practice and func -\ntion as a tool for HCPs during patient visits, but it is crucial \nthat the collected data are precise in order to assess the \naccurate disease or treatment burden\n21\n, as invalid outcomes \ncould pose a risk to patient safety and treatment effective -\nness. Validity and reliability constitute crucial psychometric \nattributes of an instrument to ensure measurements are \naccurate and dependable\n22 , 23\n.\nThe endometriosis-specific questionnaire (EQ) presented \nin this study including a severity algorithm was originally \ndeveloped prior to this study to monitor patients’ health sta -\ntus and need of outpatient follow-up. The algorithm was \napplied to determine whether the contact should be with a \nphysician or a nurse and with a focus on strengthening per -\nsonalized dialogue between patients and HCPs\n24\n. The pre -\nsent algorithm employs three colors: Red (signifying a \nsignificant problem encountered by the patient regarding \nthe item); Yellow (indicating a mild to moderate problem \nexperienced by the patient); and Green (denoting the \nabsence of any problems within the item as reported by the \npatient). However, the EQ and algorithm was never \nvalidated.\nObjective\nThe objective of this study was to evaluate the test-retest \nreliability of the severity algorithm used for triage, along \nwith single items of patients’ symptoms and general well- \nbeing, collectively forming the endometriosis-specific ques -\ntionnaire (EQ). Further, we aimed to evaluate the face- and \ncontent validity of the endometriosis-specific question -\nnaire (EQ).\nMethods\nDevelopment and preparation phase\nThe development of a first-version of the EQ and its severity \nalgorithm was done in close collaboration between patients \nand HCPs representing six different Danish hospitals from \nboth regional and University Hospitals, representing four dif -\nferent Danish regions. The initial development of the EQ was \nfacilitated by AmbuFlex, Center for Patient Reported \nOutcomes, in the Region of Central Denmark and consisted \nof both observations of consultations, interviews with \npatients, consensus meetings, and, finally, pilot testing in \nnine patients with endometriosis. Observation of patients’ \nconsultations and interviews were carried out to identify \nsymptoms and problems discussed during consultations. \nInitial questions were discussed in terms of relevance and \nunderstandability. The first-version of the EQ was then tested \nin a small group of patients to evaluate patients understand -\ning of each question, the overall questionnaire, and its \nintended purpose. Variation in the target group was sought \nby age and location of endometriosis. The methods of “Think \naloud” and “Verbal probing”\n25\nwas used to conduct cognitive \nsemi-structured interviews throughout the development pro -\ncess and analysis of interviews was inspired from thematic \nanalysis\n26\n. Prior to this study, AmbuFlex reported in-depth \ndescriptions of the initial development process elsewhere in \nDanish\n26 , 27\n, but their methods are reported internationally\n9\n. \nHowever, the EQ was never validated.\nIn our validation study, the EQ and its severity algorithm \nwas reviewed and further adjusted with input from four \ngynecologists specialized in endometriosis, along with feed -\nback from one patient. All were affiliated with the single \nstudy site; a tertiary referral endometriosis outpatient clinic \nat a Danish University Hospital. In addition, the app named \n“My Hospital” (in Danish: “Mit Sygehus”) was chosen for dis -\ntribution of the EQ. A web-version (www.mitsygehus.dk) was \nalso available. The patient pilot tested the final and digital \nversion of the EQ before validation of the EQ.\nContent of the endometriosis-specific questionnaire (EQ)\nRelevant items were either adopted from previous validated \nquestionnaires, or developed when no relevant instruments \nor items were identified. Questions about pain during peri -\nods, pain in between periods, and abdominal pain were \ndeveloped by AmbuFlex. Pain during defecation, when uri -\nnating or breathing were adopted from Riiskjær et.al.\n28\nAll \nwere measured by a numeric rating scale (NRS) from 0 to 10, \nwhere 0 indicates no pain and 10 reflects worst pain ever\n29\n. \nSymptoms reflecting feeling bloated, obstipated, having diar -\nrhea, passing blood in stool, having nausea, vomiting, prob -\nlems with urinating, frequent urination, and limitations in \nregard to work or daily activities were inspired from the \nEuropean Organization for Research and Treatment (EORTC) \nquality of life questionnaires\n30\n, since the questions are con -\nsistent with symptoms reported among patients with endo -\nmetriosis. Questions about infertility and sexual function \n(including dyspareunia) were adapted from the disease- \n308\n M. M. FEENSTRA ET AL.\n\nspecific questionnaire Endometriosis Health Profile (EHP- \n30)\n31\n. WHO well-being index (WHO-5) was used to measure \nmental well-being\n32\n, and one question from the question -\nnaire RAND Short-Form 36 (SF-36) was included in regard to \npatients’ perceptions of their health in general\n33\n.\nFurther, questions added to the EQ were related to medi -\ncation use (hormones, painkillers, and laxatives) and blood in \nthe urine. Likewise, symptoms related to pain in lower back \nor back, shoulder, and legs, as well as two items regarding \nthe number of days required to stay at home or in bed were \ncreated. Finally, two items were developed regarding part -\nner’s concern for their relationship with the patient and \nunderstanding of the patient’s challenges. Overall, the EQ \nconsisted of 54 questions in total (Supplementary Material A). \nNot all needed to be filled out, only those relevant for the \npatient.\nFive domain indicators were created, consolidating all \nitems in the EQ, and presenting an organized overview of \nthe patient’s problems by the following categories: Pain con -\ncerning menstrual bleeding (“Pain bleeding”), Pain concern -\ning breathing, urinating or defecation (“Pain other”), Other \nSymptoms including sexuality and infertility (“Symptoms”), \nWell-being and depression (“Depression”), and Need of \nContact (“Contact”).\nAdjustment of the severity algorithm\nThe purpose of the EQ was to support triage and dialogue in \noutpatient follow-up care of patients with endometriosis. The \nseverity algorithm needed to be further developed for this \npurpose. Answers to new questions added to the original EQ \nwere assigned a color by the team of endometriosis special -\nists at the study site. Each domain indicator was colored by \nthe item within the domain representing the worst color \ncode (e.g. red). Text summaries to support triage of patients \nwere developed by creating a flowchart visualizing the \ndomain indicator (worst color code of any symptom) com -\nbined with the domain indicator “Contact” reflecting the \npatient’s need of contact. A heat-map of an anonymized \ndataset guided this process. The severity algorithm could \nboth be displayed per item, per domain indicator, and as a \ntext summary for the total PRO measures; generating a sug -\ngestion as to whether the patient should see a physician or \na nurse (or displays no need to be seen at all) and selecting \nthe appropriate method (attendance at the clinic, video- or \nphone consultation). Visualization of the EQ in the system \n“My Hospital” can be seen in Supplementary Material B.\nClinical purpose of the endometriosis-specific \nquestionnaire (EQ)\nIntroducing the EQ for use in general routine outpatient follow- \nup of all patients with endometriosis implied that every patient \nbooked for a follow-up consultation received the EQ automatic -\nally through the app “My Hospital”. The EQ was available to be \nanswered 14 days prior to their consultation. All responses were \nstored in the medical record and a graphical PRO-overview of \nthe patient responses was generated. Further, responses could \nbe displayed over time (See Supplementary Material B). In add -\nition, patients could contact the clinic through a message-func -\ntion in the app, requesting consultation. This was followed by a \ndistribution of the EQ to be completed by the patient through \nthe app. After reviewing the patient’s answers, the suggestions \ngenerated by the severity algorithm, and reviewing the latest \nnotes in the electronic patient journal; the patient would be \nbooked for an outpatient consultation based on her current \nsymptom status to be consulted by either a physician or nurse, \nand by relevant method, e.g. attendance at the clinic, video- or \nphone. Nurses specialized in endometriosis were responsible \nfor reviewing patients’ answers ensuring relevant consultation \nfor the patient. The new PRO-based follow-up regime was \nnamed tele-patient-reported outcome measures (telePROM).\nStudy population\nPatients diagnosed with endometriosis from the study site \nwere invited to practice video consultations with the HCPs as a \nnew initiative. In addition, patients should fill in the EQ for the \nHCPs to practice using the EQ for dialogue support. Patients \nwere screened for eligibility by a gynecologist specialized in \nendometriosis. Inclusion criteria were diagnosis of endometri -\nosis registered in the electronic patient journal, age � 18 years \nand, in a stable period of endometriosis as they should not \nneed a physical examination. Exclusion criteria was patients, \nwho were not able to speak or read Danish. Patients for this \nstudy were recruited after their video consultation, which all \nwere conducted in September 2019 as part of testing the new \ninitiative of video consultations.\nAll patients that fulfilled inclusion criteria were provided \nwith written information about the study before enrolling. \nAdditional verbal information was provided when needed. The \ninvitation was sent electronically through “e-Boks”\n34\nwhich is a \nsecure electronic mailbox for all citizens in Denmark.\nEvaluation of test-retest reliability\nAll patients received the questionnaire (test 1) through an \napp or web-version of “My Hospital” to be completed prior \nto the video consultation with the physician and nurse. \nPatients accepting the re-test assessment (test 2) were asked \nto answer the same questionnaire again as to assess test- \nretest reliability\n35\n. To avoid patients seeing previous answers, \nthe second questionnaire (re-test) was distributed through \nSurveyXact. Non-responders were contacted two times; first \nthrough e-Boks and secondly by telephone. No sample size \nwas calculated, as we approached all patients, striving to \nmeet the recommended number of 50 participants\n36\n.\nTest-retest reliability of domain indicators\nThe EQ severity algorithm applied to each item a color code \n(green, yellow, red) and summarized answers according to \nthe items in the five domain indicators. The primary analysis \nconcerned the five domain indicators. Test and re-test classi -\nfications were tabulated, and the reliability quantified using \nthe intraclass (IC) kappa coefficient\n37 , 38\n. Confidence intervals \nCURRENT MEDICAL RESEARCH AND OPINION\n 309\n\n95% for the IC kappa were calculated using non-parametric \nbootstrapping based on 1000 replications. Kappa values \nwere interpreted as follows �0.2 (slight), 0.21–0.4 (fair), 0.41– \n0.6 (moderate), 0.61–0.8 (substantial), 0.81–1.0 (almost per -\nfect)\n39\n. The analysis was performed by implementing the \nintra-class kappa formula from Kraemer et al.\n37\n(section 2.1). \nThe code was tested by comparing to the SAS-procedure \ngiven in Fisher et al.\n38\nand the two implementations were \nfound to coincide in a large, simulated data set. As a sensitiv -\nity analysis, the primary analyses were repeated after exclud -\ning participants, who changed their hormonal treatment \nbetween the test and re-test.\nTest-retest reliability of items\nSecondary analyses were performed on the EQ item level. \nReliability of binary items was quantified using the IC kappa \nas described above for the overall indicators. For continuous \nand ordinal items, reliability is quantified as the intra-class \ncorrelation coefficient (ICC), which was estimated from a \nmixed model. The model included no fixed effects and a ran -\ndom effect on the patient level (its variance component sig -\nnifying the between-patient variation). The ICC was \ncalculated as the ratio of between-patient variance to the \ntotal variance (between- plus within-patient variance), and a \nconfidence interval was constructed on logistic scale by the \ndelta method and then backtransformed. Model validation \nwas performed by inspection of residuals, fitted values and \nBLUP estimates of random effects. Normality assumptions \nwere found to be violated for some items. Note that for \nordinal items, the ICC is equivalent to a weighted kappa \nusing quadratic weights\n40\n.\nEvaluation of face- and content validity\nWe evaluated the face- and content validity of the EQ as \nwell as intended use and relevance by performing a focus \ngroup interview (FGI) with a sub-sample of patients, who \nhad answered both questionnaires. A semi-structured inter -\nview guide was developed to explore the relevance of the \nquestions in the EQ, the comprehensiveness as to no impor -\ntant aspects of the construct should be missing, and finally \nthe comprehensibility as to the questions should be under -\nstood by patients as intended\n23\n. Additional questions regard -\ning intended use of the questionnaire in clinical practice and \ndistribution options were added. The interview guide is pro -\nvided in Supplementary Material C.\nBefore beginning the interview, patients read the ques -\ntionnaire again (paper version). The interview was carried out \nby the first author in November 2019, it lasted 58 min, was \naudio-recorded and verbatim transcribed. First-author had \nsome experience in interviewing patients, and a nurse col -\nleague observed the participants during the interview as to \nnon-verbal communication and if everyone seemed to \nexpress their opinion freely. Afterwards the first-author and \nthe nurse evaluated the overall quality of the interview \nbased on the observations. Final pilot testing on patients \nfrom the endometriosis outpatient clinic was planned to \nreassess face- and content validity regarding any adjustments \nmade based on the results of the FGI.\nOther analyses\nDescriptive summaries of baseline variables are presented as \nmedian with interquartile range (IQR) for continuous variables \nand as numbers (%) with percentage of total for categorical \nvariables. The Consensus-based Standards for the Selection of \nhealth Measurement Instruments (COSMIN) Risk of bias check -\nlist was used for content validity and reliability\n41–43\n. All analysis \nwere performed by Stata 17 (StataCorp. 2021. Stata Statistical \nSoftware: Release 17. College Station, TX: StataCorp LLC).\nResults\nReliability\nA total of 48 out of 291 screened patients were found eli -\ngible and invited to fill in the EQ and participate in a video \nconsultation ( Figure 1 ). Overall, 26 patients accepted and \nfilled in the first questionnaire (test 1). Two patients were \nunable to attend their video consultation, resulting in 24 \npatients invited for this study. A total of 14 patients \nanswered both the test and re-test questionnaire and were \nincluded for data analysis. Questionnaires were answered \nwith a median duration of 34 days (IQR 13 days) and with a \nrange between patients’ answers of 27 days to 50 days. \nPatients’ characteristics at baseline are displayed in Table 1 .\nKappa values estimates for the domain indicators in the \nEQ ranged from slight to almost perfect\n39\n( Figure 2 ). Except \nfor the domain indicators “Contact” with a kappa value of \n−0.01, 95% CI (-0.35–0.69), all disclosed kappa value esti -\nmates above 0.41, 95% CI (-0.05 to 1.0) categorized as mod -\nerate to almost perfect. When excluding patients, who \nreported a change in hormonal treatment (n ¼ 5), lower \nkappa value estimates between 0.30–0.60, 95% CI (-0.17–1) \nwere observed in three domain indicators (“Depression”, \n“Symptoms”, “Pain bleeding”). Domain indicators “Pain other” \nand “Contact” showed higher kappa value estimates of 0.30 \nand 1, 95% CI (-0.17–1) ( Figure 3 ).\nKappa value estimates of single items in the EQ are illus -\ntrated in Figure 4 . Low kappa value estimates were observed \nin relation to questions about reporting blood in the urine \n(0.00) or in the feces (0.143) and to having experienced pain \nwhile bleeding during the last four weeks (-0.12). In total, \n65% of items showed substantial reliability as kappa values \nwere 0.61 or above.\nFace- and content validity\nA total of six patients accepted to participate in a FGI. One \npatient was unable to participate at the day of the FGI, leav -\ning five participants, aged 25–47. Time of diagnosis varied \nfrom within the last year to 9–10 years ago. Patients were \ngenerally satisfied with the questionnaire and its content. It \nwas easy to fill out digitally, and notifications were a relevant \nmethod for distribution of reminders to fill out the EQ. A \n310\n M. M. FEENSTRA ET AL.\n\ntotal of seven answer categories were modified. A single \nopen-ended question was added asking patients “Is there \nanything else that is important for you to talk about at the \nconsultation?” . Patients stressed the importance of HCPs \nusing the EQ actively during consultation and having a clear \npurpose of using the tool. The full results of the FGI are dis -\nplayed in Supplementary Material D. Afterwards, the EQ was \npilot tested in two rounds at the study site with, respectively, \nsix and eight patients with endometriosis. Minor editorial \nadjustments were made, and face- and content validity was \nfound acceptable for implementation in the endometriosis \noutpatient clinic.\nDiscussion\nStudies of PRO measures for routine follow-up of endometri -\nosis are sparse. This study shows that the EQ had acceptable \nface- and content validity amongst Danish patients with \nFigure 1. Inclusion of study participants.\nCURRENT MEDICAL RESEARCH AND OPINION\n 311\n\nendometriosis, and that the EQ demonstrated substantial reli -\nability estimates in three out of five domain indicators as well \nas 65% of items with kappa value estimates above 0.60. Yet, \nthe wide confidence intervals (CI) indicate that results should \nbe interpreted with caution due to the small sample size and \nthat no confirmatory conclusions as to the test-retest reliability \nof the EQ can be made.\nPRO measures during endometriosis follow-up\nFace- and content validity is the most important property of \nan instrument\n23\n. Our study pointed out that patients found \nthe questions relevant as well as appropriate for use during \noutpatient follow-up. Several items are inspired from already \nvalidated instruments; e.g. the NRS is relevant for pain meas -\nurement in endometriosis\n29\n, Rand SF-36 and its psychometric \nproperties are well documented\n44\nand EHP-30 has been vali -\ndated in Danish\n45\n. Patients addressed the need of the possi -\nbility to add further information unique for the individual \npatient, e.g. that bleeding persists even after a hysterectomy; \nreflecting that some items needed a possibility for further \nelaboration with a focus on individual considerations. \nCollectively, these findings contribute to the groundwork, \nserving as the basis for future adaptation and integration of \nthe EQ in outpatient follow-up of endometriosis.\nTest-retest of a hormone-dependent disease\nAs endometriosis is a hormone-dependent disease, symptoms \nmight vary during a short period of time, making it complex to \nensure a stable study population in the test-retest period. In a \ntest-retest study among patients with endometriosis\n45\n, an inter -\nval of more than four weeks (i.e. more than a menstrual period) \nwas chosen with our study having a range between patients’ \nanswers of 27 days to 50 days.\nOur results might reflect patients had a benefit of receiving \na video-consultation during the test-retest period as not all \ndomain indicators and items demonstrated fair kappa values. \nNegative kappa values could indicate that the study population \nhad been too homogeneous as to all having symptoms from \nendometriosis in the same degree, and the study population \nnot reflecting much variance in disease severity. This would \naffect the kappa value. In respect to patients’ need of contact \nwith the outpatient clinic, seven out of ten patients did not \nreport having a need for contact in test 2. This result might \nreflect the fact that patients had a video consultation with an \nHCP between test 1 and test 2, and the study design could \nTable 1. Participants’ characteristics at time of test 1, N ¼ 14.\nAge, median (IQR) 40.6 (18.5)\nHormonal treatment\na\n, n (%) 11 (78.57)\nHormone spiral (Mirena, Kyleena or Jaydess) 5 (35.71)\nMini-pills 4 (28.57)\nBirth control pills 3 (21.43)\nAnti-hormones (e.g. Zoladex) 3 (21.43)\nEstrogen (for example together with hormones) 2 (14.29)\nAlgorithm domain indicators in test 1\nPain bleeding, n (%)\nGreen\nb\n6 (42.86)\nYellow\nb\n4 (28.57)\nRed\nb\n4 (28.57)\nPain other, n (%)\nGreen\nb\n4 (28.57)\nYellow\nb\n4 (28.57)\nRed\nb\n6 (42.86)\nSymptoms, n (%)\nGreen\nb\n0 (0)\nYellow\nb\n4 (28.57)\nRed\nb\n10 (71.43)\nDepression, WHO-5, n (%)\nGreen\nb\n4 (28.57)\nYellow\nb\n10 (71.43)\nRed\nb\n0 (0)\nContact, n (%)\nGreen\nb\n4 (28.57)\nYellow\nb\n0 (0)\nRed\nb\n10 (71.43)\nDepression, WHO-5, median (IQR) 48 (44)\nPatient perceived health (SF-36), n (%)\nExcellent/very good 4 (28.57)\nGood 3 (21.43)\nFair/poor 7 (50)\na\nPatients could report more than one treatment option.\nb\nGreen indicating no problem, yellow indicating mild to moderate problem \nand red indicating significant problem.\nFigure 2. Test-retest reliability of domain indicators.\n312\n M. M. FEENSTRA ET AL.\n\nhave caused a bias in patients feeling better supported at test \n2 due to the video consultation with a HCP.\nUse of PRO measures in clinical practice\nIn this study, patients underlined the importance of knowing \nthe actual purpose of the questionnaire. Like other studies\n46 , 47\n, \npatients advocated for HCPs to use their answers actively dur -\ning follow-up consultations. To do so, special attention must \nbe given to both patients and HCPs when implementing \ntelePROM in outpatient follow-up. A special focus should be \non information and training in using the EQ. When the EQ sup -\nported by the severity algorithm is used as a decision aid for \npatients’ actual need and choice of consultation method, it \nshould not stand alone but include additional data from the \nelectronic patient journal in evaluating the patient’s actual dis -\nease burden\n48\n. It adds to the safety of the initiative, as each \npatient is thoroughly evaluated by combining relevant data \nfrom different sources, which should be described in local \nguidelines to support HCPs during implementation and in \ndaily clinical practice. Further, if PRO measures are to be used \nfor clinical research it is important to acknowledge that PRO \nmeasures represent a snapshot of patients’ experiences, which \ncan fluctuate rapidly due to the hormonal influence upon \nendometriosis as a disease. Moreover, awareness of the data \ncollection timing should be considered as it may influence the \ndata, whether it is collected, e.g. before, on the day of consult -\nation or in the following days.\nFigure 3. Test-retest reliability of domain indicators sens hormones.\nFigure 4. Test-retest reliability of single items.\nCURRENT MEDICAL RESEARCH AND OPINION\n 313\n\nLimitations\nThere are limitations to consider in this study. The study \npopulation represents only Danish-speaking patients as the \nquestionnaire was in Danish, and thus; for use in other lan -\nguages the questionnaire needs to be translated and \nvalidated according to relevant COSMIN guidelines. Today, \nnon-Danish speaking patients have the possibility to bring a \nrelative to translate and/or fill in the questionnaire in the \napp but this study cannot be valid nor feasible regarding \nsuch patients. The participant rate was below COSMIN rec -\nommendations of a minimum of 50 patients as sample \nsize\n36\n, wherefore there could be a risk of selection bias and \nthus a reduced generalization of the results. Reasons for \ndropouts were not recorded, but may be influenced by the \nfact that video consultation took place on a Saturday.\nMore research is needed to further validate and test the \nreliability of the instrument in patients with endometriosis. \nTo determine change over time, each construct or item \ncould be provided with a filter question as to whether there \nhas been any changes since the last questionnaire was filled \nout. In telePROM, additional information about the patient \nwere available in the follow-up procedure; e.g. the patient’s \nelectronic journal, messages in-between consultations with \nthe endometriosis team sent through the app as well as \npatient’s own assessment of the need of a consultation self- \nreported in the EQ. Therefore, patients’ PRO measures \nbecome an integrated part of different data sources used for \ntriage of the patient. Finally, the EQ has been further revised \nsince data collecting in 2019, why more tests of validity and \nreliability are recommended, which are also of great impor -\ntance of improving data quality\n49\n.\nConclusion\nOur study is one of the first to test psychometric properties of \na Danish EQ for use in routine outpatient follow-up. Results \ndemonstrated an acceptable face- and content validity of the \nEQ, and patients reported the instrument as relevant during \noutpatient follow-up. Implementation is feasible as the EQ \nwould be supplemented by several data sources, when decid -\ning upon patients’ need and type of consultation. Due to the \nsmall sample size further psychometric validation of the EQ and \nin a broader sample of patients is recommended.\nTransparency \nDeclaration of funding \nThis study is funded by the Department of Clinical Research, University \nof Southern Denmark and Department of Gynecology and Obstetrics, \nOdense University Hospital. The funding body had no role in the design \nof the study, choice of methods, data collection, analysis or reporting of \nthe results. \nDeclaration of financial/other relationships\nThe authors have no relevant affiliations or financial involvement with \nany organization or entity with a financial interest in or financial conflict \nwith the subject matter or materials discussed in the manuscript. This \nincludes employment, consultancies, honoraria, stock ownership or \noptions, expert testimony, grants or patents received or pending, or roy -\nalties. Peer reviewers on this manuscript have no relevant financial or \nother relationships to disclose.\nAuthor contributions\nThe first- and last author contributed to the study conception, design, \nmaterial preparation and data collection. Data analysis were performed \nby Simon Bang Kristensen, Maria Monberg Feenstra and Martin Rudnicki. \nThe first draft of the manuscript was written by Maria Monberg Feenstra \nand all authors commented on previous versions of the manuscript. All \nauthors read and approved the final manuscript.\nAcknowledgements\nWe would like to send a special thank you to the patients participating \nin this study and providing us their unique perspectives and valuable \ndata on symptoms and general well-being. An acknowledgement to Dr \nPoul Bak Thorsen, MD, PhD for skillful assistance and discussion during \nthe preparation phase and data collection of this study. Lastly, thank \nyou to all funding organizations contributing to the realization of this \nstudy.\nData availability statement\nThe data supporting the findings of this study are not publicly available \ndue to restrictions, e.g. patients have not consented hereto, and data \ncontain personal information.\nEthical statement\nApproval from the Data Protection Agency of the Region of Southern \nDenmark was granted (jrn.nr 20/35407) for this study. Approval from the \nlocal Ethics Committee was not required according to Danish law. All \npatients in this study gave informed consent to participate.\nORCID\nMaria Monberg Feenstra \n http://orcid.org/0000-0002-3645-0543 \nAnne Sidenius \n http://orcid.org/0000-0002-8680-8769 \nCharlotte Nielsen \n http://orcid.org/0000-0002-8920-7758 \nSimon Bang Kristensen \n http://orcid.org/0000-0002-5533-6922 \nMartin Rudnicki \n http://orcid.org/0000-0002-5115-1445 \nReferences\n0[1] Horne AW, Saunders PTK. SnapShot: endometriosis. Cell. 2019; \n179(7):1677–1677.e1. doi: 10.1016/j.cell.2019.11.033.\n0[2] Kennedy S, Bergqvist A, Chapron C, et al. ESHRE guideline for the \ndiagnosis and treatment of endometriosis. Hum Reprod. 2005; \n20(10):2698–2704. doi: 10.1093/humrep/dei135.\n0[3] Lemaire GS. More than just menstrual cramps: symptoms and \nuncertainty among women with endometriosis. J Obstet Gynecol \nNeonatal Nurs. 2004;33(1):71–79. doi: 10.1177/0884217503261085.\n0[4] European Society of Human Reproduction and Embryology \nGuideline Development Group. Endometriosis. Guideline of \nEuropean Society of Human Reproduction and Embryology; 2022 \n[cited 2024 Jul 4]. Available from: https://www.eshre.eu/ \nGuidelines-and-Legal/Guidelines/Endometriosis-guideline.\n0[5] Horne AW, Missmer SA. Pathophysiology, diagnosis, and manage -\nment of endometriosis. BMJ. 2022;379:e070750. doi: 10.1136/bmj- \n2022-070750.\n314\n M. M. FEENSTRA ET AL.\n\n0[6] Saunders PTK, Horne AW. Endometriosis: etiology, pathobiology, \nand therapeutic prospects. Cell. 2021;184(11):2807–2824. doi: 10. \n1016/j.cell.2021.04.041.\n0[7] US Department of Health and Human Services Food and Drug \nAdministration. Guidance for industry: patient-reported outcome \nmeasures: use in medical product development to support label -\ning claims; 2009 [cited 2023 Sep 9]. Available from: https://www. \nfda.gov/media/77832/download.\n0[8] de Vet Ht CB, Mokkink BL, Knol DL. Measurement in medicine. \n1st ed. New York Cambridge University Press; 2011.\n0[9] Hjollund NHI, Larsen LP, de Thurah AL, et al. Patient-reported \noutcome (PRO) measurements in chronic and malignant diseases: \nten years’ experience with PRO-algorithm-based patient-clinician \ninteraction (telePRO) in AmbuFlex. Qual Life Res. 2023;32(4): \n1053–1067. doi: 10.1007/s11136-022-03322-9.\n[10] Videns-og Kompetencecenter for Brugerinddragelse i \nSundhedsvæsenet (ViBIS). Program PRO. Anvendelse af PRO-data \ni kvalitetsudviklingen af det danske sundhedsvæsen – anbefalin -\nger og vidensgrundlag. [Program PRO. Use of PRO data in the \nquality development of the Danish healthcare system – recom -\nmendations and knowledge base]; 2016 [cited 2024 July 4]. \nAvailable from: https://danskepatienter.dk/sites/danskepatienter. \ndk/files/media/Publikationer%20-%20Egne/B_ViBIS/A_Rapporter% \n20og%20unders%C3%B8gelser/program_pro-rapport.pdf.\n[11] Chen A, V €ayrynen K, Schmidt A, et al. The impact of implement -\ning patient-reported measures in routine maternity care: a sys -\ntematic review. Acta Obstet Gynecol Scand. 2022;101(11):1184– \n1196. doi: 10.1111/aogs.14446.\n[12] Valderas JM, Kotzeva A, Espallargues M, et al. The impact \nof measuring patient-reported outcomes in clinical practice: a \nsystematic review of the literature. Qual Life Res. 2008;17(2):179– \n193. doi: 10.1007/s11136-007-9295-0.\n[13] Ishaque S, Karnon J, Chen G, et al. A systematic review of rando -\nmised controlled trials evaluating the use of patient-reported out -\ncome measures (PROMs). Qual Life Res. 2019;28(3):567–592. doi: \n10.1007/s11136-018-2016-z.\n[14] Howell D, Molloy S, Wilkinson K, et al. Patient-reported outcomes \nin routine cancer clinical practice: a scoping review of use, impact \non health outcomes, and implementation factors. Ann Oncol. \n2015;26(9):1846–1858. doi: 10.1093/annonc/mdv181.\n[15] Ballard KD, Seaman HE, de Vries CS, et al. Can symptomatology \nhelp in the diagnosis of endometriosis? Findings from a national \ncase-control study–Part 1. BJOG. 2008;115(11):1382–1391. doi: 10. \n1111/j.1471-0528.2008.01878.x.\n[16] Lavallee DC, Chenok KE, Love RM, et al. Incorporating patient- \nreported outcomes into health care to engage patients and \nenhance care. Health Aff (Millwood). 2016;35(4):575–582. doi: 10. \n1377/hlthaff.2015.1362.\n[17] Lancaster K, Abuzour A, Khaira M, et al. The use and effects of \nelectronic health tools for patient self-monitoring and reporting \nof outcomes following medication use: systematic review. J Med \nInternet Res. 2018;20(12):e294. doi: 10.2196/jmir.9284.\n[18] Aiyegbusi OL, Hughes SE, Peipert JD, et al. Reducing the pres -\nsures of outpatient care: the potential role of patient-reported \noutcomes. J R Soc Med. 2023;116(2):44–64. doi: 10.1177/ \n01410768231152222.\n[19] Nicolas-Boluda A, Oppenheimer A, Bouaziz J, et al. Patient- \nreported outcome measures in endometriosis. J Clin Med. 2021; \n10(21):5106. doi: 10.3390/jcm10215106.\n[20] Chainrai M, Kershaw VF, Gray TG, et al. Patient-initiated follow-up \nin gynaecology: patient and clinician views. Eur J Obstet Gynecol \nReprod Biol. 2024;298:18–22. doi: 10.1016/j.ejogrb.2024.04.032.\n[21] Kottner J, Audig �e L, Brorson S, et al. Guidelines for reporting reli -\nability and agreement studies (GRRAS) were proposed. J Clin \nEpidemiol. 2011;64(1):96–106. doi: 10.1016/j.jclinepi.2010.03.002.\n[22] Mokkink LB, Boers M, van der Vleuten CPM, et al. COSMIN risk of \nbias tool to assess the quality of studies on reliability or measure -\nment error of outcome measurement instruments: a Delphi study. \nBMC Med Res Methodol. 2020;20(1):293. doi: 10.1186/s12874-020- \n01179-5.\n[23] Terwee CB, Prinsen C, Chiarotto A, et al. COSMIN methodology \nfor assessing the content validity of PROMs–user manual. \nAmsterdam: VU University Medical Center; 2018.\n[24] Region of Central Denmark. Endometriose [Endometriosis]. 2024. \nhttps://www.fagperson.sundhed.rm.dk/til-ansatte-og-samarbejd -\nspartnere/ambuflex/pro-losninger/kvindesygdomme-og-fodsler/ \nendometriose/.\n[25] Beatty PC, Willis GB. Research synthesis: the practice of cognitive \ninterviewing. Public Opinion Quarterly. 2007;71(2):287–311. doi: \n10.1093/poq/nfm006.\n[26] Ambuflex. Test af spørgeskema om endometriose [Test of endo -\nmetriosis questionnaire]; 2019. p. 1–45.\n[27] Ambuflex. Spørgeskema om endometriose. Patientinddragelse \n[Questionnaire about endometriosis. Patientinvolvement]; 2018. \np. 1–17.\n[28] Riiskjær M, Egekvist AG, Hartwell D, et al. Bowel endometriosis \nsyndrome: a new scoring system for pelvic organ dysfunction \nand quality of life. Hum Reprod. 2017;32(9):1812–1818. doi: 10. \n1093/humrep/dex248.\n[29] Bourdel N, Alves J, Pickering G, et al. Systematic review of endo -\nmetriosis pain assessment: how to choose a scale? Hum Reprod \nUpdate. 2015;21(1):136–152. doi: 10.1093/humupd/dmu046.\n[30] Aaronson NK, Ahmedzai S, Bergman B, et al. The European \nOrganization for Research and Treatment of Cancer QLQ-C30: a \nquality-of-life instrument for use in international clinical trials in \noncology. J Natl Cancer Inst. 1993;85(5):365–376. doi: 10.1093/ \njnci/85.5.365.\n[31] Jones G, Kennedy S, Barnard A, et al. Development of an endo -\nmetriosis quality-of-life instrument: the Endometriosis Health \nProfile-30. Obstet Gynecol. 2001;98(2):258–264. doi: 10.1016/ \ns0029-7844(01)01433-8.\n[32] Topp CW, Østergaard SD, Søndergaard S, et al. The WHO-5 Well- \nBeing Index: a systematic review of the literature. Psychother \nPsychosom. 2015;84(3):167–176. doi: 10.1159/000376585.\n[33] Bjorner JB, Thunedborg K, Kristensen TS, et al. The Danish SF-36 \nHealth Survey: translation and preliminary validity studies. J Clin \nEpidemiol. 1998;51(11):991–999. doi: 10.1016/s0895-4356(98) \n00091-2.\n[34] Postnord Na. eBoks. 2023 [cited 2023 Nov 28]. Available from: \nhttps://private.e-boks.com/danmark/en.\n[35] Mokkink LB, Terwee CB, Patrick DL, et al. The COSMIN study \nreached international consensus on taxonomy, terminology, and \ndefinitions of measurement properties for health-related patient- \nreported outcomes. J Clin Epidemiol. 2010;63(7):737–745. doi: 10. \n1016/j.jclinepi.2010.02.006.\n[36] Mokkink LB, Prinsen CA, Bouter LM, et al. The consensus-based \nstandards for the selection of health measurement instruments \n(COSMIN) and how to select an outcome measurement instru -\nment. Braz J Phys Ther. 2016;20(2):105–113. doi: 10.1590/bjpt-rbf. \n2014.0143.\n[37] Chmura Kraemer H, Periyakoil VS, Noda A. Kappa coefficients in \nmedical research. Stat Med. 2002;21(14):2109–2129. doi: 10.1002/ \nsim.1180.\n[38] Fisher Glr DG, Neri E, Art N, et al. Measuring test-retest reliability: \nthe Intraclass Kappa. SAS Conf Proc. 2019:1–9.\n[39] Landis JR, Koch GG. The measurement of observer agreement for \ncategorical data. Biometrics. 1977;33(1):159–174. doi: 10.2307/ \n2529310.\n[40] Fleiss JL, Cohen J. The equivalence of weighted Kappa and the \nintraclass correlation coefficient as measures of reliability. Educ \nPsychol Measure. 1973;33(3):613–619. doi: 10.1177/0013164 \n47303300309.\n[41] Mokkink LB, de Vet HCW, Prinsen CAC, et al. COSMIN risk of bias \nchecklist for systematic reviews of patient-reported outcome \nmeasures. Qual Life Res. 2018;27(5):1171–1179. doi: 10.1007/ \ns11136-017-1765-4.\n[42] Prinsen CAC, Mokkink LB, Bouter LM, et al. COSMIN guideline for \nsystematic reviews of patient-reported outcome measures. Qual \nLife Res. 2018;27(5):1147–1157. doi: 10.1007/s11136-018-1798-3.\nCURRENT MEDICAL RESEARCH AND OPINION\n 315\n\n[43] Terwee CB, Prinsen CAC, Chiarotto A, et al. COSMIN methodology \nfor evaluating the content validity of patient-reported outcome \nmeasures: a Delphi study. Qual Life Res. 2018;27(5):1159–1170. \ndoi: 10.1007/s11136-018-1829-0.\n[44] Bjorner JB, Damsgaard MT, Watt T, et al. Tests of data quality, \nscaling assumptions, and reliability of the Danish SF-36. J Clin \nEpidemiol. 1998;51(11):1001–1011. doi: 10.1016/s0895- \n4356(98)00092-4.\n[45] Hansen KE, Lambek R, Røssaak K, et al. Health-related quality of \nlife in women with endometriosis: psychometric validation of the \nEndometriosis Health Profile 30 questionnaire using confirmatory \nfactor analysis. Hum Reprod Open. 2022;2022(1):hoab042. doi: 10. \n1093/hropen/hoab042.\n[46] Grove BE, Valen Schougaard LM, Ivarsen P, et al. Remote follow- \nup based on patient-reported outcomes in patients with chronic \nkidney disease: a qualitative study of patient perspectives. PLoS \nOne. 2023;18(2):e0281393. doi: 10.1371/journal.pone.0281393.\n[47] Mejdahl CT, Schougaard LMV, Hjollund NH, et al. Patient-reported \noutcome measures in the interaction between patient and clin -\nician – a multi-perspective qualitative study. J Patient Rep \nOutcomes. 2020;4(1):3. doi: 10.1186/s41687-019-0170-x.\n[48] Schougaard LMV, Friis RB, Grytnes R, et al. Exploring the nurses’ \nperspective on using remote electronic symptom monitoring in \nclinical decision-making among patients with metastatic lung \ncancer. Semin Oncol Nurs. 2023;39(6):151517. doi: 10.1016/j. \nsoncn.2023.151517.\n[49] Schougaard LM, Larsen LP, Jessen A, et al. AmbuFlex: tele- \npatient-reported outcomes (telePRO) as the basis for follow-up in \nchronic and malignant diseases. Qual Life Res. 2016;25(3):525– \n534. doi: 10.1007/s11136-015-1207-0.\n316\n M. M. FEENSTRA ET AL.","source_license":"CC0","license_restricted":false}