Online information on dysmenorrhoea: An evaluation of readability, credibility, quality and usability.

This was a descriptive study to evaluate online information on dysmenorrhea. Our study team searched, screened, and evaluated webpages on dysmenorrhea. We followed the STROBE checklist (See Supplementary File 1 ) for reporting this study. The first step of the study was to search for webpages containing health information about dysmenorrhea. Because we intended to evaluate online information to which the lay public is exposed, the aim of the search was to locate webpages as the lay public do rather than to include as many webpages as possible. To achieve this aim, we imitated common online search habits used by the lay public. We used the three most popular search engines in the United States and worldwide: Google, Yahoo, and Bing ( Statista, 2018 ). In addition, we applied the popular search terms used by the lay public to search online information on dysmenorrhea: “period pain” and “menstrual cramps” ( Chen, Groves, Miller, and Carpenter, 2018 ). We sequentially entered two search terms “period pain” and “menstrual cramps” in each search engine. After entering both search terms into each of the three search engines, six sets of webpage search results were generated. To imitate the general public’s search habits, we selected webpages from the first page of all six sets of search results. Results from the first pages were selected, because 71.33% of the public only look at the first page of search results while only 5.59% of the public look at the second and third page of search results ( Petrescu, 2014 ). The web search was conducted in January 2018 and generated 60 webpages for screening. For webpage selection, the inclusion criteria were webpages that appeared on the first page of each search, published in English, and contained relevant information about dysmenorrhea. The exclusion criteria were duplicate webpages, commercial sites or advertisements with the sole purpose of selling products, webpages that only included commentary, and webpages that did not contain any relevant information about dysmenorrhea or its treatment options. Webpages were considered irrelevant if they did not present the reader with information on the definition, causes, symptoms, or treatments of dysmenorrhea. Based on the exclusion criteria, we removed duplicate webpages (n=30), ads and commercial sites with the sole purpose of selling products (n=3), webpages that only included commentary (n=1), and webpages that did not contain any relevant information about dysmenorrhea or its treatment options (n=1). After excluding these 35 webpages, 25 met the inclusion criteria. Our study team completed the webpage evaluation using an umbrella tool. The umbrella tool was designed by combining established and reliable tools and indicators (See Table 1 for evaluation criteria, indicators and measurement tools). Tools included the Flesch Reading Ease Formula, Flesch-Kincaid Grade Formula ( Readability formulas, 2017a ), Health on the Net (HON) Code of Conduct certification seal ( Health on the net foundation, 2017 ), Sandvik tool ( Sandvik, 1999 ), DISCERN instrument ( Charnock, 1998 ), and the LIDA instrument ( Minervation, 2007 ). A systematic review of health information evaluation tools found these tools, when used in combination, comprehensively evaluate the most important criteria for webpage evaluation, such as readability, credibility, reliability, quality, and usability ( Zhang et al., 2015 ). When creating the umbrella tool, team members assessed criteria and indicators on all tools. We removed overlap among items and created the umbrella tool covering unique criteria and indicators. Each measurement tool and the included indicators is described below. The Flesch Reading Ease Formula and Flesch-Kincaid Grade Formula were used to evaluate readability based on word and sentence length ( Readability Formulas, 2017b ). The Flesch Reading Ease Formula generates a number to indicate how easily the material can be read and understood: 0 indicates the passage is very difficult and confusing to read and 100 indicates the passage can be easily read and understood. Second, the Flesh-Kincaid Grade Level Formula generates the American average grade level at which readers can understand the information. These two scores were calculated based on sentence and word length using Microsoft Word ( Readability formulas, 2017a ). Credibility was assessed using items from three tools. First, we evaluated the presence or absence of a seal representing the Health on the Net (HON) Code of Conduct Certification. The certification was created by the HON Foundation, a nonprofit organization in official relations with the World Health Organization to promote credible and transparent online health information. In order to be granted HON certification, a website must apply for evaluation by the HON Foundation. The HON evaluates webpages based on the following criteria: authoritativeness, complementarity, privacy, attribution, justifiability, transparency, financial disclosure, and sponsorship. If certified, a site will display the HON code seal ( Health on the Net Foundation, 2017 ). Second, we selected three items from the Sandvik tool to assess credibility (ownership, authorship, source of information) ( Sandvik, 1999 ). The Sandvik tool was created when Sandvik performed a study on the online health information available for female urinary incontinence. Each item is scored from 0 to 2, with higher scores indicating higher quality for that evaluation criterion. Sandvik acknowledged potential subjectivity of scoring. To promote reliability of scoring, Sandvik encouraged scoring from more than one evaluator ( Sandvik, 1999 ). Third, we used one item from the LIDA instrument to assess credibility. The LIDA instrument was produced by Minervation, a company associated with the University of Oxford. It is a reliable and validated instrument that provides a comprehensive list of indicators to assess the quality of online health information ( Minervation, 2007 ; Zhang et al., 2015 ). The item we selected determines the presence of a declaration of objectives from the site owners. This LIDA question has three response options: yes, no, or partially ( Minervation, 2007 ). Quality and reliability were assessed using items from three tools. In addition, we noted if each webpage had been updated in the last two years. First, we used the DISCERN instrument, a valid and reliable tool to assess the quality of written information on treatment choices for health problems ( Charnock, 1998 ). It was developed by an “Expert Panel” which included clinical specialists, self-help group representatives, general practitioners, a consumer health information expert, a lay medical publisher, a health journalist and consumer, and other professionals. The instrument contains 16 questions divided into three subsections. Section 1 contains 8 questions and addresses reliability of the publication as a source of information about treatment choices. Section 2 contains 7 questions focusing on the quality of information about treatment choices. Items from each section are averaged to create section scores ranging from 1 to 5, with 5 being the most positive score. Section 3 has one question addressing the overall quality of the publication. It is scored 1–5 with 1 indicating a webpage with serious shortcomings and 5 indicating minimal shortcomings ( Charnock, 1998 ). Second, we selected two items from the previously described Sandvik tool to assess currency and balance ( Sandvik, 1999 ). Each item is scored 0 to 2, with higher scores indicating more current information and greater balance of information. Third, we used eight questions from the LIDA instrument to assess quality and reliability from the perspectives of content production. Specifically, these items address the webpage content production (whether the content was user-driven, whether the content production process was clearly described), use of evidence-based guidelines, review by experts, comprehensive literature search for content production, and content accuracy ( Minervation, 2007 ). The content accuracy question was answered based on the clinical guidelines provided by the “Primary Dysmenorrhea Consensus Guideline” ( Burnett & Lemyre, 2017 ). The proven interventions we searched for in each webpage were NSAIDS, hormonal therapy, continuous/extended contraceptives, exercise, heat, and ginger ( Burnett & Lemyre, 2017 ). A webpage received a “no” response if it discussed 0–2 of these interventions, a “partially” response for 3–4 interventions, and a “yes” response if it contained 5–6 proven interventions. Usability was assessed using items from the Sandvik and LIDA tools. We used the interactivity and navigability sections of the Sandvik tool. Evaluators searched for a clear invitation to comment or ask questions and evaluated the ease of following links from the home page to find information ( Sandvik, 1999 ). We also used one item from LIDA to assess for the presence and accuracy of non-textual media ( Minervation, 2007 ). Using the umbrella tool we created based on the above measures, two team members independently evaluated the webpages to reduce subjectivity in scoring. When discrepancies occurred, disagreements were resolved by discussion until team members reached consensus. Data were summarized using descriptive statistics, including means, standard deviations, and frequencies. Microsoft Excel Version 2016 and IBM SPSS Statistics Version 25 were used to generate descriptive statistics.

Of the 25 included webpages, 16 were commercial, four were published by professional organizations or academic medical centers, three were published by federal agencies, and two were wiki sites. Table 1 summarizes the corresponding results for each evaluation criteria. The average readability score, calculated using the Flesch Reading Ease Formula, was 52.17 (range: 20.3 – 79, standard deviation: 11.74). The average reading level, calculated using the Flesch-Kincaid Grade Level Readability Formula, was 9.83 (range: 4.5 – 17.9, standard deviation: 2.48). These results indicate that the webpages were written at an average of a 10 th grade reading level. The HON Certification Seal was not identified in 40% of webpages. Although the remaining 60% of webpages were HON Certified, the Sandvik scores indicated that the webpages had variable credibility. For ownership, 92% of webpages clearly stated the name and type of content provider, while only 8% of webpages included either the name or type of content provider. For authorship, only 28% of webpages stated the author’s name and qualifications; 48% of webpages showed no indication of authorship, while 24% of webpages included all other indication of authorship (e.g. author name was listed but not the author’s qualifications). For sources of information, only 8% of webpages had references from scientific literature. Other sources of referenced information included commercial websites, opinions, and wiki sites. Webpages also varied in the method of citation. Only 8% of webpages included both in-text citations and a reference list; 60% of webpages included only a reference list, while 32% of webpages did not include any source of information. Considering the conflict of interest statement evaluated with the LIDA instrument, 44% of webpages declared the objectives of those who run the website. Common conflicts of interest included competing interests and financial rewards. Using the DISCERN instrument, we found that webpages varied greatly on information quality regarding treatment options. Only 4% of webpages promoted shared decision-making with more than one other individual (e.g., healthcare provider, family, partner). Approximately 76% of webpages promoted shared decision-making with one other individual, but 20% did not support shared decision-making. For explanation of risks, 52% of webpages partially explained risks of treatment while 48% did not mention risks of treatment. Furthermore, only 16% of webpages described benefits for each treatment; 72% of webpages partially described benefits and the remaining 12% did not include benefits of treatment. For Sandvik scores on currency, approximately 76% of webpages were updated in the last two years; however, it is unclear whether webpages update the information obtained through literature searches. Regarding balance, 8% of webpages focused on promoting products, while 24% were biased in favor of their own products or services. The remaining 68% of webpages provided balanced information. For LIDA content production items, the information on the evaluated webpages were not user driven. None of the webpages mentioned taking user needs into account while producing webpage content. The method of content production was also of variable clarity; 32% of webpages did not provide clear information on the content production process. Also, 92% of webpages did not reference evidence-based guidelines. Of all 25 webpages, 56% were authored or reviewed by health professionals. Approximately 84% of webpages did not perform a comprehensive literature search when creating webpage content. Content was fully accurate in only 32% of webpages and partially accurate in 52% of webpages. For interactivity and navigability, most webpages scored well. While 8% of webpages were not interactive, 60% of webpages received the highest score possible (i.e., 2 points). For navigability, 80% of webpages received the highest score (i.e., 2 points) for providing information easily found by following links from the home page. The remaining webpages contained scattered information or lacked a search engine on the webpage. Webpages were also evaluated for non-textual media. Non-textual media was included in 60% of the 25 webpages; however, only 73% of the media provided factual information. Incorrect non-textual information was noted. For example, on one of the webpages, there was a picture of a copper intrauterine device (IUD) in the dysmenorrhea treatment section. This is misleading because a copper IUD can actually exacerbate dysmenorrhea symptoms. Instead, it is a hormonal IUD that can improve dysmenorrhea symptoms.

Our study has important implications for clinical practice. We found that online information on dysmenorrhea could be difficult for the general public to read. In addition, the information varied in credibility and quality. Healthcare providers should be aware of inaccurate and confusing health information on the Internet. To address gaps in online information on dysmenorrhea, healthcare providers should utilize strategies to provide accurate and easily understandable information to patients. Healthcare providers can promote women’s self-care by educating women about dysmenorrhea, evidence-based treatment, and the risks and benefits of treatment options. When educating patients on menstrual pain, healthcare providers should try to avoid medical jargon. Patient education should be individualized based on patients’ prior knowledge to enhance their understanding of dysmenorrhea. In addition, healthcare providers should encourage patients to ask questions. Patients’ questions enable healthcare providers to determine if they have knowledge gaps or concerns about dysmenorrhea treatment. Such strategies will enhance patient-centered communication. The public considers the Internet an important source of information. However, the Internet often contains advertisements for non-evidence-based remedies, seemingly credible webpages masking the purpose of selling, and personal opinions. Healthcare providers should teach patients ways to effectively discern reliable information from less-reliable information. In addition, they could direct their patients to reliable online information, such as webpages published by a credible medical organization, academic institution or an official government health agency. Finally, as nurses work closely with patients and individuals in the community setting, nurses are uniquely positioned to educate patients and communities about how to navigate online health information.

Despite the high prevalence of dysmenorrhea and its negative impacts on women’s lives, many women do not seek professional help for dysmenorrhea ( Chen, Shieh, Drauker, & Carpenter, 2018 ; Tanaka et al., 2014 ). Some women believe their symptoms are a normal part of being a woman ( Chen et al., 2018 ; Wong, 2011 ). Others feel that dysmenorrhea is not worth a costly trip to their healthcare provider ( Chen et al., 2018 ). Some women feel uncomfortable or embarrassed about discussing menstrual pain with their healthcare provider ( Chen et al., 2018 ; Tanaka et al., 2014 ). Therefore, many women turn to other information sources, including the Internet, to identify the cause of their pain and methods to manage it ( Chen et al., 2018 ; Subasinghe et al., 2016 ). Online health information provides an outlet for people to find current information regarding their symptoms in an easily accessible and private manner ( Eysenbach & Kohler, 2002 ). Many people seek health information online. It is reported that 72% of people in the United States turn to the Internet to find medical information ( Fox & Duggan, 2013 ). Similarly, individuals from many countries seek information related to their health on the Internet ( World Health Organization, 2018 ). Many women look online for dysmenorrhea-related information. In addition, some men also search information on dysmenorrhea for their family, partner, or friend ( Chen et al., 2018 ). Even though dysmenorrhea is one of the most common health conditions among women and many people seek online information on dysmenorrhea, little is known about how reliable the online information is regarding dysmenorrhea. Important criteria to evaluate online health information include readability, credibility, quality, and usability. Readability is a measure of how easily a reader can understand the written text. It is important to evaluate webpage readability to determine whether the information is accessible to lay people without medical training ( Aleksova, Kuczynska-Burggraf, Ranasinha, & Vincent, 2017 ). Credibility refers to whether the information is trustworthy based on the website owner, author qualifications, and the sources used to obtain the information. Credibility determines if a webpage contains information that users can trust ( Zhang, Sun, & Xie, 2015 ). The quality of a webpage refers to the accuracy, currency, and comprehensiveness of the content it provides ( Zhang et al., 2015 ). It is important to evaluate the quality of the online health information available, because how accurate, current, and comprehensive the health information is plays a vital role to effective decision-making in health care. Usability is evaluated to determine if users are able to find the information they need and if the webpage is engaging and interactive. It is important for webpages to be easily navigable so that users can easily find the information they need. Additionally, webpages that are engaging and interactive tend to be more appealing to users ( Zhang et al., 2015 ). The purpose of this study was to evaluate online health information regarding dysmenorrhea, including readability, credibility, quality, and usability of the information. Through the study, we tried to identify gaps in online information on dysmenorrhea. For healthcare providers, it is important to understand the quality of information individuals have read on the Internet in order to properly educate them on the management of dysmenorrhea.

Online information on dysmenorrhea has generally low readability, mixed credibility, and variable quality. Some webpages contained inaccurate or biased information that was not evidence-based. In addition, content production was not user-driven. Finally, there was little support for selecting among different treatment options or shared decision-making. Strategies to improve health information on dysmenorrhea include avoiding complex terms, incorporating visual aids, presenting evidence-based information, and developing a decision aid to support shared-decision making.

The Internet has become an important tool for individuals seeking health information. Although easily accessible, online health information is not regulated and often contains unreliable or inaccurate content. In this study, using standardized evaluation tools, we examined the readability, credibility, quality, and usability of online health information on dysmenorrhea. Our findings are consistent with previous research evaluating online health information. One study on online information regarding ear tubes showed that the webpages had low readability, varying accuracy, and insufficient support for informed decision-making ( McKearney & McKearney, 2013 ). Similarly, another study on online health information for menopause concluded that most webpages were written above the suggested reading level. Most webpages lacked adequate content and approximately half of webpages did not identify authors or references ( Aleksova et al., 2017 ). We found the webpages’ we evaluated were likely too complex for comprehension by a substantial portion of dysmenorrhea information consumers. The average US resident reads at an eighth-grade level ( Stossel, Segar, Gliatto, Fallar, & Karani, 2012 ) and the Joint Commission recommends that all patient education materials be written at or below the fifth-grade reading level in order to meet the health literacy needs of the general public ( Stossel et al., 2012 ). However, the reading level of the evaluated webpages (i.e., tenth grade on average) was significantly higher than the recommendation. It should also be noted that adolescent girls, who suffer disproportionately with dysmenorrhea are at the sixth, seventh or eighth grade levels. Considering the varying literacy of the information users, it is important to use plain language, substitute complex medical terms with simpler terms, shorten sentences, and incorporate visual illustrations. Most webpages we evaluated contained inaccurate information. For example, webpages recommended specific yoga poses and specific dietary supplements that have not been well tested. Yoga may improve quality of life for women with dysmenorrhea, but scientific evidence is limited ( McGovern & Cheung, 2018 ). Further, scientific evidence is limited to support the effectiveness and safety of dietary supplements for dysmenorrhea. ( Pattanittum et al., 2016 ). Inaccurate non-textual information also existed, which can be particularly misleading (e.g., a picture of copper IUD). Recommending treatments that lack adequate evidence may frustrate women rather than reduce their pain and discomfort. Only a small number of webpages contained accurate, comprehensive, and evidence-based information about dysmenorrhea. Examples of evidence-based treatments include nonsteroidal anti-inflammatory medications ( Marjoribanks, Aveleke, Farquhar, & Proctor, 2015 ) and high intensity transcutaneous electrical nerve stimulator (TENS) ( Proctor, Smith, Farquhar, & Stones, 2002 ). It is worth noting that the majority of the webpages we evaluated were commercial. The high prevalence of commercial health websites was also reported in previous research ( Aleksova et al., 2017 ). Among the commercial webpages we evaluated, each of them contained ads and half of them promoted products (e.g., certain dietary supplements). These webpages may be more of marketing channels than promoters of accurate health information ( Aleksova et al., 2017 ). The potential conflict of interest may result in biased information. Out of the four categories of websites (commercial, professional organization or an academic medical center, federal agency, and wiki sites), commercial websites presented the most inaccurate content. None of the webpages employed user-driven content production. Individuals seeking information on dysmenorrhea want to know which treatments can be used at school and work, the benefits and risks of each treatment, speed of pain relief, and ease of intervention administration ( Chen et al., 2018 ), but such information is rarely offered on the webpages we evaluated. Specific information should be offered that meets women’s lifestyle needs and promotes shared decision-making between women, and their partner or parent, and provider. It is important to involve information consumers in the content production process. There are several strengths of this study. First, we imitated popular search strategies to generate webpages commonly read by the public. Specifically, we used popular search engines and lay search terms. Second, two evaluators independently reviewed the webpages and compared results to decrease subjectivity. Third, we evaluated the online health information based on multiple criteria (i.e., readability, credibility, quality, and usability). The combination of multiple established tools allowed us to evaluate the webpages in a comprehensive way. We acknowledge the limitations of this study. First, our webpage searches were by no means exhaustive. It was impossible to utilize every search term or search engine. In addition, we only reviewed one page of search results, because approximately 71% of Internet searches result in a page one click ( Petrescu, 2014 ). Second, the study results were limited to webpages in English. The authors had limited command of other languages and limited access to translation resources. Thus, we conducted the search from the United States and used only English language search engines and English search terms. Researchers from other countries may evaluate online information on dysmenorrhea in other languages. Third, the study was limited to websites that were active in January 2018. It is worth noting that the public will generate different results as webpages are published or updated. Finally, we did not evaluate any linked webpages included on the reviewed webpages. Despite these limitations, our study has implications for developing educational information on dysmenorrhea. First, readability should be improved by avoiding difficult terminology when possible, simplifying sentence structure, and providing visual aids ( U.S. National Library of Medicine, 2017 ). Second, online information should be evidence-based and authored or reviewed by an expert in that subject area. This will provide individuals with treatment options that are accurate and clinically proven to improve dysmenorrhea. Third, information users’ needs can be addressed by incorporating a decision aid that compares the relative benefits and risks of treatment. A decision aid can also improve shared-decision making between information users and their providers when selecting a personalized treatment. Development of the decision aid should be user-driven and user-centered.

Characterized by menstrual pain, dysmenorrhea affects 45% to 95% of women of reproductive age globally ( Iacovides, Avidon, & Baker, 2015 ). Dysmenorrhea is categorized as either primary or secondary. Primary dysmenorrhea is pain without the presence of underlying pathologies, while secondary dysmenorrhea is pain associated with an identifiable pathologic condition (e.g., endometriosis, pelvic inflammatory disease, uterine fibroids) ( Latthe, Champaneria, & Khan, 2012 ; Janssen et al., 2013 ). Dysmenorrhea is the leading cause of absence from school and lost work hours in women ( Davis, Kennedy, Moore, & Prentice, 2007 ; Iacovides et al., 2015 ). Even though dysmenorrhea is not life threatening, it can negatively affect women’s sleep, mood, physical activity, and quality of life ( Iacovides et al., 2015 ). An increasing amount of evidence suggests that dysmenorrhea may increase women’s risk for developing chronic pain later in the lifespan ( Iacovides et al., 2015 ; Wei et al., 2016 ). For example, women with dysmenorrhea are at a greater risk for developing chronic pain conditions such as fibromyalgia, irritable bowel syndrome, and noncyclic chronic pelvic pain ( Iacovides et al., 2015 ).