Assessing the quality, readability and reliability of online information on COVID-19: an infoveillance observational study

29

This study aimed to assess the quality, reliability and readability of internet-based information 30 on COVID-19 available on Brazil’ most used search engines. Methods: A total of 68 websites were selected 31 through Google, Bing, and Yahoo. The websites content quality and reliability were evaluated using the 32 DISCERN questionnaire, the Journal of American Medical Association (JAMA) benchmark criteria, and 33 the presence of th e Health on Net (HON) certification. Readability was assessed by the Flesch Reading 34 Ease adapted to Brazilian Portuguese (FRE-BP). Results: The web contents were considered moderate to 35 low quality according to DISCERN and JAMA mean scores. Most of the samp le presented very difficult 36 reading levels and only 7.4% displayed HON certification. Websites of Governmental and health -related 37 authorship nature showed lower JAMA mean scores and quality and readability measures did not correlate 38 to the webpages content type. Conclusion: COVID-19 related contents available online were considered 39 of low to moderate quality and not accessible. 40

COVID-19; Internet; Consumer Health Information; Health Education; Information 41 Dissemination. 42 43 . CC-BY-NC-ND 4.0 International licenseIt is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 30, 2020. ; https://doi.org/10.1101/2020.05.30.20117614doi: medRxiv preprint 3

44 Health care is rapidly transitioning from a paternalistic approach to a person -centered model. This 45 process aims to improve health outcomes by building a shared decision-making process between healthcare 46 professionals and patients, characterized by the gre ater involvement of people in resolutions and actions 47 concerning their own health (Lee et al. 2018; Petersen et al. 2019). However, the effectiveness of this new 48 model can be hindered by a considerable number of barriers, such as low education, inadequate access to 49 knowledge and social and economic deprivation (Lee et al. 2017). 50 The internet offers a large amount of information, although the quality of health and sanitary 51 information offered is highly variable, ranging from scientific and evidence-based data to home remedies 52 or information of very questionable origin that can be dangerous to health (Eysenbach et al. 2002) . The 53 biggest barrier on the internet is not the difficulty of finding health care information, but identifying those 54 that are valid and reliable (Berland et al. 2001; Lopez-Jornet and Camacho-Alonso 2009; Lopez-Jornet and 55 Camacho-Alonso 2010; Passos et al. 2020). 56 During public health emergencies, people must be aware about the health risks they face, and what 57 measures can be taken to protect their health and lives. Reliable information provided early, often, and in 58 accessible language standards , enables individuals to make choices and act to protect themselves, their 59 families and communities from health hazards (WHO 2017). Fake news and misinformation concerning 60 health on the internet represents a threat to global health (Carrieri et al. 2019) . The World Health 61 Organization (WHO) warned that the COVID-19 outbreak had been accompanied by a massive abundance 62 of information, some of which was accurate and some of which was not, which made it difficult for people 63 to find reliable sources and trustworthy information when they needed it (Kouzy et al. 2020; Pulido et al. 64 2020). The consequences of disinformation overload are the spread of uncertainty, fear, anxiety and racism. 65 Thus, monitoring the quality of information available to the population is of great importance to control the 66 spread of the disease itself and to mitigate its socioeconomic impacts (Hua and Shaw 2020). Therefore, the 67 WHO is dedicating tremendous efforts aimed at providing evidence-based information and advice to the 68 population through its social media channels and a new information platform called WHO Information 69 Network for Epidemics (Zarocostas 2020). 70 Online-available information has been increasingly employed as a surrogate tool for estimating 71 epidemiology (Cervellin et al. 2017) . Web-based sources are been used in the analysis, detection, and 72 forecasting of diseases and epidemics, and in predicting human behavior toward several health topics. In 73 this context, infoveillance studies have become an integral part of health sciences that focuses on scanning 74 the Internet for user -contributed health-related content, aiming to improve public health, measuring and 75 predicting the quality of health information on the Web (Eysenbach et al. 2009). 76 Several studies have already assessed the quality of the information available on the internet related 77 to different health conditions (Cuan-Baltazar et al. 2020; Jo et al. 2018; Lee et al. 2017; Passos et al. 2020; 78 Priyanka et al. 2018), however, there is no evidence about the quality of COVID -19 contents available in 79 . CC-BY-NC-ND 4.0 International licenseIt is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 30, 2020. ; https://doi.org/10.1101/2020.05.30.20117614doi: medRxiv preprint 4 Brazil. This study aimed to assess the reliability, readability and quality of COVID-19 related information 80 retrieved from Brazilian websites. 81

82 The search strategy was designed with regard to the relevance of terms employed by the internet 83 users. A query was performed on Google Trends to confirm the link of Brazilian Portuguese words to 84 COVID-19 issues. Google Trends enables researchers to study the trends and patterns of Google search 85 queries (Effenberger et al. 2020). The search term “coronavírus” held the most popularity among internet 86 users in Brazil, in April 2020. 87 Sites on the internet were identified using the three most accessed search engines by internet users 88 in Brazil: Google (www.google.com), Bing (www.bing.com) and Yahoo (www.yahoo.com), respectively, 89 97.59%, 1.2% and 1.04% of accesses in April 2020 (Statcounter 2020). In April 2020, the searches were 90 performed using computers connected to the internet, previously set up by clearing the cookies and search 91 history of each browser. The first 100 consecutive sites in each search were visited and classified. The 92 search was not restricted in terms of file format or domain. The search was limited to the Portuguese 93 language. Duplicate sites were excluded, as were non -operative sites or sites with denied direct access 94 through password requirements, book review sites, or sites offering journal abstracts, and those sites that 95 did not offer information on COVID-19. Websites that could be modified by the general population were 96 also not considered in this investigation. 97 The quality of website information was assessed by four evaluators, who were previously trained in 98 the analysis tools used. Concerning the scientific accuracy and reliability of websites information, WHO 99 official reports and technical guidelines were used as standards. The websites that were divergently 100 qualified by the examiners were reassessed to the achievement of a consensus score. As this was a study of 101 published information and involved no participants, no ethics approval was required. In order to avoid any 102 changes that may be made to the eligible websites during the period of analysis, the sites were assessed in 103 the same day by the evaluators. 104 The sites were classified in terms of affiliation as commercial, news portal, non-profit organization, 105 university or health center and government. The type of content was classified as corresponding to medical 106 facts, human experiences of interest, questions and answers and socioeconomic related content. 107 The quality of information of the selected websites was assessed using criteria of the Journal of the 108 American Medical Association (JAMA) benchmarks (Silberg et al. 1997). These are a display of authorship 109 of medical content, display of attribution or references, display of currency (date of update), and disclosure 110 of ownership, sponsorship, advertising policies or conflicts of interest. This tool lets the reader easily decide 111 if the site has the basic components like transparency and reliability. For each fulfilled criterion, 1 point 112 was given, with a total score ranging from 0 to 4. 113 The DISCERN instrument (Discern) is a valid and reliable tool to evaluate health information. It is 114 the first standardized quality index and was creat ed by the Division of Public Health and Primary Health 115 Care at Oxford University, London. The instrument comprises 16 questions, each representing a different 116 quality criterion. The DISCERN questions are organized into three sections as follows: Questions 1–8 117 . CC-BY-NC-ND 4.0 International licenseIt is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 30, 2020. ; https://doi.org/10.1101/2020.05.30.20117614doi: medRxiv preprint 5 address the reliability of the publication and help users to decide whether it can be trusted as a source of 118 information relating to treatment choice. Questions 9–15 address specific details of the information relating 119 to treatment alternatives. In thi s context, questions 9 –11 refer to the active treatments described in the 120 publication (possibly including self-care), while the options without treatment are addressed separately in 121 question 12. In turn, question 16 corresponds to the global quality assessment at the end of the instrument. 122 Each question is scored on a scale of 1–5 (where 1, the publication is poor; and 5, the publication is of good 123 quality). In the present study, only the first section of the questionnaire was used for reliability assessment. 124 The readability (RE) of the websites was assessed by the Flesch Reading Ease adapted to Brazilian 125 Portuguese (FRE) (Lee et al. 2017). This method classifies the readability of a text on a scale from 0 (very 126 difficult) to 100 (very easy) based on a calculation that considers the number of syllables per word and 127 words per sentence. The adapted formula is given by the following equation: RE = 248.835 − (1.015 × 128 ASL) − (84.6 × ASW). Where: ASL = mean number of words per sentence; ASW = mean number of 129 syllables per word. 130 Those metrics were calculated using the online tool Readable.io (Readable.io, Bolney, England) 131 (Readable.io) through the information of the respective Uniform Resource Locator (URL) of each website. 132 All analyses were performed based on the overall written content downloaded from these links. The reading 133 difficulty of a text is presented according to the following scores: very easy (75-100), easy (50-75), difficult 134 (25-50), and very difficult (0-25). 135 The existence of the Health on the Net (HON) Foundation seal was also recorded. HON is a code of 136 conduct for medical and healthcare sites, defining a series of norms allowing users to know the source and 137 the purpose of the medical information presented. The HON contemplates compliance with the following 138 eight basic criteria: 1. authorship; 2. complementarity; 3. privacy; 4. attribution, references and currency; 139 5. justifiability; 6. author transparency; 7. sp onsor transparency (financial disclosure); and 8. honesty in 140 advertising policy. The website may display the HON code seal if they agree to comply with the standards 141 listed, and they are subjected to random audits for compliance(HON). 142 Data were submitted to statistical analysis, all tests were applied considering an error of 5% and the 143 confidence interval of 95%, and the analyzes were carried out using SPSS software version 23.0 (SPSS Inc. 144 Chicago, IL, USA). Descriptive analysis was performed to characterize the Web pages selected for the 145 study. Although the hypothesis of normal distribution of data was not confirmed by t he Kolmogorov-146 Smirnov test, the statistical analysis was performed by the application of nonparametric tests. The 147 correlations between distinct measures were demonstrated by the Spearman rank correlation coefficients. 148 Distinct websites according to the type of content were compared by Kruskal-Wallis test. Mann-Whitney 149 U test was employed to assess the differences between the natures of websites, for this test, the affiliation 150 criteria used in data collection were dichotomized into governmental and health-related authorship nature 151 (grouping university or health center and government affiliation) and nongovernmental nor health-related 152 authorship nature (grouping news portals, commercial sites and non-profit organization affiliation)(Lee et 153 al. 2017). 154

155 . CC-BY-NC-ND 4.0 International licenseIt is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 30, 2020. ; https://doi.org/10.1101/2020.05.30.20117614doi: medRxiv preprint 6 Over 2.5 billion results were retrieved from the search engines. Of the 300 webpages assessed, 68 156 fulfilled the inclusion and exclusion criteria. The search retrieval flow diagram is presented in Figure 1. 157 According to affiliation most websites were from news portals (51.5%), followed by government 158 (29.4%), commercial sites (13.2%), university or health center (4.4%) and non-profit organization (1.5%). 159 Considering the type of content, the majority of the sites displayed medical facts (88.2%) followed by 160 socioeconomic related content (5.9%), questions and answers (4.4%) and human experiences of interest 161 (1.5%). 162 None of the evaluated websites met all four criteria of JAMA benchmarks, 35.3% had a single 163 criterion, 30.9% did not in clude any criteria, 23.5% had 2 criteria and 10.3% presented 3 criteria. Only 164 7.4% of the sites had HON certification. The DISCERN instrument identified that 75% of the websites had 165 moderate reliability, 17.6% showed high reliability and 7.4% low reliabili ty. For the Flesch Index, over 166 half of the sample were classified as very difficult (57.4%), while 41.2% were considered difficult. Only 1 167 website (1.5%) was classified as easy and none as very easy. Sample means of JAMA, DISCERN and FRE-168 BP scores are shown in table 1. 169 The correlation between distinct measures assessed through the instruments was analyzed. 170 Spearman rank correlation coefficients showed a positive significant correlation between JAMA and 171 Discern scores (p<0.001), and between HON certification presence and JAMA (p=0.006) and Discern 172 (p<0.001) scores, as shown in table 2. 173 No significant differences were observed among the mean scores of DICERN, JAMA and FRE-BP 174 according with the type of websites content (table 3). As the human experience of interest type of content 175 represents a single occurrence in the sample it was disregarded in this step of the statistical analyzes. 176 The mean JAMA scores were different according to the dichotomized affiliation categorization. 177 Higher mean was observed in nongovernmental nor health -related authorship nature. For DISCERN and 178 FRE-BP mean scores no significant statistical difference was observed, as shown in table 4. 179

180 The internet has great potential for spreading health information (Berland et al. 2001). Whatever the 181 communication channel used, it is known that its content is able to influence the decisions of an individual 182 about his health, including changes in lifestyle (Afshin et al. 2016). Web-based information influences how 183 patients comply with advices, clinical diagnoses, and treatment regimens recommended by health 184 professionals (Lu et al. 2018; Lu and Zhang 2019).Treatment adherence and compliance relies on trust and 185 good professional -patient communication (Wahl et al. 2005 ). In this context, internet -based new 186 technologies are gaining growing global attention and becoming increasingly available for predicting, 187 preventing and monitoring emerging infectious diseases, such as COVID-19 (Effenberger et al. 2020; Yang 188 et al. 2020). However, misinformation spread through the internet can hinder the communication of health 189 entities and professionals with the general population (Lu and Zhang 2019) and so reduce adherence to the 190 confrontations proposed to contain the pandemic, as social distancing (Farooq et al. 2020). Research on the 191 role of internet content, social media messages and dominant discourses that are communicated to the public 192 . CC-BY-NC-ND 4.0 International licenseIt is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 30, 2020. ; https://doi.org/10.1101/2020.05.30.20117614doi: medRxiv preprint 7 is an emerging topic of public health interest in scientific work that requires further investigation (Pulido 193 et al. 2020). 194 Despite the harm that misinformation may pose, especially during a pandemic in which the 195 population's reaction to health measures imposed by governments is of crucial importance to combat the 196 spread of the disease (US Medicine Institute 2002), scientific evidence on the topic is scarce. Besides WHO 197 efforts to monitor and improve the quality of information available online on this subject (Hua and Shaw 198 2020), the few available evidence warns of the low quality and reliability of data (Abd-Alrazaq et al. 2020; 199 Febres-Cordero et al. 2018; Kouzy et al. 2020). These works are, however, limited to social media content 200 analysis and lacking standard parameters to data evaluation. In the present study, for all parameters used in 201 the data analysis, the quality of information ranged from low to moderate. 202 The HON seal was displayed in only 7.5% of the sample. Other studies carried on Brazilian websites 203 revealed even a small number of sites with this certification (Lee et al. 2017; Passos et al. 2020) . The 204 absence of thi s seal on sites of important institutions indicates that concern for certifying the quality of 205 information in the internet is still scarce. Another possible cause for low seal adhesion is the fact that the 206 annual review required for seal maintenance is not free (Passos et al. 2020). 207 Besides the quality of information, the amount of it provided for an individual is also a concern. 208 Previous research suggests that the vast amount of available information can be confusing, potentially 209 resulting in over -concern and information overload (Farooq et al. 2020 ). Information overload is being 210 associated to mental health problems during COVID -19 outbreak. These findings inspire the need for 211 greater government involvement to prevent information overload while facing a public health emergency 212 (Gao et al. 2020). However, government represents only 29.4% of the sources of information, while news 213 portals represent 54.5% of the website’s affiliation. In addition, governmental and health-related sources of 214 information showed lower JAMA mean scores when compared to n ongovernmental nor health -related 215 sources. This demonstrates the need for better articulation between health entities and government to stand 216 out as the main provider of reliable content, directing users to good quality sources and avoiding 217 information overload. 218 According to the readability scores, the websites were considered difficult and very difficult for 219 most of population. In addition to this finding it is relevant to consider the low level of health literacy, 220 which is the degree to which people have the capacity to understand health information, reported for 221 Brazilian Portuguese speakers (Batista et al. 2018). This may result in a communication gap for laypersons. 222 In addition to ensure the quality and reliability of information, it is important that these quality 223 contents are presented in a comprehensible and accessible manner (Miguens-Vila et al. 2018) . Studies 224 reported a negative correlation of readability scores with JAMA (Sobota and Ozakinci 2015) and DISCERN 225 (Lee et al. 2017) scores. These findings can be considered as exacerbating factors of the impact of the low 226 quality of information on internet users, as it demonstrates that more accessible content is of even worse 227 quality. In the present study such correlation was not found possibly due the growing concern about the 228 quality of health-related information available online (Farooq et al. 2020), however, no positive correlation 229 . CC-BY-NC-ND 4.0 International licenseIt is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 30, 2020. ; https://doi.org/10.1101/2020.05.30.20117614doi: medRxiv preprint 8 was found, which demonstrates the need for further efforts on improving the accessibility of high -quality 230 health related information available online. 231 The internet and search engines are dynamic processes that constantly change. The sites evaluated 232 in this investigation may not necessarily reflect the information available to patients at another time. This 233 was a limitation for this investigation. However, the search engines used for the consultation represent 234 99.8% of the access of Brazilian internet users (Statcounter 2020). In addition, to cover a reasonable amount 235 of data, the first 100 consecutive websites of each search engine were accessed. 236 Regarding the present sample of Brazilian websites, COVID-19 contents were considered of low to 237 moderate quality and low readabili ty based on the parameters adopted. This pattern just reasonably 238 correlated with the nature of websites’ authorship. These findings indicate the need for further efforts on 239 improving the quality of health-related content on internet. Health authorities might apply this evidence to 240 measure the effect of the transmission of information on the population and define better risk 241 communication strategies. 242 Conflict of interest 243 The authors declare no conflict of interest related to the present study. 244

245 1. Abd-Alrazaq A, Alhuwail D, Househ M et al. (2020) Top Concerns of Tweeters During the COVID-246 19 Pandemic: Infoveillance Study. J Med Internet Res. 22:e19016. https:// doi: 247 10.3390/diagnostics10040224 248 2. Afshin A, Babalola D, McLean M et al. (2016) In formation Technology and Lifestyle: A Systematic 249 Evaluation of Internet and Mobile Interventions for Improving Diet, Physical Activity, Obesity, 250 Tobacco, and Alcohol Use. J Am Heart Assoc. 5:e1-18. http://doi: 10.1161/JAHA.115.003058 251 3. Batista MJ, Lawrence HP, Sousa MLR (2018) Oral health literacy and oral health outcomes in an adult 252 population in Brazil. BMC Public Health. 18:60. https://doi:10.1186/s12889-017-4443-0 253 4. Berland GK, Elliott MN, Morales LS et al (2001) Health information on the Internet: accessib ility, 254 quality, and readability in English and Spanish. Jama 285:2612 -2621 255 https://doi.org/10.1001/jama.285.20.2612 256 5. Carrieri V, Madio L, Principe F (2019) Vaccine hesitancy and (fake) news: Quasi experimental 257 evidence from Italy. Health Econ 28:1377-1382. https://doi.org/10.1002/hec.3937 258 6. Cervellin G, Comelli I, Lippi G (2017) Is Google Trends a Reliable Tool for Digital Epidemiology? 259 Insights From Different Clinical Settings. Journal of epidemiology and global health 7:185 -189. 260 https://doi.org/10.1016/j.jegh.2017.06.001 261 7. Cuan-Baltazar JY, Muñoz-Perez MJ, Robledo-Vega C et al (2020) Misinformation of COVID-19 on 262 the Internet: Infodemiology Study. JMIR Public Health Surveill 6:e18444. 263 https://doi.org/10.2196/18444 264 8. Discern DISCERN TOOL. Division of Public Health and Primary Health Care at Oxford University. 265 http://www.discern.org.uk/. Accessed 20 April 2020 266 9. Effenberger M, Kronbichler A, Shin JI et al (2020) Association of the COVID -19 pandemic with 267 Internet Search Volumes: A Google Trends(TM) Analysis. Int J Infe ct Dis 95:192 -197. 268 https://doi.org/10.1016/j.ijid.2020.04.033 269 10. Eysenbach G, Powell J, Kuss O et al (2002) Empirical studies assessing the quality of health 270 information for consumers on the world wide web: a systematic review. Jama 287:2691 -2700. 271 https://doi.org/10.1016/j.jegh.2017.06.001 272 . CC-BY-NC-ND 4.0 International licenseIt is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 30, 2020. ; https://doi.org/10.1101/2020.05.30.20117614doi: medRxiv preprint 9 11. Eysenbach G (2009) Infodemiology and Infoveillance: Framework for an Emerging Set of Public 273 Health Informatics Methods to Analyze Search, Communication and Publication Behavior on the 274 Internet. Journal of medical Internet research 11:e11. https://doi.org/10.2196/jmir.1157 275 12. Farooq A, Laato S, Islam AKMN (2020) Impact of Online Information on Self -Isolation Intention 276 During the COVID -19 Pandemic: Cross -Sectional Study. Journal of medical Internet research 22: 277 e19128. https://doi.org/10.2196/19128 278 13. Febres-Cordero B, Brouwer KC, Rocha-Jimenez T et al. (2018) Influence of peer support on HIV/STI 279 prevention and safety amongst international migrant sex workers: A qualitative study at the Mexico -280 Guatemala border. PLoS One. 13: 1-20. https://doi:10.1371/journal.pone.0190787 281 14. Gao J, Zheng P, Jia Y et al. (2020) Mental health problems and social media exposure during COVID-282 19 outbreak. PLoS One. 15:e0231924. https://doi.org/10.1371/journal.pone.0231924 283 15. HON T Health on the Net (HON). HON Foundation. https://www.hon.ch/en/. Accessed 20 April 2020. 284 16. Hua J, Shaw R (2020) Corona Virus (COVID -19) "Infodemic" and Emerging Issues through a Data 285 Lens: The Case of China. Int J Environ Res Public Health 17:2309. 286 https://doi.org/10.3390/ijerph17072309 287 17. Institute of Medicine (US) Committee on Assuring the Healt h of the Public in the 21st Century. The 288 Future of the Public's Health in the 21st Century. Washington (DC): National Academies Press (US); 289 2002. Available from: https://www.ncbi.nlm.nih.gov/books/NBK221239/ doi: 10.17226/10548 290 18. Jo JH, Kim EJ, Kim JR et al (2018) Quality and readability of internet-based information on halitosis. 291 Oral Surg Oral Med Oral Pathol Oral Radiol 125:215-222. https://doi.org/10.1016/j.oooo.2017.12.001 292 19. Kouzy R, Abi Jaoude J, Kraitem A et al (2020) Coronavirus Goes Viral: Quantifying the COVID-19 293 Misinformation Epidemic on Twitter. Cureus 12:e7255. https://doi.org/10.7759/cureus.7255 294 20. Lee K, Robins S, Bragazzi N et al (2017) Evaluating the Dental Caries -Related Information on 295 Brazilian Websites: Qualitative Study. J Med Internet Res 19:e415. https://doi.org/10.2196/jmir.7681 296 21. Lee H, Chalmers NI, Brow A et al (2018) Person-centered care model in dentistry. BMC Oral Health 297 18:198. https://doi.org/10.1186/s12903-018-0661-9 298 22. López-Jornet P, Camacho-Alonso F (2009) The quality of internet sites providing information relating 299 to oral cancer. Oral Oncol 45:e95-98. https://doi.org/10.1016/j.oraloncology.2009.03.017 300 23. Lopez-Jornet P, Camacho -Alonso F (2010) The quality of internet information relating to oral 301 leukoplakia. Med Oral Patol Oral Cir Bucal 15:e727-731. http://doi:10.4317/medoral.15.e727 302 24. Lu X, Zhang R, Wu W et al. (2018) Relationship Between Internet Health Information and Patient 303 Compliance Based on Trust: Empirical Study. J Med Internet Res. 20:e253. https:// doi: 304 10.2196/jmir.9364 305 25. Lu X, Zhang R (2019) Impact of Physician-Patient Communication in Online Health Communities on 306 Patient Compliance: Cross -Sectional Questionnaire Study. J Med Internet Res. 21:e12891. https:// 307 doi: 10.2196/12891 308 26. Miguéns-Vila R, Ledesma-Ludi Y, Rodríguez-Lozano F et al. (2018) Disparities between English and 309 Spanish in readability of online endodontic information for laypeople. J Am Dent Assoc. 149:960-966. 310 https://doi: 10.1016/j.adaj.2018.07.003 311 27. Passos KK, Leonel AC, Bonan PR et al (2020) Q uality of information about oral cancer in Brazilian 312 Portuguese available on Google, Youtube, and Instagram. Med Oral Patol Oral Cir Bucal 25:e346 -313 e352. http://doi:10.4317/medoral.23374 314 28. Petersen CL, Weeks WB, Norin O et al. (2019) Develop ment and Implementation of a Person -315 Centered, Technology-Enhanced Care Model For Managing Chronic Conditions: Cohort Study. JMIR 316 Mhealth Uhealth 7:e11082. http://doi:10.2196/11082 317 29. Priyanka P, Hadi YB, Reynolds GJ (2018) Analysis of the Patient Information Quality and Readability 318 on Esophagogastroduodenoscopy (EGD) on the Internet. Can J Gastroenterol Hepatol 2018:2849390. 319 http:// doi:10.1155/2018/2849390 320 30. Pulido CM, Ruiz-Eugenio L, Redondo-Sama G et al (2020) A New Application of Social Impact in 321 Social Medi a for Overcoming Fake News in Health. Int J Environ Res Public Health 17:2430. 322 http://doi:10.3390/ijerph17072430 323 31. Readable.io Readibility Score. https://readable.io/. Accessed 20 April 2020 324 . CC-BY-NC-ND 4.0 International licenseIt is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 30, 2020. ; https://doi.org/10.1101/2020.05.30.20117614doi: medRxiv preprint 10 32. Silberg WM, Lundberg GD, Musacchio RA (1997) Assessing, controlling, and assuring the quality of 325 medical information on the Internet: Caveant lector et viewor--Let the reader and viewer beware. Jama 326 277:1244-1245 327 33. Sobota A, Ozakinci G (2015) The quality and readability of online consumer information about 328 gynecologic cancer. Int J Gynecol Cancer. 25:537-541. https://doi: 10.1097/IGC.0000000000000362 329 34. Statcounter (2020) Search Engine Market Share Brazil. GlobalStats. https://gs.statcounter.com/search-330 engine-market-share/all/brazil. Accessed 20 April 2020 331 35. Wahl C, Gregoire JP , Teo K et al. (2005) Concordance, compliance and adherence in healthcare: 332 closing gaps and improving outcomes. Healthc Q. 8:65-70. https://doi: 10.12927/hcq..16941 333 36. World Health Organization (2017) Communicating risk in public health emergencies: a WHO 334 guideline for emergency risk communication (ERC) policy and practice Switzerland ISBN 978-92-4-335 155020-8 336 37. Yang T, Gentile M, Shen CF et al. (2020) Combining Point -of-Care Diagnostics and Internet of 337 Medical Things (IoMT) to Combat the COVID-19 Pandemic. Diagnostics (Basel). 10:e224. https://doi: 338 10.3390/diagnostics10040224 339 38. Zarocostas J (2020) How to fight an infodemic. Lancet 395:676. http://doi:10.1016/s0140 -340 6736(20)30461-x 341 342 . CC-BY-NC-ND 4.0 International licenseIt is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 30, 2020. ; https://doi.org/10.1101/2020.05.30.20117614doi: medRxiv preprint 11 Table 1 Descriptive statistics of the scores of DISCERN, the Journal of American Medical Association 343 benchmark, and Flesch Reading Ease 344 DISCERN JAMAa FRE-BPb Mean (SD) 1.90 (0.493) 1.13 (0.976) 22.32 (12.325) Median 2.00 1.00 23.00 Minimum 1.00 0.00 1.00 Maximum 3.00 3.00 72.00 aJAMA: Journal of American Medical Association. bFRE-BP: Flesch Reading Ease adapted to Brazilian Portuguese. 345 . CC-BY-NC-ND 4.0 International licenseIt is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 30, 2020. ; https://doi.org/10.1101/2020.05.30.20117614doi: medRxiv preprint 12 Table 2 Correlations between distinct quality measures 346 Quality measures Correlation coefficient (p-value) JAMAa 0.464* (<0.001)ab -0.045 (0.716)ac 0.329* (0.006)ad DISCERNb 0.464* (<0.001)ab -0.182 (0.138)bc 0.531* (<0.001)bd FRE-BPc -0.045 (0.716)ac -0.182 (0.138)bc -0.018 (0.882)cd HONd 0.329* (0.006)ad 0.531* (<0.001)bd -0.018 (0.882)cd aJAMA: Journal of American Medical Association. cFRE-BP: Flesch Reading Ease adapted to Brazilian Portuguese. dHON: Health on Net. Superscript letters indicate correlation groups *Significant statistical difference between the groups (Spearman correlation coefficients, p<0.05). 347 . CC-BY-NC-ND 4.0 International licenseIt is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 30, 2020. ; https://doi.org/10.1101/2020.05.30.20117614doi: medRxiv preprint 13 Table 3 Descriptive statistics of different websites type of content for DISCERN, the Journal of American 348 Medical Association benchmark, and Flesch Reading Ease adapted to Brazilian Portuguese 349 350 Site content DISCERN JAMAa FRE-BPb Medical facts (n=60) Mean (SD) 1.90 (0.511) 1.10 (0.986) 22.38 (12.806) Median 2.00 1.00 23.00 Minimum 1.00 0.00 1 Maximum 3.00 3.00 72 Questions and answers (n=3) Mean (SD) 1.67 (0.577) 1.33 (1.115) 25.00 (2.646) Median 2.00 2.00 24.00 Minimum 1.00 0.00 23.00 Maximum 2.00 2.00 28.00 Socioeconomics (n=4) Mean (SD) 2.00 (0.000) 1.50 (1.000) 23.00 (9.274) Median 2.00 2.00 23.50 Minimum 2.00 0.00 14.00 Maximum 2.00 2.00 31.00 p-value* 0.818 0.780 0.555 aJAMA: Journal of American Medical Association. bFRE-BP: Flesch Reading Ease adapted to Brazilian Portuguese. *Kruskal-Wallis test (p<0.05). . CC-BY-NC-ND 4.0 International licenseIt is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 30, 2020. ; https://doi.org/10.1101/2020.05.30.20117614doi: medRxiv preprint 14 Table 4 Descriptive statistics for both affiliation website groups for DISCERN, the Journal of American 351 Medical Association benchmark, and Flesch Reading Ease adapted to Brazilian Portuguese 352 Quality scores Websites Governmental and health-related nature (n=23) Nongovernmental nor health-related nature (n=45) p- value Mean (SD) Median Minimum Maximum Mean (SD) Median Minimum Maximum DISCERN 1.87 (0.548) 2 1 3 1.91 (0.468) 2 1 3 0.691 JAMAa 0.70 (1.063) 0 0 3 1.36 (0.857) 1 0 3 0.002* FRE-BPb 18.87 (9.725) 22 1 40 24.09 (13.213) 25 3 72 0.083 aJAMA: Journal of American Medical Association. bFRE-BP: Flesch Reading Ease adapted to Brazilian Portuguese. *Significant statistical difference between the groups (Mann-Whitney U test, P<0.05). 353 . CC-BY-NC-ND 4.0 International licenseIt is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 30, 2020. ; https://doi.org/10.1101/2020.05.30.20117614doi: medRxiv preprint 15 Fig. 1 Search retrieval flow diagram 354 355 . CC-BY-NC-ND 4.0 International licenseIt is made available under a is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. (which was not certified by peer review) The copyright holder for this preprint this version posted May 30, 2020. ; https://doi.org/10.1101/2020.05.30.20117614doi: medRxiv preprint