{"paper_id":"00cbb95b-88b9-40ab-b355-bba61d9e1a93","body_text":"Last update: August 1st, 2018 \n  \nPrepared by \nQuan Nha HONGa, Pierre PLUYEa,, Sergi FÀBREGUESb, Gillian BARTLETTa, Felicity BOARDMANc,  \nMargaret CARGOd, Pierre DAGENAISe, Marie‐Pierre GAGNONf, Frances GRIFFITHSc, Belinda NICOLAUa, \nAlicia O’CATHAINg, Marie‐Claude ROUSSEAUh, & Isabelle VEDELa  \n \n \n \naMcGill University, Montréal, Canada; bUniversitat Oberta de Catalunya, Barcelona, Spain; cUniversity of Warwick, Coventry, England;  \ndUniversity of Canberra, Canberra, Australia; eUniversité de Sherbrooke, Sherbrooke, Canada; fUniversité Laval, Québec, Canada;  \ngUniversity of Sheffield, Sheffield, England; hInstitut Armand‐Frappier Research Centre, Laval, Canada \n \nMIXED METHODS APPRAISAL TOOL (MMAT) \nVERSION 2018 \nUser guide\n\nFor dissemination, application, and feedback: Please contact mixed.methods.appraisal.tool@gmail.com \nFor more information: http://mixedmethodsappraisaltoolpublic.pbworks.com/                                                                               1 \nWhat is the MMAT?  \nThe MMAT is a critical appraisal tool that  is designed for the appraisal stage of \nsystematic mixed studies reviews, i.e., revi ews that include qualitat ive, quantitative and \nmixed methods studies. It permits to appr aise the methodologica l quality of five \ncategories to studies: qualitative research, randomized controlled trials, non-randomized \nstudies, quantitative descriptive studies, and mixed methods studies. \n \n \nHow was the MMAT developed?  \nThe MMAT was developed in 2006 (Pluye et al., 2009a) and was revised in 2011 (Pace \net al., 2012). The present version 2018 was developed on the basis of findings from a \nliterature review of critical appraisal tools, interviews with MMAT users, and an e-\nDelphi study with intern ational experts (Hong, 2018). The MMAT developers are \ncontinuously seeking for improvement and testing of this tool. Users’ feedback is always \nappreciated.  \n \n \nWhat the MMAT can be used for?  \nThe MMAT can be used to appraise the qu ality of empirical studies, i.e., primary \nresearch based on experiment, observation or  simulation (Abbott, 1998; Porta et al., \n2014). It cannot be used for non-empirical papers  such as review and theoretical papers. \nAlso, the MMAT allows the appraisal of most common types of  study methodologies \nand designs. However, some specific design s such as economic and diagnostic accuracy \nstudies cannot be assessed w ith the MMAT. Other critical  appraisal tools might be \nrelevant for these designs.  \n \n \nWhat are the requirements?  \nBecause critical appraisal is ab out judgment making, it is a dvised to have at least two \nreviewers independently involved in the ap praisal process. Also, using the MMAT \nrequires experience or training in these domains. For instance, MMAT users may be \nhelped by a colleague with specific expertise when needed.  \n \n \n \nHow to use the MMAT?  \nThis document comprises two parts: checklis t (Part I) and explanation of the criteria \n(Part II).  \n \n1. Respond to the two screening questions. Res ponding ‘No’ or ‘Can’t tell’ to one or \nboth questions might indicate that the pape r is not an empirical study, and thus \ncannot be appraised using the MMAT. MMAT users might decide not to use these \nquestions, especially if the selection criteria of their review are limited to empirical \nstudies.  \n2. For each included study, choose the appropriate category of studies to appraise. Look \nat the description of the methods used in  the included studies. If needed, use the \nalgorithm at the end of this document.  \n3. Rate the criteria of the c hosen category. For example, if  the paper is a qualitative \nstudy, only rate the five criteria in the qua litative category. The ‘Can’t tell’ response \ncategory means that the paper do not report appropriate information to answer ‘Yes’ \nor ‘No’, or that report unclear information related to the criterion. Rating ‘Can’t tell’ \ncould lead to look for compan ion papers, or contact authors to ask more information \nor clarification when needed. In Part II of  this document, indicators are added for \nsome criteria. The list is not exhaustive and not all indicators are necessary. You \nshould agree among your team which ones ar e important to consider for your field \nand apply them uniformly across all included studies from the same category. \n \nHow to score?  \nIt is discouraged to calculate an overall scor e from the ratings of each criterion. Instead, \nit is advised to provide a more detailed presen tation of the ratings of each criterion to \nbetter inform the quality of the included studie s. This may lead to perform a sensitivity \nanalysis (i.e., to consider the quality of st udies by contrasting their results). Excluding \nstudies with low methodological quality is usually discouraged.  \n \nHow to cite this document?  \nHong QN, Pluye P, Fàbregues S, Bartlett G, Boardman F, Cargo M, Dagenais P, Gagnon \nM-P, Griffiths F, Nicolau B, O’Cathain A,  Rousseau M-C, Vedel I. Mixed Methods \nAppraisal Tool (MMAT), version 2018. Registration of Copyright (#1148552), Canadian \nIntellectual Property Office, Industry Canada. \n \n\n         2 \nPart I: Mixed Methods Appraisal Tool (MMAT), version 2018 \n \nCategory of study \ndesigns Methodological quality criteria Responses \nYes No Can’t tell Comments \nScreening questions  \n(for all types) \nS1. Are there clear research questions?     \nS2. Do the collected data allow to address the research questions?      \nFurther appraisal may not be feasible or appropriate when the answer is ‘No’ or ‘Can’t tell’ to one or both screening questions. \n1. Qualitative 1.1. Is the qualitative approach appropr iate to answer the research question?     \n1.2. Are the qualitative data collection methods adequate to address the research question?     \n1.3. Are the findings adequately derived from the data?     \n1.4. Is the interpretation of results sufficiently substantiated by data?      \n1.5. Is there coherence between qualitative data sources, collection, analysis and interpretation?     \n2. Quantitative \nrandomized controlled \ntrials \n2.1. Is randomization appropriately performed?     \n2.2. Are the groups comparable at baseline?     \n2.3. Are there complete outcome data?     \n2.4. Are outcome assessors blinded to the intervention provided?     \n2.5 Did the participants adhere to the assigned intervention?     \n3. Quantitative non-\nrandomized  \n3.1. Are the participants representative of the target population?     \n3.2. Are measurements appropriate regarding both the outcome and intervention (or exposure)?     \n3.3. Are there complete outcome data?     \n3.4. Are the confounders accounted for in the design and analysis?     \n3.5. During the study period, is the intervention administered (or exposure occurred) as intended?     \n4. Quantitative \ndescriptive \n4.1. Is the sampling strategy relevant to address the research question?     \n4.2. Is the sample representative of the target population?     \n4.3. Are the measurements appropriate?     \n4.4. Is the risk of nonresponse bias low?     \n4.5. Is the statistical analysis appropriate to answer the research question?     \n5. Mixed methods 5.1. Is there an adequate rationale for using a mixed methods design to address the research question?     \n5.2. Are the different components of the study effectively integrated to answer the research question?     \n5.3. Are the outputs of the integration of qualitative and quantitative components adequately interpreted?     \n5.4. Are divergences and inconsistencies between quantitative and qualitative results adequately addressed?     \n5.5. Do the different components of the study adhere to the quality criteria of each tradition of the methods involved?     \n\n3 \nPart II: Explanations  \n \n1. Qualitative studies Methodological quality criteria \n“Qualitative research is an approach for exploring and understanding the \nmeaning individuals or groups ascribe to a social or human problem” \n(Creswell, 2013b, p. 3). \n \nCommon qualitative research approaches include (this list if not \nexhaustive): \n \nEthnography \nThe aim of the study is to describe and interpret the shared cultural \nbehaviour of a group of individuals. \n \nPhenomenology \nThe study focuses on the subjective experiences and interpretations of a \nphenomenon encountered by individuals. \n \nNarrative research \nThe study analyzes life experiences of an individual or a group. \n \nGrounded theory \nGeneration of theory from data in the process of conducting research (data \ncollection occurs first). \n \nCase study \nIn-depth exploration and/or explanation of issues intrinsic to a particular \ncase. A case can be anything from a decision-making process, to a person, \nan organization, or a country. \n \nQualitative description \nThere is no specific methodology, but a qualitative data collection and \nanalysis, e.g., in-depth interviews or focus groups, and hybrid thematic \nanalysis (inductive and deductive). \n \nKey references: Creswell (2013a); Sandelowski (2010); Schwandt (2015) \n1.1. Is the qualitative approach appropriate to answer the research question? \n \nExplanations  \nThe qualitative approach used in a study (see non-exhaustive list on the left side of this table) should be appropriate for the \nresearch question and problem. For example, the use of a grounded theory approach should address the development of a \ntheory and ethnography should study human cultures and societies.  \n \nThis criterion was considered important to add in the MMAT since there is only one category of criteria for qualitative studies \n(compared to three for quantitative studies).  \n1.2. Are the qualitative data collection methods adequate to address the research question? \n \nExplanations  \nThis criterion is related to data collection method, including data sources (e.g., archives, documents), used to address the \nresearch question. To judge this criterion, consider whether the method of data collection (e.g., in depth interviews and/or \ngroup interviews, and/or observations) and the form of the data (e.g., tape recording, video material, diary, photo, and/or field \nnotes) are adequate. Also, clear justifications are needed when data collection methods are modified during the study. \n1.3. Are the findings adequately derived from the data? \n \nExplanations  \nThis criterion is related to the data analysis used. Several data analysis methods have been developed and their use depends on \nthe research question and qualitative approach. For example, open, axial and selective coding is often associated with grounded \ntheory, and within- and cross-case analysis is often seen in case study.  \n1.4. Is the interpretation of results sufficiently substantiated by data? \n \nExplanations  \nThe interpretation of results should be supported by the data collected. For example, the quotes provided to justify the themes \nshould be adequate.  \n1.5. Is there coherence between qualitative data sources, collection, analysis and interpretation? \n \nExplanations  \nThere should be clear links between data sources, collection, analysis and interpretation. \n \n  \n\n4 \n2. Quantitative \nrandomized \ncontrolled trials \nMethodological quality criteria \nRandomized controlled \nclinical trial: A clinical \nstudy in which individual \nparticipants are allocated \nto intervention or control \ngroups by randomization \n(intervention assigned by \nresearchers). \n \nKey references: Higgins \nand Green (2008); \nHiggins et al. (2016); \nOxford Centre for \nEvidence-based \nMedicine (2016); Porta \net al. (2014) \n2.1. Is randomization appropriately performed? \n \nExplanations  \nIn a randomized controlled trial, the allocation of a participant (or a data collection unit, e.g., a school) into the intervention or control group is based solely on chance. \nResearchers should describe how the randomization schedule was generated. A simple statement such as ‘we randomly allocated’ or ‘using a randomized design’ is insufficient \nto judge if randomization was appropriately performed. Also, assignment that is predictable such as using odd and even record numbers or dates is not appropriate. At minimum, \na simple allocation (or unrestricted allocation) should be performed by following a predetermined plan/sequence. It is usually achieved by referring to a published list of random \nnumbers, or to a list of random assignments generated by a computer. Also, restricted allocation can be performed such as blocked randomization (to ensure particular allocation \nratios to the intervention groups), stratified randomization (randomization performed separately within strata), or minimization (to make small groups closely similar with \nrespect to several characteristics). Another important characteristic to judge if randomization was appropriately performed is allocation concealment that protects assignment \nsequence until allocation. Researchers and participants should be unaware of the assignment sequence up to the point of allocation. Several strategies can be used to ensure \nallocation concealment such relying on a central randomization by a third party, or the use of sequentially numbered, opaque, sealed envelopes (Higgins et al., 2016). \n2.2. Are the groups comparable at baseline? \n \nExplanations  \nBaseline imbalance between groups suggests that there are problems with the randomization. Indicators from baseline imbalance include: “(1) unusually large differences \nbetween intervention group sizes; (2) a substantial excess in statistically significant differences in baseline characteristics than would be expected by chance alone; (3) imbalance \nin key prognostic factors (or baseline measures of outcome variables) that are unlikely to be due to chance; (4) excessive similarity in baseline characteristics that is not \ncompatible with chance; (5) surprising absence of one or more key characteristics that would be expected to be reported” (Higgins et al., 2016, p. 10). \n2.3. Are there complete outcome data? \n \nExplanations  \nAlmost all the participants contributed to almost all measures. There is no absolute and standard cut-off value for acceptable complete outcome data. Agree among your team \nwhat is considered complete outcome data in your field and apply this uniformly across all the included studies. For instance, in the literature, acceptable complete data value \nranged from 80% (Thomas et al., 2004; Zaza et al., 2000) to 95% (Higgins et al., 2016). Similarly, different acceptable withdrawal/dropouts rates have been suggested: 5% (de \nVet et al., 1997; MacLehose et al., 2000), 20% (Sindhu et al., 1997; Van Tulder et al., 2003) and 30% for a follow-up of more than one year (Viswanathan and Berkman, 2012).  \n2.4. Are outcome assessors blinded to the intervention provided? \n \nExplanations  \nOutcome assessors should be unaware of who is receiving which interventions. The assessors can be the participants if using participant reported outcome (e.g., pain), the \nintervention provider (e.g., clinical exam), or other persons not involved in the intervention (Higgins et al., 2016).  \n2.5 Did the participants adhere to the assigned intervention? \n \nExplanations  \nTo judge this criterion, consider the proportion of participants who continued with their assigned intervention throughout follow-up. “Lack of adherence includes imperfect \ncompliance, cessation of intervention, crossovers to the comparator intervention and switches to another active intervention.” (Higgins et al., 2016, p. 25).  \n\n5 \n3. Quantitative non-randomized studies Methodological quality criteria \nNon-randomized studies are defined as any quantitative \nstudies estimating the effectiveness of an intervention or \nstudying other exposures that do not use randomization to \nallocate units to comparison groups (Higgins and Green, \n2008). \n \nCommon designs include (this list if not exhaustive): \n \nNon-randomized controlled trials \nThe intervention is assigned by researchers, but there is no \nrandomization, e.g., a pseudo-randomization. A non-\nrandom method of allocation is not reliable in producing \nalone similar groups.  \n \nCohort study  \nSubsets of a defined population are assessed as exposed, \nnot exposed, or exposed at different degrees to factors of \ninterest. Participants are followed over time to determine if \nan outcome occurs (prospective longitudinal). \n \nCase-control study \nCases, e.g., patients, associated with a certain outcome are \nselected, alongside a corresponding group of controls. \nData is collected on whether cases and controls were \nexposed to the factor under study (retrospective). \n \nCross-sectional analytic study \nAt one particular time, the relationship between health-\nrelated characteristics (outcome) and other factors \n(intervention/exposure) is examined. E.g., the frequency of \noutcomes is compared in different population subgroups \naccording to the presence/absence (or level) of the \nintervention/exposure. \n \nKey references for non-randomized studies: Higgins and \nGreen (2008); Porta et al. (2014); Sterne et al. (2016); \nWells et al. (2000) \n3.1. Are the participants representative of the target population? \n \nExplanations  \nIndicators of representativeness include: clear description of the target population and of the sample (inclusion and exclusion criteria), reasons \nwhy certain eligible individuals chose not to participate, and any attempts to achieve a sample of participants that represents the target \npopulation. \n3.2. Are measurements appropriate regarding both the outcome and intervention (or exposure)? \n \nExplanations  \nIndicators of appropriate measurements include: the variables are clearly defined and accurately measured; the measurements are justified and \nappropriate for answering the research question; the measurements reflect what they are supposed to measure; validated and reliability tested \nmeasures of the intervention/exposure and outcome of interest are used, or variables are measured using ‘gold standard’. \n3.3. Are there complete outcome data? \n \nExplanations  \nAlmost all the participants contributed to almost all measures. There is no absolute and standard cut-off value for acceptable complete outcome \ndata. Agree among your team what is considered complete outcome data in your field (and based on the targeted journal) and apply this \nuniformly across all the included studies. For example, in the literature, acceptable complete data value ranged from 80% (Thomas et al., 2004; \nZaza et al., 2000) to 95% (Higgins et al., 2016). Similarly, different acceptable withdrawal/dropouts rates have been suggested: 5% (de Vet et \nal., 1997; MacLehose et al., 2000), 20% (Sindhu et al., 1997; Van Tulder et al., 2003) and 30% for follow-up of more than one year \n(Viswanathan and Berkman, 2012). \n3.4. Are the confounders accounted for in the design and analysis? \n \nExplanations  \nConfounders are factors that predict both the outcome of interest and the intervention received/exposure at baseline. They can distort the \ninterpretation of findings and need to be considered in the design and analysis of a non-randomized study. Confounding bias is low if there is \nno confounding expected, or appropriate methods to control for confounders are used (such as stratification, regression, matching, \nstandardization, and inverse probability weighting).  \n3.5 During the study period, is the intervention administered (or exposure occurred) as intended? \n \nExplanations  \nFor intervention studies, consider whether the participants were treated in a way that is consistent with the planned intervention. Since the \nintervention is assigned by researchers, consider whether there was a presence of contamination (e.g., the control group may be indirectly \nexposed to the intervention) or whether unplanned co-interventions were present in one group (Sterne et al., 2016).  \n \nFor observational studies, consider whether changes occurred in the exposure status among the participants. If yes, check if these changes are \nlikely to influence the outcome of interest, were adjusted for, or whether unplanned co-exposures were present in one group (Morgan et al., \n2017).  \n\n6 \n4. Quantitative descriptive studies Methodological quality criteria \nQuantitative descriptive studies are “concerned with and \ndesigned only to describe the existing distribution of \nvariables without much regard to causal relationships or \nother hypotheses” (Porta et al., 2014, p. 72). They are used \nto monitoring the population, planning, and generating \nhypothesis (Grimes and Schulz, 2002). \n \nCommon designs include the following single-group \nstudies (this list if not exhaustive): \n \nIncidence or prevalence study without comparison \ngroup \nIn a defined population at one particular time, what is \nhappening in a population, e.g., frequencies of factors \n(importance of problems), is described (portrayed). \n \nSurvey \n“Research method by which information is gathered by \nasking people questions on a specific topic and the data \ncollection procedure is standardized and well defined.” \n(Bennett et al., 2011, p. 3). \n \nCase series  \nA collection of individuals with similar characteristics are \nused to describe an outcome. \n \nCase report  \nAn individual or a group with a unique/unusual outcome is \ndescribed in detail. \n \nKey references: Critical Appraisal Skills Programme \n(2017); Draugalis et al. (2008) \n4.1. Is the sampling strategy relevant to address the research question? \n \nExplanations  \nSampling strategy refers to the way the sample was selected. There are two main categories of sampling strategies: probability sampling \n(involve random selection) and non-probability sampling. Depending on the research question, probability sampling might be preferable. Non-\nprobability sampling does not provide equal chance of being selected. To judge this criterion, consider whether the source of sample is \nrelevant to the target population; a clear justification of the sample frame used is provided; or the sampling procedure is adequate.  \n4.2. Is the sample representative of the target population? \n \nExplanations  \nThere should be a match between respondents and the target population. Indicators of representativeness include: clear description of the target \npopulation and of the sample (such as respective sizes and inclusion and exclusion criteria), reasons why certain eligible individuals chose not \nto participate, and any attempts to achieve a sample of participants that represents the target population.  \n4.3. Are the measurements appropriate? \n \nExplanations  \nIndicators of appropriate measurements include: the variables are clearly defined and accurately measured, the measurements are justified and \nappropriate for answering the research question; the measurements reflect what they are supposed to measure; validated and reliability tested \nmeasures of the outcome of interest are used, variables are measured using ‘gold standard’, or questionnaires are pre-tested prior to data \ncollection. \n4.4. Is the risk of nonresponse bias low? \n \nExplanations  \nNonresponse bias consists of “an error of nonobservation reflecting an unsuccessful attempt to obtain the desired information from an eligible \nunit.” (Federal Committee on Statistical Methodology, 2001, p. 6). To judge this criterion, consider whether the respondents and non-\nrespondents are different on the variable of interest. This information might not always be reported in a paper. Some indicators of low \nnonresponse bias can be considered such as a low nonresponse rate, reasons for nonresponse (e.g., noncontacts vs. refusals), and statistical \ncompensation for nonresponse (e.g., imputation). \n \nThe nonresponse bias is might not be pertinent for case series and case report. This criterion could be adapted. For instance, complete data on \nthe cases might be important to consider in these designs.  \n4.5. Is the statistical analysis appropriate to answer the research question? \n \nExplanations  \nThe statistical analyses used should be clearly stated and justified in order to judge if they are appropriate for the design and research question, \nand if any problems with data analysis limited the interpretation of the results. \n  \n\n7 \n5. Mixed methods studies Methodological quality criteria \nMixed methods (MM) research involves combining qualitative \n(QUAL) and quantitative (QUAN) methods. In this tool, to be \nconsidered MM, studies have to meet the following criteria (Creswell \nand Plano Clark, 2017): (a) at least one QUAL method and one QUAN \nmethod are combined; (b) each method is used rigorously in accordance \nto the generally accepted criteria in the area (or tradition) of research \ninvoked; and (c) the combination of the methods is carried out at the \nminimum through a MM design (defined a priori, or emerging) and the \nintegration of the QUAL and QUAN phases, results, and data.  \n \nCommon designs include (this list if not exhaustive): \n \nConvergent design \nThe QUAL and QUAN components are usually (but not necessarily) \nconcomitant. The purpose is to examine the same phenomenon by \ninterpreting QUAL and QUAN results (bringing data analysis together \nat the interpretation stage), or by integrating QUAL and QUAN \ndatasets (e.g., data on same cases), or by transforming data (e.g., \nquantization of qualitative data).  \n \nSequential explanatory design \nResults of the phase 1 - QUAN component inform the phase 2 - QUAL \ncomponent. The purpose is to explain QUAN results using QUAL \nfindings. E.g., the QUAN results guide the selection of QUAL data \nsources and data collection, and the QUAL findings contribute to the \ninterpretation of QUAN results. \n \nSequential exploratory design \nResults of the phase 1 - QUAL component inform the phase 2 - QUAN \ncomponent. The purpose is to explore, develop and test an instrument \n(or taxonomy), or a conceptual framework (or theoretical model). E.g., \nthe QUAL findings inform the QUAN data collection, and the QUAN \nresults allow a statistical generalization of the QUAL findings. \n \nKey references: Creswell et al. (2011); Creswell and Plano Clark, \n(2017); O'Cathain (2010) \n5.1. Is there an adequate rationale for using a mixed methods design to address the research question? \n \nExplanations  \nThe reasons for conducting a mixed methods study should be clearly explained. Several reasons can be invoked such as to \nenhance or build upon qualitative findings with quantitative results and vice versa; to provide a comprehensive and complete \nunderstanding of a phenomenon or to develop and test instruments (Bryman, 2006).  \n5.2. Are the different components of the study effectively integrated to answer the research question? \n \nExplanations  \nIntegration is a core component of mixed methods research and is defined as the “explicit interrelating of the quantitative and \nqualitative component in a mixed methods study” (Plano Clark and Ivankova, 2015, p. 40). Look for information on how \nqualitative and quantitative phases, results, and data were integrated (Pluye et al., 2018). For instance, how data gathered by both \nresearch methods was brought together to form a complete picture (e.g., joint displays) and when integration occurred (e.g., \nduring the data collection-analysis or/and during the interpretation of qualitative and quantitative results).  \n5.3. Are the outputs of the integration of qualitative and quantitative components adequately interpreted? \n \nExplanations  \nThis criterion is related to meta-inference, which is defined as the overall interpretations derived from integrating qualitative and \nquantitative findings (Teddlie and Tashakkori, 2009). Meta-inference occurs during the interpretation of the findings from the \nintegration of the qualitative and quantitative components, and shows the added value of conducting a mixed methods study \nrather than having two separate studies.  \n5.4. Are divergences and inconsistencies between quantitative and qualitative results adequately addressed? \n \nExplanations  \nWhen integrating the findings from the qualitative and quantitative components, divergences and inconsistencies (also called \nconflicts, contradictions, discordances, discrepancies, and dissonances) can be found. It is not sufficient to only report the \ndivergences; they need to be explained. Different strategies to address the divergences have been suggested such as reconciliation, \ninitiation, bracketing and exclusion (Pluye et al., 2009b). Rate this criterion ‘Yes’ if there is no divergence.  \n5.5. Do the different components of the study adhere to the quality criteria of each tradition of the methods involved? \n \nExplanations  \nThe quality of the qualitative and quantitative components should be individually appraised to ensure that no important threats to \ntrustworthiness are present. To appraise 5.5, use criteria for the qualitative component (1.1 to 1.5), and the appropriate criteria for \nthe quantitative component (2.1 to 2.5, or 3.1 to 3.5, or 4.1 to 4.5). The quality of both components should be high for the mixed \nmethods study to be considered of good quality. The premise is that the overall quality of a mixed methods study cannot exceed \nthe quality of its weakest component. For example, if the quantitative component is rated high quality and the qualitative \ncomponent is rated low quality, the overall rating for this criterion will be of low quality.  \n \n\n*Adapted from National Institute for Health Care Excellence. (2012). Methods for the development of nice public health guidance. London: National Institute for Health and Care Excellence; and Scottish Intercollegiate \nGuidelines Network. (2017). Algorithm for classifying study design for questions of effectiveness. Retrieved December 1, 2017, from http://www.sign.ac.uk/assets/study_design.pdf.                    8 \nAlgorithm for selecting the study categories to rate in the MMAT* \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n. \n\n\n9 \nReferences \nAbbott, A. (1998). The causal devolution. Sociological Methods & Research, 27(2), 148-181. \nBennett, C., Khangura, S., Brehaut, J. C., Graham, I. D., Mohe r, D., Potter, B. K., et al. (2011). Reporting guidelines for sur vey research: An analysis of published guidance and reporting practices. PLoS \nMedicine, 8(8), e1001069. \nBryman, A. (2006). Integrating quantitative and qualitative research: How is it done? Qualitative Research, 6(1), 97-113. \nCreswell, J. W. (2013a). Qualitative inquiry and research design: Choosing among five approaches (3rd ed.). Thousand Oaks, CA: SAGE Publications. \nCreswell, J. W. (2013b). Research design: Qualitative, quantitative, and mixed methods approaches. Thousand Oaks, CA: SAGE Publications. \nCreswell, J. W., Klassen, A. C., Pla no Clark, V. L., Smith, K. C. (2011). Best practices for mixed methods research in the health sciences . Bethesda, MD: Office of Behavioral and Social Sciences \nResearch, National Institutes of Health. http://obssr.od.nih.gov/mixed_methods_research. \nCreswell, J. W., & Plano Clark, V. (2017). Designing and conducting mixed methods research (3rd ed.). Thousand Oaks, CA: SAGE Publications. \nCritical Appraisal Skills Programme. (2017). CASP checklists. Retrieved December 1, 2017, from http://www.casp-uk.net/casp-tools-checklists.  \nde Vet, H. C., de Bie, R. A., van der Heijden, G. J., Verhagen, A. P., Sijpkes, P., & Knipschild, P. G. (1997). Systematic reviews on the basis of methodological criteria. Physiotherapy, 83(6), 284-289. \nDraugalis, J. R., Coons, S. J., & Plaza, C. M. (2008). Best practices for survey research reports: A synopsis for authors and reviewers. American Journal of Pharmaceutical Education, 72(1), Article 11. \nFederal Committee on Statistical Methodology. (2001). Measuring and reporting sources of error in surveys . Washington DC: Statistical Policy Office, O ffice of Information and Regulatory Affairs, \nOffice of Management and Budget. \nGrimes, D. A., & Schulz, K. F. (2002). Descriptive studies: What they can and cannot do. The Lancet, 359(9301), 145-149. \nHiggins, J. P., & Green, S. (2008). Cochrane handbook for systematic reviews of interventions. Chichester, UK: Wiley Online Library. \nHiggins, J. P. T., Sterne, J. A. C., Savovi ć, J., Page, M. J., Hróbjartsson, A., Boutron, I., et al. (2016). A revised tool for assessing risk of bias in randomized trials . In Chandler, J., McKenzie, J., Boutron, \nI. & Welch, V. (Eds.), Cochrane Methods. Cochrane Database of Systematic Reviews, Issue 10 (Suppl 1). \nHong, Q. N. (2018). Revision of the Mixed Methods Appraisal Tool (MMAT): A mixed methods study (Doctoral dissertation). Department of Family Medicine, McGill University, Montréal.  \nMacLehose, R. R., Reeves, B. C., Harvey, I. M., Sheldon, T. A., Russell, I. T., & Black, A. M. (2000). A systematic review of c omparisons of effect sizes derive d from randomised and non-randomised \nstudies. Health Technology Assessment, 4(34), 1-154. \nMorgan, R., Sterne, J., Higgins, J., Thayer, K ., Schunemann, H., Rooney, A., et al. (2017). A new instrument to assess Risk of Bias in Non-r andomised Studies of Exposures (ROBINS-E): Application to \nstudies of environmental exposure.  Abstracts of the Global Evidence Summit, Cape Town, South Africa. Cochrane Datab ase of Systematic Reviews 2017, Issue 9 (Suppl  1). \nhttps://doi.org/10.1002/14651858.CD201702. \nO'Cathain, A. (2010). Assessing the quality of mixed methods resear ch: Towards a comprehe nsive framework. In Tashakkori, A. & T eddlie, C. (Eds.), Handbook of Mixed methods in social and \nbehavioral research (pp. 531-555). Thousand Oaks, CA: SAGE Publications. \nOxford Centre for Evidence-based Medicine. (2016). Levels of evidence. Retrieved February 19, 2018, from https://www.cebm.net/2016/05/ocebm-levels-of-evidence/.  \nPace, R., Pluye, P., Bartlett, G., Macaula y, A. C., Salsberg, J., Jagosh, J., et al. (2012). Testing the reliability and effici ency of the pilot Mixed Methods Appraisal Tool (MMAT) for systematic mixed \nstudies review. International Journal of Nursing Studies, 49(1), 47-53. \nPlano Clark, V. L., & Ivankova, N. V. (2015). Mixed methods research: A guide to the field. Thousand Oaks, CA: SAGE Publications. \nPluye, P., Gagnon, M. P., Griffiths, F., Johnson-Lafleur, J. (2009a). A scoring system for appraising mixed methods research, a nd concomitantly appraising qualitati ve, quantitative and mixed methods \nprimary studies in mixed studies reviews. International Journal of Nursing Studies, 46(4), 529-546. \nPluye, P., Grad, R. M., Levine, A., & Nicolau, B. (2009b). Understanding divergence of quantitative and qualitative data (or results) in mixed methods studies. International Journal of Multiple Research \nApproaches, 3(1), 58-72. \nPluye, P., Garcia Bengoechea, E., Granikov, V., Kaur, N., & Ta ng, D. L. (2018). A world of possibilities in mixed methods: Revi ew of the combinations of strategies used to integrate the phases, results, \nand qualitative and quantitative data. International Journal of Multiple Research Approaches, 10(1), 41-56. \nPorta, M. S., Greenland, S., Hernán, M., dos Santos Silva, I., Last, J. M. (2014). A dictionary of epidemiology. New York: Oxford University Press. \nSandelowski, M. (2010). What's in a name? Qualitative description revisited. Research in Nursing and Health, 33(1), 77-84. \n\n10 \nSchwandt, T. A. (2015). The SAGE dictionary of qualitative inquiry. Thousand Oaks, CA: SAGE Publications. \nSindhu, F., Carpenter, L., & Seers, K. (1997). Development of a tool to rate the quality assessment of randomized controlled tr ials using a Delphi technique. Journal of Advanced Nursing, 25 (6), 1262-\n1268. \nSterne, J. A., Hernán, M. A., Reeves, B. C., Savovi ć, J., Berkman, N. D., Viswanathan, M., et al. (2016). ROBINS-I:  A tool for assessing risk of bias in non-randomised studies of interventions. British \nMedical Journal, 355(i4919). \nTeddlie, C., & Tashakkori, A. (2009). Foundations of mixed methods research: In tegrating quantitative and qualitative approach es in the social and behavioral science s. Thousand Oaks, CA: SAGE \nPublications. \nThomas, B. H., Ciliska, D., Dobbins, M., & Micucci, S. (2004). A process for systematically reviewing the literature: Providing  the research evidence for public health nursing interventions. Worldviews \non Evidence-Based Nursing, 1(3), 176-184. \nVan Tulder, M., Furlan, A., Bombardier, C., Bouter, L., & Editorial Board of the Cochrane Collaboration Back Review Group. (2003). Updated method guidelines for systematic reviews in the Cochrane \nCollaboration Back Review Group. Spine, 28(12), 1290-1299. \nViswanathan, M., & Berkman, N. D. (2012). Development of the RTI item bank on risk of bias and precision of observational studies. Journal of Clinical Epidemiology, 65(2), 163-178. \nWells, G., Shea, B., O’connell, D., Peterson, J., Welch, V., Losos, M., et al. (2000). The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Retrieved \nApril 16, 2016, from http://www.ohri.ca/programs/clinical_epidemiology/nosgen.pdf.  \nZaza, S., Wright-De Agüero, L. K., Briss, P. A., Truman, B. I ., & Hopkins, D. P. (2000). Data collection instrument and procedu re for systematic reviews in the guide to community preventive services. \nAmerican Journal of Preventive Medicine, 188(Suppl 1), 44-74.","source_license":"CC0","license_restricted":false}