Meta-analysis models relaxing the random effects normality assumption: methodological systematic review and simulation study

Meta-analysis models relaxing the random effects normality assumption: methodological systematic review and simulation study | Research Square window.SnipcartSettings = { analytics: { enabled: false } }; (function() { var accessVector = localStorage.getItem('access_vector') || ''; window.dataLayer = window.dataLayer || []; if (accessVector) { window.dataLayer.push({ user: { profile: { profileInfo: { snid: accessVector } } } }); } })(); (function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src='https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);})(window,document,'script','dataLayer','GTM-K279D39R'); Browse Preprints In Review Journals COVID-19 Preprints AJE Video Bytes Research Tools Research Promotion AJE Professional Editing AJE Rubriq About Preprint Platform In Review Editorial Policies Our Team Advisory Board Help Center Sign In Submit a Preprint Cite Share Download PDF Research Article Meta-analysis models relaxing the random effects normality assumption: methodological systematic review and simulation study Kanella Panagiotopoulou, Theodoros Evrenoglou, Christopher H Schmid, and 2 more This is a preprint; it has not been peer reviewed by a journal. https://doi.org/ 10.21203/rs.3.rs-5722159/v1 This work is licensed under a CC BY 4.0 License Status: Published Journal Publication published 16 Oct, 2025 Read the published version in BMC Medical Research Methodology → Version 1 posted 10 You are reading this latest preprint version Abstract Background Random effects meta-analysis is widely used for synthesizing the studies of a systematic review assuming a normal between-study distribution. However, this assumption might not always be plausible. Alternative options have been suggested but not used in published meta-analyses. Methods We conducted a systematic review to identify articles that proposed alternative meta-analysis models assuming non-normal distributions for the random effects, such as skewed or semi-parametric distributions. Subsequently, we performed a simulation study to evaluate the performance of the identified models and to compare them with the normal model. We considered 22 scenarios varying the amount of between study variance, the number of included studies, and the shape of the true distribution: normal, skew-normal, and mixture of two normal distributions. For each scenario, we generated 1000 meta-analyses datasets. To investigate additional aspects of the alternative models, we also applied them at three extracted simulated datasets representing three scenarios with different true distributions. Results We identified in total 27 articles suggesting 24 alternative models that can be classified into three broad categories: models based on long-tail and skewed distributions, on mixtures of distributions, and on Dirichlet process priors (DP). We compared 15 models in our simulation study implemented in the Frequentist or Bayesian framework. Results revealed small differences in bias between the different models but larger differences in the level of coverage probability. In scenarios with large between-study variance, all models were substantially biased in the estimation of the mean treatment effect. However, mixture and semi-parametric models revealed latent underlying clustering of studies and assisted to form subgroups of common characteristics. The three simulated datasets demonstrated similar patterns with the simulation study for the bias of the mean treatment effect. Conclusion Focusing only on the mean treatment effect of the random effects meta-analysis can be misleading when substantial heterogeneity is suspected or outliers are present. In such cases, identifying the factors that differentiate the studies and looking at the prediction intervals can be very informative. Based on our simulation, investigators could have the normal model as their starting point and consider alternative models as sensitivity analysis in view of seemingly non-normal data. Clinical trial number: not applicable evidence synthesis semi-parametric models skewed data outliers heterogenous studies Figures Figure 1 Figure 2 Figure 3 Figure 4 Figure 5 Figure 6 Full Text Additional Declarations No competing interests reported. Supplementary Files NonnormalmetaanalysismodelsSupplrev.docx Cite Share Download PDF Status: Published Journal Publication published 16 Oct, 2025 Read the published version in BMC Medical Research Methodology → Version 1 posted Editorial decision: Revision requested 02 Jun, 2025 Reviews received at journal 25 May, 2025 Reviews received at journal 19 May, 2025 Reviewers agreed at journal 22 Apr, 2025 Reviews received at journal 18 Apr, 2025 Reviewers agreed at journal 14 Apr, 2025 Reviewers agreed at journal 14 Apr, 2025 Reviewers invited by journal 14 Apr, 2025 Submission checks completed at journal 11 Apr, 2025 First submitted to journal 07 Apr, 2025 You are reading this latest preprint version Research Square lets you share your work early, gain feedback from the community, and start making changes to your manuscript prior to peer review in a journal. As a division of Research Square Company, we’re committed to making research communication faster, fairer, and more useful. We do this by developing innovative software and high quality services for the global research community. Our growing team is made up of researchers and industry professionals working together to solve the most critical problems facing scientific publishing. Also discoverable on Platform About Our Team In Review Editorial Policies Advisory Board Help Center Resources Author Services Accessibility API Access RSS feed Manage Cookie Preferences © Research Square 2026 | ISSN 2693-5015 (online) Privacy Policy Terms of Service Do Not Sell My Personal Information {"props":{"pageProps":{"initialData":{"identity":"rs-5722159","acceptedTermsAndConditions":true,"allowDirectSubmit":false,"archivedVersions":[],"articleType":"Research Article","associatedPublications":[],"authors":[{"id":442770926,"identity":"6d88504e-1b62-431f-8e1d-2ff3cf8f3a82","order_by":0,"name":"Kanella Panagiotopoulou","email":"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAZAAAAAyAQMAAABI0h/eAAAABlBMVEX///8AAABVwtN+AAAACXBIWXMAAA7EAAAOxAGVKw4bAAAA/ElEQVRIiWNgGAWjYBACCRDxoADK+wDEbOzEaEkwgHAYZ4C0MJOihZkHTBLQItl+9uCDBAO7PP7ZzY9f2/zaJs/HzMD44WMObi3SPHnJBgkGycUSd46ZWef23TZsY2Zglpy5DbcWOYYcM4kEA+bEhhsJZsa5PbcZgVrYmHnxaeF/Y/4jwaA+cf6N9G/Glj237QlqkZbIMQN6/3Dihhs5xo8ZftxOJKhFcsYbY6DDjhcb3sgpY+xtuJ3cxszYjNcvEudzDD98qKjOk7uRvvnDjz+3bee3Nx/88BGPFhhIAGI2CcY2EJuxgbB6qBbmDwx/iFI8CkbBKBgFIwwAAGyVUPgDTLfeAAAAAElFTkSuQmCC","orcid":"","institution":"Université Paris Cité, Inserm","correspondingAuthor":true,"prefix":"","firstName":"Kanella","middleName":"","lastName":"Panagiotopoulou","suffix":""},{"id":442770927,"identity":"da9b9a57-91d4-46fa-82fe-17f55c2223a1","order_by":1,"name":"Theodoros Evrenoglou","email":"","orcid":"","institution":"University of Freiburg","correspondingAuthor":false,"prefix":"","firstName":"Theodoros","middleName":"","lastName":"Evrenoglou","suffix":""},{"id":442770928,"identity":"b4641f8f-7269-4e37-bb8c-b07235a4378a","order_by":2,"name":"Christopher H Schmid","email":"","orcid":"","institution":"Brown University","correspondingAuthor":false,"prefix":"","firstName":"Christopher","middleName":"H","lastName":"Schmid","suffix":""},{"id":442770929,"identity":"ef3db9d0-844c-41db-8710-785e1546e58f","order_by":3,"name":"Silvia Metelli","email":"","orcid":"","institution":"Université Paris Cité, Inserm","correspondingAuthor":false,"prefix":"","firstName":"Silvia","middleName":"","lastName":"Metelli","suffix":""},{"id":442770930,"identity":"ea4bd04e-5876-4dd1-860d-cd7f665567cf","order_by":4,"name":"Anna Chaimani","email":"","orcid":"","institution":"Université Paris Cité, Inserm","correspondingAuthor":false,"prefix":"","firstName":"Anna","middleName":"","lastName":"Chaimani","suffix":""}],"badges":[],"createdAt":"2024-12-27 14:23:12","currentVersionCode":1,"declarations":"","doi":"10.21203/rs.3.rs-5722159/v1","doiUrl":"https://doi.org/10.21203/rs.3.rs-5722159/v1","draftVersion":[],"editorialEvents":[{"content":"https://doi.org/10.1186/s12874-025-02658-3","type":"published","date":"2025-10-16T15:58:13+00:00"}],"editorialNote":"","failedWorkflow":false,"files":[{"id":80599891,"identity":"9a086df0-54bc-4fef-845c-be09fb077c7b","added_by":"auto","created_at":"2025-04-15 05:07:14","extension":"jpg","order_by":1,"title":"Figure 1","display":"","copyAsset":false,"role":"figure","size":772548,"visible":true,"origin":"","legend":"\u003cp\u003eMeta-analysis of two simulated sets of studies generated from two normal distributions with the same mean but different variances. The dataset of panel (a) was generated from \u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;\u0026nbsp;and that of panel (b) from\u003c/p\u003e","description":"","filename":"Figure1.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5722159/v1/7cc6c19ffd1b30b3db43396b.jpg"},{"id":80601478,"identity":"2f0e3211-39a6-484a-862b-7cd3146ea60e","added_by":"auto","created_at":"2025-04-15 05:31:51","extension":"jpg","order_by":2,"title":"Figure 2","display":"","copyAsset":false,"role":"figure","size":960204,"visible":true,"origin":"","legend":"\u003cp\u003eSimulation results in terms of mean of absolute bias for the mean of the random effects distribution. The names of the models are explained in Table 2. (NC=Non-convergence)\u003c/p\u003e","description":"","filename":"Figure2.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5722159/v1/16ab26339158362d8279bf36.jpg"},{"id":80601520,"identity":"b0b7185e-2e17-4623-9c18-916df7132e40","added_by":"auto","created_at":"2025-04-15 05:32:42","extension":"jpg","order_by":3,"title":"Figure 3","display":"","copyAsset":false,"role":"figure","size":759290,"visible":true,"origin":"","legend":"\u003cp\u003eSimulation results in terms of coverage probability for the mean of the random effects distribution. The horizontal lines represent the upper and lower bounds of the 95% confidence interval for the nominal level. The names of the models are explained in Table 2. (NC=Non-convergence)\u003c/p\u003e","description":"","filename":"Figure3.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5722159/v1/cd95b9729e8cd0dc4681fe19.jpg"},{"id":80599295,"identity":"e18a11f4-3676-4607-8b39-8d6fb98ba6ff","added_by":"auto","created_at":"2025-04-15 04:59:14","extension":"jpg","order_by":4,"title":"Figure 4","display":"","copyAsset":false,"role":"figure","size":800902,"visible":true,"origin":"","legend":"\u003cp\u003eSimulation results in terms of mean of absolute bias for the between-study standard deviation. The names of the models are explained in Table 2. (NC=Non-convergence)\u003c/p\u003e","description":"","filename":"Figure4.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5722159/v1/1d39d46356ce4e8c1727bc28.jpg"},{"id":80601481,"identity":"f6d5476a-079a-4daf-8a5f-e9cdfe0d030d","added_by":"auto","created_at":"2025-04-15 05:31:53","extension":"jpg","order_by":5,"title":"Figure 5","display":"","copyAsset":false,"role":"figure","size":760297,"visible":true,"origin":"","legend":"\u003cp\u003eSimulation results in terms of percent relative bias for the between-study standard deviation. The names of the models are explained in Table 2. (NC=Non-convergence)\u003c/p\u003e","description":"","filename":"Figure5.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5722159/v1/4a7ddf33e3086078586408d8.jpg"},{"id":80599297,"identity":"b3bdb670-f630-4e82-a0c5-1f2015bb5d80","added_by":"auto","created_at":"2025-04-15 04:59:14","extension":"jpg","order_by":6,"title":"Figure 6","display":"","copyAsset":false,"role":"figure","size":1052042,"visible":true,"origin":"","legend":"\u003cp\u003eSimulation results in terms of mean absolute bias for the study-specific treatment effects averaged within meta-analyses and across meta-analyses. The names of the models are explained in Table 2. (NC=Non-convergence)\u003c/p\u003e","description":"","filename":"Figure6.jpg","url":"https://assets-eu.researchsquare.com/files/rs-5722159/v1/661f8fd889c52af39b5afcdd.jpg"},{"id":93956051,"identity":"8575af30-ed96-47cc-a6ef-ae98e7180d6a","added_by":"auto","created_at":"2025-10-20 16:09:49","extension":"pdf","order_by":1,"title":"","display":"","copyAsset":false,"role":"manuscript-pdf","size":5768103,"visible":true,"origin":"","legend":"","description":"","filename":"manuscriptNonnormalmetaanalysismodelsrev.pdf","url":"https://assets-eu.researchsquare.com/files/rs-5722159/v1_covered_d3349698-59cc-464c-8c98-9720e2311c97.pdf"},{"id":80601476,"identity":"ec2a5072-bec8-4553-a9a3-e13a92840800","added_by":"auto","created_at":"2025-04-15 05:31:48","extension":"docx","order_by":0,"title":"","display":"","copyAsset":false,"role":"supplement","size":5379650,"visible":true,"origin":"","legend":"","description":"","filename":"NonnormalmetaanalysismodelsSupplrev.docx","url":"https://assets-eu.researchsquare.com/files/rs-5722159/v1/11c0e0814ca9a46280a7cbb4.docx"}],"financialInterests":"No competing interests reported.","formattedTitle":"Meta-analysis models relaxing the random effects normality assumption: methodological systematic review and simulation study","fulltext":[],"fulltextSource":"","fullText":"","funders":[],"hasAdminPriorityOnWorkflow":false,"hasManuscriptDocX":false,"hasOptedInToPreprint":true,"hasPassedJournalQc":"","hasAnyPriority":false,"hideJournal":false,"highlight":"","institution":"","isAcceptedByJournal":true,"isAuthorSuppliedPdf":true,"isDeskRejected":"","isHiddenFromSearch":false,"isInQc":false,"isInWorkflow":false,"isPdf":true,"isPdfUpToDate":true,"isWithdrawnOrRetracted":false,"journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-research-methodology","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"bmrm","sideBox":"Learn more about [BMC Medical Research Methodology](http://bmcmedresmethodol.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/bmrm/default.aspx","title":"BMC Medical Research Methodology","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true},"keywords":"evidence synthesis, semi-parametric models, skewed data, outliers, heterogenous studies","lastPublishedDoi":"10.21203/rs.3.rs-5722159/v1","lastPublishedDoiUrl":"https://doi.org/10.21203/rs.3.rs-5722159/v1","license":{"name":"CC BY 4.0","url":"https://creativecommons.org/licenses/by/4.0/"},"manuscriptAbstract":"\u003cp\u003e\u003cstrong\u003eBackground\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eRandom effects meta-analysis is widely used for synthesizing the studies of a systematic review assuming a normal between-study distribution. However, this assumption might not always be plausible. Alternative options have been suggested but not used in published meta-analyses.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMethods\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe conducted a systematic review to identify articles that proposed alternative meta-analysis models assuming non-normal distributions for the random effects, such as skewed or semi-parametric distributions. Subsequently, we performed a simulation study to evaluate the performance of the identified models and to compare them with the normal model. We considered 22 scenarios varying the amount of between study variance, the number of included studies, and the shape of the true distribution: normal, skew-normal, and mixture of two normal distributions. For each scenario, we generated 1000 meta-analyses datasets. To investigate additional aspects of the alternative models, we also applied them at three extracted simulated datasets representing three scenarios with different true distributions.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eResults\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eWe identified in total 27 articles suggesting 24 alternative models that can be classified into three broad categories: models based on long-tail and skewed distributions, on mixtures of distributions, and on Dirichlet process priors (DP). We compared 15 models in our simulation study implemented in the Frequentist or Bayesian framework. Results revealed small differences in bias between the different models but larger differences in the level of coverage probability. In scenarios with large between-study variance, all models were substantially biased in the estimation of the mean treatment effect. However, mixture and semi-parametric models revealed latent underlying clustering of studies and assisted to form subgroups of common characteristics. The three simulated datasets demonstrated similar patterns with the simulation study for the bias of the mean treatment effect.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eConclusion\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eFocusing only on the mean treatment effect of the random effects meta-analysis can be misleading when substantial heterogeneity is suspected or outliers are present. In such cases, identifying the factors that differentiate the studies and looking at the prediction intervals can be very informative. Based on our simulation, investigators could have the normal model as their starting point and consider alternative models as sensitivity analysis in view of seemingly non-normal data.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eClinical trial number: \u003c/strong\u003enot applicable\u003c/p\u003e","manuscriptTitle":"Meta-analysis models relaxing the random effects normality assumption: methodological systematic review and simulation study","msid":"","msnumber":"","nonDraftVersions":[{"code":1,"date":"2025-04-15 04:59:09","doi":"10.21203/rs.3.rs-5722159/v1","editorialEvents":[{"type":"communityComments","content":0},{"type":"decision","content":"Revision requested","date":"2025-06-02T13:24:50+00:00","index":"","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-05-25T13:53:54+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-05-19T21:15:45+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"172144901053553297086677829647146076190","date":"2025-04-22T07:47:48+00:00","index":"hide","fulltext":""},{"type":"editorInvitedReview","content":"","date":"2025-04-18T16:21:25+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"328119498419816107308267703240155215918","date":"2025-04-14T14:13:37+00:00","index":"hide","fulltext":""},{"type":"reviewerAgreed","content":"178132623080248916249662817324302227383","date":"2025-04-14T13:27:16+00:00","index":"hide","fulltext":""},{"type":"reviewersInvited","content":"","date":"2025-04-14T12:21:11+00:00","index":"","fulltext":""},{"type":"checksComplete","content":"","date":"2025-04-11T09:58:43+00:00","index":"","fulltext":""},{"type":"submitted","content":"BMC Medical Research Methodology","date":"2025-04-07T14:39:32+00:00","index":"","fulltext":""}],"status":"published","journal":{"display":true,"email":"[email protected]","identity":"bmc-medical-research-methodology","isNatureJournal":false,"hasQc":true,"allowDirectSubmit":false,"externalIdentity":"bmrm","sideBox":"Learn more about [BMC Medical Research Methodology](http://bmcmedresmethodol.biomedcentral.com/)","snPcode":"","submissionUrl":"https://www.editorialmanager.com/bmrm/default.aspx","title":"BMC Medical Research Methodology","twitterHandle":"BMC_series","acdcEnabled":true,"dfaEnabled":false,"editorialSystem":"em","reportingPortfolio":"BMC Series","inReviewEnabled":true,"inReviewRevisionsEnabled":true}}],"origin":"","ownerIdentity":"574734fd-b31e-44ee-8214-12ff601abefd","owner":[],"postedDate":"April 15th, 2025","published":true,"recentEditorialEvents":[],"rejectedJournal":[],"revision":"","amendment":"","status":"published-in-journal","subjectAreas":[],"tags":[],"updatedAt":"2025-10-20T16:03:36+00:00","versionOfRecord":{"articleIdentity":"rs-5722159","link":"https://doi.org/10.1186/s12874-025-02658-3","journal":{"identity":"bmc-medical-research-methodology","isVorOnly":false,"title":"BMC Medical Research Methodology"},"publishedOn":"2025-10-16 15:58:13","publishedOnDateReadable":"October 16th, 2025"},"versionCreatedAt":"2025-04-15 04:59:09","video":"","vorDoi":"10.1186/s12874-025-02658-3","vorDoiUrl":"https://doi.org/10.1186/s12874-025-02658-3","workflowStages":[]},"version":"v1","identity":"rs-5722159","journalConfig":"researchsquare"},"__N_SSP":true},"page":"/article/[identity]/[[...version]]","query":{"redirect":"/article/rs-5722159","identity":"rs-5722159","version":["v1"]},"buildId":"8U1c8b4HqxoKbykW_rLl7","isFallback":false,"isExperimentalCompile":false,"dynamicIds":[84888],"gssp":true,"scriptLoader":[]}