Variance Estimation for Assessing Healthcare Providers’ Performance using log Standardized Incidence Ratio

Introduction In healthcare providers’ performance assessment, standardized incidence ratios (SIRs) are essential tools used to assess whether observed event rates deviate from expected values. Accurate estimation of variance in these ratios is crucial as it affects decision-making regarding providers’ performance. There is little data on how the choice of these variance estimation methods affects decision-making. In this paper, we compared three methods, namely, delta-method, bootstrapping and Bayesian approaches, to estimate the variance of the logarithm of SIR (Log-SIR).

and analysis Using patient-level data from Australia and New Zealand Dialysis and Transplant Registry (ANZDATA) for 2005-2023, we used a random effects model to predict treatment at home one year after starting treatment. We compared the three approaches (with over 5000 iterations for bootstrapping and MCMC sampling) using bias, variance and mean square errors (MSE) as performance measures. Using the three methods, funnel plots were used to compare the hospitals’ performance in treating Indigenous and non-Indigenous patients close to home, as a service-level measure of equity.

The bias values across all methods are similar, with Bayesian narrowly having the lowest bias (0.01922), followed by the delta-method (0.01927) and Bootstrap (0.02567). In addition, the Bayesian exhibits the lowest variance (0.00005), indicating more stable and less dispersed estimates. The delta-method has a higher variance (0.00016), while Bootstrap has the highest variance (0.00027), meaning it introduces more uncertainty. Finally, the Bayesian has the lowest MSE (0.00042), indicating better overall accuracy while the Bootstrap has the highest MSE (0.00094), showing it is the least reliable method.

We demonstrate that these methods can be used to measure equity for patient-centred outcomes, both within and between service providers simultaneously. The choice of variance estimation method is critical and heavily affects the interpretation of the performance of health service providers. We favour the Bayesian MCMC method. The Bayesian MCMC method found to be better approach. Competing Interest Statement The authors have declared no competing interest. Funding Statement This study is funded by the National Health and Medical Research Council of Australia (GNT1158075) Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Yes The details of the IRB/oversight body that provided approval or exemption for the research described are given below: Ethics approval has been obtained so far from the Human Research Ethics Committee (HREC) of the Northern Territory Department of Health and Menzies School of Health Research (2019-3530), Far North Queensland HREC (2023/QCH/99606 (Nov ver 4)-1732), the Central Adelaide Local Health Network HREC (2023/HRE00209), the Aboriginal Health Council of South Australia (AHREC Protocol \#: 04-23-1078), the Aboriginal Health and Medical Research Council of New South Wales (AH\&MRC HREC reference: 2230/24) and the Far North Queensland Human Research Ethics Committee (FNQ HREC reference: HREC/2023/QCH/99606 (Nov ver 4)-1732). I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Yes I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Yes I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Yes Data Availability All data produced in the present study are available upon reasonable request to the authors Abbreviations - SIR - Standardised Incidence Ratio - SMR - Standardised Mortality Ratio - Log-SIR - Logarithm of the Standardised Incidence Rati - SD - Standard Deviations - MSE - Mean Square Errors - MCMC - Markov Chain Monte Carlo - ANZDATA - Australia and New Zealand Dialysis and Transplant Registry - KRT - Kidney Replacement Therapy - FDR - False Discovery Rate