Penalized empirical likelihood estimation and EM algorithms for closed-population capture--recapture models

Abstract Capture--recapture experiments are widely used to estimatethe abundance of a finite population.Based on capture--recapture data, the empirical likelihood (EL) methodhas been shown to outperform the conventionalconditional likelihood (CL) method.However, the current literature on EL abundance estimationignores behavioral effects,and the EL estimates may not be stable,especially when the capture probability is low.We make three contributions in this paper.First, we extend the EL method to capture--recapture models that account for behavioral effects.Second, to overcome the instability of the EL method,we propose a penalized EL (PEL) estimation methodthat penalizes large abundance values.We then investigate the asymptotics of the maximum PEL estimator and the PEL ratio statistic.Third, we develop %novelstandard expectation--maximization (EM) algorithmsfor PEL to improve its practical performance.The EM algorithm is also applicable to EL and CL with slight modifications.Our simulation and a real-world data analysis demonstrate that the PEL method successfully overcomes the instability of the EL methodand the proposed EM algorithm produces more reliableresults than existing optimization algorithms.