On correctness of gene tree tagging under a unified model of gene duplication, loss, and coalescence

Abstract ASTRAL-pro is the leading method for reconstructing species trees under complex evolutionary scenarios involving gene duplication, loss, and coalescence, commonly modeled by DLCoal. A unique aspect of A-pro is that it utilizes rooted gene trees, with internal vertices labeled as duplications or speciations, to modify its objective function compared to the traditional ASTRAL method. Although there is a natural event-based definition of correct tagging when genes evolve with only gene duplications and losses, it cannot be applied when there is deep coalescence. Here, we introduce a definition of correct tagging that is broadly applicable, proposing that a gene tree vertex is correctly tagged as a duplication if it is the most recent common ancestor of at least one pair of gene copies related via a duplication event. Using this definition, we study some statistical properties of ASTRAL-pro’s objective function under the DLCoal model and evaluate the accuracy of ASTRAL-pro’s tagging algorithm in simulations. Competing Interest Statement The authors have declared no competing interest. Footnotes After submitting to bioRxiv, we noticed that one of our claims was incorrect and a more delicate argument is needed to prove the consistency of Apro. This revision makes it clear that the consistency of Apro is still an open conjecture; instead focuses on our contributions of the correctness of gene tree tagging.