Adaptive Cluster-Count Autoencoders with Dirichlet Process Priors for Geometry-Aware Single-Cell Representation Learning

Standard autoencoders for single-cell transcriptomics learn latent spaces whose cluster structure emerges only post hoc through K -means or community detection, leaving cluster count and boundary quality uncontrolled during training. Here we ask whether imposing an adaptive nonparametric prior can shift this balance. We equip a feedforward autoen-coder with an online Dirichlet Process Mixture Model (DPMM) prior that refits cluster assignments throughout training and directly regularizes latent compactness and separation. Across 56 scRNA-seq datasets the DPMM prior produces a pronounced geometry– concordance trade-off : cluster compactness (ASW) improves by 127% and Davies–Bouldin overlap drops by 47%, but label-recovery metrics decline (NMI −17%, ARI −21%) and downstream k NN accuracy falls from 0.784 to 0.725. Wilcoxon signed-rank tests confirm that the geometry gains are significant with large Cliff’s δ effects while concordance losses remain bounded and non-significant. A second-stage conditional-flow refinement (DPMM-FM) further improves projection fidelity (DRE 0.751, LSE 0.695, DREX 0.873) at additional concordance cost, revealing a three-tier operating regime: prior-free for label recovery, DPMM for manifold geometry, and DPMM-FM for visualization fidelity. Against 18 external baselines DPMM-Base wins 70.5% of core-metric comparisons ( p <0.05). Gene Ontology enrichment confirms that geometry-improved latent components recover coherent biological programs. Rather than claiming universal superiority, this study characterizes the operating envelope of nonparametric mixture priors and identifies the task contexts— trajectory analysis, manifold visualization, and program-level annotation—where adaptive geometric structure outweighs label-counting accuracy.