Beyond Agreement: Standardizing Crowdsourced Synapse Annotations through Proofreading in EM Connectomics

A bstract Reliable synapse identification in volumetric EM is hampered by subtle, 3D cues that yield variable human judgments. We present a standardized proofreading protocol that pairs explicit, operational criteria with machine-learning candidate generation and a two-stage calibration of annotators. In two larval Drosophila melanogaster volumes imaged at 8×8×8 nm, five raters (expert + 4 calibrated annotators) reviewed model-proposed candidates using efficient node-based labels. Multi-rater judgments were aggregated with a probabilistic Dawid–Skene (DS) model to produce consensus labels with calibrated uncertainty. Post-calibration, individual annotator accuracy versus the expert improved (McNemar p < 0.05 for all raters), DS–expert agreement increased, and DS posterior entropy decreased for true positives/negatives, indicating more decisive consensus; gains were modest and dataset-dependent in chance-corrected agreement (Krippendorff’s α ). By making uncertainty explicit, this protocol converts noisy judgments into auditable supervision suitable for training and evaluation, while honestly communicating residual ambiguity essential for reliable and robust connectomics at scale.