LEMMIv2: Benchmarking Framework for Metagenomic and 16S Amplicon Profilers with a Catalogue of Evaluated Tools

31 Next-generation sequencing has become the state-of-the-art method for studying 32 microorganisms, enabling the profiling of entire microbial communities through shotgun 33 metagenomics, including species that have not been cultivated and without prior knowledge of 34 the sample's contents. Over the past decade, data production has increased exponentially. In 35 response, researchers have developed numerous computational methods to assign taxonomic 36 labels to sequencing reads more efficiently and at scale, while improving classification quality 37 (e.g., Wood et al., (2019), Blanco-Míguez et al., (2023)). Along with the growing volume of data, 38 the evolution of sequencing technologies has driven innovation in the field. Illumina platforms, 39 which produce short reads, remain the most widely used due to their accuracy and throughput. 40 However, novel long-read sequencing technologies from PacBio and Oxford Nanopore (ONT) are 41 increasingly being used for metagenomics (Agustinho et al., 2024), leading to the development 42 of computational methods fine-tuned to their specific characteristics. 43 Individual research groups and larger communities have surveyed and evaluated these tools to 44 help users make informed decisions. Self-benchmarking has also become a standard practice 45 when publishing new tools. While informative, one-off benchmarking papers provide snapshots 46 that quickly become outdated. When authors introduce new metrics and innovative dataset 47 designs, they often make comparisons with existing data difficult or impossible. Given the 48 proliferation of benchmarking methods in various areas of genomics, including metagenomic 49 profiling, there is a need for more sustainable and reproducible approaches in the long term 50 (Mangul et al., 2019). 51 In this context, we developed the LEMMI benchmarking platform for metagenomics profilers 52 (Seppey et al., 2020), available at https://lemmi-v1.ezlab.org/. Our objective was to offer a 53 framework for continuous assessment of candidate tools under stable conditions, using public 54 data to simulate microbial communities and Illumina datasets, while controlling the reference 55

available to the tools. This platform can incorporate tools that did not exist when the 56 benchmark was first designed, maintaining a consistent challenge for newcomers. The 57 benchmark results are presented on an interactive web app. The platform ensures access to the 58 evaluated methods through publicly available software containers, which persist even if 59 developers discontinue the maintenance of the tools. Since launching the platform, we have 60 enabled submissions of new tools via software containers, provided they followed a specific 61 design procedure that allowed our team to conduct the evaluation. While this approach fulfilled 62 our initial goal of maintaining an incremental benchmark, we later aimed to provide a LEMMI 63 framework that allows developers and users to run evaluations independently. This would enable 64 them to replicate our analyses, design their own benchmarks, and more easily prepare their tools 65 to be compatible with the LEMMI procedure we run. This manuscript presents version 2 of LEMMI 66 (LEMMIv2, https://lemmi.ezlab.org/) for metagenomics profilers, highlighting key innovations such 67 as a standalone pipeline and the addition of long-read technology. This version builds on our 68 learnings from the initial release and represents a complete refactoring of both the benchmarking 69 methodology and the results presentation interface. 70 The second objective of this manuscript is to introduce LEMMI16S (https://lemmi16S.ezlab.org/), 71 a dedicated LEMMI benchmark for amplicon sequencing. Amplicon sequencing remains popular 72 for profiling microbial communities due to its cost-effectiveness, well-established protocols, and 73 reduced computational power requirements compared to shotgun metagenomics. In amplicon 74 sequencing, the entire 16S ribosomal RNA gene or subsets of its hypervariable regions (the 16S 75 rRNA gene contains nine hypervariable regions) are used for microbial identification. Although 76 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint traditionally considered limited to genus or species-level resolution, new developments are 77 challenging this limitation (Johnson et al., 2019; Buetas et al., 2024). An amplicon sequencing 78 analysis typically includes preliminary steps to remove primers, low-quality reads, chimeras, and 79 other PCR and sequencing artifacts. Subsequently, the remaining reads are either clustered into 80 Operational Taxonomic Units (OTUs), grouping sequences at 95% or 97% nucleotide identity as 81 a proxy for genus or species level (Schloss & Handelsman, 2005), or arranged into Amplicon 82 Sequence Variant (ASV) tables through denoising and correction procedures without clustering 83 (Callahan et al., 2016). Finally, these resulting OTU groups or ASV tables are used in 84 classification methods to assign taxonomic levels. These tasks and their advancements have led 85 to the development of new software suites, which also require thorough evaluation. LEMMI16S 86 addresses this need by providing a standardised benchmark for assessing the performance and 87 accuracy of various amplicon sequencing tools across various scenarios. 88

89 The LEMMIv2/16S framework 90 A workflow that promotes sharing while centralizing knowledge. 91 The LEMMIv2/16S workflow centres around an online catalogue of evaluated tools that has been 92 significantly refactored compared to LEMMIv1. In the first version, datasets for evaluation had to 93 be manually generated or sourced from public datasets. In contrast, the LEMMIv2/16S framework 94 is designed to generate new benchmarking material on demand, using public data to simulate 95 scenarios that mimic real-life problems. These scenarios, referred to as LEMMI instances, help 96 users identify the most relevant tools for their needs. Users can set up multiple parameters to 97 generate these instances, including the target clade (PROK for prokaryotes, EUK for eukaryotes, 98 and VIR for viruses), the taxonomic composition of the microbial community, the sequencing 99 technology, and the taxonomy (NCBI (Schoch et al., 2020) or the Genome Taxonomy Database 100 (GTDB; Parks et al., 2018), a new feature in LEMMIv2). Based on these definitions, in-silico reads 101 are produced. For example, the “2023_12_PROK_NCBI_clean_v220” instance in the LEMMIv2 102 catalogue includes 2 million reads (Illumina HiSeq 2500, read length 150bp) representing 155 103 species, simulating a clinical sample from dental plaque. The “Various human pathogens, regions 104 V1‑V2” instance in the LEMMI16S catalogue includes 845,000 reads (Illumina MiSeq v3, read 105 length 200bp) from the V1-V2 region of the 16S gene, representing 12 species of various human 106 pathogens. As in LEMMIv1, each scenario corresponds to an online report with metrics such as 107 precision, recall, runtime, and memory consumption of the tools benchmarked. Alongside 108 benchmark results, the catalogue provides links to obtain the tools and the necessary information 109 to understand and use them. A major improvement in LEMMIv2/16S is the introduction of a 110 standalone pipeline, similar to the one used to create the online catalogue, which method 111 developers can use to set up local evaluations. This makes it much easier to create a LEMMI-112 compliant software container for their tools compared to LEMMIv1. Developers can generate their 113 own LEMMI instances or easily obtain existing datasets or configuration files to replicate an 114 existing benchmark. Any tool container compatible with LEMMI can be obtained through the 115 LEMMI website or a third-party repository and used in personal benchmarks with minimal effort. 116 Finally, developers who choose to publish their method can have their LEMMI-compliant 117 container directly evaluated and included in the ongoing public benchmark maintained by the 118 LEMMIv2/16S development team (Figure 1). 119 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint An online catalogue of tools evaluated across a wide range of samples 120 The landing pages of LEMMI16S (Figure 2A) and LEMMIv2 (Figure 2B) showcase a variety of 121 benchmark instances available in the current catalogue. These instances represent a range of 122 scenarios, each differing in terms of community composition, simulated sequencing technologies, 123 and reference taxonomies. In the LEMMIv2 catalogue, users can explore instances that simulate 124 clinical samples, reflecting bacterial communities found in human patients, as well as microbial 125 communities from environmental niches such as alpine lakes. Additionally, users can examine 126 scenarios utilising short reads, as with LEMMIv1, alongside long reads simulated with Oxford 127 Nanopore Technology. These instances are analysed using the NCBI taxonomy, as in the 128 previous version, or the GTDB taxonomy as an alternative. 129 A new feature of the LEMMIv2 benchmark is the inclusion of instances where host and 130 contaminant taxa (non-targets) are present alongside the organisms of interest (targets). For 131 example, reads representing the host genome should ideally not be included in predictions. This 132 feature tests the tools’ ability to focus on relevant content, encouraging strategies, potentially 133 involving the combination of tools into pipelines, to filter out or ignore irrelevant data before 134 classification. In the LEMMI16S catalogue, users can explore instances simulating the 135 amplification of various regions of the 16S gene from target organisms. Each LEMMIv2/16S 136 instance encompasses multiple replicates representing different sampling of similar environments 137 in the form of sequencing reads provided to candidate tools for generating predictions. 138 The online evaluation (Supplementary Figure 1) presents several widgets displaying various 139 metrics summarizing prediction quality and resource consumption for each evaluated method. 140 The top section presents averages across all samples, while subsequent sections allow users to 141 explore the data sample by sample (Supplementary Figure 2). A widget consolidates key metrics 142 for a user-selected shortlist of tools into a single spider plot (Figure 3). Another section of the 143 website ("All Evaluated Tools") offers useful information about each tool, including links to the 144 code, relevant publications, and the container used for evaluation (Supplementary Figure 3). 145 The standalone pipeline 146 The standalone pipelines allow users to replicate the online benchmarks or create their own local 147 instances with minimal effort. These pipelines can be easily deployed by cloning the code from 148 the Git repositories (https://gitlab.com/ezlab/lemmi-v2/, https://gitlab.com/ezlab/lemmi16S/) and 149 installing the required packages using the Conda installer, along with setting up a container 150 engine (details provided below). Once set up, the LEMMIv2/16S pipeline automatically downloads 151 and prepares the necessary reference repositories, generates benchmark instances, evaluates 152 candidate tools, and displays the results on a web page. The only task left for the user is to 153 complete the configuration files. Additionally, because LEMMIv2/16S are built on Snakemake 154 (Köster & Rahmann, 2012), jobs can be efficiently distributed across CPU cores on a local 155 machine or multiple nodes in a computing cluster. 156 Docker and Apptainer containers 157 LEMMIv2/16S ensures the long-term availability of exact tool versions for evaluation through the 158 use of containers. A container functions as a self-contained evaluation unit, allowing the 159 combination of multiple steps or even different methods within it. It receives input files from 160 LEMMIv2/16S and generates the required outputs for evaluation. Resource usage, such as 161 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint runtime and memory consumption, is measured to include all processes that occur when the 162 container is deployed for a benchmarking task. 163 164 In LEMMIv1, Docker (https://www.docker.com/ last accessed 10.02.2015) was the only supported 165 container engine. However, LEMMIv2/16S now supports both Docker and Apptainer. Apptainer 166 (formerly Singularity) (Kurtzer et al., 2017) is widely adopted in HPC environments for its 167 enhanced security features, avoiding some vulnerabilities found in Docker (Zhou et al., 2022), 168 and its native integration with Snakemake (Köster & Rahmann, 2012). Additionally, Graphical 169 Processing Units (GPUs), which are increasingly used in machine learning applications, can be 170 utilised in LEMMIv2/16S with both container technologies, further expanding the platform's 171 capabilities. 172 173 All software containers that encapsulate candidate tools can either be sourced from online 174 directories or accessed locally. Compatible containers, along with corresponding configuration 175 files or datasets, are available at https://quay.io/user/ezlab/ and 176 https://zenodo.org/communities/lembench/. Other repositories, such as https://hub.docker.com 177 (last accessed 10.02.2025), can also be used to share and access compatible software 178 containers, ensuring flexibility and broad accessibility for users to find the necessary resources 179 for LEMMIv2/16S evaluations. 180 Sequences management 181 The LEMMIv2/16S pipeline manages microbial genomes for benchmarking through well-182 structured repositories. These repositories serve both as sources for generating in-silico reads 183 and as references during tool evaluation. 184 Repositories 185 Support for GTDB is a new feature in LEMMIv2. Users can now build the LEMMIv2 sequence 186 directory using prokaryotic, eukaryotic, and viral genome assemblies from GenBank, annotated 187 with both NCBI taxonomy and GTDB metadata for prokaryotes. Each genome is validated to 188 ensure it includes the necessary metadata; those lacking this information are discarded. LEMMIv2 189 ensures reproducibility by applying a user-defined cut-off date to filter the input material. 190 Additionally, in this version, users can specify a host genome separately by providing a link to an 191 assembly in the configuration, such as the human genome GRCh38. 192 193 Unlike LEMMIv1, LEMMIv2 does not require each genome to have a corresponding protein 194 equivalent to be included in the repertoire. This previous requirement excluded many assemblies 195 and limited LEMMIv1's ability to represent the full diversity available in public databases. In 196 LEMMIv2, control over the reference material during the benchmarking process is restricted to 197 nucleotide-based approaches. However, tools that rely on protein references can still be 198 evaluated by using a reference provided with the tool, similar to how marker gene-based tools 199 are assessed, though this comes with some limitations (as detailed below). 200 201 In the case of LEMMI16S, the sequence repository is constructed using either the SILVA 202 database (Pruesse et al., 2007) or a GTDB16S release (Parks et al., 2018) containing bacterial 203 sequences. As with LEMMIv2, each amplicon is validated to ensure that the appropriate metadata 204 is included. Filters are also applied to remove low-quality or redundant sequences, ensuring a 205 well-defined reference repository. 206 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint Flexible control of the reference database 207 For all tools capable of creating a reference on demand, LEMMIv2/16S manages and provides 208 the sequences that can be used as references (Figure 4A). In LEMMIv1, a genome exclusion 209 approach was implemented: any genome used to generate sequencing reads was removed from 210 the reference provided to the tool to avoid overfitting (Figure 4B). LEMMIv2 continues to utilise 211 this approach. 212 213 However, this method is not applicable to tools that require an embedded reference, such as 214 marker gene-based methods, where the reference database cannot be altered. Consequently, 215 LEMMIv1 allowed all genomes to be kept as references during benchmarks including tools 216 requiring embedded references, resulting in uniform overfitting bias by possibly including the 217 exact source of the reads in the reference. This less-than-ideal strategy has been replaced in 218 LEMMIv2. The new approach involves splitting the reference repository using a cut-off release 219 date: genomes older than this date are used as references, while the more recent genomes are 220 used to generate the reads (Figure 4C). This method allows tools with embedded references to 221 be evaluated alongside other tools, provided that a cut-off date older than the release date of the 222 embedded reference is set, ensuring that genomes used to generate the benchmarking reads 223 were not available when the reference was created. 224 225 None of the tools evaluated with LEMMI16S require an embedded reference, so the cut-off 226 strategy is not currently applicable, although it could be implemented in the future. The current 227 version of LEMMI16S always returns the entire 16S marker as a reference, regardless of the 228 specific region present in the reads. As with LEMMIv2, LEMMI16S excludes any amplicon used 229 in the read-generation from the reference database for methods requiring model training. 230 Clade exclusion 231 In most samples, particularly those from new environmental niches, some taxa often lack close 232 relatives in the reference database. This scenario can be effectively simulated using 233 LEMMIv2/16S. 234 235 To provide a more realistic challenge, LEMMIv2/16S can designate unknown organisms, which 236 helps distinguish tools that accurately handle reads from undiscovered taxa from those that 237 generate numerous false positives in such contexts. To simulate the presence of unknown taxa 238 using public data, LEMMIv2/16S allows users to designate certain lineages (at any taxonomic 239 rank, such as species or family) as unknown during sample generation. Genomes or sequences 240 from these designated lineages are excluded from the reference provided to the tools, regardless 241 of their use in the read generation process (Figure 4D). Consequently, any reads belonging to 242 these lineages must be reported as unknown to provide a correct prediction. It is important to note 243 that this option should not be used when evaluating tools with embedded references, as the 244

cannot be modified in those cases. 245 Minimum filter threshold to consider a taxon 246 Previous benchmarks have highlighted the importance of applying a minimum threshold of reads 247 to validate the presence of a taxon with methods that classify all reads, in order to reduce false 248 positives (McIntyre et al., 2017). This threshold may vary depending on the method used, but in 249 real-life analyses and without calibration values, it can only be set arbitrarily and be the same of 250 all methods. However, in a simulated environment, the known contents of the samples allow for 251 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint the definition of an optimal threshold that maximises each tool's performance. This threshold can 252 then potentially be applied to real samples, provided that the simulation parameters (such as the 253 number of reads, species richness, and abundance distribution) are representative enough of the 254 actual sample. 255 256 The LEMMIv2/16S pipeline employs a strategy based on the aforementioned rationale to optimise 257 the expected performance of each evaluated tool and provide guidance on setting an appropriate 258 threshold for each. It achieves this by simulating two types of samples: "calibration samples" 259 (indicated by the suffix c00x) and "evaluation samples" (indicated by the suffix e00x), using 260 identical parameters. The taxa specified in the configuration file of the benchmarking instance are 261 assigned random abundances according to the same distribution. While the dominant taxa may 262 differ between the samples, the taxa present, their number, and the total number of reads remain 263 consistent. The filtering threshold that maximises the quality of predictions in the calibration 264 samples (by optimising the F1 score for taxa presence detection) is applied to the evaluation 265 samples prior to calculating the metrics reported in the benchmark, ensuring that the results reflect 266 the tool's optimal performance. Additionally, the threshold for each tool is displayed in a dedicated 267 widget, helping users understand calibration differences between tools. 268 Comparison of simulation and real samples 269 To ensure realistic benchmarks, all LEMMIv2 and LEMMI16S instances are designed based on 270 actual biological samples. To assess how well the simulation aligns with the real-world sample, 271 we compared specific features of both datasets for a given instance. For example, a dental plaque 272 sample from a healthy patient was used to guide the simulation process for LEMMIv2. The 273 parameters for this pipeline were estimated from an initial analysis of the sample, reported in the 274 LEMMIv2 configuration file https://lemmi.ezlab.org/data/2023_05_PROK_NCBI.yaml. The 275 number of reads, read lengths, distributions, compositions (e.g., number of organisms and taxa), 276 sequencing technology and related settings were carefully tailored to ensure that the resulting 277 LEMMIv2 instance (https://lemmi.ezlab.org/2023_05_PROK_NCBI) accurately mirrored the 278 characteristics of the original dental plaque sample. 279 A similar method was employed for LEMMI16S, where parameters for the simulation were derived 280 from samples collected from the respiratory tract, specifically the dataset NCBI SRA reference 281 SRR22879376. 282 Supplementary Table 1 and Supplementary Figure 4 illustrate the similarities between the 283 simulated and real samples in terms of reads composition, species distribution, and GC content. 284 These results demonstrate the ability of the instance generation process to faithfully replicate key 285 characteristics of the original biological material. 286 The evaluation results 287 LEMMIv2 for metagenomics 288 LEMMIv1 was limited to generating samples representing Illumina's short read technology. To 289 expand the scope of tool evaluations, particularly for those dedicated to long-read analysis, 290 LEMMIv2 now includes the ability to simulate Oxford Nanopore Technology data. This 291 enhancement allows the evaluation of tools specifically designed for long-read applications, as 292 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint well as the assessment of methods originally developed for short-read inputs when they are 293 applied to long-read datasets. 294 295 The prokaryotic profilers primarily designed for short reads that have been evaluated at the time 296 of writing include MetaPhlAn v4.0.2 (Blanco-Míguez et al., 2023) and v3.0.13 (Beghini et al., 297 2021), mOTUs v3.0.1 (Ruscheweyh et al., 2022), Sourmash v4.2.3 (Brown & Irber, 2016), KMCP 298 v0.9.0 (Shen et al., 2023), Kraken v2.1.2 (Wood et al., 2019), Centrifuge v1.0.4 (Kim et al., 2016), 299 Ganon v2.0.0 (Piro & Reinert, 2023) and v1.1.0 (Piro et al., 2020), CCMetagen v1.4.0 (Marcelino 300 et al., 2020), Metacache v2.2.0 (Müller et al., 2017), Centrifuger v1.0.0 (Song & Langmead, 2024), 301 and Metabuli v1.0.2 (Kim & Steinegger, 2024). Where applicable, these tools were also evaluated 302 on long reads and viral datasets. The viral prediction pipeline VirMet v1.0.0 303 (https://github.com/medvir/VirMet last accessed 10.02.2025) was evaluated only on viral 304 datasets. Metamaps v0.1.0 (Dilthey et al., 2019) and MMseqs v2.12 (Steinegger & Söding, 2017) 305 were assessed using Oxford Nanopore Technology simulations. 306 Certain tools were considered for evaluation but did not scale efficiently in terms of runtime under 307 the conditions tested; LEMMI reports these entries as they are benchmarking results. These 308 include kASA (Weging et al., 2021) and MTSv (Furstenau et al., 2022), which are mentioned on 309 the LEMMI website but not yet evaluated due to current performance limitations. Deepmicrobes 310 (Liang et al., 2020) was evaluated but its predictions were limited to the genus level and above. 311 Clinical samples 312 Dental plaque samples were analysed using several candidate tools to identify approximately fifty 313 bacterial species per sample (https://lemmi.ezlab.org/2023_12_PROK_NCBI_v220). These 314 samples simulated Illumina short reads and were classified under the NCBI taxonomy. The 315 simulated mix also included human host and fungal reads as non-target organisms. To avoid 316 overfitting, the repository was split into reference and sample using a cut-off date set later than 317 the release of MetaPhlAn 4.0.2, the tool with the most recent embedded reference (as depicted 318 in Figure 4C). All methods performing read-level classification were given the opportunity to filter 319 out human host and fungal reads before analysing the remaining sequences to classify the target 320 organisms (Supplementary Table 2). 321 At the species level, marker-based tools delivered the most accurate predictions of taxa presence, 322 despite their inability to remove host and contaminant reads prior to analysis. MetaPhlAn 4 323 emerged as the top performer in terms of F1 score, with the highest precision and recall 324 (Supplementary Figure 5). Among k-mer-based methods processing the LEMMI-provided 325 reference, Kraken, Ganon, Sourmash, and KMCP showed similar performance levels. To manage 326 resource limits (memory, disk space, and runtime), each species in the reference included only 327 one representative genome. However, some tools capable of handling up to five reference 328 genomes per species were re-evaluated with a more comprehensive reference, and these results 329 were marked as (toolname)_all. Interestingly, this led to a negative impact on recall for some tools 330 like Kraken, as previously observed by (Nasko et al., 2018). 331 Centrifuge showed a slightly lower F1 score than the top performers, while Metabuli and 332 Centrifuger ranked lowest due to low precision values. LEMMIv2 automatically applied a filtering 333 threshold derived from calibration samples, optimised to maximise the F1 score at the species 334 level; the values are reported in Table 1. Regarding the L2 distance to the truth (measuring the 335 distance between predicted and actual abundances), KCMP performed the best, followed by 336 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint Kraken and mOTUs, while MetaPhlAn 4 had the least accurate abundance estimates 337 (Supplementary Table 3). 338 At higher taxonomic levels (genus and above), the performance gap between MetaPhlAn 4 and 339 the other tools narrowed. mOTUs improved its precision significantly, becoming the best 340 performer in terms of F1 score at the genus level (Supplementary Figure 6). 341 Additional tools such as Metacache, CCMetagen, and Deepmicrobes were evaluated alongside 342 many of the previously mentioned tools in an earlier LEMMI instance, which is available at 343 https://lemmi.ezlab.org/2023_05_PROK_NCBI. This instance represents the same dental plaque 344 sample type. These tools ranked lower in F1 scores for species detection and were not 345 reevaluated on the latest instance. 346 Runtime and memory evaluation 347 To fairly assess runtime without penalizing tools that first filter out host and contaminant reads, 348 compared to those that do not, a clean version of the previous LEMMI instance containing only 349 target bacterial reads was created (https://lemmi.ezlab.org/2023_12_PROK_NCBI_clean_v220). 350 This benchmark revealed that Ganon was the fastest tool, processing 3 million Illumina 150 base 351 pair paired-end reads in just half a minute. On the other hand, MetaPhlAn 4 and KMCP were the 352 slowest, each taking approximately five minutes to complete the analysis (Supplementary Figure 353 7A). 354 In terms of memory usage, Sourmash and mOTUs stood out as the most efficient tools 355 (Supplementary Figure 7B). Interestingly, when evaluating tools with one representative genome 356 per species versus five genomes per species, Sourmash maintained its memory footprint, even 357 with a more comprehensive reference (e.g., sourmash_423 vs. sourmash_423_all on 358 https://lemmi.ezlab.org/2023_12_PROK_NCBI_v220). This contrasts with other tools, which 359 generally increased their memory requirements when using more extensive references. 360 Unknown taxa 361 The benchmarking instance presented above only includes taxa that are currently represented in 362 the database, making it easier to solve than real-world samples. To address this, a similar 363 instance was created (https://lemmi.ezlab.org/2023_11_PROK_GTDB_v220), where one in 364 every three taxa was marked as unknown (as shown in Figure 4D). This instance is suitable only 365 for evaluating tools that can construct their reference from the provided material, and the target 366 taxonomy is set to GTDB. 367 In this more challenging scenario, Sourmash achieved the best results, outperforming other top 368 tools such as Kraken, KMCP, and Ganon by maintaining high recall and precision, leading to the 369 highest F1 score. KMCP was notably affected by the inclusion of unknown taxa, showing a drop 370 in precision at the species level (Supplementary Figure 7C). Despite this, KMCP still minimized 371 the L2 error in abundance estimation. This is because its false predictions tended to involve low-372 abundance taxa, while highly abundant taxa were accurately reported. 373 Environmental samples 374 A benchmarking instance (https://lemmi.ezlab.org/2022_03_PROK_NCBI_2) was created based 375 on a study of a meromictic lake (Saini et al., 2022). The aim was to estimate the prokaryotic 376 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint fraction, consisting of about 20 species at the species rank. However, all tools tested showed 377 poor precision and recall (Supplementary Figure 7D). The poor recall could be attributed to the 378 lack of reference genomes for niche environmental taxa in tools with embedded references, such 379 as MetaPhlAn and mOTUs. But designing a LEMMI instance with a reference genome for every 380 species should have enabled other tools to perform similarly well as they did with human samples. 381 A likely explanation for the poor results is either the greater genomic distance between species 382 in this environmental instance and their reference compared to clinical samples or an improper 383 labelling of some reference genomes under the NCBI taxonomy. This mismatch between the 384 NCBI taxonomy and the true genomic distances between the organisms placed in the reads and 385 in the reference may have prevented tools from correctly associating sequences with the 386 appropriate taxonomic identifiers more frequently than with human microbiome simulations. 387 388 To test this hypothesis, two instances with identical parameters were created 389 (https://lemmi.ezlab.org/2024_01_PROK_NCBI_v220 and 390 https://lemmi.ezlab.org/2024_01_PROK_GTDB_v220), differing only in the taxonomic system 391 used (NCBI vs. GTDB). GTDB is a taxonomic system constructed with genomic distances. When 392 tested with KMCP, the F1-score at the species level was significantly higher for the GTDB 393 taxonomy instance (0.8) compared to the NCBI taxonomy instance (0.34). This improvement was 394 also reflected in abundance estimation, where the L2 distance dropped from 0.6 (NCBI taxonomy) 395 to 0.03 (GTDB taxonomy) for sample e001, highlighting the impact of the taxonomic 396 representation based on genomic distances when analysing understudied niche compared to well 397 characterised and highly sampled human microbiomes. 398 399 We also generated an instance inspired by the same study (Saini et al., 2022), focusing on the 400 eukaryote fraction of the sample (https://lemmi.ezlab.org/2022_01_EUK_NCBI). In this case, 401 Sourmash demonstrated the best performance in species identification, recovering an average of 402 3.5 out of 4 species while maintaining a much lower false prediction rate than other high-recall 403 tools (Supplementary Figure 7E). Marker-based tools were not included in this instance. 404 Viral samples 405 Samples containing eight low-abundance viruses (approximately 300 reads each among 5 million 406 bacterial reads) were generated to simulate conditions similar to pathogen detection 407 (https://lemmi.ezlab.org/2021_9_VIR_NCBI). In this scenario, Sourmash performed poorly 408 (Supplementary Figure 7F), while Kraken outperformed all other tools, including VirMet, which 409 was the only dedicated viral pipeline evaluated. Kraken achieved the best F1 score at the species 410 level, with VirMet ranking second. 411 Long reads 412 LEMMIv2 also supports the simulation of Oxford Nanopore long reads. The clinical samples 413 discussed earlier were generated in an additional LEMMI instance 414 (https://lemmi.ezlab.org/2022_04_PROK_NCBI_LR2) consisting of 20,000 reads with lengths 415 ranging from 57 to 32,389 base pairs, averaging 1,096 base pairs, and covering approximately 416 70 bacterial species. Several tools originally designed for short reads were evaluated, and only 417 those that successfully completed the analysis with meaningful results were reported. Centrifuge 418 demonstrated solid performance, though it did not match Metamaps in terms of recall and 419 precision at the species level (Supplementary Figure 7G). However, Centrifuge was significantly 420 faster, completing the analysis hundreds of times quicker. 421 422 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint LEMMI16S 423 LEMMI16S currently focuses on Illumina short reads, with long reads considered for future 424 benchmarks. In the current version, four candidate tools have been included: QIIME 2 (version 425 v2022.8) (Bolyen et al., 2019), DADA2 (version v1.26) (Callahan et al., 2016), Kraken2 (version 426 v2.13) (Wood et al., 2019), and MAPseq (version v2.1.1) (Matias Rodrigues et al., 2017). These 427 tools were subjected to comprehensive evaluation under different scenarios, with simulated reads 428 being generated from various amplicon repositories (i.e., SILVA release_138.1 and GTDB release 429 214.0), hypervariable regions (i.e., V1-V2, V4, and V7-V9), and bacterial compositions. Other 430 tools, including mothur (version v1.47) (Schloss et al., 2009) and SPINGO (version v1.3) (Allard 431 et al., 2015), were also considered for evaluation, but their runtime and prediction accuracy fell 432 below expectations, requiring further investigation before public release. 433 The next section describes the instances in which these candidate tools were evaluated. Note 434 that alternative 16S rRNA databases, such as GreenGenes (DeSantis et al., 2006) and RDP 435 (Cole et al., 2009), were not considered, primarily because they have not seen updates in recent 436 years. Analytic methods requiring complete amplicons or hypervariable regions exceeding 400 437 bp were excluded from consideration primarily as a result of constraints imposed by short-read 438 sequencing technology, in which read length is restricted to 250 bp for both forward and reverse 439 directions, and several tools mandate a minimum overlap between these read pairs. Furthermore, 440 the analysis at the species level was omitted, as it has been suggested that species assignment 441 for short 16S sequences necessitates 100% identity and distinguishing between species is 442 inherently challenging due to the identical or highly homologous nature of 16S rRNA genes among 443 certain distinct species (Edgar, 2018). 444 445 Human pathogens samples 446 The dataset was inspired by the inter-laboratory study proposed by O’Sullivan et al. (O’Sullivan 447 et al., 2021). It contains fifteen popular human pathogens species, with their abundance being 448 randomly represented in five instances (3 for calibration and 2 for evaluation). The species 449 amplicons were sourced from the SILVA repository. For each instance, 845,000 reads were 450 generated by LEMMI16S (length=250bp, mean=400bp, std=10bp) from the V1V2 region. All 451 candidate tools were evaluated with this dataset, considering their default stages and parameters 452 (https://lemmi16S.ezlab.org/alfa_v1v2_SILVA_gcn 453 , https://zenodo.org/records/8343988). Predictions at the family and genus levels were reported, 454 as they indicated the most variations in the results. At these levels, all candidate tools were found 455 to perform suitably, with subtle differences between all the metrics. QIIME 2_v2022.8 achieved 456 the highest F1 score, DADA2_v1.26 excelled abundance estimation (with the lowest L2 distance 457 to truth), and Kraken2_v213 was the fastest and most memory-efficient among the evaluated 458 tools. 459 460 Human Oral Microbiome samples with unknown organisms 461 This dataset includes 74 genera from the Human Oral Microbiome Database (eHOMD) (Chen et 462 al., 2010) are represented. The LEMMI16S workflow was configured to utilize the GTDB SSU 463 database and generate five samples (3 for calibration and 2 for evaluation), each with 500,000 464 reads (length=250bp, mean=400bp, std=10bp) from the V4 region (a characteristic region used 465 for studying HOM composition (Dashper et al., 2019)). Furthermore, an additional sample was 466 created by randomly designating 14 organisms as unknown while retaining the other parameters 467 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint (available at https://lemmi16S.ezlab.org/HOMD_v4_GTDB_gcn, 468 https://lemmi16S.ezlab.org/HOMD_v4_unknown_GTDB_gcn, and 469 https://zenodo.org/records/8344037). Supplementary Figure 8 illustrates that this condition 470 slightly impacts the performance of the tools, with DADA2_v1.26 leading in F1 score and L2 471 distance in both scenarios, while Kraken excels in computational speed and memory efficiency. 472 473 Respiratory samples 474 This dataset comprises a taxa list derived from the experiment conducted on respiratory samples 475 from patients with bronchiectasis by López-Aladid et al (López-Aladid et al., 2023). The samples 476 (3 for calibration and 2 for evaluation) were subjected to the generation of 500,000 reads (length 477 = 250bp, mean = 400bp, std = 10bp) from the V7–V9 region, representing 20 genera with varying 478 abundance levels. The amplicons were sourced from the GTDB repository. At the Family and 479 genus levels, low performance is demonstrated by DADA2_v1.26, while MAPseq_v2.1.1 exhibits 480 clear superiority in terms of the F1-score. Remarkably, QIIME 2_v2022.8 showcases the best 481 abundance estimation. In contrast, Kraken stands out as the fastest and the least RAM-intensive 482 tool but performs poorly in other metrics, including F1-score and L2 distance. 483

484 485 Metagenomics practitioners, both users and developers, will find LEMMIv2/16S to be a versatile 486 framework that enables the simulation of sequencing reads corresponding to samples they are 487 working on or interested in. This functionality allows users to easily evaluate existing tools 488 compatible with LEMMI, including their own developments, gaining insights into which features 489 influence a tool's behaviour, runtime, and resource consumption. One such feature is the 490 proportion of unknown lineages present in the sample being analysed. Our benchmarks, based 491 on public datasets, enable the rapid creation of new data, and the ability to selectively exclude 492 specific genomes from the reference, which allows for accurate simulation of the challenges 493 posed by the incomplete sampling of microbial diversity. 494 495 In addition, the benchmark maintained by the LEMMI team at https://lemmi.ezlab.org and 496 https://lemmi16S.ezlab.org simplifies the process of centralising performance data for these tools, 497 allowing for a quickly updated and widely shared list of evaluated methods. These platforms offer 498 the community an ongoing and incremental evaluation resource. As of this writing, the 499 benchmarks have included both widely popular and well-established tools, as well as newer, less-500 known options. Many tools are still awaiting evaluation, and we encourage developers to utilise 501 LEMMI and submit their containers for public benchmarking. 502 503 When designing an experiment to study a microbial community, it's essential to consider the 504 classification algorithm and reference database independently to optimize their fit to the target 505 material. A poorly chosen database can undermine the validity of the results (Gihawi et al., 2023). 506 If resources and time permit, selecting a custom genome set is recommended rather than relying 507 on default options, unless a comprehensive third-party database like 508 https://benlangmead.github.io/aws-indexes/k2 (last accessed 10.02.2025) is preferred. In all 509 cases, to accurately evaluate a method's predictive power, the evaluation should be independent 510 of any prepackaged reference constructed at an arbitrary date. 511 In line with this principle, LEMMIv2/16S ensures that, where possible, all tools are evaluated with 512 the same reference material. This approach can reduce performance gaps between tools with 513 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint similar methodologies, possibly indicating that transitioning to a new method may be unnecessary 514 unless specific dataset features justify it (e.g., Supplementary Figure 5 shows Kraken, Ganon, 515 KMCP, and Sourmash producing comparable results, while Supplementary Figure 7C 516 demonstrates Sourmash's superior performance when unknown organisms are present). This 517 setup does not oversell minor progress but ensures that significant advancement in classification 518 capabilities by a new tool will be clearly apparent through LEMMIv2/16S's benchmarks in the 519 future. 520 The evaluation of tools with embedded references (e.g., marker-based methods such as those 521 included in this report) is essential for offering a comprehensive view of available options. To 522 avoid an overfit of the benchmark, a cut-off date is used to ensure that the reads do not directly 523 match the genomes likely used to build the markers. However, with the LEMMI approach, these 524 tools cannot be fully tested in scenarios where the presence of unknown organisms simulates 525 more realistic samples. 526 527 The work invested in the selection of marker genes for tools like MetaPhlAn and mOTUs 528 undoubtedly results in high accuracy, with much of their strength tied to their reference datasets. 529 It is important for LEMMI users to recognize that comparisons between tools with embedded 530

and those using controlled, external references serve different purposes. While the 531 former demonstrates the effectiveness of a tool’s internal reference set, the latter emphasises the 532 tool’s general classification performance. Therefore, when assessing tools, users should 533 differentiate between these two approaches to make informed decisions based on their specific 534 research needs. 535 536 In this paper, we emphasise that using the GTDB taxonomy to structure the reference database 537 significantly enhances the analysis of niche environments compared to well-characterised 538 human-related communities. For researchers working with these less-characterised 539 environments, it is advisable to choose methods that can directly handle GTDB taxonomy or 540 utilise tools that can convert GTDB data into an NCBI-like taxonomy (e.g., using 541 https://github.com/nick-youngblut/gtdb_to_taxdump last accessed 10.02.2025). Additionally, 542 marker-based methods should ensure compatibility with the GTDB framework to maximise their 543 effectiveness in diverse and novel environments. 544 545 In addition to considering the time and memory required for sample analysis, the time required to 546 process a reference may also be a criterion if frequent updates are needed. LEMMIv2/16S reports 547 this information. Note that some tools, such as Ganon, can update their reference without 548 reprocessing the existing part, so users will need to judge whether the initial construction as 549 reported on LEMMI and the update process are different. We have found that increasing the size 550 of the reference, for example, from one representative genome per species to five 551 representatives, affects memory usage during sample analysis in different ways across tools. A 552 tool performing well with 10,000 genomes may not scale effectively and remain the best when the 553

expands to 50,000 genomes. 554 555 One might wonder whether deep learning approaches will bring significant advances or whether 556 a plateau in classification performance has been reached. In this work, we evaluated 557 Deepmicrobes with LEMMIv2, but its performance is not yet satisfactory. The containerization 558 approach used by the pipeline allows for the use of GPUs in both reference construction and 559 sample analysis, which will certainly cover future methods to be evaluated. 560 561 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint All tools evaluated in LEMMIv2/16S come with a public container, ensuring they can operate in a 562 standardised environment (e.g., UNIX machines with dozens of CPUs, hundreds of GB of RAM, 563 and terabytes of storage). The mandatory use of containers enhances reproducibility and 564 simplifies the deployment of the benchmark. However, some software may require specific 565 hardware or libraries for optimal performance, and portability cannot always be guaranteed 566 without rebuilding the containers from source. Some tools have claimed high performance on 567 desktop hardware (e.g., laptops) and LEMMIv2/16S can be used on such machines to test these 568 claims against realistically sized datasets. As of now, we have not identified any examples in our 569 benchmarks that behave well in these conditions. 570 571 Long read technologies represent a significant advancement in sample profiling, as they capture 572 more information within each read and reduce the risk of classification errors caused by short, 573 indistinguishable sequences. LEMMIv2's simulation of Oxford Nanopore technologies has shown 574 that MetaMaps, a method specifically designed for long reads, outperforms Centrifuge, which was 575 originally developed for short reads but still managed to provide valuable results. 576 Future LEMMIv2 developments should include simulations for PacBio. Additionally, the 577 LEMMI16S platform could benefit from integrating long read simulations, as long reads are 578 increasingly relevant in the field of amplicon sequencing (Buetas et al., 2024). 579 LEMMIv1 included evaluations of read binning because tools like Kraken were initially designed 580 for read classification with companion scripts for profiling. However, to maintain a focus on 581 profiling in LEMMIv2, read binning aspects are no longer evaluated, as not all tools support read-582 level analysis. A future platform dedicated to read binning, leveraging the existing software 583 containers, could be established to address these tools and their specific needs. 584 585

586 In this report, we introduced the latest version of LEMMI: LEMMIv2 for metagenomics and its 587 sister platform, LEMMI16S, for evaluating amplicon sequencing methods. We detailed their key 588 features and shared some of the results obtained, which were used to build tool catalogues on 589 https://lemmi.ezlab.org and https://lemmi16S.ezlab.org. We demonstrate how LEMMI enables re-590 evaluation of established methods and new software releases in unbiased metagenomics 591 scenarios within an independent computational environment. The goal of LEMMI is to maintain 592 an up-to-date catalogue and encourage both tool users and developers to visit these websites, 593 access the standalone pipelines, design their own benchmarks, and submit their developments 594 for public evaluation. This collaborative effort will contribute to a comprehensive knowledge base 595 on metagenomics and 16S amplicon sequencing methods. 596

597 The LEMMIv2/16S standalone pipelines are built around several Snakemake (Köster & 598 Rahmann, 2012) workflows. The features not specifically attributed to one of the pipelines in this 599 section are common to both. Both have parameters defined in different YAML configuration files 600 for the global configuration (e.g. https://gitlab.com/ezlab/lemmi-v2/-/raw/v2.2.0/ 601 config/config.yaml.default), the creation of instances (e.g. https://gitlab.com/ezlab/lemmi-v2/-602 /raw/v2.2.0/benchmark/yaml/instances/demo_prok_gtdb.yaml) and the execution of candidate 603 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint

(e.g. https://gitlab.com/ezlab/lemmi-v2/-/raw/v2.2.0/benchmark/ 604 yaml/runs/kraken_212.demo.yaml). 605 Containerization 606 Several pre- and post-processing tasks in the pipeline, as well as all evaluated methods, are 607 wrapped in software containers and run either on Singularity (Kurtzer et al., 2017) through its 608 native integration with Snakemake, or on Docker (https://www.docker.com last accessed 609 10.02.2025) through a custom integration with Snakemake. The resources used by the candidate 610

are reported by tracking the time the container is loaded and the peak RSS memory 611 reached by any process running inside the container. 612 Repository construction 613 LEMMIv2 is based on the genbank list of assemblies which is obtained from 614 https://ftp.ncbi.nlm.nih.gov/genomes/ASSEMBLY_REPORTS/assembly_summary_genbank.txt 615 (last accessed 01.11.2023). It can be filtered to remove entries older and newer than the date 616 range specified in the configuration file. The results presented in this manuscript are based on all 617 genomes older than 2023/11/01. Each species lineage is validated using the ETE Toolkit 3 library 618 (Huerta-Cepas et al., 2016) set with NCBI dmp files corresponding to a user-defined date. The 619

presented in this manuscript use 2023/11/01. Entries without a valid lineage and without 620 an FTP URL are dropped. In addition, the metadata corresponding to a release of the GTDB 621 taxonomy is downloaded and parsed, and the genomes present in that taxonomy are annotated 622 with extra information. The results presented in this manuscript are based on the GTDB release 623 207. All remaining genomes matching the lineages specified in the config file are then downloaded 624 up to a user-defined limit of genomes per species (results in this manuscript: five genomes per 625 species covering all Bacteria and Archaea as well as microbial eukaryotes including all Fungi) to 626 form the LEMMIv2 repository. An equivalent of the NCBI dmp files are generated for GTDB using 627 https://github.com/nick-youngblut/gtdb_to_taxdump (last accessed 10.02.2025). A host species 628 is also provided. For the work presented in this manuscript, the human genome version GRCh38 629 was used. 630 631 In the case of LEMMI16S, the sequence repository is built using either the SILVA database 632 (Glöckner et al., 2017) or a GTDB16S release (Parks et al., 2018), both of which contain bacterial 633 16S sequences. Poor-quality, redundant, and short sequences are removed from the final 634 repository using the RESCRIPt tool and tutorial (Robeson et al., 2021). The resultant sequences 635 and their taxonomy are conserved for future stages. In this case, all sequences in the repository 636 always represent the complete 16S genes. The hyper-variable regions to be amplified are defined 637 and extracted from the repository sequences during the generation of the LEMMI16S instance. 638 Instance creation 639 The LEMMIv2 configuration file defining an instance is parsed to obtain the target clade 640 (prokaryotes, eukaryotes or viruses) and the contaminant clades (prokaryotes, eukaryotes, and/or 641 viruses). The relative proportion of each clade plus the host is defined, and the number of 642 genomes to use for each clade is also taken from the configuration file. If specified, the genomes 643 are filtered to match the specific lineages before the expected number of genomes are sampled 644 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint from the repository. For each clade (prokaryotes, eukaryotes, viruses), a lognormal distribution is 645 used to randomly define the individual contribution of each genome to the relative abundance. 646 This sampling operation is performed for each sample defined in the configuration file, except for 647 those labelled as negative samples which are generated without the target clade. The list of 648 genomes remaining as a reference is created by excluding those selected in the previous step, 649 as well as all genomes belonging to the lineages marked as "unknown" in the configuration file. If 650 a cut-off date is defined for selecting the reference, the genomes for creating the reads are 651 sampled within the genomes more recent than this date, and the reference is made up of all 652 genomes older than the cut-off date. For Illumina sequencing, the ART reads simulator (Huang 653 et al., 2012) is used to generate the samples via the readSimulator library 654 (https://github.com/wanyuac/readSimulator last accessed 10.02.2025); several parameters such 655 as the number of reads or the Illumina sequencing chemistry need to be specified. If the 656 sequencing technology is Nanopore, NanoSim (Yang et al., 2017) is used. 657 658 The LEMMI16S pipeline applies two filters to select amplicons and produce simulated reads. A 659 taxon filter allows the selection of amplicons from a specific lineage, and a region filter enables 660 the conservation of only those that contain the particular zone in the 16S gene that is mentioned 661 in the instance config file. The resultant amplicons, which come from the specific taxa and include 662 the indicated region, are split into two datasets (query and reference). The reference dataset is 663 reserved for building the model reference. The query dataset is used as the input for the simulator 664 tool, which produces the 'synthetic' reads. The instance configuration file allows setting the 665 sample composition (organisms or taxonomic rank), the target 16S regions (e.g. V1-V2), the 666 query/reference percentage, the total number of reads to be generated, and the simulator 667 properties. If a particular taxonomic rank is marked as unknown, all its sequences are removed 668 from the reference dataset but retained in the query dataset. 669

processing by the candidate methods 670 Before running an evaluation, the LEMMIv2 pipeline places all genomic and taxonomic files 671 needed to build the references for the selected instance into a dedicated temporary folder, 672 organised into three sections: the host genome, reference genomes for target organisms, and 673

genomes for unwanted organisms to be excluded from the report. The candidate 674 container is loaded and the scripts LEMMI_process_ref_host.sh, LEMMI_process_ref_targets.sh 675 and LEMMI_process_ref_contaminants.sh are called in sequence. The candidate method 676 container must produce three databases compatible with its classification procedure for the next 677 step. Methods that cannot fulfil some or all of these tasks will instead execute a sleep command 678 for 30 seconds to acknowledge the completion of this step. 679 680 LEMMI16S replaces the LEMMI_process scripts with the LEMMI16S_training.sh script. It 681 contains the candidate method’s instructions to train a machine learning model or build a 682

database (according to the method strategy) from the reference dataset. Similar to 683 LEMMIv2, a sleep command for 30 seconds is executed when no reference needs to be built. 684 Analysis of samples by the candidate methods 685 To obtain the predictions necessary to evaluate a candidate tool, the LEMMIv2 pipeline places all 686 in-silico reads in the dedicated temporary folder containing the references built in the previous 687 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint steps. It then loads the candidate container and invokes the LEMMI_analysis.sh script. The 688 candidate method container must respond to this call by providing a taxonomic profile of the 689 sample based on the references available in the folder, unless the method comes with an 690 embedded reference and did not create any at the previous step. If applicable, the method will 691 first filter out the host reads, then filter out the contaminant reads, and then classify the remaining 692 reads to produce a report with four columns: organism name (binomial for species, name for the 693 parent clades), taxid, number of reads, and relative abundance of genome copies; the abundance 694 includes the unclassified fraction. The reported parent taxonomic ranks may or may not include 695 the sum of the child ranks, and LEMMIv2 will calculate the missing values depending on the option 696 selected. 697 698 Similarly, LEMMI16S invokes the LEMMI16S_analysis.sh script to copy the in-silico reads 699 generated from the query dataset into a temporary folder. It then uses the reference model to 700 classify them. If necessary, the method will call auxiliary tools to perform preliminary tasks before 701 generating the final classification. The four-columns format presented above is used to produce 702 the final report. For the tools that report OTU or ASV classification, the number of reads is 703 computed by consulting the size of the OTU or ASV. 704 Evaluation of the predictions 705 The predictions provided by the candidate methods are summed and filtered to ensure that they 706 contain only entries consistent with the taxonomic system used by LEMMI and provided during 707 the reference construction step. The samples labelled 'calibration' are then used to find the 708 filtering threshold that maximises the F1 score for taxa presence/absence predictions at each 709 taxonomic rank considered (the results presented in this manuscript cover phylum, class, order, 710 family, genus and species). This method-specific threshold is then applied to the taxonomic 711 profiles reported for all samples labelled 'Evaluation'. The following metrics are calculated from 712 the filtered taxonomic profiles: precision in taxa presence/absence predictions, recall in taxa 713 presence/absence predictions, F1 score in taxa presence/absence predictions, and L2 error in 714 relative abundance estimation of genome copy number. The numbers of true positives and false 715 positives in taxa presence/absence predictions are also reported. Bray-Curtis dissimilarity is used 716 to create a heatmap of the dissimilarity of each tool's predictions. 717 Web app to explore the results 718 A web application presents all the results, including the prediction accuracy metrics mentioned 719 above, as well as the runtime and RSS memory peak of the different tasks performed by the 720 candidate methods. The web app is built with Vue.js (https://vuejs.org/ last accessed 30.06.2024), 721 deployed using npm (https://www.npmjs.com/ last accessed 30.06.2024), and uses Apache 722 ECharts (https://echarts.apache.org/ last accessed 30.06.2024) to generate the plots. The web 723 app is packed in the LEMMI master container, which is also required to perform other tasks 724 throughout the benchmarking lifecycle. 725 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint Companion scripts 726 Several candidate methods natively report read abundance rather than genome copy abundance. 727 To make all methods comparable in this respect, a post-processing script, identical for all 728 methods, was added to candidate containers where necessary to normalise the profile of reads 729 to a profile of genome copies, using the information provided by LEMMIv2 to the method to build 730 the reference. 731 Computational resources 732 The results presented in this manuscript were produced using Singularity version 3.8.1-1.el8 733 running on a dedicated server with 48 Intel® Xeon® w5-3425 CPUs, 500 GB DDR5 RAM, and 734 250 TB disk storage. 735 736

737 This research was funded in part by the Swiss National Science Foundation (SNSF) 738 310030_189062 to EZ. AB acknowledges support from the Federal Commission for Scholarships 739 for Foreign Students for the Swiss Government Excellence Scholarship (ESKAS No. 2022.0531). 740 Author contributions 741 MS, AB, and EZ conceived the study. MS and AB coded the pipeline. MS and MB coded the 742 webapp. MS, AB, and MM conducted the analyses. EZ led the project. All authors wrote the 743 manuscript. 744 Competing interests 745 The authors declare no competing interests 746 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint

747 Agustinho, D. P., Fu, Y., Menon, V. K., Metcalf, G. A., Treangen, T. J., & Sedlazeck, F. J. (2024). 748 Unveiling microbial diversity: harnessing long-read sequencing technology. Nature 749 methods, 1-13. 750 Allard, G., Ryan, F. J., Jeffery, I. B., & Claesson, M. J. (2015). SPINGO: a rapid species-classifier 751 for microbial amplicon sequences. BMC Bioinformatics, 16, 1-8. 752 Beghini, F., McIver, L. J., Blanco-Míguez, A., Dubois, L., Asnicar, F., Maharjan, S., Mailyan, A., 753 Manghi, P., Scholz, M., & Thomas, A. M. (2021). Integrating taxonomic, functional, and 754 strain-level profiling of diverse microbial communities with bioBakery 3. elife, 10, e65088. 755 Blanco-Míguez, A., Beghini, F., Cumbo, F., McIver, L. J., Thompson, K. N., Zolfo, M., Manghi, 756 P., Dubois, L., Huang, K. D., & Thomas, A. M. (2023). Extending and improving 757 metagenomic taxonomic profiling with uncharacterized species using MetaPhlAn 4. 758 Nature biotechnology, 41(11), 1633-1644. 759 Bolyen, E., Rideout, J. R., Dillon, M. R., Bokulich, N. A., Abnet, C. C., Al-Ghalith, G. A., 760 Alexander, H., Alm, E. J., Arumugam, M., & Asnicar, F. (2019). Reproducible, interactive, 761 scalable and extensible microbiome data science using QIIME 2. Nature biotechnology, 762 37(8), 852-857. 763 Brown, C. T., & Irber, L. (2016). sourmash: a library for MinHash sketching of DNA. Journal of 764 open source software, 1(5), 27. 765 Buetas, E., Jordán-López, M., López-Roldán, A., D’Auria, G., Martínez-Priego, L., De Marco, G., 766 Carda-Diéguez, M., & Mira, A. (2024). Full-length 16S rRNA gene sequencing by PacBio 767 improves taxonomic resolution in human microbiome samples. BMC genomics, 25(1), 310. 768 Callahan, B. J., McMurdie, P. J., Rosen, M. J., Han, A. W., Johnson, A. J. A., & Holmes, S. P. 769 (2016). DADA2: High-resolution sample inference from Illumina amplicon data. Nature 770 methods, 13(7), 581-583. 771 Chen, T., Yu, W.-H., Izard, J., Baranova, O. V., Lakshmanan, A., & Dewhirst, F. E. (2010). The 772 Human Oral Microbiome Database: a web accessible resource for investigating oral 773 microbe taxonomic and genomic information. Database, 2010. 774 Cole, J. R., Wang, Q., Cardenas, E., Fish, J., Chai, B., Farris, R. J., Kulam-Syed-Mohideen, A., 775 McGarrell, D. M., Marsh, T., & Garrity, G. M. (2009). The Ribosomal Database Project: 776 improved alignments and new tools for rRNA analysis. Nucleic acids research, 777 37(suppl_1), D141-D145. 778 Dashper, S., Mitchell, H., Lê Cao, K.-A., Carpenter, L., Gussy, M., Calache, H., Gladman, S., 779 Bulach, D., Hoffmann, B., & Catmull, D. (2019). Temporal development of the oral 780 microbiome and prediction of early childhood caries. Scientific Reports, 9(1), 19732. 781 DeSantis, T. Z., Hugenholtz, P., Larsen, N., Rojas, M., Brodie, E. L., Keller, K., Huber, T., Dalevi, 782 D., Hu, P., & Andersen, G. L. (2006). Greengenes, a chimera-checked 16S rRNA gene 783 database and workbench compatible with ARB. Applied and environmental microbiology, 784 72(7), 5069-5072. 785 Dilthey, A. T., Jain, C., Koren, S., & Phillippy, A. M. (2019). Strain-level metagenomic 786 assignment and compositional estimation for long reads with MetaMaps. Nature 787 communications, 10(1), 3066. 788 Edgar, R. C. (2018). Accuracy of taxonomy prediction for 16S rRNA and fungal ITS sequences. 789 PeerJ, 6, e4652. 790 Furstenau, T. N., Schneider, T., Shaffer, I., Vazquez, A. J., Sahl, J., & Fofanov, V. (2022). MTSv: 791 rapid alignment-based taxonomic classification and high-confidence metagenomic 792 analysis. PeerJ, 10, e14292. 793 Gihawi, A., Ge, Y., Lu, J., Puiu, D., Xu, A., Cooper, C. S., Brewer, D. S., Pertea, M., & Salzberg, 794 S. L. (2023). Major data analysis errors invalidate cancer microbiome findings. MBio, 795 14(5), e01607-01623. 796 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint Glöckner, F. O., Yilmaz, P., Quast, C., Gerken, J., Beccati, A., Ciuprina, A., Bruns, G., Yarza, P., 797 Peplies, J., & Westram, R. (2017). 25 years of serving the community with ribosomal RNA 798 gene reference databases and tools. Journal of biotechnology, 261, 169-176. 799 Huang, W., Li, L., Myers, J. R., & Marth, G. T. (2012). ART: a next-generation sequencing read 800 simulator. Bioinformatics, 28(4), 593-594. 801 Huerta-Cepas, J., Serra, F., & Bork, P. (2016). ETE 3: reconstruction, analysis, and visualization 802 of phylogenomic data. Molecular biology and evolution, 33(6), 1635-1638. 803 Johnson, J. S., Spakowicz, D. J., Hong, B.-Y., Petersen, L. M., Demkowicz, P., Chen, L., Leopold, 804 S. R., Hanson, B. M., Agresta, H. O., & Gerstein, M. (2019). Evaluation of 16S rRNA gene 805 sequencing for species and strain-level microbiome analysis. Nature communications, 806 10(1), 5029. 807 Kim, D., Song, L., Breitwieser, F. P., & Salzberg, S. L. (2016). Centrifuge: rapid and sensitive 808 classification of metagenomic sequences. Genome research, 26(12), 1721-1729. 809 Kim, J., & Steinegger, M. (2024). Metabuli: sensitive and specific metagenomic classification via 810 joint analysis of amino acid and DNA. Nature methods, 1-3. 811 Köster, J., & Rahmann, S. (2012). Snakemake—a scalable bioinformatics workflow engine. 812 Bioinformatics, 28(19), 2520-2522. 813 Kurtzer, G. M., Sochat, V., & Bauer, M. W. (2017). Singularity: Scientific containers for mobility 814 of compute. PLoS One, 12(5), e0177459. 815 Liang, Q., Bible, P. W., Liu, Y., Zou, B., & Wei, L. (2020). DeepMicrobes: taxonomic 816 classification for metagenomics with deep learning. NAR Genomics and Bioinformatics, 817 2(1), lqaa009. 818 López-Aladid, R., Fernández-Barat, L., Alcaraz-Serrano, V., Bueno-Freire, L., Vázquez, N., 819 Pastor-Ibáñez, R., Palomeque, A., Oscanoa, P., & Torres, A. (2023). Determining the most 820 accurate 16S rRNA hypervariable region for taxonomic identification from respiratory 821 samples. Scientific Reports, 13(1), 3974. 822 Mangul, S., Martin, L. S., Hill, B. L., Lam, A. K.-M., Distler, M. G., Zelikovsky, A., Eskin, E., & 823 Flint, J. (2019). Systematic benchmarking of omics computational tools. Nature 824 communications, 10(1), 1393. 825 Marcelino, V. R., Clausen, P. T., Buchmann, J. P., Wille, M., Iredell, J. R., Meyer, W., Lund, O., 826 Sorrell, T. C., & Holmes, E. C. (2020). CCMetagen: comprehensive and accurate 827 identification of eukaryotes and prokaryotes in metagenomic data. Genome biology, 21, 1-828 15. 829 Matias Rodrigues, J. F., Schmidt, T. S., Tackmann, J., & von Mering, C. (2017). MAPseq: highly 830 efficient k-mer search with confidence estimates, for rRNA sequence analysis. 831 Bioinformatics, 33(23), 3808-3810. 832 McIntyre, A. B., Ounit, R., Afshinnekoo, E., Prill, R. J., Hénaff, E., Alexander, N., Minot, S. S., 833 Danko, D., Foox, J., & Ahsanuddin, S. (2017). Comprehensive benchmarking and 834 ensemble approaches for metagenomic classifiers. Genome biology, 18, 1-19. 835 Müller, A., Hundt, C., Hildebrandt, A., Hankeln, T., & Schmidt, B. (2017). MetaCache: context-836 aware classification of metagenomic reads using minhashing. Bioinformatics, 33(23), 837 3740-3748. 838 Nasko, D. J., Koren, S., Phillippy, A. M., & Treangen, T. J. (2018). RefSeq database growth 839 influences the accuracy of k-mer-based lowest common ancestor species identification. 840 Genome biology, 19, 1-10. 841 O’Sullivan, D. M., Doyle, R. M., Temisak, S., Redshaw, N., Whale, A. S., Logan, G., Huang, J., 842 Fischer, N., Amos, G. C., & Preston, M. D. (2021). An inter-laboratory study to investigate 843 the impact of the bioinformatics component on microbiome analysis using mock 844 communities. Scientific Reports, 11(1), 10590. 845 Parks, D. H., Chuvochina, M., Waite, D. W., Rinke, C., Skarshewski, A., Chaumeil, P.-A., & 846 Hugenholtz, P. (2018). A standardized bacterial taxonomy based on genome phylogeny 847 substantially revises the tree of life. Nature biotechnology, 36(10), 996-1004. 848 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint Piro, V. C., Dadi, T. H., Seiler, E., Reinert, K., & Renard, B. Y. (2020). ganon: precise 849 metagenomics classification against large and up-to-date sets of reference sequences. 850 Bioinformatics, 36(Supplement_1), i12-i20. 851 Piro, V. C., & Reinert, K. (2023). ganon2: up-to-date and scalable metagenomics analysis. bioRxiv, 852 2023.2012. 2007.570547. 853 Pruesse, E., Quast, C., Knittel, K., Fuchs, B. M., Ludwig, W., Peplies, J., & Glöckner, F. O. (2007). 854 SILVA: a comprehensive online resource for quality checked and aligned ribosomal RNA 855 sequence data compatible with ARB. Nucleic acids research, 35(21), 7188-7196. 856 Robeson, M. S., O’Rourke, D. R., Kaehler, B. D., Ziemski, M., Dillon, M. R., Foster, J. T., & 857 Bokulich, N. A. (2021). RESCRIPt: Reproducible sequence taxonomy reference database 858 management. PLoS computational biology, 17(11), e1009581. 859 Ruscheweyh, H.-J., Milanese, A., Paoli, L., Karcher, N., Clayssen, Q., Keller, M. I., Wirbel, J., 860 Bork, P., Mende, D. R., & Zeller, G. (2022). Cultivation-independent genomes greatly 861 expand taxonomic-profiling capabilities of mOTUs across various environments. 862 Microbiome, 10(1), 212. 863 Saini, J. S., Hassler, C., Cable, R., Fourquez, M., Danza, F., Roman, S., Tonolla, M., Storelli, N., 864 Jacquet, S., & Zdobnov, E. M. (2022). Bacterial, phytoplankton, and viral distributions and 865 their biogeochemical contexts in meromictic Lake Cadagno offer insights into the 866 Proterozoic Ocean microbial loop. MBio, 13(4), e00052-00022. 867 Schloss, P. D., & Handelsman, J. (2005). Introducing DOTUR, a computer program for defining 868 operational taxonomic units and estimating species richness. Appl Environ Microbiol, 869 71(3), 1501-1506. https://doi.org/10.1128/AEM.71.3.1501-1506.2005 870 Schloss, P. D., Westcott, S. L., Ryabin, T., Hall, J. R., Hartmann, M., Hollister, E. B., Lesniewski, 871 R. A., Oakley, B. B., Parks, D. H., & Robinson, C. J. (2009). Introducing mothur: open-872 source, platform-independent, community-supported software for describing and 873 comparing microbial communities. Applied and environmental microbiology, 75(23), 874 7537-7541. 875 Schoch, C. L., Ciufo, S., Domrachev, M., Hotton, C. L., Kannan, S., Khovanskaya, R., Leipe, D., 876 Mcveigh, R., O’Neill, K., & Robbertse, B. (2020). NCBI Taxonomy: a comprehensive 877 update on curation, resources and tools. Database, 2020, baaa062. 878 Seppey, M., Manni, M., & Zdobnov, E. M. (2020). LEMMI: a continuous benchmarking platform 879 for metagenomics classifiers. Genome research, 30(8), 1208-1216. 880 Shen, W., Xiang, H., Huang, T., Tang, H., Peng, M., Cai, D., Hu, P., & Ren, H. (2023). KMCP: 881 accurate metagenomic profiling of both prokaryotic and viral populations by pseudo-882 mapping. Bioinformatics, 39(1), btac845. 883 Song, L., & Langmead, B. (2024). Centrifuger: lossless compression of microbial genomes for 884 efficient and accurate metagenomic sequence classification. Genome biology, 25(1), 106. 885 Steinegger, M., & Söding, J. (2017). MMseqs2 enables sensitive protein sequence searching for 886 the analysis of massive data sets. Nature biotechnology, 35(11), 1026-1028. 887 Weging, S., Gogol-Döring, A., & Grosse, I. (2021). Taxonomic analysis of metagenomic data with 888 kASA. Nucleic acids research, 49(12), e68-e68. 889 Wood, D. E., Lu, J., & Langmead, B. (2019). Improved metagenomic analysis with Kraken 2. 890 Genome biology, 20, 1-13. 891 Yang, C., Chu, J., Warren, R. L., & Birol, I. (2017). NanoSim: nanopore sequence read simulator 892 based on statistical characterization. Gigascience, 6(4), gix010. 893 Zhou, N., Zhou, H., & Hoppe, D. (2022). Containerization for high performance computing 894 systems: Survey and prospects. IEEE Transactions on Software Engineering, 49(4), 2722-895 2740. 896 897 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint Figures and Tables 898 899 FIGURE 1. The LEMMIv2/16S workflow. The benchmarking pipeline is publicly available and can 900 be used by any tool developer or user in their own computing environment. It is also used by the 901 LEMMIv2/16S development team to maintain a continuously updated catalogue of evaluated tools 902 that can be consulted online. All methods are packaged in software containers that can be shared 903 in public repositories and are available for re-evaluation or use in real-world analyses. Additional 904 compatible datasets can be shared on Zenodo and reused with the pipeline. 905 906 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint 907 908 FIGURE 2. A) The home page of LEMMI16S. Different benchmark instances are presented with 909 a brief description of theirs contents, highlighting key features such as the taxonomy used or the 910 presence of unknown lineages. The version of LEMMI16S used to generate the benchmark is 911 specified for each instance. B) Home page of LEMMIv2 for metagenomics with a similar layout. 912 913 A) B) .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint 914 915 916 FIGURE 3. The LEMMI16S widget summarising key metrics. Users can select a shortlist of tools 917 to display together.918 Recall Comparative L2-distance based on ranking F1 Score Precision Comparative Memory Usage Comparative Runtime Efficiency .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint 919 920 FIGURE 4. LEMMIv2 reads and references sampling. A) GenBank genome assemblies with 921 sufficient metadata serve as starting material. Four species are represented by four colours 922 (white, grey, black, purple). B) When creating an instance of LEMMIv2, genomes are randomly 923 selected. At least two representatives are required for a species to be retained. These serve as 924 the source of the reads, while the rest form the reference provided to the tools. C) If controlling 925 the reference for all tools is not possible, (e.g. for tools based on curated marker genes), a cut-926 off date can be used to separate the reference from the reads. By defining a date that corresponds 927 to the publication date of the most recent custom reference, LEMMIv2 will select only more recent 928 genomes for the reads and keep all older genomes for the reference provided to the other tools. 929 D) To increase complexity, entire species (in this case taxon C in black), are made unknown by 930 removing all representatives from the reference provided to the tools. 931 932 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint Table 1. Minimal number of reads required to consider a taxon as being present at the species 933 level in the evaluation samples of the instance 2023_12_PROK_NCBI_v220. This lower detection 934 limit maximizes the F1-score on the calibration samples and is applied to the evaluation samples. 935 936 937 Tool name Value metaphlan_402 0 motus_301 0 kraken_212 1530 kraken_212_all 2548 ganon_200 1054 sourmash_423_all 988 ganon_200_all 1126 kmcp_090 2722 sourmash_423 1141 centrifuge_104 4341 metabuli_102 5638 centrifuger_100 5190 938 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint Supplementary Material 939 940 941 942 Supplementary Figure 1. Detail page of a LEMMIv2 instance, showing different widgets 943 reporting average values for all the samples or metrics that are not specific to a sample such as 944 resources for building the reference. 945 946 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint 947 948 949 Supplementary Figure 2. Detail page of a LEMMIv2 instance, showing different widgets 950 reporting values for a specific evaluation sample, including a heatmap on how dissimilar the 951 predictions of each tool are for that sample. 952 953 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint 954 955 956 957 958 Supplementary Figure 3. Extract from the LEMMI16S catalogue. Each tool has a section where 959 different versions can be presented, with a link to the source of the container, the built container 960 available in a repository, and links to a website and publication for the method where applicable. 961 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint 962 Supplementary Figure 4. Relative abundances of taxa predicted by Sourmash and Metaphlan 963 on LEMMIv2 samples 2023_05_PROK_NCBI-c001, c002 and the corresponding real sample. 964 Includes predictions at all taxonomic ranks. 965 966 967 968 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint 969 970 Supplementary Figure 5. Performance of methods for the identification of organisms at species 971 level for the instance 2023_12_PROK_NCBI_v220. Methods marked with _all used up to five 972 genomes for each species as a reference. The others used only one genome as reference. 973 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint 974 975 976 Supplementary Figure 6. Performance of methods for the identification of organisms at genus 977 level for the instance 2023_12_PROK_NCBI_v220. Methods marked with _all used up to five 978 genomes for each species as a reference. The others used only one genome as reference. 979 980 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint 981 982 Supplementary Figure 7. A) Average runtime (minutes) to perform the analysis of a sample from 983 the 2023_12_PROK_NCBI_clean_v220 instance. B) Average peak memory (GB) to perform the 984 analysis of a sample from the 2023_12_PROK_NCBI_clean_v220 instance. C) F1 score of 985

for the identification of organisms at species level for the instance 986 2023_11_PROK_GTDB_v220. D) F1 score of methods for the identification of organisms at 987 species level for the instance 2022_03_PROK_NCBI_2. E) F1 score of methods for the 988 identification of organisms at species level for the instance 2022_01_EUK_NCBI. F) F1 score of 989

for the identification of organisms at species level for the instance 2021_9_VIR_NCBI. 990 G) F1 score of methods for the identification of organisms at species level for the instance 991 2022_04_PROK_NCBI_LR2. 992 A) B) C) D) E) F) G) .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint 993 Supplementary Figure 8. Results taken from LEMMI16S instance representing Human Oral 994 Microbiome samples with unknown organisms. 995 996 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint Supplementary Table 1. Characteristics of simulated samples compared to the real samples on 997 which they are based. 998 999 Reads length Number of paired-end reads Number of 31- mers %GC LEMMIv2 Plaque from a healthy patient 151 21'153'159 2,059,607,399 45 Simulation 2023_05_P ROK_NCBI- c001 150 20'015'528 2,206,174,136 41 Simulation 2023_05_P ROK_NCBI- c002 150 20'055'700 2,224,328,630 41 LEMMI16S Human pathogens samples 300 654,196 354,293,576 53.7 Simulation Human pathogens samples_gc n-c001 250 867,086 380,472,649 52.4 Simulation Human pathogens samples_gc n-c002 250 892,299 392,439,207 53.6 1000 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint Supplementary Table 2. Status of the tools in terms of the ability of the container to filter out off-1001 target reads prior to final classification. In some cases, the tool does not classify at the read level, 1002 making such pre-filtering impossible, and in other cases, the process was considered too heavy 1003 in terms of runtime and memory. 1004 1005 Tool name Filter host Filter contaminants ccmetagen_140 Yes No centrifuge_104 Yes Yes centrifuge_100 Yes Yes deepmicrobes_g No No ganon_200, _160, _100 Yes Yes kmcp_090, _081 Yes Yes kraken_212 Yes Yes metabuli_101 Yes Yes metacache_220 Yes Yes metamaps_01 Yes Yes metaphlan_402, _3013 No No mmseqs_12113 Yes Yes motus_301 No No sourmash_423 Yes Yes virmet_100 Yes Yes 1006 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint Supplementary Table 3. Estimation of true abundance (L2 distance to truth) at species level. 1007 Average ranks in minimising error across all evaluation samples from instance 1008 2023_12_PROK_NCBI_v220. 1009 1010 Tool name Value metaphlan_402 12 motus_301 3.25 kraken_212 4.75 kraken_212_all 2.75 ganon_200 6.5 sourmash_423_all 10 ganon_200_all 5.25 kmcp_090 1.15 sourmash_423 11 centrifuge_104 4.5 metabuli_102 7.75 centrifuger_100 9 1011 .CC-BY-NC-ND 4.0 International licensemade available under a (which was not certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is The copyright holder for this preprintthis version posted March 11, 2025. ; https://doi.org/10.1101/2025.03.06.641904doi: bioRxiv preprint