OmniOta validation dossier

The better the chemistry, the more reads resolved to species at the same accuracy: 99 in 2015, 15,549 in 2019, 28,774 in 2023, always above 97% correct. That is the behaviour expected of an engine that respects the limit of the data: it does not overclaim when the material is poor, and it exploits the material when it is good.

Each accuracy is conditioned on the reads the system does classify, and that denominator changes between rows: 89.5% of reads in 2023 against 53.7% in 2019. That is why the reads column always travels next to the percentage, in all five rows.

Brown et al. (2017) sequenced four pure cultures and several mixtures of declared composition on MinION, and published the accuracy of MG-RAST, Kraken and One Codex on them. Paytuví-Gallart et al. (2019) reprocessed those same files with GAIA and added their column to the table. It is the only point where an OmniOta figure can be placed beside a competitor's on known ground truth: the composition of the samples is not a matter of opinion, and the competitor's figure is published.

The other third-party papers that used GAIA — dog faeces, air in Ghana, pig whey, dust in Moscow — are real samples with no known composition: they show whether two tools agree, not which one is right.

Shigella is polyphyletic within Escherichia coli — clones that acquired the invasion plasmid — and its average nucleotide identity exceeds the species threshold. The name is retained for clinical, not taxonomic, relevance. At genus rank, counting a read of E. coli assigned to Shigella as an error measures an artefact of nomenclature, not a mistake by the classifier. It affects 36 of the 78 reads in the Ecoli dataset: without the merge the figure would be 39.7% instead of 94.6%, and that is why it is declared.

• It is not a bench table: it derives from the complexes the product already declares and shows to the user, citing ISO 13136:2012, so it can only merge what it merges in production.
• There are exactly two cases in the whole catalogue: Shigella → Escherichia and Borrelia → Borreliella, both of nomenclature conserved against phylogeny.
• It is applied equally to all five columns whenever another tool's rule is reproduced, never to ours alone.
• It does not touch species names: Shigella flexneri is reported as Shigella flexneri.

The dataset staggered es HM-783D from BEI Resources, el Microbial Mock Community B of the Human Microbiome Project: twenty certified species spread between 0.03% and 41.25% of genomic DNA. It demands telling S. aureus de S. epidermidis, three Streptococcus apart from each other, and detecting members three orders of magnitude below the dominant one — which is where a classifier really breaks, as against four well-separated genera that are almost got right by elimination.

Four of the twenty members have changed genus since 2010 and the current reference knows only the new name. Without declaring that equivalence the bench would have counted 20% of the mock as failure when it was a perfect hit. The ground truth is transcribed with the source's name and the equivalence lives separately: rewriting the truth with today's nomenclature would make it match our reference by construction, and the comparison would stop checking anything. A test guard pins both directions — that the renamed ones match and that congeners are not merged — because a synonym table is the convenient place to slip in exactly the merges that suit you.

It is a mixed community, so accuracy is an upper bound: any read assigned to any member counts as correct. The reads come from the run's prefix, which over-represents the flow cell's first hours; for comparing between platforms the bias acts the same way in both, and for an absolute figure it comes from the best stretch of the run and is stated as such. The two yeasts come out at exactly 0.00% on both platforms: the reference was built from RefSeq's bacterial catalogue and contains no fungi, and the figure being identical on both confirms it.

The same community, the same preparation, two different sequencers. The question is answered separately for the certified community and for everything emitted, because the answers differ and giving only one of the two would hide something.

The composition is invariant between GridION and PromethION. The per-genus differences run from 0.05 to 0.43 points and the global divergence is practically the one sampling itself produces: not "close to zero", but statistically indistinguishable from chance, which is what had to be shown.

Between the material previously used and the current chemistry there are 8.5 points of identity. That is not a nuance: it is the difference between being near the wall of an exact k-mer method and being far from it. Percentiles of the modern material: p25 = 95.14%, p75 = 98.16%, p95 = 98.92%, p99 = 99.33%.

On the 16S/amplicon axis the bench already covers R10.4.1 chemistry with material from September 2025 (MSPlus community, ont-open-data/zymo_16s_2025.09): 12 of 12 members detected, zero regulated false positives, one over-claim. Measured and recorded in the taxonomic-resolution documentation, and cited here as what it is: a measurement from another session, not re-run in this one.

Between Brown 2017 and Zymo D6300 everything changes at once: the year, the chemistry, the sequencer, the community, the laboratory and the identity of the reads. With two points differing in six variables, attributing a collapse to one of them is not justified, and any reviewer's obvious retort would be the right one: you picked an old dataset.

This experiment exists so that sentence stops being arguable. It takes the same file from Zymo and injects error at increasing rates, with a fixed seed and three depths to separate identity from depth. Everything else is identical by construction: the only free variable is identity. And it could have gone against us: had the collapse failed to reproduce, the published explanation would be false and would have to be withdrawn. That is why it counts.

The most degraded level lands at 74,23 %, which is the median identity of Brown 2017. The experiment reaches by construction the very point it set out to explain, starting from modern, certified material: no 2015 dataset was needed; degrading the 2019 one is enough.

sylph column at 5,000 reads; at 20,000 and 50,000 the collapse shifts but reproduces just the same. The ground truth is 8 members at all seven levels.

All eight organisms of the community are recovered at every level, including the most degraded — the same one where the competitor does not return a single line. It is the result that matters for a real food sample: DNA fragmented by thermal processing, a fatty matrix, difficult extraction.

Both tools find the complete community, and each is stronger on a different axis. Per read OmniOta is highly precise: 99.18%, and that is the figure governing an abundance table and everything computed on top of it. Per named taxon, sylph decides better: it emits 8 where we emit 61, because it models genome coverage with k-mer statistics and requires the genome to be genuinely present before naming it, whereas we name any taxon that receives a read assignment.

The two times are not directly comparable: the ~75 min include aligning against the nine parts of the reference while building the index on every pass, whereas sylph reuses an index built once. With a persistent index the difference would shrink considerably, and that measurement is pending. sylph is a profiler: it gives the list of genomes present with their abundance and does not assign each read, which is why it has no per-read accuracy column. Shigella boydii in its output is not a false positive: it is the Escherichia/Shigellacomplex, counted as correct by the same criterion applied to OmniOta in §3.1.

Kraken2/Bracken is not measured: its database with these same 22,514 genomes would require on the order of 170 GB of RAM, as measured, so the reason is published instead of giving it a trimmed database that would distort the comparison.

The certified composition of the HMW standard is equimolar between bacteria: 14.3% of genomic DNA each. Counting reads, Salmonella gathered 47.1% and Bacillus 3.2%. It is not subset bias — the profiles of the first and last 10,000 reads are practically identical — it is the metric. Counting reads is not counting DNA mass. Abundance is mean coverage, aligned bases divided by genome size, which is what CoverM, sylph and CAMI hold to.

Measuring DNA mass instead of counting reads halves the error — 51.8% less — on material with certified composition. Counting reads, Salmonella appeared to be 3.3 times its real mass and Bacillus a quarter of its own. The rule is not a matter of style.

The comparison with MetaPhlAn counts precisely because the two tools share nothing: we align against 22,514 complete genomes, MetaPhlAn profiles against its clade-specific UniRef markers. The two methods agree to within 2.14 points on average and on the ordering of the three largest, so the sequenced material is not equimolar, and that is established by two independent routes. And on what it does compete on, OmniOta lands 18% closer to the certified composition: L1 0.411 against 0.501.

For the remaining divergence there are at least two known reference-side causes before invoking any from the method: the "E. coli" in the panel is the genome of Shigella boydii and measuring coverage against a surrogate genome underestimates it; the same applies to P. aeruginosa PAO1. There is room for real extraction and lysis bias, which this experiment does not separate from the above. MetaPhlAn detects S. cerevisiae and we do not, because our reference is bacterial: it is a capability of theirs that we lack.

The full detail of each axis — extended caveats, per-dataset results, the bench's version history and the figures withdrawn for not being reproducible — lives in the repository's canonical documentation, available under agreement for technical evaluation.

1. Brown BL, Watson M, Minot SS, Rivera MC, Franklin RB (2017). MinION nanopore sequencing of environmental metagenomes. GigaScience 6(3):1-10. 10.1093/gigascience/gix007
2. Paytuví A, Battista E, Scippacercola F, Aiese Cigliano R, Sanseverino W (2019). GAIA: an integrated metagenomics suite. bioRxiv 804690. 10.1101/804690 — Table S2 of the supplementary: accuracy on Brown 2017.
3. Nicholls SM, Quick JC, Tang S, Loman NJ (2019). Ultra-deep, long-read nanopore sequencing of mock microbial community standards. GigaScience 8(5):giz043. 10.1093/gigascience/giz043
4. Shaw J, Yu YW (2024). Rapid species-level metagenome profiling with sylph. Nature Biotechnology. 10.1038/s41587-024-02412-y
5. Richter M, Rosselló-Móra R (2009). Shifting the genomic gold standard for the prokaryotic species definition. PNAS 106(45):19126-31. 10.1073/pnas.0906412106
6. Jain C, Rodriguez-R LM, Phillippy AM, Konstantinidis KT, Aluru S (2018). High throughput ANI analysis of 90K prokaryotic genomes. Nature Communications 9:5114. 10.1038/s41467-018-07641-9
7. Dunlap CA et al. (2020). Bacillus spizizenii as a species. Int J Syst Evol Microbiol.
8. Altman DG, Bland JM (1995). Absence of evidence is not evidence of absence. BMJ 311:485.
9. ISO 13136:2012 — Microbiology of the food chain: horizontal method for the detection of Shiga toxin-producing E. coli .
10. Sequencing material: European Nucleotide Archive PRJEB8672, PRJEB8716, ERR3152364, ERR3152365; NCBI PRJNA934154; ZymoBIOMICS D6300 manufacturer datasheet; BEI Resources HM-783D.

HelixCore · BIOTECNO.org · Vitoria-Gasteiz
OmniOta validation dossier · v1.4.0 · 2026-08-10