Research
How Accurate Is AI Beatmap Timing? ManiaGAI vs Mapperatorinator
We timed 2,501 ranked osu! mania songs automatically and compared the result with the mappers' own timing — ManiaGAI v2, ManiaGAI v1 and Mapperatorinator, at four tolerances. Where each method wins, and what the benchmark can't tell you.
Players forgive a lot in a beatmap. They don’t forgive timing. A chart whose notes sit a few milliseconds off the beat feels wrong from the first measure, however good its patterns are — so before an AI mapper can place a single note, it has to answer two questions a human mapper answers by ear: what is the song’s BPM, and where exactly does the beat fall?
This post is about how well ManiaGAI answers them. We ran its timing detection on every ranked osu! mania song we could evaluate and compared the result with the timing the mapper wrote — next to ManiaGAI v1 and the open-source Mapperatorinator. The short version: ManiaGAI v2 gets the most songs right at the tolerance players actually hear, Mapperatorinator is more precise at the tightest one, and the benchmark leaves real questions open. The details are below.
What was measured
Automatic timing has to produce what a mapper produces in the editor: a BPM and an offset, written as timing lines. For a song with one steady tempo, that is one line. So the test set is every ranked or approved osu! mania mapset whose mapper timed the song with exactly one line — 2,507 mapsets, after removing rate-edit packs and duplicate mapsets. All three methods produced a result on 2,501 of them, and every number below is over those 2,501 songs.
The three methods:
- ManiaGAI v2 — the timing detection behind ManiaGAI since version 2 (October 1, 2026): beat activations from the open-source Beat This model, turned into a timing grid by our own fitting method. The benchmark ran a configuration close to, but not identical with, the one that shipped.
- ManiaGAI v1 — the detector the site used before version 2.
- Mapperatorinator — the timing mode of OliBomby’s open-source AI mapper, run as released. It is the only open-source timing baseline we benchmarked.
A song counts as correct when the detected BPM matches the mapper’s — half or double tempo allowed, since a song timed at 85 or 170 BPM can be charted identically — and the beat phase lands within a tolerance of the mapper’s. We report four tolerances, from 10 to 40 ms.
The results
| Method | ≤10 ms | ≤20 ms | ≤30 ms | ≤40 ms | BPM correct | One timing line |
|---|---|---|---|---|---|---|
| ManiaGAI v2 | 71.4% | 92.3% | 96.5% | 97.1% | 98.0% | 100% |
| Mapperatorinator | 78.0% | 90.9% | 91.7% | 91.7% | 92.6% | 11.0% |
| ManiaGAI v1 | 74.8% | 88.0% | 90.3% | 90.7% | 94.0% | 92.8% |
Share of 2,501 songs. “BPM correct” allows half/double tempo. “One timing line” = the song was timed with a single line, as its mapper did.
Which method leads depends on the tolerance, and that is the most informative thing in the table. At 10 ms, Mapperatorinator is ahead. From 20 ms on, ManiaGAI v2 is. At 40 ms ManiaGAI v2 is correct on 97.1% of songs, against 91.7% for Mapperatorinator and 90.7% for ManiaGAI v1.
Why 40 ms is the headline number
A tight tolerance measures precision; a looser one measures whether the timing is usable. We pick 40 ms as the headline because of a listening check: of 31 songs that the 10 ms criterion marked wrong, 30 sounded correctly timed when played. Errors of 10–40 ms are real, and the table reports them — but at that size the BPM and the beat are right, and that is what decides whether a chart plays on the beat.
Song by song
Percentages hide which songs each method gets right. Counting songs where exactly one of the two methods was correct:
- At 40 ms, ManiaGAI v2 was right where Mapperatorinator was wrong on 187 songs, and the reverse happened on 51. Both were right on 2,242; both were wrong on 21.
- At 20 ms the margin is much closer: 187 songs to 153.
- Against its own predecessor at 40 ms, ManiaGAI v2 won 190 songs and lost 30.
Where Mapperatorinator is better
Mapperatorinator places the beat more precisely when it gets the tempo right:
- Within 10 ms it is correct on 78.0% of songs, against 71.4% for ManiaGAI v2.
- Its beat-phase error is smaller: a median of 5.0 ms against 5.8 ms.
Where ManiaGAI is better
ManiaGAI v2 gets the tempo right more often (98.0% against 92.6% with half/double allowed), and it has a much smaller problem with false tempo changes. Every song in this set has one steady tempo, so any extra timing line is a tempo change that isn’t in the music — and in a chart it means a sudden shift of the beat grid in the middle of a song.
- ManiaGAI v2 timed all 2,501 songs with a single line.
- ManiaGAI v1 added lines on 181 songs (7.2%).
- Mapperatorinator added lines on 2,226 songs (89.0%) — a median of 6 lines and a mean of 7.9 per song.
That needs context: Mapperatorinator doesn’t assume a song has one tempo, and on songs that genuinely change tempo, writing several lines is the right behaviour. This benchmark only contains steady-tempo songs, so it can show the cost of that design and not its benefit.
How close to the ceiling?
At 40 ms, 16 songs (0.6%) defeated all three methods. Even a perfect judge that picked the best of the three for every song would land at 99.4% — so the distance between ManiaGAI v2’s 97.1% and that ceiling is small. Of those 16 songs, 12 were tempo errors for ManiaGAI v2 and 4 were phase errors.
What this benchmark doesn’t tell you
- It is not a held-out test. 344 of the songs had been used while the ManiaGAI v2 detector was being tuned. We checked whether that inflated the result: at 40 ms, ManiaGAI v2 scores 97.1% on those songs and 97.1% on the 2,157 it never saw; at 10 ms, 70.3% against 71.5%. The tuning didn’t buy accuracy that fails to generalise — but a clean held-out set would be stronger evidence.
- Only steady-tempo songs. Songs that change tempo are harder for every method, and they aren’t here. In the data ManiaGAI v2 learned from, 15.8% of charts change tempo mid-song.
- Only ranked maps, with the mapper as ground truth. Ranked timing is checked by people, which makes it the best reference available — but it is still one person’s timing, and the population is the music that gets ranked.
- One external baseline. Mapperatorinator is the only other method we ran. We haven’t benchmarked beat trackers that weren’t built for osu!, and we don’t claim anything about them.
- Timing, not charts. The benchmark scores the timing grid alone, not how good the generated charts are.
- Close to the shipped configuration, not identical. The production detector differs in details from the arm measured here.
What’s next for timing
Three things stand out. The remaining steady-tempo errors are mostly about tempo, not phase. At the tightest tolerance there is still precision to gain. And the biggest open question is the one this benchmark can’t answer: how every method — ours included — handles songs whose tempo really changes. That is the gap a future benchmark needs to close.
If you want to see what the timing means in practice, BPM and timing in osu! explained covers the basics, the Technology page shows where timing sits in the whole pipeline, and the fastest test is a song you know well.