Building OsuGAI: AI-Generated osu! standard Beatmaps

OsuGAI is DanceGAI’s first model for osu! standard, osu!’s original mode: hit circles, sliders and spinners spread across the playfield. It launched today as a beta, next to ManiaGAI for osu! mania. This report explains how it represents a map, what it learned from, how it was trained, what we measured — and, just as plainly, what it still gets wrong.

Why osu! standard is harder than osu! mania

An osu! mania chart is notes in columns: a generator decides when a note comes and which of 4 to 9 columns it lands in. In osu! standard every object also has a place on a 512 × 384 playfield, and the player has to move to it. That adds three problems mania doesn’t have:

• Two-dimensional movement. Spacing, jump angles and flow between objects are most of what makes a standard map feel good or bad — and they are continuous, not a choice between a few columns.
• Slider geometry. A slider is a curve with a length, a speed and sometimes repeats; it has to fit the music and the objects around it.
• Readability. Objects appear before you hit them, overlap and stack. A map can be perfectly timed and still be impossible to read.

The representation: gameplay first, editor objects second

The obvious way to teach a model osu! standard is to give it the editor’s objects — “circle at (x, y)”, “slider with these control points”. We tried that family first. OsuGAI v1 instead describes a map the way it is played, and turns that back into objects afterwards. Every beat is five tokens:

• One interaction token: the button state across the beat, sampled at 48 cells per beat — pressed or released, nothing else. A tap is a short press; a slider or spinner is a long one. The token is one of the 4,096 most frequent 48-cell patterns in the training charts, which cover 99.87% of training beats and 99.69% of held-out beats.
• Four spatial tokens: where the cursor is at up to four moments in the beat — every press, every release, and then the longest holds split evenly. Positions are on a 4-pixel grid (128 × 96 cells), so the largest rounding error is 2.83 pixels; an unused slot is empty.

That is five slots per beat and a single vocabulary of 16,385 tokens (4,096 button patterns plus 12,289 positions). The moments the spatial tokens describe are derived from the button pattern alone, so the model never has to learn object types.

Why separate intended gameplay from the editor’s representation? Two reasons, both measured on human replays before we trained anything. First, the editor’s choices are largely invisible in play: from the cursor alone, a short slider could be told apart from a circle only 43–45% of the time — so object-level details are decisions the decoder can make, not something the model must guess. Second, the gameplay description loses almost nothing that matters: maps rebuilt from an idealised button state plus a human cursor path were hit by other players’ replays as often as maps rebuilt from an object-aware description (97.9% vs 98.0%). Using the players’ actual button states instead dropped that to 78.5% — real presses are too noisy to learn from, which is why the button token is idealised.

The training data

The corpus is ranked, approved and loved osu! standard maps — 117,918 of 117,920 charts converted successfully. Validation and holdout are song-disjoint: no song in them appears in training. As for ManiaGAI, every chart’s timing was first moved onto our own timing detector’s grid, so the model trains on the same kind of grid it gets when it generates. No chart was dropped for being difficult.

Where the cursor positions come from: osu!lazer’s Auto replay

A map file says where objects are, not how the cursor moves between them. We derive the spatial tokens from the Auto replay of osu!lazer — the built-in perfect player. We ported its cursor logic to Python and checked it against the real game on 20 maps: 264,716 frames, the same frame counts, zero button or stacking mismatches, 0.014 pixels mean difference. Maps decoded back from Auto cursor paths were hit by other players’ human replays 99.7% of the time on a 91-map benchmark.

One honest caveat: Auto is not a human. It follows each slider’s exact path, so its cursor is close to “the original slider geometry written as positions” — 72% of original slider paths can be read back from Auto within 4 pixels, versus 1% from a single human replay. OsuGAI v1 therefore learns where a mapper puts things, not how people actually move.

The architecture

1. Music
2. Timing grid
3. Planner
4. Noter
5. Decoder
6. .osu / .osz

OsuGAI runs on the same engine as ManiaGAI (described in Planner and Noter), with the mode-specific parts swapped:

• Planner (reused). A small diffusion model, unchanged from our earlier osu! standard work, plans each part of the song before any object exists: its star level, and a style description — slider share, angle character, spacing irregularity, stream spacing and rhythm.
• Noter (new). A MaskGIT-style transformer of about 64.1 million parameters fills in the five tokens of every beat, following the plan, the audio and the difficulty settings (circle size, approach rate, overall difficulty and slider velocity). Besides exact-position accuracy, its training gives partial credit for a position that lands within 16 pixels of the target.
• Decoder (rule-based, repeat-aware). Turns gameplay back into objects: a press becomes a hit circle; a long press becomes a spinner when the cursor circles the centre for at least 1.5 beats, otherwise a slider — the simplest line, arc or curve that keeps the slider ball close to the predicted path. Clear, evenly timed reversals become a repeat slider. New combos follow a fixed rule.

Training: four rounds

The Noter was warm-started from our previous osu! standard model and trained in four rounds of four time-capped epochs (about 46 minutes each) — roughly 14 hours in total on a single RTX 4070 Ti SUPER (16 GB), peaking at about 6 GB of GPU memory. After every round we generated an evaluation set: 8 held-out songs × 22 settings = 176 maps.

• Validation loss fell from 3.1389 to 2.9570 across the four rounds.
• Structural validity: all 176 evaluation maps were valid osu! standard maps in every round — legal objects, no slot violations.
• Difficulty and style respond: measured star rating rose with the target on 8 of 8 songs, and each of the four style axes moved the map in the right direction on 8 of 8 songs — although following the plan section by section is still weak.
• Deterministic: the same song with the same seed gives byte-identical .osu files.

Speed. On that development GPU, eight difficulties of Flower Dance (4:22) took about 16–18 seconds through the timed stages — audio preparation, timing, audio features, planning and about 11–12 seconds of note generation; decoding and writing the files add about 0.6 seconds per map on average, on the CPU. This is a local benchmark: production runs on a different, cloud GPU, and we haven’t measured OsuGAI’s production latency yet.

Limitations

A valid map is not the same as a fun map, and none of the numbers above says OsuGAI’s maps are good to play competitively. What we know is wrong:

• Difficulty overshoot. Maps come out harder than the target, and much harder at the top: on Flower Dance, targets of 7.43, 8.71 and 10 measured 9.28, 13.49 and 13.99 stars. The website therefore offers targets ★1–★6 only and always shows the measured rating.
• Sliders that cross themselves. 3.3% of generated sliders in the final round self-intersect (2.4% after the first round), against 0.1% in the original maps.
• Slider geometry artifacts. Some sliders hit the speed clamp (2.8% in the final round), and repeat sliders are rarer than in human maps (6.1% of sliders vs 21%).
• Auto supervision. The cursor paths are Auto’s, not people’s: the model learns object placement, not human movement.
• Overflow and vocabulary fallback. 0.58% of training beats need more than four positions; only the first four are kept. 0.13% of beats fall outside the 4,096 button patterns and use the nearest one. Very dense maps are affected the most.
• No broad human playtesting yet. Every check so far is automatic or based on recorded replays. The beta is where that starts.

What’s next

• Better difficulty calibration — targets you can trust, and the full star range back.
• Cleaner slider geometry — fewer self-crossings and clamped speeds, more natural repeats.
• Human gameplay data — learning how people move, not only where objects are.
• Stronger style control — bars that the map follows section by section, not only on average.
• More osu! modes — osu! taiko and osu! catch are next on the roadmap.

New to the mode? What is osu! standard? explains the game itself; the technology page covers the engine OsuGAI shares with ManiaGAI.