Why DanceGAI Uses a Planner + Noter to Generate Beatmaps

The most natural way to describe an AI beatmap generator is “audio in, neural network, notes out”. ManiaGAI doesn’t work that way. It splits the job in two: a Planner decides what each part of the song should feel like to play, and a Noter decides exactly where the notes go. This post explains why, what each model does, and what the split costs.

A beatmap is an arrangement, not a transcription

A chart that follows the audio closely isn’t necessarily a good chart. A good osu! mania map is closer to an arrangement: the mapper decides which sounds deserve notes, how dense a section should feel, where a jack or a long note belongs, how patterns flow between hands — and how all of that changes between an Easy and an Expert of the same song.

A single model that maps audio straight to notes has to make every one of those decisions at once, at every moment: structure, difficulty, style and exact column placement. That kind of model tends to get the local details right while losing the shape of the song — a chorus that doesn’t hit harder than the verse, a breakdown that doesn’t breathe.

Two questions, two models

1. Music
2. Understanding
3. Planner
4. Noter
5. Validation
6. Beatmap

Both models read the same musical picture of the song: a timing grid, and a representation of the audio aligned to it beat by beat (from the open-source MERT model). From there:

• The Planner answers what should happen. Before any note exists, it lays out the gameplay structure of the whole song, section by section: how hard each part should be, and the mix of long notes, jacks and other pattern types.
• The Noter answers how the notes go. It turns the plan into actual placement — which columns, which notes are taps and which are long notes, how patterns connect — following both the plan and the audio.

Each model gets a narrower job, and the plan gives note generation something a note-by-note model lacks: a view of the whole song that has already been decided.

The Planner: Gaussian diffusion over a whole song

The Planner is a Gaussian diffusion model of about 6 million parameters. It generates the plan for every section of the song jointly, conditioned on the music, the key count and the requested difficulty and style.

Diffusion fits this job for two reasons. A plan is a smooth, continuous description of a song — local difficulty and pattern mix rising and falling over time — which is the kind of signal diffusion models handle well. And generating it as a whole, rather than section after section, lets the Planner balance the entire song at once: it can make the chorus peak and the bridge rest because it sees both.

The Noter: MaskGIT-style refinement

The Noter is a MaskGIT-style non-autoregressive transformer of about 51 million parameters. Charts are represented as discrete pattern tokens. Instead of writing them one at a time from the start of the song to the end, the Noter fills in the whole chart and refines it over several passes — committing the notes it is confident about and revisiting the rest. How those passes are scheduled across a song is our own method, and one of the things we keep private.

Why not generate the chart left to right, the way a language model writes text?

• Writing strictly left to right means early decisions can’t take later music into account, and mistakes compound over a long song.
• In a refinement model, every pass sees context on both sides of every section, which keeps patterns consistent across a song.
• The whole chart is produced in a fixed number of passes instead of one token at a time — which also matters for speed.
• A section can be revised without regenerating everything after it (more on that below).

Difficulty and style are inputs, not afterthoughts

The split is also what makes generation controllable. ManiaGAI takes explicit controls, and the models were trained to follow them:

• Key count — 4K to 9K.
• Difficulty — any selection of star ratings from ★1 to ★10; one run produces a whole mapset.
• Style — five dimensions: long-note ratio, inverse long notes, jacks, short long notes and delay. Each difficulty can get its own style, calibrated from real charts at that key count and star rating.

These controls act on the plan, where they matter: “harder”, “more long notes” or “more jacks” are decisions about the shape of a section, not about one note. Density isn’t a separate knob — it follows from the target difficulty and the plan. And because every difficulty is generated from the same understanding of the song, a mapset reads as one family of charts rather than unrelated ones.

Local regeneration falls out of the design

Generated charts are rarely wrong everywhere. More often one passage is off — a pattern that doesn’t fit, a stretch that’s too dense. Regenerating the whole chart to fix it throws away everything that was already good.

1. Select
2. Mask
3. Regenerate
4. Refine

Because the Noter refines a whole chart rather than writing it left to right, it can regenerate a selected region while holding everything around it fixed, using the surrounding notes as context — so the new section joins the chart instead of clashing with it. That is the foundation for section-level editing: a map editor where you select the part you don’t like and regenerate only that section, or edit the notes yourself and test-play the result, is in internal testing. It is not available on the site yet.

Training them separately

The two models are trained separately, each on the representation it works with: the Planner as a diffusion model over a description of each section’s gameplay, the Noter by masked-token prediction over pattern tokens. Samples are balanced across key counts and difficulty levels so rare combinations, like 8K at ★10, aren’t drowned out by common ones. The data behind both is described in Inside the Dataset Behind an AI osu! mania Mapper.

What the split costs

A hierarchy isn’t free. The trade-offs we live with:

• Two models to train and keep in sync. The plan is an interface between them; when its definition changes, both sides have to follow.
• The Noter can only be as good as the plan it is given. A plan that misjudges a section gives the Noter the wrong brief.
• A plan only describes what we chose to describe. Section-level difficulty and pattern mix capture a lot of what makes a chart feel the way it does, but not everything a mapper intends.

What we don’t claim

We haven’t benchmarked this design against a single end-to-end model trained on the same data, so we don’t claim it is better than one. What we can say is that it gives us a cleaner way to model long-range structure and local note placement separately, a natural place to put difficulty and style controls, and a model that can revise one section of a chart without touching the rest. Mapping style is still controlled through a handful of dimensions; describing a style by example rather than by sliders is a direction we are working toward.

The Technology page shows where both models sit in the full pipeline, and how we made generation fast covers what running them takes.