Abstract
Music creation is an iterative process: composers repeatedly revise what has been written and decide how the music should develop next. Supporting this process requires modeling how music evolves under user instructions. We introduce the Music World Model (MWM), a hierarchical latent world model for controllable long-form symbolic music generation. MWM represents musical content as latent states and requested changes as explicit actions. A phrase-level planner predicts structural context, while a segment-level executor recursively generates musical states under this context and the requested action. Users can update actions at phrase boundaries, and the planner refreshes its predictions from generated history to account for how the music has evolved. However, mismatches between new plans and preceding context can enter this history and influence subsequent plans, causing structural drift. We therefore introduce Boundary-Aware History Correction (BHC), an inference-time mechanism that combines context anchoring with persistent correction memory to reconcile planned slow trajectories with preceding context. BHC carries correction information across phrase boundaries while keeping learned weights and external actions fixed, without future reference music. To support learning these action-conditioned dynamics, we construct Slakh-Action from Slakh2100 Redux, aligning multi-track symbolic music with segment-level states, phrase-level summaries, and structural action annotations.
Model Overview
View full size
Represent
Encode musical context into a compact latent state.
Plan
Predict phrase-level structure from the evolving history.
Generate
Roll out segment states under actions, then decode the music.
Music Continuation
Five musical contexts, each followed by a 32-bar multi-track continuation. The same contexts are used in the control experiments below.
Listen from the beginning, or skip the observed context to hear the continuation.
Context GeneratedExample selection and verification
We screened the existing 1,760 validation windows from 233 songs for lower repetition while retaining clear control responses. Every selected window passes all 11 control groups and all 16 individual checks, has an active melody and nonempty restoration targets, and keeps its continuation active through 64 bars. Exported MIDI notes also meet precision, recall, restoration and background-preservation thresholds.
Among eligible windows, we prioritized lower overlap between adjacent eight-bar phrases in both the arrangement and melody, including the later continuation, and selected one window per song. The examples share a fixed inference configuration and the same control settings across songs. This is a retrospective qualitative selection, not an unbiased evaluation or a human listening-quality ranking. Recurring motifs remain.
Selection protocol and provenance · Source-window ranking · Verification records
| Context | BPM | Repeat · 32 bars | Repeat · 64 bars | Control / 100 | Checks passed |
|---|
Model Comparison
Listen to the same five musical contexts: real ground truth, our continuation, and six baseline models prompted with GT music.
All players share the same 8-bar GT prelude, playback tempo, SoundFont, and gain. Baselines receive the observed music only; the actual next 32 bars are withheld. Recordings end at the last generated note with a short release tail. Native output lengths and generation limits are listed on each card. Generation and listening protocol · Model and recording provenance · GT sources
The requested continuation is 32 bars. For Context 2, BEAT and Text2midi use the most recent 4 observed bars to fit their native context limits. Ours retains its original history of up to 48 observed bars, while baseline inputs contain 8 bars (or the noted 4-bar fallback). MIDI and ABC adapters differ in quantization and expressive detail. These fixed qualitative examples use model-specific inference settings; they are not a controlled model ranking.
Long-Horizon Generation
Compare 32- and 64-bar excerpts from the same autonomous rollout for each song.
Listen for recurring motifs, evolving texture, and how the arrangement develops at phrase boundaries.
Control Examples
Compare control settings on the same five musical contexts, each paired with an autonomous reference.
Action Sequences
Apply one action every eight bars and compare the response against the same-context autonomous reference.
Shape the musical density
Click a phrase to listen. Switch players to compare the same moment.
Listening Guide & Recording Setup
What to listen for. Compare musical continuity, phrase transitions, density, instrument roles, and unintended changes. Use the same context across controls, and switch players to keep the listening position aligned.
Recording setup. Generation, horizons, and controls use five songs selected for lower phrase repetition among windows that passed every control check. The same settings are used across songs. The comparison pairs each generated continuation with the actual 32-bar ground truth. Six baseline models provide recordings conditioned on the corresponding observed music. BHC and component comparisons are reported in the experimental results below. Selection scores do not measure human listening quality.
Control meanings. Role restoration uses an explicit MIDI guide from the corresponding autonomous reference. Mute and density use output constraints; constrained music feeds subsequent generation. Guidance strength measures target following, not volume. Recordings use one SoundFont and common gain, with no per-clip normalization, fixed prefix tempo, and BHC disabled.
Playback. Only one recording plays at a time. The floating player follows the active recording. MP3 and MIDI links download the files associated with each slot. Verification records
BHC Analysis
Compare predicted plans, boundary-aware correction, and a teacher-plan diagnostic reference.
Predicted planRecursive generation under the model’s own structural plan.
Boundary-aware correctionAnchor the slow trajectory and carry a decaying boundary correction forward.
Teacher-plan referenceUse target-derived slow structure for diagnosis while retaining recursive execution.
The paper compares matched prefixes, commanded actions, and frozen model weights. BHC uses generated history without future reference music. The 32-bar evaluation contains 102 held-out songs and 1,024 windows; the 64-bar evaluation contains 44 songs and 396 windows. TT is a diagnostic reference that requires target-derived plans.
Component Ablations
Full BHC and five controls compare anchoring, propagation, smoothing, and correction direction.
All settings share the frozen checkpoint, prefixes, action schedules, and decoder. Six deterministic inference configurations are averaged within song; they are not independent training seeds. Confidence intervals use 10,000 paired song-bootstrap replicates.
| Correction setting | F1 gain ↑ | 95% CI | Latent reduction ↑ | Logit reduction ↑ |
|---|---|---|---|---|
| Full BHC | +0.0031 | [-0.0021,+0.0085] | 0.058 | 0.024 |
| Anchoring only | +0.0026 | [-0.0024,+0.0072] | 0.061 | 0.027 |
| Propagation only | +0.0010 | [-0.0030,+0.0046] | 0.032 | 0.013 |
| EMA slow-plan | +0.0012 | [-0.0028,+0.0051] | 0.036 | 0.015 |
| Reverse propagation | +0.0005 | [-0.0031,+0.0034] | 0.025 | 0.011 |
| Norm-matched random | -0.0009 | [-0.0045,+0.0014] | -0.006 | -0.002 |
| Correction setting | F1 gain ↑ | 95% CI | Latent reduction ↑ | Logit reduction ↑ |
|---|---|---|---|---|
| Full BHC | +0.0080 | [+0.0030,+0.0178] | 0.151 | 0.081 |
| Anchoring only | +0.0068 | [+0.0005,+0.0132] | 0.119 | 0.062 |
| Propagation only | +0.0045 | [+0.0007,+0.0093] | 0.072 | 0.038 |
| EMA slow-plan | +0.0023 | [-0.0012,+0.0059] | 0.051 | 0.024 |
| Reverse propagation | +0.0012 | [-0.0026,+0.0050] | 0.035 | 0.019 |
| Norm-matched random | -0.0002 | [-0.0013,+0.0011] | 0.012 | 0.005 |
F1 gain is measured at the terminal step relative to uncorrected MWM. Latent and logit reductions are decreases in rollout-averaged RMS distance to TT. At 32 bars, all F1-gain intervals include zero. At 64 bars, full BHC has the largest mean F1 gain and distance reductions; these comparisons do not establish either component’s necessity or a statistically significant advantage over anchoring alone.