Paper Archive

Expressive piano performance generation needs symbolic pitch, timing, and dynamics. We evaluate six tokenization strategies for a Transformer that generates classical piano performances. Our tokenizations add velocity, beat annotations, and sustain pedal, from note-only to full representations. We pretrain on MAESTRO, then finetune on ASAP with beat-level annotations. The model uses anticipatory-style note encoding with cross-attention on composer and genre. FAD on the ASAP test set shows that note + velocity + pedal and full modes achieve the lowest mean FAD (1.76 and 1.97). Both beat the note-only baseline (3.10). Beat tokens show mixed, category-dependent effects and do not improve the best modes on average.

Download

We present InstructFX2FX, a system for sequential audio effect refinement through multi-turn natural-language instructions. Existing text-to-effect systems are largely single-shot, mapping one textual descriptor to one preset. Real audio engineering is instead sequential: engineers refine an existing effect chain through successive instructions. This poses a stateful problem that single-shot systems do not address: given the current effect parameters state and a new instruction, update the sound while preserving what earlier instructions already achieved. InstructFX2FX addresses this with a hybrid architecture that divides labor between a language model and CLAP-guided optimization. The LLM serves as a high-level planner that selects effects and proposes the initial parameter state, motivated by recent evidence that LLMs can outperform CLAP-based optimization for single-turn text-to-effect mapping; CLAP-guided optimization then refines the existing parameter state, providing a more stable and robust refinement mechanism than LLM reprompting. In the demo, attendees drive a dry recording through successive natural-language instructions: after each turn, they choose how strongly the effect is applied, then issue the next instruction based on what still differs from the sound they intend. In a preliminary evaluation on SocialFX-derived descriptor pairs, CLAP-guided refinement achieves lower DSP-feature MMD than an LLM+LLM initialize-then-reprompt baseline on 9 of 10 pairs. Trajectory analysis further shows that, for differentiable effects, optimization tends to gradually move the audio toward the new target while retaining the effects of the previous instruction, highlighting the potential for gradual refinement. Audio demo and source code are available online.

Download