No Fine-Tuning, Real-Time Jam: Sampling Control for Pretrained Music Transformers

Core Idea: Don't Train the Model—Steer Its Sampling

A developer on Reddit shared a fascinating experiment: having a pretrained symbolic music model (the Anticipatory Music Transformer, specifically music-medium-800k) jam in real time with a human musician. The key point: no fine-tuning at all—instead, they "steer" the model's output by modifying the logits at sampling time. The author calls this approach "steering the sampling."

Concretely: the author plays a MIDI keyboard while the model plays a second piano. During the performance, the system tracks the current key, the chords being played, and tempo in real time, then applies biases to the logits before sampling. This method rests on a simple but effective intuition: the model already "understands" music; all it lacks is real-time contextual constraint.

Logits and sampling: In autoregressive generative models, at each step the network outputs a real-valued vector spanning the entire vocabulary (here, a vocabulary of notes/events)—the logits. After softmax, this yields a probability distribution over the next token, and a sampling strategy (such as temperature sampling, top-k, or top-p) determines the actual output. "Modifying the logits at sampling time" means adding or subtracting a bias value dimension by dimension to the vector before softmax: positive values raise the probability that certain tokens are chosen, negative values suppress them. This approach doesn't change the model weights—it only intervenes at inference time—so it can be switched on and off in real time, tuned at any moment, and won't disrupt the overall musical structure the model has already learned.

Symbolic music and the Anticipatory Music Transformer: Symbolic music refers to music represented as discrete events (pitch, duration, velocity, instrument, etc.)—such as MIDI or event sequences—rather than waveform audio. The Anticipatory Music Transformer is an autoregressive Transformer for symbolic music generation; its distinguishing feature is an "anticipation" mechanism introduced during training that lets the model generate the current part based on the context of subsequent (or concurrent) parts, making it naturally suited to accompaniment, continuation, and multi-part interaction scenarios. music-medium-800k is one of its medium-scale checkpoints, trained on roughly 800,000 music sequences.

The Specific Mechanics of Real-Time Steering

The bias rules the author lists are very concrete, each corresponding to a musical need:

• Down-weight out-of-key notes: lower the probability of notes outside the current key, reducing "off-key" moments;
• Up-weight chord tones and tension notes: raise the probability of chord constituent notes and tension notes so the accompaniment fits the harmony more closely;
• Down-weight hand regions: down-weight the pitch ranges where the human musician's hands are currently playing, so the model doesn't "steal the show";
• Up-weight when the player stops: when the author stops playing, raise the probability in the melody region so the model naturally fills in;
• Phrase-continuation reward: give a small bonus to outputs that continue the intervallic relationships of the previous phrase, creating a sense of "call and response."

In addition, the system samples about one second ahead, and whenever the author changes chords, it discards the planned content and resamples. The whole system runs on a Jetson Orin Nano, and the author has open-sourced the code (GitHub repository duetto).

In-key notes, chord tones, and tension notes: In tonal music, a key (such as C major) specifies seven diatonic scale degrees, and out-of-key notes create a distinct "sour" feel, so down-weighting out-of-key notes is the most direct harmonic constraint. Chord tones are the constituent notes of the current chord (e.g., C, E, G for a C major triad), and up-weighting them keeps the accompaniment within the harmonic skeleton. Tension notes are notes outside the chord tones that create color and tension, such as ninths, elevenths, and thirteenths; moderately raising their probability keeps the accompaniment from being bland, but the proportion must be controlled so as not to undermine consonance. Together these three form a continuum from "safe" to "colorful," and the bias weights effectively tune how adventurous the model is along that continuum.

Why "No Fine-Tuning" Is Worth Noticing

The most valuable aspect of this project isn't the jam session itself, but the way it demonstrates a low-cost approach to adapting pretrained models. Fine-tuning requires data, compute, and time, whereas sampling steering only requires intervening on the logits at inference time—with virtually no extra training cost—and can be adjusted in real time during a performance.

For highly structured domains like symbolic music, this approach is especially feasible: key, chords, and pitch range can all be computed and constrained explicitly, making them a perfect fit for rule-based biases. The author also admits that all the bias parameters were "hand-tuned by ear," and that they themselves find this a bit crude.

The trade-off between fine-tuning and inference-time steering: Fine-tuning modifies model parameters via gradient updates, shifting the model's distribution as a whole toward the target data; the results are usually more natural and more general, but it requires labeled data, compute, and training cycles, and it's hard to change behavior on the fly during a performance. Inference-time steering only intervenes on the output distribution—zero training cost, interpretable, and hot-adjustable—but the rules are human-designed, so their coverage and generalization are limited by the rules themselves. The two are not mutually exclusive: a common approach is to first train a lightweight conditioning module (such as a conditioning head or adapter) on a small amount of data, then add its output as a logits bias at inference time—combining learning capability with real-time controllability. This is precisely a realistic path toward the "more principled method" the author asks about.

The Author's Question: Can It Be More "Principled"?

At the end of the post, the author poses an open question: is there a more systematic way to do this—for example, a small learned conditioning head, or guidance-style techniques borrowed for symbolic models?

This points to a real problem in generative music control today: manual tuning is interpretable and debuggable, but it transfers poorly; learned steering is more general but requires additional training data and models. For developers who want real-time interaction on local devices, striking a balance between the two may be the direction worth exploring next.

Implications for Real-Time AI Jamming

From an engineering perspective, this project proves that consumer-grade edge devices (the Jetson Orin Nano) are sufficient to support real-time symbolic music generation and interaction. The real bottleneck isn't compute—it's "how to make the model understand the human," that is, how to express the control signals.

The author's approach essentially translates music theory rules into logits biases—a lightweight, transparent path that can be iterated on quickly. It may not be the endgame, but for developers who want to rapidly validate ideas for interactive music AI, it's an excellent starting point.

One-line Summary

No fine-tuning required—just real-time modification of sampling logits enables human-AI improvised jamming.

Paragraph Summary

A Reddit developer demonstrated an experiment in real-time jamming between a human and the pretrained symbolic music model Anticipatory Music Transformer. The core approach is no fine-tuning at all—instead, rule-based biases are applied to the logits at sampling time: suppressing out-of-key notes, boosting chord tones and tension notes, avoiding the pitch range the musician is playing, raising the melody-region probability when the musician stops, and rewarding intervallic continuation of phrases. The system samples about one second ahead, discards and resamples whenever the chord changes, runs entirely on a Jetson Orin Nano, and the code is open-sourced. The author notes that this manual tuning approach is low-cost, transparent, and iterable in real time, but transfers poorly, and raises the open question of whether learned conditioning heads or guidance-style methods could replace it.