1B Model Learns Both When to Interject and What to Say: Gating and Prompt Generation Unified

Proactive conversational assistants need to decide two things during a chat: whether to interject at this moment, and if so, what to say. Existing solutions typically split these two problems—using a gating module to decide timing, then a larger generator to produce content. A new study puts both into a single compact decoder and systematically evaluates it on over 28,000 decision points. The conclusions are both surprising and quite practical.

The Cost of Split Pipelines

The traditional approach is two-stage: gating first filters out moments that shouldn't trigger a response, and the generator is only invoked when actually needed. The appeal of this design is compute savings—ideally, many conversation turns never trigger large-model inference.

But the numbers in the paper illustrate the limits of this saving. When the generator is an 8B model, gating yields a 4.39x compute saving; with a 1B generator, the saving drops sharply to 1.38x. In other words, the smaller the generator, the weaker the "compute saving" rationale for gating, and its value lies more in accuracy—blocking moments that don't warrant a response and preventing low-quality prompts from polluting the conversation.

More critically, there's end-to-end performance. Evaluating generation scores in stages easily overestimates actual effectiveness—that's only the theoretical upper bound when gating is perfect (the oracle-gate ceiling). With real gating, end-to-end exact match is 0.2619 versus an upper bound of 0.2971; end-to-end precision is 0.2387. This means existing systems are still a notable distance from the ceiling, and jointly optimizing both stages is itself a worthwhile direction.

End-to-end exact match in this type of task typically means the system-generated prompt must be completely identical at the text level to the human-annotated reference prompt to count as correct. This is a very strict metric: even semantically equivalent prompts with slightly different wording or punctuation are marked wrong. It therefore gives a strict lower bound on performance, suitable for conservative comparisons but prone to underestimating practical usability. The gap between the paper's end-to-end exact match of 0.2619 and the oracle upper bound of 0.2971 reflects the real loss from imperfect gating, not the entire gap in generation quality itself.

Oracle-gate ceiling (theoretical upper bound with perfect gating) refers to the score obtained by assuming the gating module judges with 100% accuracy whether each moment should trigger a response, then letting the generator produce content. It strips out gating error and measures only the generator's capability under "ideal trigger conditions." Staged evaluations often report generation scores close to this ceiling, but this does not represent real deployment performance; real systems must count gating misses and false positives in end-to-end metrics, and the difference between the two is the room that joint optimization can capture.

Memory Is the Key Variable for Prompt Quality

In ablation experiments, the research team found a counterintuitive result: prompt quality is driven primarily by the memory profile, not by retrieval or generator scale.

Removing the memory profile causes exact match to drop by 24.7 points, and recalibrated BERTScore falls from 0.57 to 0.35. By contrast, adding a retrieval module or switching to a larger generator brought no gain in exact match. This finding has direct implications for product design—if an assistant is to produce short prompts that "fit the current conversation context," modeling users' long-term memory well is more cost-effective than stacking model parameters or bolting on retrieval.

Memory profile refers to the long-term structured information the system maintains for each user, such as preferences, habits, historical topics, and identity background, typically stored as summaries or key-value pairs and fed as conditioning input at each decision. It differs from a retrieval module in that retrieval looks up relevant snippets in the current conversation or an external knowledge base, whereas the memory profile is a user-level representation accumulated across sessions. In the ablation experiments, removing the memory profile caused exact match to drop 24.7 points, indicating that the model relies heavily on these stable user contexts when generating short prompts; without them, prompts tend to degrade into generic responses. This also explains why simply scaling up the generator or adding retrieval cannot compensate—they provide local information, not persistent user state.

How a Single 1B Decoder Handles Both Tasks

The proposed approach performs gating and generation within the same decoder: reading the gating signal from an intermediate layer and continuing to generate the prompt in the same forward pass. The benefit is eliminating the second inference and letting the gating decision and generated content share the same representation.

At the final checkpoint, this single model matches or exceeds the two-model pipeline on automatic end-to-end metrics, with wall-clock time 1.31 to 1.76 times less than an equivalent 1B two-model pipeline. Experiments across three random seeds show that its deep exit performs comparably to an independent gating module.

Note, however, the difference model scale makes: human evaluation (32 dialogues, 1,144 ratings) supports the memory-effect conclusion while showing the single model is on par with a matched 1B pipeline but below the 8B pipeline. In other words, the compact model solves the "efficiency and engineering complexity" problem, not the "absolute quality ceiling" problem.

Mid-layer gating means not waiting until the decoder's final layer to make a decision, but instead tapping a lightweight classification head at some intermediate layer of the network to output a "should respond now" signal. Since Transformer layers progressively aggregate context representations, intermediate layers often already contain enough decision information; reading early avoids running the full generation process for turns that shouldn't respond. More importantly, the gating signal shares the same hidden representation as subsequent generation, so the gating decision can directly influence the direction of the generated content, rather than being independent as in a two-model pipeline. This design lets a single forward pass perform both judgment and generation, saving the latency and memory overhead of a second inference.

The Evaluation Methodology Itself Deserves Discussion

The paper points out that exact match is a strict lower bound that underestimates practically usable prompt quality. Graded metrics and LlamaPIE's built-in judge correlate more strongly with human ratings than exact match. This is a reminder for teams building conversational assistants: optimizing solely against exact match may cause you to miss improvements that genuinely affect user experience.

Overall, the value of this work is not in claiming that small models win across the board, but in evaluating "when to interject" and "what to say" as a joint problem and providing empirical evidence that memory profiles matter more than retrieval and model scale. For teams looking to deploy proactive assistants on-device or in low-latency scenarios, this is an engineering route worth considering.

BERTScore is a text similarity metric based on pretrained language model embeddings, computing the degree of match between generated and reference text in semantic space through token-by-token alignment, outputting precision, recall, and F1. Compared with exact match, it is more tolerant of synonymous rewording and word-order changes, so it is often used as a "semantic-level" supplementary metric. In the paper, recalibrated BERTScore fell from 0.57 to 0.35, showing that removing the memory profile not only affects literal matching but also substantially harms semantic fit. LlamaPIE's built-in judge refers to the method bundled with that dataset that uses a large model to score the quality of generated prompts, belonging to the LLM-as-judge paradigm; it correlates more strongly with human ratings than exact match and is better suited to capturing prompts that are worded differently but equally valid.

One-line Summary

A single 1B decoder jointly handles gating and generation, with empirical evidence that memory profiles matter more than retrieval and model scale.

Paragraph Summary

The paper studies joint modeling of "when to interject" and "what to say" in proactive conversational assistants. Traditional two-stage pipelines use gating to filter timing and then a generator to produce content, but experiments show the compute saving weakens as the generator shrinks: 4.39x at 8B, only 1.38x at 1B; end-to-end exact match is 0.2619, far below the oracle upper bound of 0.2971. Ablations find prompt quality is driven primarily by the memory profile—removing it drops exact match by 24.7 points—while adding retrieval or a larger generator yields no gain. The authors propose reading the gating signal from an intermediate layer of a single 1B decoder and generating in the same forward pass, with wall-clock time 1.31 to 1.76 times less than a two-model pipeline and automatic metrics matching or exceeding it; human evaluation shows it is on par with a matched 1B pipeline but below the 8B pipeline. The paper also notes exact match is a strict lower bound, and graded metrics correlate more strongly with human ratings.