SimRec: Reinforcement Learning Reshapes the Search and Follow-up Strategies of Recommendation Agents

SimRec uses reinforcement learning to train the complete interaction strategy of a recommendation agent, with a small model surpassing larger training-free agents.
SimRec is a reinforcement learning framework that optimizes the complete interaction strategy of a tool-integrated recommendation agent as a whole, rather than separately tuning isolated modules like query rewriting and ranking. It introduces a task-anchored user simulator (providing feedback conditioned on task requirements rather than user profiles) and task-aligned rewards (using GRPO to optimize the entire trajectory, with training-only goal injection to alleviate sparse rewards). On ESCI and Amazon C4, Qwen3-4B's NDCG@100 improves from 32.23% to 35.42% and from 22.94% to 24.72%, respectively, surpassing larger training-free agents. The trained agent more frequently verifies candidates, leverages preferences, and recovers from rejected recommendations. The code is open-sourced.
Recommendation systems are moving from "guess what you like" toward "search as you chat." When large models are given the ability to search the product catalog, view details, ask users follow-up questions, and even re-recommend after being rejected, a more fundamental question emerges: who decides these behaviors? Most current approaches rely on manually designed workflows, and during training they only optimize isolated components like query rewriting or list ranking—the interaction strategy as a whole goes unmanaged. SimRec attempts to fill this gap.

From Component Optimization to Policy Optimization
SimRec is a reinforcement learning framework whose goal is to train the complete interaction strategy of a tool-integrated recommendation agent, rather than separately tuning the search, ranking, and follow-up modules.
The problem with existing training methods is fragmentation: the module responsible for rewriting queries is only accountable for rewrite quality, and the module responsible for ranking is only accountable for list quality—but in a recommendation conversation, decisions like "when to ask a follow-up question, when to give results directly, and how to adjust after being rejected" have no unified optimization objective. SimRec treats the entire trajectory as the optimization target, letting the agent learn how to combine these tool calls within a complete interaction.
This shift in thinking means the reward signal must also be redesigned—it can't just look at whether the final hit occurred, but must also cover whether the retrieval process and tool usage are reasonable.
Two Key Designs: A Task-Anchored User Simulator and Task-Aligned Rewards
SimRec introduces two core components.
The task-anchored user simulator is responsible for giving feedback on each recommendation. Its distinguishing feature is that it does not rely on user profiles; instead, it responds conditioned on the current task requirements. This means the simulator judges "whether this recommendation satisfies the need" rather than "whether this recommendation matches some user's historical preferences." The simulator can be an external LLM, or in self-play mode, the recommender itself can play the role.
Task-aligned rewards use GRPO to optimize the entire trajectory. The reward looks not only at the final hit result but also factors in natural retrieval quality and effective tool usage. Because hit signals are often sparse in large catalogs, SimRec adopts a training-only goal injection mechanism to alleviate this problem.
Together, the two designs address two sources of training signal: immediate feedback from the user side and overall task-level evaluation.
GRPO (Group Relative Policy Optimization) is a reinforcement learning algorithm, a variant of PPO (Proximal Policy Optimization) designed specifically for aligning large language models. Unlike PPO, which requires training a separate value network (critic), GRPO samples a group of outputs for the same input and uses the mean and standard deviation of rewards within the group to normalize the advantage estimate, thereby eliminating the value model and significantly reducing memory and compute overhead. In SimRec, GRPO is used to optimize the entire recommendation interaction trajectory: multiple complete dialogue paths are sampled each time, relative advantages are computed based on task-aligned rewards, and then the policy is updated. This within-group relative comparison approach is naturally suited to sparse-reward settings—even if absolute reward values are low, as long as there is variation within the group, the policy can still obtain effective learning signals.
Goal injection is a mechanism SimRec introduces during training to alleviate sparse rewards. In large product catalogs, the probability that the agent ultimately hits the correct product through multiple rounds of interaction is very low, making the reward signal based on the final hit result extremely sparse and the policy difficult to learn effectively. Goal injection works by providing the agent or simulator with partial information about the target product during training (e.g., category, attribute keywords), artificially increasing task reachability so that positive signals appear more frequently in trajectories. The key is that this mechanism is used only during training and is completely removed at inference time, so it does not cause information leakage or reduce the task difficulty in actual deployment. This idea shares common ground with curriculum learning: first let the model learn effective behaviors on easier tasks, then generalize to harder scenarios.
Results: A Small Model Outperforms Larger Training-Free Agents
On the ESCI and Amazon C4 datasets, SimRec improves a Qwen3-4B agent's NDCG@100 from 32.23% to 35.42%, and on Amazon C4 from 22.94% to 24.72%.
More noteworthy is the comparison result: this trained 4B model surpasses much larger training-free agents in NDCG and also outperforms task-aligned baseline methods. Parameter scale is not the only variable—the training quality of the interaction strategy can also bring considerable gains.
The analysis also shows that the trained agent is significantly more frequent in three behaviors: verifying candidate products, leveraging user preferences, and recovering from rejected recommendations to offer new options. These are precisely the concrete manifestations of "knowing how to search and knowing how to ask follow-up questions," and they are the details that handwritten workflows struggle to cover.
NDCG@100 (Normalized Discounted Cumulative Gain at 100) is a ranking quality evaluation metric commonly used in information retrieval and recommendation systems. It measures the relevance ranking quality of the top 100 recommendation results: the relevance score at each position is discounted by position (the further back, the lower the weight), then divided by the discounted cumulative value of the ideal ranking (sorted by relevance in descending order) for normalization, yielding a final value between 0 and 1. The higher the NDCG@100, the more highly relevant products are ranked closer to the front. In SimRec's experiments, NDCG@100 is used to evaluate the quality of the final recommendation list produced by the agent, so an improvement from 32.23% to 35.42% means that among the Top-100 recommendations, relevant products are overall presented closer to the front, which has direct significance for user experience.
Implications for Recommendation Agent Development
The value of SimRec is not just in metric improvements. It suggests a path: rather than writing increasingly complex interaction flows for agents, it is better to treat the interaction strategy itself as the learning object and train it with simulated feedback and trajectory-level rewards.
The code is open-sourced at https://github.com/ZJU-REAL/SimRec, making it a directly usable starting point for teams that wish to reproduce it or build recommendation agent training on top of it. Of course, the gap between simulators and real users, as well as the feedback quality in self-play mode, remain aspects that such methods need to continuously validate during deployment.
One-line Summary
SimRec uses reinforcement learning to train the complete interaction strategy of a recommendation agent, with a small model surpassing larger training-free agents.
Paragraph Summary
SimRec is a reinforcement learning framework that aims to optimize the complete interaction strategy of a tool-integrated recommendation agent as a whole, rather than separately tuning isolated modules such as query rewriting and ranking. It introduces a task-anchored user simulator (which provides feedback conditioned on task requirements rather than user profiles) and task-aligned rewards (using GRPO to optimize the entire trajectory, along with training-only goal injection to alleviate sparse rewards). On the ESCI and Amazon C4 datasets, Qwen3-4B's NDCG@100 improves from 32.23% to 35.42% and from 22.94% to 24.72%, respectively, surpassing larger training-free agents. The trained agent more frequently verifies candidates, leverages preferences, and recovers from rejected recommendations. The code is open-sourced.
Related articles

Gemini 4 Neon Leak: A Potential Strong Contender Against Opus 5.5
A Hacker News thread suggests Gemini 4 Neon looks like an Opus 5.5 competitor. This article unpacks the product positioning behind this community signal, shifts in the top-model competitive landscape, and what it means for developers' model selection strategy.

Claude Fable 5.2's Self-Authored Short Film 'Fable' Sparks Discussion
A short film 'Fable,' credited to Claude Fable 5.2 for script, visuals, and directing, appears on Hacker News. This article analyzes the significance of full-pipeline AI creation, community reactions, and what it means for content creators.

Anthropic IPO Pricing: A Signal of AI Giants Going Public
Anthropic's IPO pricing draws attention. This article analyzes its potential impact on AI industry valuation, commercialization paths, and the competitive landscape, and highlights public-market expectations and challenges for AI companies.