Scale AI AgentEnv Evaluation Framework Now Available on Vercel Sandbox

Scale AI's AgentEnv framework can now run on Vercel Sandbox. This framework is designed to test agents in concrete tasks and score their results. The Vercel provider is included in AgentEnv 0.9.1334 and later versions, and it runs the agent and its task environment in two separate Sandboxes.

What is AgentEnv?

AgentEnv is an agent evaluation framework from Scale AI. Its core idea is to put agents in front of real tasks and then score their performance. Unlike traditional evaluations that only examine the text a model outputs, AgentEnv places the agent in an interactive environment, records the actions it takes, and ultimately assigns a task score.

This evaluation approach is closer to real deployment scenarios: the agent needs to understand the task, plan steps, call tools, and leave a traceable behavioral trajectory in the environment. For developers and researchers, AgentEnv provides a reproducible way to compare differences between different agent implementations.

Agent Evaluation is fundamentally different from traditional large language model evaluation. Traditional evaluations (such as MMLU and GSM8K) usually only check whether the text from a single model output is correct—a static "input-output" mapping. Agent evaluation, by contrast, focuses on the complete trajectory of the model as a decision-making entity: calling tools, reading environment feedback, and adjusting strategy across multiple rounds of interaction. As a result, evaluation metrics expand from accuracy to dimensions such as task completion rate, step efficiency, tool-call correctness, and behavioral safety. Scale AI's AgentEnv is one of the representative frameworks of this paradigm, requiring agents to actually "do things" in an executable environment rather than merely "answer questions."

Why Choose Vercel Sandbox?

Vercel Sandbox provides AgentEnv with an isolated and programmable execution environment. According to the official description, the Vercel provider runs the agent and its task environment in two separate Sandboxes. This isolation design brings several practical benefits:

• Auditable behavior: The actions recorded by the agent can be inspected separately, making it easier to pinpoint issues.
• Environment consistency: The task environment is separated from the agent's runtime environment, reducing mutual interference and making scoring more stable.
• Low onboarding cost: You can directly reuse your existing Vercel CLI login state without needing to configure a separate set of credentials.

For teams already using Vercel, this means agent evaluation can be incorporated into existing workflows without maintaining another piece of infrastructure.

Sandbox is a restricted execution environment used to isolate and run untrusted code or code that may produce side effects. In agent evaluation scenarios, a sandbox needs to meet several key requirements: first, resource isolation, preventing the agent's file operations, network requests, or processes from affecting the host machine; second, resettability, so that each evaluation can start from a clean state; third, observability, so that the commands executed by the agent, file changes, and tool calls can be captured. Vercel Sandbox implements these capabilities based on temporary cloud compute instances and supports programmatic creation, destruction, and inspection of runtime environments, making it suitable for hosting evaluation workloads such as "agent + task environment" that need to be repeatedly started and reclaimed.

How to Run an Evaluation

To try the AgentEnv evaluation workflow on Vercel Sandbox, you can follow the steps below.

Preparation

First, complete the AgentEnv setup guide, then run the evaluation command from the cloned repository. The entire process uses your existing Vercel CLI login information, so there is no need to log in again.

Run the Command

Run the following in the repository directory:

./vercel-eval --scope YOUR_TEAM --project YOUR_PROJECT

Replace YOUR_TEAM with your Vercel team slug and YOUR_PROJECT with the name of an existing project. After the command runs, you can view the task score and the sequence of actions recorded by the agent.

The value of this example is that it runs the complete evaluation pipeline end to end: from starting the agent and executing the task to obtaining the score and action records, all in one reproducible command.

The --scope and --project parameters in the command correspond to the organization and project levels of the Vercel platform. --scope specifies the team slug and determines the resource quota and permission scope used for the evaluation; --project specifies an existing Vercel project, and AgentEnv will use that project to create and manage the underlying Sandbox instances. This means the evaluation run is not a completely independent service but is mounted under an existing Vercel project, thereby reusing the project's environment variables, access control, and billing system. Before running, make sure the currently logged-in CLI account has the appropriate permissions for that team and project.

Impact on Agent Development Workflows

The combination of AgentEnv and Vercel Sandbox reflects that agent evaluation is moving from "benchmark scores" to "running workflows." Developers no longer care only about a model's score on a particular benchmark; they care more about whether the agent's behavior in a near-production environment is reliable, explainable, and reproducible.

This trend places higher demands on the toolchain: the evaluation environment needs to be sufficiently isolated, action records need to be sufficiently fine-grained, and the scoring mechanism needs to be sufficiently transparent. Vercel Sandbox's isolated runtime model satisfies the first two points, while AgentEnv's task scoring mechanism handles the third.

For teams building agent applications, what is worth noting is this idea of "evaluation as infrastructure"—treating the evaluation environment as a component as important as development and deployment, rather than something added afterward.

Quick Start Guide

If you want to try it immediately, you can follow this sequence:

1. Make sure the AgentEnv version is 0.9.1334 or later.
2. Read and complete the AgentEnv setup guide.
3. Clone the repository and log in using your existing Vercel CLI.
4. Run ./vercel-eval --scope YOUR_TEAM --project YOUR_PROJECT.
5. Check the task score and the actions recorded by the agent.

For more details, refer to Vercel's Sandbox ecosystem documentation.

Conclusion

AgentEnv's arrival on Vercel Sandbox provides an isolated, reusable, and smoothly integrated solution for agent evaluation that fits into existing workflows. Its design of placing the agent and the task environment in separate Sandboxes balances behavioral auditability and scoring stability. For teams looking to systematically evaluate agent performance, this is a new option worth trying.

One-line Summary

Scale AI's AgentEnv evaluation framework now supports Vercel Sandbox, enabling isolated agent runs with scores and action records.