AI Phone as Director, Computer as Crew: Building a Custom Video Director Agent with Zero Code

Title

AI Phone as Director, Computer as Crew: Building a Custom Video Director Agent with Zero Code

Meta Description

A Bilibili creator shares a hands-on project: using a STEPX Neo AI phone as the director and a computer as the crew, building a general-purpose AI video director agent with zero code via StepStudio—covering script parsing, storyboard design, and image/video generation—with plans to open-source it.

Article Content

The barrier to video creation is being rapidly lowered by AI, but the real pain point is: how do you get AI to understand your creative vision and automatically break it down into an executable production workflow? A Bilibili creator shared a hands-on project—using an AI phone (STEPX Neo) as the creative entry point and a computer as the compute backend, they built a general-purpose AI video director agent from scratch using the STEPX ecosystem's agent development tool, covering the full pipeline from writing scripts and breaking down storyboards to image generation and video output.

Core Philosophy: Agents Aren't Written—They're "Chatted" Into Existence

Instead of diving straight into code, the creator opened STEPX's official agent development tool, StepStudio, and first laid out the requirements clearly: build a general-purpose AI video director agent capable of handling sci-fi, animation, drama, advertising, and other genres.

The first round defined the basic workflow—starting from an idea, analyzing the story, designing characters and scenes, and planning narrative pacing. The second round layered on professional directing capabilities: storyboard arrangement, camera movement, character action design, and shot continuity. Character consistency was especially emphasized: "You can't have the character look completely different from one shot to the next."

This "conversational development" approach turns the agent-building process into a requirements-refinement process. The creator revealed that the agent has already been configured in StepCloud and can be invoked directly from the phone.

StepStudio and Conversational Agent Development

StepStudio is a visual development tool in the STEPX ecosystem for building agents, and its core idea is to replace traditional coding with natural language conversation. Developers don't need to master programming languages or framework details—they simply describe goals, constraints, and workflows in everyday language, and the tool translates those descriptions into executable agent logic and tool-call chains. This lowers the barrier to agent development and is especially suited to creators without a technical background. It's worth noting that conversational development doesn't mean structured thinking is unnecessary—the clearer the requirements and the more defined the boundaries, the more controllable the resulting agent behavior. That's why the creator adopted a phased strategy of "defining the basic workflow first, then layering on professional capabilities"—essentially using conversation to decompose requirements and iterate on design, which is the key to conversational development producing complex agents.

Phone as Director, Computer as Crew: How Cross-Device Collaboration Works

The agent on the phone receives creative input and instructions, while the actual image and video generation tasks are handed off to the computer. The creator configured image and video generation API tools on the computer, and the agent automatically calls these tools to produce assets.

The test workflow was quite complete: first, a script was sent, and the agent analyzed the story, organized characters, scenes, and key props, and converted them into image generation prompts. Next, the agent was instructed to call the image generation tool on the computer to produce reference assets needed for the film.

With reference images in hand, the process moved to video prompt design. The creator emphasized that the focus at this stage isn't what the images look like, but how characters move, how the camera follows, and how the story stays coherent. The agent strictly followed the specifications, generated prompts with motion descriptions, and ultimately produced previewable video results.

Cross-Device Collaboration and Agent Tool-Calling Mechanism

Cross-device collaboration means tasks within the same workflow are split across different devices: the phone handles lightweight interaction (receiving instructions, showing progress, triggering workflows), while the computer handles compute-heavy tasks (image generation, video rendering). Its technical foundation is the agent's "tool-calling" capability—after parsing requirements, the agent automatically determines which tool is needed and sends requests to the computer via pre-configured API interfaces. This means API tools must be registered and authenticated on the computer in advance for the agent to call them reliably. The advantage of cross-device collaboration is decoupling the interaction entry point from the compute backend, allowing creators to capture inspiration in mobile scenarios without being limited by the phone's own compute power. However, this also imposes requirements on network stability, task queue management, and result-return mechanisms—engineering details that need attention when implementing such solutions.

Hands-On Breakdown: Four Key Stages from Script to Video

The entire process can be broken down into four stages:

• Script Parsing and Asset Extraction: The agent reads the script, automatically identifies characters, scenes, and props, and converts them into structured image prompts.
• Image Asset Generation: Calls the computer-side image tool to batch-produce reference images, ensuring visual consistency of characters and scenes.
• Video Prompt Design: Focuses on motion and camera language, generating descriptions that include character actions and camera trajectories to ensure shot-to-shot continuity.
• Video Generation and Preview: Calls the video generation API to complete final rendering and preview the results.

The creator admitted that this test was "only aimed at getting the pipeline working," but the results were already "very interesting." The plan is to continue refining the agent and eventually open-source it.

Character Consistency and Shot Continuity

Character consistency means the same character maintains stable visual features—appearance, clothing, demeanor—across different shots, and it's one of the toughest challenges in AI video generation. Since diffusion models produce random output each time, without constraints the same character can look noticeably different from one shot to the next. Common solutions include: using reference images or character design sheets as fixed inputs, locking key feature descriptions in prompts, and maintaining a unified character asset library for reuse across stages. Shot continuity focuses on whether the spatial relationships and motion logic between shots are natural, involving camera trajectories, character action transitions, and narrative pacing. Together they form the core of "director thinking" and are the rules this project's agent was required to follow closely—by explicitly writing these professional requirements into the agent's workflow, the generation tools can maintain overall consistency even during batch production.

Ecosystem Value: STEPX's Cross-Device Synergy Is the Biggest Highlight

Compared to stacking standalone tools, the real value of this project lies in ecosystem integration. The phone handles lightweight interaction and creative triggering, the computer handles compute-heavy tasks, and the agent acts as the orchestration hub, connecting both ends into a complete workflow.

At the end of the video, the creator commented: "STEPX's AI ecosystem is very well-developed, and the connectivity between products is excellent." For content creators, this "phone + computer + agent" combination means you can toss inspiration to AI anytime, anywhere, and see usable assets and storyboard plans when you return to your computer.

Takeaways for AI Video Creators

This case demonstrates another possibility for AI video production: not pursuing one-click generation, but building an agent that understands creative logic and acts as a translation layer between director thinking and generation tools. Challenges traditional in film production—character consistency, shot continuity—are being broken down into rules an agent can execute.

For developers wanting to replicate a similar workflow, the creator's path is worth referencing: first define the workflow in natural language, then gradually layer on professional capabilities, and finally connect the generation APIs. The open-source plan also means this solution could be reused and iterated on by more people.

One-line Summary

A Bilibili creator used conversational development to build a cross-device AI video director agent, running the full pipeline from script to video.

Paragraph Summary

The article introduces a hands-on project by a Bilibili creator: using the STEPX Neo AI phone as the creative entry point and a computer as the compute backend, they built a general-purpose AI video director agent from scratch through conversational development with STEPX's StepStudio tool. The agent covers the full pipeline—script parsing, asset extraction, image prompt generation, video prompt design, and video generation—with a focus on solving the two major challenges of character consistency and shot continuity. The phone receives creative instructions, the computer executes image and video generation tasks, and the agent serves as the orchestration hub connecting both ends. The creator admitted the test was aimed at getting the pipeline working, but the results were already quite interesting, with plans to refine and open-source it. The article argues this case demonstrates the ecosystem value of "phone + computer + agent" cross-device collaboration in AI video production.