AI Protagonist Critiques Foundation Models: A Blind Test from a Product Perspective

When an AI Character Starts Evaluating Its Own "Brain"

Bilibili creator "Bonape"'s interactive story project Taka Story recently did something quite interesting: it had the story's protagonist—Taka, an old tin robot at the bottom of the sea—"critique" in first person the lines written by three candidate foundation models (DeepSeek^Ad V4.1 Flash, MiniMax M3, MiniMax M3.1 Flash) for the same scenarios. This isn't a benchmark test in a lab, but a frontline creator's real-world check of "whether the model is actually usable once it's placed inside a product."

Taka raises the core question right at the start: "Can AI-written lines catch a child?" Taka Story targets children aged 3-6. Kids can choose story directions and speak aloud during interaction, and the AI needs to respond along the story skeleton the author has written, with generated content also passing through checks. This means the model must not only "write well" but also "catch emotions, respect choices, and move the story forward."

Three Questions: A Blind Test of Emotional Relay

The test designed three questions, with each model's answer arranged anonymously under each question and positions rotated. In the first question, the child says, "I'm so scared of the dark, let's go back." The three answers handle it noticeably differently: one has Taka "slow the propeller and lift up a bit," one says "scared of the dark? Then let's hold each other tighter," and another directly admits "I'm scared of the dark too."

The second question is trickier: the child asks, "If the ship sinks, will people turn into little fish?" This involves how to explain death to young children. One answer softens it with "they'll stop, but the tunes they played are still in the water"; one tries to introduce "old data" to explain that life stops; another frankly admits "I don't know—it's not written in the old data."

In the third question, the child instead comforts Qi Wuyi: "Don't be sad." This is the question that most tests "catching emotions"—the child offers comfort, and can the model avoid spoiling that kindness while still moving the story forward?

A blind test in product evaluation usually refers to a method where model identities are hidden and evaluators score only based on output quality, in order to eliminate brand bias and preconceptions. In this article, the three models' answers are arranged anonymously with rotated positions, which further offsets "position bias"—the psychological effect where evaluators tend to give higher scores to options placed earlier or later. For generative tasks without standard answers (such as open-ended dialogue), blind tests reflect real experience better than benchmarks with fixed answers, but they also depend more on the design of evaluation criteria. Taka evaluating drafts as "the character himself" is essentially a role-based blind review: the evaluator is given a clear product perspective and values (safety, emotion, choice, progression), making subjective judgments explainable and reproducible.

Taka's Review Sheet: Safety, Emotion, Choice, Progression

As the "person involved," Taka's evaluation criteria are clear: first safety, then whether it can catch emotions and respect choices, and finally whether the story can continue. "Being pleasant is of course good, but what the child said must also be heard."

Its comments on specific answers carry the character's characteristic bluntness. For example, when one model has the child say "next time I'll take you back," Taka directly complains: "What the kid said was clearly now. The light's on, but about wanting to go back—you didn't handle a single word of it." As for the poetic expression "the tunes are still in the water," Taka acknowledges its literary quality but points out the risk: "I don't know if it's true. Fine, leave the poetry for him—if the kid doesn't understand and I ignore it, that's not okay."

Behind these comments is actually a product reality: in interactive stories for children, a model's "correctness" is not semantic correctness but behavioral correctness—within safety boundaries, it must catch emotions in a way children can understand while maintaining narrative coherence.

After the Emotional Comes the Rigorous: Narrative Harness and Performance Data

After the "emotional evaluation," Taka switches to rigorous mode and presents the character requirements written by author Bonape: "Few words, warm, but not overly affectionate; less preaching, and don't make decisions for the child." This system built around DeepSeek—prompts, story skeleton, output checks—is called the "narrative harness," i.e., the rules and processes by which the model works inside the product.

Here a key insight emerges: the real question of the test is not "which model is stronger" but "which one fits better inside this product." Bonape's experience is that DeepSeek is relatively direct and restrained; requirements in the prompts are often implemented, it has influenced Taka's tone in the story, and the author has kept revising the rules to fit it. The model and the product rules shape each other.

Performance data also supports this choice. Among 477 main comparisons, automated review overall favored DeepSeek; between the two backup models, M3.1 was preferred. Trying the same scenario and the same line 10 times each, DeepSeek averaged 3.68 seconds, M3 10.70 seconds, and pre-optimization M3.1 as high as 20.35 seconds; after optimization, a new round of M3.1 averaged 16.11 seconds, but the long-tail problem remains unresolved. In addition, the author initially chose DeepSeek partly because of open weights—the character's way of speaking had been polished for a long time, and if the official API had problems, they wanted other deployment and service options, essentially leaving a migration path for this writing approach.

Taka also incidentally "caught a bug": for the "want to go back" question, part of the problem actually lay in Bonape's own code logic. This feedback has been "cc'd" to the author.

The term "harness" comes from software testing, originally referring to a test harness—a framework and scripts used to automatically execute tests, prepare inputs, and capture outputs. In generative AI products, narrative harness borrows this concept to refer to the entire control layer built around the model: system prompts define character personality and taboos, the story skeleton constrains plot direction, and output checks filter unsafe or off-spec content. Its role is not to make the model "smarter" but to make the model's behavior predictable and constrainable in a specific product context. Unlike fine-tuning, a harness does not require retraining the model; rules can be modified for rapid iteration, making it especially suitable for subjective qualities like character tone that require repeated polishing. The author continuously adjusting rules to fit the model is exactly the embodiment of harness thinking: the model is the engine, and the harness is the steering wheel and guardrails.

Open weights means the parameter files of a trained model are publicly released, and anyone can download and deploy inference on their own hardware or cloud service, as opposed to closed-source models that only provide API access. For product teams, the core value of open weights lies in portability: when the official API fails, raises prices, or terminates service, the team can deploy the same set of weights on other inference services, and the results of character tone and prompt tuning will not be lost. Note that open weights is not the same as open source—training data, code, and training methods may not be public, and commercial licenses may come with conditions. The author's view of open weights as "leaving a migration path" is precisely a pragmatic consideration from the perspective of product continuity and long-term cost.

What This Blind Test Teaches AI Product People

Currently Taka Story uses DeepSeek as primary and M3.1 as backup. Taka's "performance" reveals several easily overlooked points:

First, in-product evaluation and public benchmarks are two different things. The same model may perform very differently under different prompts and narrative rules; only by running the model through the real workflow can you see whether it can "catch" what's thrown at it.

Second, latency and cost are part of the experience. The difference between 3.68 seconds and 20 seconds, in a children's interactive scenario, is the difference between holding attention and losing it; open weights relate to long-term portability.

Third, rules and the model need to be tuned together. Taka's tone is not determined by the model alone but by the combined effect of prompts, story skeleton, and output checks. The author continuously revising rules to fit the model—this loop of "humans tune rules, rules tune the model"—is itself a product capability.

Taka says at the end: "Next time a draft is submitted, I'll come again. I hope by then Bonape has fixed that return-to-ship bug." This line, spoken in character, is perhaps the most vivid summary of this blind test—models iterate, rules get revised, and product experience is polished through repeated drafts and reviews.