llm-echo 0.5a0: A Testing Powerhouse for Simulating Chain-of-Thought LLM Output

Overview

Simon Willison has released version 0.5a0 of the llm-echo plugin, introducing a new -o thinking 1 option that simulates chain-of-thought (thinking) output from large language models. While seemingly simple, this update reflects the ongoing maturation of testing infrastructure in the LLM toolchain.

What Is llm-echo and Why Do Developers Need It?

llm-echo is a testing plugin developed by Simon Willison for the LLM command-line tool. It provides a mock model called "echo" that doesn't call any LLM API — instead, it echoes back the input prompt in JSON format.

Simon Willison's LLM command-line tool is an open-source project that allows users to interact with various large language models through the terminal. It uses a plugin architecture supporting multiple backends including OpenAI, Anthropic, and local models. The tool's design philosophy is deeply influenced by Unix pipe philosophy — each tool does one thing well, and they combine through standard input/output into complex workflows. This architecture allows LLM to be embedded in shell scripts, CI/CD pipelines, and automation tasks, rather than being just an interactive chat tool.

Its core use case is automated testing. When developers test workflows or applications built on LLM tools, they don't need to consume real API calls every time (which is both time-consuming and expensive) — the echo model can verify whether data flows correctly.

Mock objects are a classic pattern in software testing, originating from Test-Driven Development (TDD) practices. In scenarios involving external service interactions, mock objects replace real dependencies, enabling tests to run quickly and deterministically in isolated environments. For LLM applications, real API calls have three problems: high cost (per-token billing), non-deterministic responses (the same prompt may return different results), and slow speed (network latency). As a mock model, llm-echo perfectly solves all three problems, letting developers focus on validating application logic rather than model output quality.

What's New in 0.5a0: Simulated Chain-of-Thought Output

Chain-of-Thought Technical Background

Chain-of-Thought (CoT) is a technique that has large language models show intermediate reasoning steps before giving a final answer. A 2022 Google paper first systematically demonstrated that CoT prompting significantly improves model performance on complex reasoning tasks. In 2024-2025, this concept was further productized: OpenAI's o1/o3 series models have built-in reasoning processes, Anthropic's Claude introduced extended thinking functionality, and DeepSeek^Ad-R1 provides complete reasoning chain output in an open-source manner. These models' API response structures differ from traditional models, typically containing a thinking/reasoning field and a final output field, requiring the application layer to handle these two parts separately.

Usage

The core update in this version is support for the -o thinking 1 option, used as follows:

uvx --with llm==0.32a1 --with llm-echo==0.5a0 llm -m echo hi -o thinking 1

The uvx command used in this example is part of Astral's uv package manager, used to run Python command-line tools in temporary virtual environments. The --with parameter allows specifying additional dependency packages with exact versions. The advantage of this approach is that it requires no global installation or manual virtual environment management, making it particularly suitable for testing compatibility of specific version combinations. Written in Rust, uv is 10-100x faster than traditional pip and has quickly become one of the mainstream tools in the Python ecosystem.

When this command is executed, the echo model will:

First output a simulated reasoning block to standard error (stderr)
Then return the JSON result echoing the prompt

This feature is specifically designed to work with LLM 0.32a0 and higher, which introduced native support for model chain-of-thought/reasoning output.

The Engineering Design of stdout vs. stderr Separation

In Unix system design, standard output (stdout, file descriptor 1) is used for a program's normal output, while standard error (stderr, file descriptor 2) is used for diagnostic information, logs, and metadata. This separation allows users to pipe stdout to downstream programs while redirecting stderr to log files or displaying it directly in the terminal.

The LLM tool outputs chain-of-thought to stderr and the final response to stdout, meaning users can write commands like llm -m o3 'analyze this code' | jq . — the reasoning process displays in real-time on screen for human reading, while the structured final result flows cleanly into downstream tools for processing. This design makes chain-of-thought an "observable but non-interfering" output, perfectly aligning with the composability philosophy of command-line tools.

Why Simulated Chain-of-Thought Matters for Developers

With the rise of reasoning models like OpenAI o1/o3, Anthropic Claude extended thinking, and DeepSeek-R1, thinking output has become an important component of modern LLM interactions. When building applications, developers need to verify:

Whether the application correctly handles chain-of-thought output
Whether stderr and stdout separation works properly
Whether downstream components in the pipeline can correctly parse responses with reasoning blocks

Without mock tools, these testing scenarios either rely on real API calls or require developers to write complex simulation logic themselves. llm-echo simplifies this problem to a single command-line option.

Design Philosophy: Test-First AI Toolchain

llm-echo embodies Simon Willison's consistent tool design principles:

Composability: Integrates as a plugin into the existing LLM tool ecosystem
Test-first: Provides first-class support for automated testing
Progressive enhancement: As the main tool evolves (e.g., adding thinking support), testing tools keep pace

This emphasis on testing infrastructure is uncommon in the open-source AI tools space, but is crucial for building reliable AI application workflows. In traditional software engineering, test coverage and the maturity of testing tools are often key indicators of project maturity. AI application development is undergoing a transition from "rapid prototyping" to "production-grade engineering," and tools like llm-echo are the infrastructure enabling this transition.

Summary

While llm-echo 0.5a0 is just a minor version update, it reflects the maturing of the AI toolchain — focusing not only on features themselves but also on developer experience and engineering practices. For developers building automated workflows with LLM CLI tools, this is a practical update worth adding to your toolbox.

Key Takeaways

llm-echo 0.5a0 adds the -o thinking 1 option to simulate reasoning output blocks from large models
The echo model doesn't call real APIs, designed specifically for automated testing to save development costs
This update works with LLM 0.32a0 and above, supporting testing scenarios for chain-of-thought output
Simulated reasoning blocks output to stderr while normal responses output to stdout, achieving output separation
Reflects the ongoing maturation of AI toolchain testing infrastructure