[KongchangAI]
· 2 min read· 1,168 words

Open-Source Tool: Track Tokens, Cost, and Latency for Every LLM Call

Open-Source Tool: Track Tokens, Cost, and Latency for Every LLM Call

An open-source tool records tokens, cost, latency, and Time to First Token per LLM call, filling the observability gap.

A developer posted an open-source tool on Hacker News' Show HN that records four metrics per LLM call: tokens, cost, latency, and Time to First Token. The article notes most teams stop at API success rates after integrating LLMs, lacking per-call cost and latency observability. The tool consolidates four data points per call, refining cost from monthly bills to each call and separating total latency from Time to First Token to distinguish slow model generation from slow infrastructure. TTFT is especially critical for streaming interactive apps like chatbots. The project is early-stage with limited community validation, but the direction is clear: LLM observability is becoming production infrastructure.

Why You Need to Record Every LLM Call

Once an application starts relying on large language models, a call is no longer as simple as "pressing a button"—every request generates token consumption, the real money cost that comes with it, end-to-end latency, and the Time to First Token (TTFT) that users perceive most directly. A developer posted an open-source tool on the Show HN section of Hacker News with a straightforward goal: record the token count, cost, latency, and Time to First Token for every LLM call.

The post received 4 points on Hacker News with no comments yet—a typical "early project exposure." But it points to a very real pain point: after most teams integrate LLMs, their monitoring granularity tends to stop at API success rates, lacking per-call observability into cost and latency. This tool puts four categories of metrics into a single call record, effectively adding an "itemized ledger" to LLM calls.

Time to First Token (TTFT) is a key latency metric in streaming output scenarios, referring to the time elapsed from when the client initiates a request to when it receives the first output token returned by the model. In non-streaming calls, users must wait for the complete response to be generated before seeing any content; with streaming output (like ChatGPT-style character-by-character display), TTFT directly determines the "responsiveness" users perceive. TTFT is mainly composed of three parts: request queuing, network round-trip, and prompt processing (the prefill stage), and together with the subsequent per-token generation speed (the decode stage) it affects total latency. For interactive applications like chatbots and code completion, an excessively long TTFT makes users feel "lag," even if total generation time is short. Therefore, monitoring TTFT separately helps distinguish whether it's high first-token latency (an infrastructure or queuing problem) or slow generation speed (a model or output-length problem).

What the Tool Actually Records

According to the original description, the tool collects four categories of data around a single LLM call:

  • Tokens: how many tokens are consumed for input and output respectively, the basis for cost accounting;
  • Cost: converting tokens into a monetary amount, making "how much this call cost" visible;
  • Latency: the total time from when a call is sent to when it finishes;
  • Time to First Token: the time from when a request is sent to when the first output token is received, directly determining the response speed users perceive.

Singling out Time to First Token for recording shows the author understands streaming output scenarios: short total latency doesn't mean a good experience—users are already waiting before the first piece of text appears. For interactive applications like chatbots and code completion, Time to First Token is often more worth watching than total duration.

Token counting and billing are the foundation of large language model cost accounting. Mainstream LLM APIs (such as OpenAI and Anthropic) price input tokens and output tokens separately, and output tokens are usually more expensive. The cost of a single call = number of input tokens × input unit price + number of output tokens × output unit price. Since prompt length, context window, and output length all significantly affect token count, only by recording token consumption per call can monthly bills be attributed to specific features, users, or request types. In addition, different model providers use different tokenizers, so the same piece of text may have very different token counts; when comparing costs across models, note the difference in tokenization methods.

The Practical Value Behind the Metrics

Cost Goes from "Monthly Bill" to "Per Call"

LLM billing is based on tokens. If you only see the end-of-month bill, it's hard to determine which feature, which user, or which type of request the cost went to. After recording tokens and cost per call, teams can pinpoint whether long contexts or long outputs are driving up expenses, and thereby optimize prompts or adopt truncation strategies.

Latency Viewed in Two Segments

Recording total latency and Time to First Token separately helps distinguish two types of problems: whether the model generation itself is slow, or whether request queuing and network round-trips are slowing down the appearance of the first token. The former requires switching models or shortening output; the latter is an infrastructure problem—the optimization directions are completely different.

Streaming responses refer to an LLM returning results token by token via Server-Sent Events (SSE) or similar mechanisms, rather than waiting for the complete response and returning it all at once. Under streaming responses, total latency is split into Time to First Token (TTFT) and subsequent generation time (the time to generate the remaining tokens). Monitoring tools need to support parsing streaming responses to accurately record TTFT and total latency. If a tool only supports non-streaming calls, it cannot obtain the TTFT metric or reflect the real user experience. Therefore, when evaluating such observability tools, whether they support parsing streaming responses is an important engineering consideration.

Providing a Basis for Capacity and Experience Decisions

With the latency distribution of per-call records, teams can answer questions like "should we route latency-sensitive requests through a separate channel" or "which model is more stable in Time to First Token," rather than choosing based on gut feeling.

Engineering Signals Worth Noting

The fact that this project appeared on Show HN itself shows that LLM observability is shifting from "a feature of platform vendors" to "a small tool developers build themselves." Early projects get very little exposure, lack comments and community validation, and their maturity must be assessed on your own—whether they support streaming responses, how to integrate different model providers, and where data is stored are all unaddressed in the original description.

But the direction is clear: once LLM calls become a core path of the application, recording tokens, cost, latency, and Time to First Token is no longer optional—it's the infrastructure for troubleshooting and budget control. For teams bringing LLMs into production, such lightweight tools deserve a place on the evaluation list—at least to see how others define metrics before building your own monitoring solution.

One-line Summary

An open-source tool records tokens, cost, latency, and Time to First Token for every LLM call, filling the observability gap.

Paragraph Summary

A developer posted an open-source tool on Hacker News' Show HN section that records four categories of metrics per LLM call: token count, cost, latency, and Time to First Token. The article notes that after most teams integrate LLMs, monitoring granularity stops at API success rates, lacking per-call observability into cost and latency. The tool puts four categories of data into a single call record, refining cost from a monthly bill down to each call, and records total latency separately from Time to First Token, helping distinguish slow model generation from slow infrastructure. Time to First Token is especially critical for streaming interactive applications like chatbots. The project is at an early exposure stage, with insufficient community validation and implementation details, but the direction is clear: LLM observability is becoming infrastructure for production environments.

#LLM observability#Time to First Token#LLM cost tracking#token counting#open-source tool#Show HN
Share:

Related articles