LLM Gateway

Compared with traditional API gateways (such as Kong or APIGateway), LLM gateways also need to handle streaming responses, token metering, model routing, and other specific requirements.

The billing dimension of LLM gateways has changed from "number of requests" to "number of input and output tokens," requiring precise metering and quota management.

The response mode of LLM gateways has changed from one-time returns to SSE streaming output, imposing new requirements on proxies' timeout and buffering strategies.

LLM gateways introduce prompt-level policy checkpoints, including content filtering, sensitive information scanning, and injection attack protection.

LLM gateways are becoming a key component of enterprise AI infrastructure alongside vector databases and orchestration frameworks.

If a vendor offers a discount of more than 23% on both cache reads and cache writes, its "50% off" badge can match a uniform 35%-off plan.

A fully informative LLM gateway pricing page should display side by side the official price and the gateway price for the four token categories—input, output, cache read, and cache write—and note whether a uniform multiplier is applied to all four.

Method for verifying a supplier's real discount: collect four types of unit prices, write down your own usage composition, multiply and add the four items and compare the total, rather than looking at the discount badge.

In the session assumed by the article, a gateway labeled "50% off" but with cache reads at only a 20% discount and cache writes at full price actually gives a 28% discount, costing $12.13.