SLO

Inference latency is typically decomposed into two core metrics: time to first token (TTFT) and subsequent token generation throughput.

SRE (Site Reliability Engineering) is an engineering practice proposed and popularized by Google, whose core idea is to solve operational problems using software engineering methods and to quantify system reliability as SLOs for continuous measurement.

SLI is a specific metric for measuring service health, SLO is an internally set target value, and SLA is an external commercial commitment that entails financial penalties when violated.

"Elevated errors" means the error rate is significantly above the normal baseline but has not yet reached a full outage, typically corresponding to an SLO violation and manifesting as intermittent failures.

The Goodput metric measures the volume of valid requests or valid tokens that meet SLOs (such as TTFT and ITL) per unit time, and can reflect the economic efficiency actually paid for in production environments.

The industry typically uses SLO (Service Level Objective) and Error Budget to quantify impacts such as performance degradation, for example using '99.9% of requests return within 500ms' as a target.

Five steps for setting reliability goals for AI features: choose only one quality SLI per feature, version the eval set, set the SLO from a week of baseline data, tag AI-written PRs, and define explicit failure blocking rules.

For AI systems, attaching the SLO only to availability means model quality degradation will never exhaust the error budget, and thus will never trigger the brake.

An SLI is the actual measured value, an SLO is the internal target the team sets for the measured value, and the error budget is the room for failure allowed by 100% minus the SLO; the three form a closed loop.

SLOs (Service Level Objectives) and error budgets: mature platforms usually set quantitative targets for availability and use them to drive capacity planning and failure drills.

P99 latency is generally a key metric for assessing the true experience of production systems; Google's SRE framework calls it tail tolerance and recommends explicitly specifying P95 or P99 latency upper bounds in SLOs.

Existing P/D disaggregation systems typically employ fixed prefill/decode worker ratios combined with request routing for load distribution, which leads to latency SLO violations when workload demand ratios fluctuate.

SLO cannot be obtained once through administrative procedures, but must be continuously maintained through ongoing transparent information disclosure, community engagement, and responsiveness to local concerns.

Amazon's move to abandon NDAs is essentially a proactive management of the risk of SLO deficiency.

The core task of benchmarking is to find the maximum sustainable throughput under SLO constraints, rather than the system's absolute peak throughput.

A Service Level Objective (SLO) is a quantified commitment to service quality, which in LLM inference scenarios typically encompasses both latency and availability dimensions.

Goodput refers to the proportion or throughput of requests successfully completed under SLO constraints, treating requests that exceed the latency budget as failures and excluding them from effective output.

Agent Reliability introduces SLO and error-budget semantics, and provides SLO assertions that can be used in tests/CI.

NVIDIA recommends a top-down capacity planning approach: define SLOs, run benchmark tests, and calculate the total number of GPUs needed.

Error budgets are a management tool between SLOs and SLAs that transforms reliability from an absolute target into a flexibly allocatable resource.