GPU Benchmarks Are Lying: How to Measure LLM Inference Under Thermal Throttling

Most teams evaluate LLM inference performance by running a 30-second benchmark, grabbing a tokens/s number, and calling it done. But when that number gets carried into production, actual throughput often comes in noticeably lower than expected. The problem isn't the model or the inference framework—it's that the measurement itself ignores how GPUs thermally throttle under sustained load.

Why a 30-second benchmark is almost meaningless

Modern GPUs don't run at a fixed frequency. Take the NVIDIA T4: TDP is only 70W, boost clock tops out at 1590 MHz, and it uses passive cooling—the card has no fan of its own and relies on the server's airflow. When you first start running, the card is cool and the driver pushes clocks to the max; a few minutes later, temperature and power hit their limits, and the driver starts downclocking. That's throttling.

There are three common causes of downclocking:

• SW Power Cap: hitting the power limit—very frequent on the T4
• HW Slowdown / HW Thermal Slowdown: hardware-level overheat protection
• SW Thermal Slowdown: the driver proactively downclocking to control temperature

For LLM inference, the decode phase is mainly limited by memory bandwidth, so a drop in SM frequency doesn't cause a proportional slowdown; the prefill phase is compute-limited and is hit much harder by downclocking. If you only measure a single blanket tokens/s number from the first 30 seconds, you've measured nothing but the moment when the GPU is in its best state—and production is never in that state.

Boost clock is the highest frequency a GPU can briefly reach within the headroom allowed by cooling and power, not a sustained rated frequency. NVIDIA GPUs follow a dynamic frequency-scaling policy by default: when the chip is cool and power is under the cap, the driver pushes SM frequency to the boost ceiling; once temperature or power approaches the threshold, frequency gradually falls back toward base clock or even lower. So the vendor's advertised boost clock is essentially an "instantaneous peak," not a stable operating point under sustained load. For long-running, fully loaded workloads like LLM inference, what really determines tokens/s is the steady-state frequency, not the boost frequency of the first few dozen seconds. Understand this, and you'll see why any short test aimed at "just running it to see how fast it is" only measures the GPU's most ideal instantaneous state.

Step 1: Watch the GPU during the benchmark

Before measuring tokens/s, you need to know what the GPU is doing. nvidia-smi has a built-in continuous logging mode—open another terminal and capture in parallel:

**TDP (Thermal Design Power)** is the thermal design power the GPU vendor specifies, representing the upper bound of heat the card is expected to dissipate under sustained load. The T4's TDP is only 70W, and it uses a single-slot passive cooling design—no fan on the card, relying entirely on the server chassis's front-to-back airflow to carry heat away. The actual cooling capacity of a passively cooled card depends heavily on chassis airflow, intake temperature, and heat from adjacent cards, so the same T4 installed in different chassis and different rack positions may sustain noticeably different steady-state frequencies. This also explains why same-model cloud instances differ by a few percent in throughput: different thermal environments mean throttling kicks in at different times and to different depths. When evaluating this class of card, TDP and cooling method are worth looking at before boost clock.
# Output CSV every second, run in parallel with the benchmark
nvidia-smi \
  --query-gpu=timestamp,temperature.gpu,clocks.sm,clocks.mem,power.draw,power.limit,clocks_throttle_reasons.active \
  --format=csv,nounits -l 1 > gpu_log.csv &

# View details of current throttle reasons
nvidia-smi -q -d PERFORMANCE

# View the maximum clocks supported by the card
nvidia-smi -q -d CLOCK | grep -A3 "Max Clocks"

Driver versions 535 and above renamed this field to clocks_event_reasons.active; the old name still works on most drivers. If you hit a Field ... is not a valid field error, just switch to the new name. The field is a bitmask: 0x0000000000000004 corresponds to SW Power Cap, 0x0000000000000020 to SW Thermal Slowdown.

When reading gpu_log.csv, you'll typically see the same pattern: clocks.sm holds steady at 1590 for the first minute, then gradually drops to 1200–1350 between minutes 3 and 5 and stabilizes. That stable value is the number you should use for capacity planning.

Step 2: Measure long enough, and record data over time

llama.cpp's llama-bench is good for quick comparisons, but by default it only repeats 5 times (-r 5), and each run is short. You can strengthen it like this:

**NVML (NVIDIA Management Library)** is the low-level monitoring and management interface NVIDIA provides; the `nvidia-smi` command itself, along with `dcgm-exporter` and various GPU monitoring agents, are all built on top of it. Through the `nvidia-ml-py` Python binding, a program can read temperature, SM/memory clocks, power draw, and the throttle reasons bitmask at sub-second granularity inside the benchmark loop, aligning each request's tokens/s with the GPU state at that moment. Compared with looking at an instantaneous `nvidia-smi` snapshot after the fact, the value of NVML sampling is temporal alignment: you can directly see at which minute throttling begins and how much decode and prefill each drop afterward. This is also the key tool that turns "measuring a number" into "measuring a curve."
# Warm up + multiple iterations, output JSON for easy analysis
./llama-bench -m models/gemma-3-4b-it-Q4_K_M.gguf \
  -ngl 99 -p 512 -n 256 -r 50 -o json > bench.json

A more effective approach is to continuously send requests to llama-server while collecting GPU metrics via NVML. The following script depends on nvidia-ml-py (pip install nvidia-ml-py requests):

import time, json, statistics, requests
import pynvml as nv

URL = "http://127.0.0.1:8080/completion"
DURATION = 15 * 60  # run for 15 minutes
PROMPT = "Explain the mechanism of B-tree indexes in PostgreSQL. " * 20

nv.nvmlInit()
h = nv.nvmlDeviceGetHandleByIndex(0)

rows, start = [], time.time()
while time.time() - start < DURATION:
    r = requests.post(URL, json={"prompt": PROMPT, "n_predict": 256,
                                  "cache_prompt": False}).json()
    t = r["timings"]
    rows.append({
        "t": round(time.time() - start),
        "prefill_tps": t["prompt_per_second"],
        "decode_tps": t["predicted_per_second"],
        "temp": nv.nvmlDeviceGetTemperature(h, nv.NVML_TEMPERATURE_GPU),
        "sm_mhz": nv.nvmlDeviceGetClockInfo(h, nv.NVML_CLOCK_SM),
        "power_w": nv.nvmlDeviceGetPowerUsage(h) / 1000,
        "throttle": hex(nv.nvmlDeviceGetCurrentClocksThrottleReasons(h)),
    })
    print(rows[-1])

json.dump(rows, open("run.json", "w"))

def summary(sub, label):
    d = [x["decode_tps"] for x in sub]
    p = [x["prefill_tps"] for x in sub]
    print(f"{label}: decode median={statistics.median(d):.1f} "
          f"p10={statistics.quantiles(d, n=10)[0]:.1f} | "
          f"prefill median={statistics.median(p):.0f}")

summary([x for x in rows if x["t"] < 60], "Cold (0-60s)")
summary([x for x in rows if x["t"] > DURATION - 300], "Steady (last 5 minutes)")

There are three key points in the script:

1. cache_prompt: False: without turning this off, prefill reuses the previous request's KV cache, inflating the measured prefill number
2. Split prefill and decode: the two phases respond very differently to thermal throttling, and mixing them hides the problem
3. Compare cold and steady: this is the single most important number in the whole exercise. If the gap exceeds 5%, capacity planning must use the steady value

Step 3: Confirm your stop-loss measures once the card throttles

Once you know the card is throttling, you can take the following measures:

• Lock clocks for stable performance: sudo nvidia-smi -lgc 1200,1200 locks the GPU frequency. It sounds like handicapping yourself, but throughput becomes stable, p99 latency becomes more predictable, and in many cases average performance actually doesn't drop, because frequency no longer fluctuates. To unlock, use sudo nvidia-smi -rgc. You may not have permission for this on cloud VMs.
• Enable persistence mode: sudo nvidia-smi -pm 1 keeps the driver from unloading between runs, avoiding a slow first request.
• Choose quantization that matches your workload: decode is bandwidth-limited, and Q4_K_M is generally less sensitive to throttling than FP16 because each token requires reading less data.
• Reduce prefill batch size when the card is hot: if your workload is prompt-heavy (long-context RAG), reduce -ub (ubatch size) to lower power spikes.
• Production monitoring: export NVML metrics to Prometheus via dcgm-exporter, and alert when DCGM_FI_DEV_CLOCK_THROTTLE_REASONS stays nonzero for more than 10 consecutive minutes.

One more thing: two T4 instances of the same model in the cloud can differ by a few percent depending on rack position and server airflow. Don't measure on a single instance and apply that number to the whole cluster.

p99 latency refers to the value at the 99th percentile after sorting all request latencies from low to high—that is, the latency level of the slowest 1% of requests. For online inference services, p99 often reflects user experience and SLA risk better than the average: the average gets dragged down by a large number of fast requests and masks the few slow ones. When GPU frequency fluctuates, average throughput may look fine, but requests that fall into the throttling window slow down significantly and p99 deteriorates accordingly. Locking clocks often improves p99 without a noticeable drop in average performance precisely because it eliminates the latency long tail caused by frequency fluctuations. For capacity planning, in addition to the median, you should also watch tail metrics like p10 throughput and p99 latency.

Key points and conclusions

Running an LLM benchmark for only 30 seconds gives you a marketing number, not an operations number. The original author shared a real lesson: a team measured an 8B model on an hourly-rented GPU, got 42 tokens/s in 30 seconds, and planned capacity accordingly; after going live and running all day, they actually got only 37–38 tokens/s—a capacity shortfall of nearly 10%. The problem was neither the model nor llama.cpp—it was the measurement method.

Things to do right now:

1. Run benchmarks for at least 10–15 minutes, recording SM frequency, temperature, power, and tokens/s in parallel
2. Report prefill and decode separately, using median and p10 instead of averages
3. Use steady-state numbers for capacity planning; cold-card numbers are unusable. When the gap exceeds 5%, spell it out in the sizing document
4. Consider clock locking when you need predictable latency, rather than chasing peak throughput
5. Add throttle reasons to monitoring so you always know whether production is running below spec

The GPU doesn't lie—it just answers exactly the question you asked. If you want to know performance over a full day, you have to measure under full-day conditions.

One-line Summary

A 30-second benchmark measures the GPU's cold-state peak; production needs steady-state data for capacity planning.

Paragraph Summary

The article points out that teams often use tokens/s from a 30-second short test for capacity planning, but that number reflects the GPU's cold-state peak performance at boost frequency. Modern GPUs (like the passively cooled T4) thermally throttle and downclock under sustained load when power and temperature hit their limits, and measured steady-state throughput is often nearly 10% lower than cold-state. The author proposes a three-step method: log SM frequency, temperature, power, and throttle reasons in parallel using nvidia-smi or NVML; run benchmarks for 10–15 minutes, split prefill and decode, and compare cold vs. steady data; then use clock locking, persistence mode, quantization choices, and dcgm-exporter monitoring to trade for stable performance. The core conclusion is that capacity planning must use steady-state median and p10, not averages or cold-state numbers.