Prometheus
Prometheus is an open-source systems monitoring and alerting toolkit originally developed at SoundCloud and later donated to the Cloud Native Computing Foundation (CNCF). It uses a time-series data model and collects metrics via an HTTP pull mechanism. Prometheus includes a powerful built-in query language, PromQL, supporting multi-dimensional data analysis. It offers flexible alerting rule configuration and integrates with visualization tools such as Grafana, and is widely adopted in cloud-native and microservices architecture monitoring.
Core Facts
A typical MLOps workflow covers stages including data version management, experiment tracking, model packaging, serving interfaces, CI/CD automation, online monitoring, and drift detection.
Docker、Kubernetes、Terraform、etcd、Prometheus等云原生核心项目几乎都用Go编写
Timeline (last 90 days)
In the local observability stack, the OpenTelemetry Collector exposes metrics in Prometheus format on port 8889, Prometheus scrapes at 15-second intervals, and Grafana reads the Prometheus data source.
Claude Code has the OTel SDK built in, so the data format it outputs is not proprietary, and any OTLP-compatible backend (such as Prometheus, Datadog, Honeycomb, etc.) can receive it directly.
Never leave thinking.budget_tokens or reasoning_effort at unbounded defaults in the production SDK; set a hard cap (1,024 to 2,048 tokens) and record usage.completion_tokens_details.reasoning_tokens as a separate Prometheus metric per tenant.
In agent scenarios, observability requires complete recording of each step's inputs and outputs, tool call parameters, latency, and token consumption; common practices include structured logging, distributed tracing, and task-dimension aggregated metric dashboards, and often require coordination with components such as Prometheus, Grafana, and Loki.
It is recommended that teams export at least 90 days of GPU utilization time series with Prometheus or DCGM and use the P50 and P90 quantiles to verify whether the 65% threshold is stably exceeded.
In time series systems such as Prometheus, each additional unique label value combination generates a new time series, so high-cardinality labels such as waybill ID will cause the number of series to expand linearly with order volume, driving up storage costs and query latency.
Prometheus scrapes metrics at a fixed interval and evaluates alert rules with PromQL; when a rule matches, it enters the pending state, and only after the for clause duration is continuously satisfied does it transition to firing; Alertmanager takes over routing, grouping, inhibition, silencing and escalation.
When explicit alert rules, grouping, inhibition, silencing and notification routing are needed while retaining the open-source metrics model, Prometheus + Alertmanager is the winning solution, at the cost of operational ownership (unless a managed Prometheus service is used).
Metric labels should only use dimensions with a limited set of values, such as carrier, workflow version, and result; waybill IDs and free-form error messages should be written to logs and correlated with metrics via trace_id, rather than used as labels.
Muse Wonder was trained in a gigawatt-scale data center named Prometheus.
40 more timeline events
All Facts (20)
A typical MLOps workflow covers stages including data version management, experiment tracking, model packaging, serving interfaces, CI/CD automation, online monitoring, and drift detection.
65%UnverifiedLiteLLM的日志系统支持与Langfuse、Helicone、Datadog和Prometheus等可观测性平台集成
90%UnverifiedPrometheus最初由SoundCloud开发，2016年成为继Kubernetes之后第二个加入CNCF的毕业项目
90%UnverifiedNCCL Inspector与Prometheus监控系统深度集成，通过Prometheus Exporter暴露监控指标
90%Partially VerifiedDocker、Kubernetes、Terraform、etcd、Prometheus等云原生核心项目几乎都用Go编写
65%UnverifiedIn the local observability stack, the OpenTelemetry Collector exposes metrics in Prometheus format on port 8889, Prometheus scrapes at 15-second intervals, and Grafana reads the Prometheus data source.
50%UnverifiedClaude Code has the OTel SDK built in, so the data format it outputs is not proprietary, and any OTLP-compatible backend (such as Prometheus, Datadog, Honeycomb, etc.) can receive it directly.
50%UnverifiedNever leave thinking.budget_tokens or reasoning_effort at unbounded defaults in the production SDK; set a hard cap (1,024 to 2,048 tokens) and record usage.completion_tokens_details.reasoning_tokens as a separate Prometheus metric per tenant.
50%UnverifiedIn agent scenarios, observability requires complete recording of each step's inputs and outputs, tool call parameters, latency, and token consumption; common practices include structured logging, distributed tracing, and task-dimension aggregated metric dashboards, and often require coordination with components such as Prometheus, Grafana, and Loki.
50%UnverifiedIt is recommended that teams export at least 90 days of GPU utilization time series with Prometheus or DCGM and use the P50 and P90 quantiles to verify whether the 65% threshold is stably exceeded.
50%UnverifiedIn time series systems such as Prometheus, each additional unique label value combination generates a new time series, so high-cardinality labels such as waybill ID will cause the number of series to expand linearly with order volume, driving up storage costs and query latency.
50%UnverifiedMetric labels should only use dimensions with a limited set of values, such as carrier, workflow version, and result; waybill IDs and free-form error messages should be written to logs and correlated with metrics via trace_id, rather than used as labels.
50%UnverifiedWhen explicit alert rules, grouping, inhibition, silencing and notification routing are needed while retaining the open-source metrics model, Prometheus + Alertmanager is the winning solution, at the cost of operational ownership (unless a managed Prometheus service is used).
50%UnverifiedPrometheus scrapes metrics at a fixed interval and evaluates alert rules with PromQL; when a rule matches, it enters the pending state, and only after the for clause duration is continuously satisfied does it transition to firing; Alertmanager takes over routing, grouping, inhibition, silencing and escalation.
50%UnverifiedMuse Wonder was trained in a gigawatt-scale data center named Prometheus.
50%UnverifiedReskPoints can be installed via pip install reskpoints, and supports installing versions with optional dependencies such as datadog, prometheus, and opentelemetry.
50%UnverifiedPrometheus uses a pull model that scrapes at fixed intervals and stores timestamped samples, naturally satisfying the rate recovery requirement.
50%UnverifiedmacOS has no native, always-on HTTP endpoint for GPU metrics, unlike NVIDIA's nvidia-smi, so a custom mac_gpu_exporter is needed to periodically run powermetrics and convert it into a Prometheus-scrapable format.
50%UnverifiedIn the same year, the authors of Prometheus found that with a reference answer, the evaluator no longer needs to solve the problem itself.
50%UnverifiedThe Prometheus tool group of Grafana MCP allows executing instant and range PromQL queries, listing metric and label names, and computing p50/p90/p95/p99 from histograms.
50%