DCGM
DCGM (Data Center GPU Manager) is a GPU cluster management software framework developed by NVIDIA for data center environments. It provides system administrators and operations personnel with capabilities including GPU health monitoring, performance metrics collection, fault diagnosis, policy management, and job scheduling. DCGM supports API-based integration with mainstream cluster management and monitoring platforms such as Kubernetes and Prometheus, and is suited for unified operations management of large-scale GPU server deployments.
Core Facts
Timeline (last 90 days)
GPU telemetry is typically exposed by NVML or DCGM, covering power consumption, temperature, SM clock, memory bandwidth utilization, ECC error counts, NVLink link status and more.
The prerequisite for an autonomous recovery system to make correct decisions is aligning DCGM events, Kubernetes events and training framework logs on a timeline to form a unified fault context.
Hardware-layer observability metrics for GPU clusters are collected by NVIDIA's DCGM (Data Center GPU Manager), covering memory errors, ECC events, temperature, power and XID error codes.
It is recommended that teams export at least 90 days of GPU utilization time series with Prometheus or DCGM and use the P50 and P90 quantiles to verify whether the 65% threshold is stably exceeded.
NVIDIA's NVLink, GPU drivers, and data center management software stack (such as DCGM) provide infrastructure support for closed-loop power control.
DCGM version 3.x introduced performance counter multiplexing capability to partially mitigate the counter mutual exclusion limitation.
DCGM consists of the background daemon nv-hostengine and a client library; the daemon communicates with GPU hardware via NVML and CUPTI.
DCGM's Profiling-level metrics (identified by the DCGM_FI_PROF_ prefix) require the use of GPU hardware performance counters and may be mutually exclusive with certain third-party profiling tools.
The GKE managed solution has correct attribution but misleading and non-upgradeable utilization metrics, while self-built DCGM has incorrect attribution by default but honest and configurable utilization metrics.
All Facts (12)
DCGM (Data Center GPU Manager) is a GPU management and monitoring tool provided by NVIDIA, widely used in data center environments.
70%UnverifiedDCGM Exporter能够暴露GPU温度、功耗、SM利用率、显存使用率、ECC错误计数、PCIe带宽等数十项硬件指标
90%UnverifiedFleet Intelligence的实时遥测能力建立在NVML和DCGM等底层监控技术之上
85%UnverifiedGPU telemetry is typically exposed by NVML or DCGM, covering power consumption, temperature, SM clock, memory bandwidth utilization, ECC error counts, NVLink link status and more.
50%UnverifiedThe prerequisite for an autonomous recovery system to make correct decisions is aligning DCGM events, Kubernetes events and training framework logs on a timeline to form a unified fault context.
50%UnverifiedHardware-layer observability metrics for GPU clusters are collected by NVIDIA's DCGM (Data Center GPU Manager), covering memory errors, ECC events, temperature, power and XID error codes.
50%UnverifiedIt is recommended that teams export at least 90 days of GPU utilization time series with Prometheus or DCGM and use the P50 and P90 quantiles to verify whether the 65% threshold is stably exceeded.
50%UnverifiedNVIDIA's NVLink, GPU drivers, and data center management software stack (such as DCGM) provide infrastructure support for closed-loop power control.
50%UnverifiedDCGM consists of the background daemon nv-hostengine and a client library; the daemon communicates with GPU hardware via NVML and CUPTI.
50%UnverifiedDCGM version 3.x introduced performance counter multiplexing capability to partially mitigate the counter mutual exclusion limitation.
50%UnverifiedDCGM's Profiling-level metrics (identified by the DCGM_FI_PROF_ prefix) require the use of GPU hardware performance counters and may be mutually exclusive with certain third-party profiling tools.
50%UnverifiedThe GKE managed solution has correct attribution but misleading and non-upgradeable utilization metrics, while self-built DCGM has incorrect attribution by default but honest and configurable utilization metrics.
50%