nvidia-smi for cluster operators: from CLI to automated fleet telemetry

12 min read

Every operator running an on-prem GPU fleet starts the same way: open a terminal, run nvidia-smi, and watch utilization, power draw, and temperature scroll by. It is the most honest view of what an NVIDIA accelerator is doing, and it ships in the driver — no agent, no SaaS, no outbound connection. This guide covers how to use nvidia-smi well at the node level, where it stops scaling at the fleet level, and what to do about it.

What nvidia-smi actually is

nvidia-smi is a thin CLI on top of NVML (NVIDIA Management Library). NVML is the same library DCGM, Prometheus exporters, and orchestration platforms call into. When you read a number out of nvidia-smi you are reading it out of NVML — there is no separate "monitoring stack" between you and the GPU.

That is what makes it the right starting point for any on-prem deployment: zero dependencies, zero data egress, and a one-to-one map to the metrics your future telemetry pipeline will collect.

The four things you actually want to watch

1. Utilization

utilization.gpu is the percentage of time over the past sampling period that one or more kernels were executing on the GPU. It is not memory bandwidth, not FLOPs, not tensor-core occupancy. A workload pinned at 99% utilization.gpu can still be wildly inefficient.

2. Power draw

power.draw is the most reliable proxy for "is this GPU doing real work". An H100 at idle sits around 70W; a saturated training step pulls 650–700W. If utilization is 100% but power is 200W, the kernel is launch-bound or memory-bound, not compute-bound.

3. Memory

Track both memory.used and utilization.memory. The first is capacity, the second is HBM bandwidth utilization. Out-of-memory kills are the most common production failure on H100s under multi-tenant scheduling; alert on memory.used crossing 90% of capacity per slice.

4. Thermals

temperature.gpu is the die temperature; temperature.memory is HBM. H100 throttles the SM clock at ~87°C and memory at ~95°C. Sustained operation above 80°C is a rack airflow problem, not a workload problem.

A practical query for cluster operators

Skip the default table view. It is designed for one person looking at one box. Use --query-gpu with--format=csv,noheader,nounits for scriptable, line-oriented output:

nvidia-smi \
  --query-gpu=index,name,utilization.gpu,utilization.memory,memory.used,memory.total,power.draw,temperature.gpu,temperature.memory,ecc.errors.uncorrected.aggregate.total \
  --format=csv,noheader,nounits \
  -l 1

-l 1 polls once per second. On an otherwise idle H100 this costs a few hundred microseconds per sample and is safe to leave running. Pipe it to a file per host and you have a basic time series.

Per-process attribution

Utilization without attribution is hard to act on. Use --query-compute-apps to map workloads to slices:

nvidia-smi \
  --query-compute-apps=pid,process_name,gpu_uuid,used_memory \
  --format=csv,noheader,nounits

Cross-reference gpu_uuid with nvidia-smi -L to attach physical/MIG identity, and pid with your scheduler to attach a tenant or job ID.

ECC errors are a leading indicator

ecc.errors.uncorrected.aggregate.totalshould be zero. Anything non-zero is a candidate to drain and RMA before it corrupts a long-running fine-tune. Add it to every poll; the cost is negligible.

Where this stops working

nvidia-smi is fine for one box. The wheels come off somewhere between eight and twenty:

  • Per-host SSH is not a monitoring system. Looping ssh host nvidia-smi across a rack loses samples on any flapping link, and the latency floor is the slowest SSH session.
  • No retention. CSV files on each host get rotated, lost, or grow until disk fills. Postmortems on an OOM that happened three hours ago need a real time-series store.
  • No alerting. Nobody is watching the terminal at 03:00. Threshold breaches on power draw, temperature, ECC, or memory need to page.
  • No tenant view. A neocloud or internal platform owes each tenant a view of their slice — not the host's view of all of them.
  • No topology. NVLink, PCIe, and InfiniBand topology matter at scheduling time. nvidia-smi topo -m prints it; nothing consumes it.

The graduation path

The next layer up is NVIDIA's own DCGM, which exposes the same NVML counters as a long-running daemon with a gRPC interface, plus health checks and policy. dcgm-exporter publishes them to Prometheus. That gets you retention, alerting, and dashboards — but you still own the scheduler integration, the tenant model, the topology awareness, and the alert routing.

For a quarter-rack to a few racks, building that stack from scratch is a multi-quarter platform-engineering project that competes with the actual reason you bought the hardware.

How CLASTIQ handles this

CLASTIQ runs DCGM under the hood and pulls the same NVML counters you already read with nvidia-smi — every GPU, every host, every slice — into a sub-frame telemetry plane with ~4ms update latency. Tenant attribution is automatic, NVLink and InfiniBand topology feeds the scheduler, alerts route to PagerDuty or Opsgenie, and historical retention lives in your storage, inside your perimeter.

It runs entirely on-prem. No outbound telemetry, no SaaS control plane, no foreign jurisdiction touching your data — designed for sovereign, regulated, and air-gapped environments.

TL;DR

  • Use nvidia-smi --query-gpu with CSV output, not the default table.
  • Watch utilization, power, memory, thermals, and ECC together — any one in isolation lies.
  • It scales to one rack on a good day. Beyond that, graduate to a real telemetry plane.
  • Stay on-prem. Your GPU telemetry is operational data; do not ship it to someone else's cloud.