Running GLM-5.3 on Two DGX Stations: A vLLM Tensor Parallelism Hands-On

Why Run a Large Model Locally Across Nodes

GLM-5.3 has 753 billion parameters. Although only 40 billion are active per token, all weights must remain resident and available. The NVFP4 checkpoint is roughly 464 GB, while a single GB300 GPU has only 252 GB of VRAM—the model does not fit on one card. In the demonstration, NVIDIA technical marketing engineer Fatima Fall presents the solution: split the model across two DGX Stations, each using one GPU, and use tensor parallelism so the two GPUs jointly process every request.

The value of this scenario isn't a benchmark score—it's a reproducible path: stand up a local API service with vLLM across two machines, keep all model weights in GPU memory, and expose a single inference endpoint.

Tensor Parallelism (TP) is a model-parallel strategy that splits the weight matrices within a single Transformer layer (such as the Q/K/V projections of attention heads or the up/down projection matrices of the MLP) along specific dimensions across multiple GPUs. Each GPU holds only part of the weights; during the forward pass, the cards process the same batch of tokens in parallel and then aggregate partial results through collective communication operations such as All-Reduce, so that one request is completed cooperatively by multiple GPUs. Unlike pipeline parallelism (which splits by layer), tensor parallelism is extremely sensitive to communication bandwidth and latency and typically requires NVLink- or RDMA-class high-speed interconnect between GPUs. In this demo, Tensor Parallel Size=2 means each layer's weights of GLM-5.3 are split into two halves, placed on the single GPU of each DGX Station, and every token's computation requires the two GPUs to exchange intermediate activations.

Network and System Preparation: Connect the Nodes First

The first hurdle for cross-node tensor parallelism is the network. In the demo, a directly connected network rail is selected and its RDMA device identified; the IP address of that rail on each Station is recorded and written into an environment file. The two environment files differ only in the rank number—Station A is rank 0 and Station B is rank 1.

The link target is 400 GB bandwidth, MTU 9000 jumbo frames, and clock synchronization. Only after jumbo frames are confirmed can packets pass smoothly between nodes. Before launching, verify that both GPUs are idle, disconnect the PC-side AI network connection, and set up passwordless SSH from A to B.

At the software level, both machines must pull the same pinned version of the vLLM image. Tensor parallelism requires an identical software stack on both ends; matching image digests confirm the builds are the same. The checkpoint is downloaded on Station A and copied to B, keeping one full copy of the checkpoint on each disk.

RDMA (Remote Direct Memory Access) allows one machine to directly read and write another machine's memory, bypassing the OS kernel and CPU copies, thereby greatly reducing network latency and CPU overhead. In GPU clusters, RDMA is typically provided by InfiniBand or RoCE (RDMA over Converged Ethernet) NICs, and NCCL prefers it for cross-node GPU communication. Identifying the RDMA device for each network rail and recording its IP in the demo is precisely to let NCCL bind to the correct NIC when forming the communication ring. If the RDMA link is not established, tensor-parallel All-Reduce degrades to TCP communication, bandwidth and latency deteriorate sharply, and cross-node inference becomes nearly unusable. Therefore, the "RDMA check OK" in the logs is the key signal that direct GPU-to-GPU communication between nodes is actually working.

Launch Parameters: Tensor Parallel Size Set to 2

Both Stations run the same launch script, each starting one vLLM rank. Key configurations include:

• Tensor Parallel Size 2: split the model across two GPUs, each holding part of the weights, with every request computed jointly by both cards
• Maximum model length 98,304 tokens: covering input and output
• GPU memory utilization 93%, with CPU weight offloading disabled
• Multi-token prediction using four draft tokens to accelerate decoding
• FP8 KV cache to save VRAM with a lower-precision format

Startup order is B first, then A. Each rank loads its own share of weights, prepares kernels, and captures CUDA graphs. When Station A reports application startup complete, the service is ready.

NCCL (NVIDIA Collective Communications Library) is NVIDIA's collective communication library, responsible for implementing All-Reduce, All-Gather, Broadcast, and other operations across multiple GPUs and nodes. vLLM's tensor parallelism relies on NCCL under the hood to synchronize computation results across ranks. At startup, each vLLM rank joins the same NCCL communication group; NCCL automatically probes the topology (NVLink within a node, RDMA across nodes) and establishes the communication ring. "RDMA check OK" in the logs indicates that NCCL has successfully completed a connectivity test on the specified RDMA device. When debugging cross-node tensor parallelism, NCCL logs (enabled via NCCL_DEBUG=INFO) are the primary tool for diagnosing NIC binding, rank numbering, and link quality issues.

Multi-Token Prediction (MTP) is an inference acceleration technique: a lightweight draft head or draft model is attached alongside the main model to predict multiple future tokens in one forward pass, which the main model then verifies in parallel, accepting matching tokens and rejecting mismatches. This follows the same idea as speculative decoding and can reduce decoding steps without changing the output distribution. In the demo, setting four draft tokens means each decoding attempt additionally predicts 4 tokens; if verification passes, the model advances multiple positions at once. This technique incurs some extra VRAM overhead but can significantly improve tokens/s in interactive scenarios, making it especially suitable for latency-sensitive agent applications in local deployments.

Verification: One API, Two GPUs

Once the service is up, confirm that the communication link is truly working. First request the API on Station A; a "half OK" response means A is responding. Then check the NCCL logs on both machines—"RDMA check OK" indicates GPU-to-GPU communication between the two ranks over the RDMA link. In nvidia-smi, Station A shows worker TP0 and B shows TP1, i.e., one tensor-parallel rank per GPU. Total VRAM usage includes model weights, KV cache, and runtime buffers.

In the actual request test, the API was asked to summarize quarterly sales data and count requests; the returned result showed sales growing from 120 to 180, a 50% increase, with finish reason "stop", indicating the response ended normally. The two GPUs jointly complete the computation behind a single API, and agent frameworks can use this endpoint directly.

Performance Baseline: Throughput Varies with Concurrency

The demo uses NVIDIA AI Perf in a separate virtual environment on Station A for load testing, with the speedbench coding dataset; the full dataset contains 80 conversations, and only a small sample is used in the video. Two cases were tested, concurrency 1 and 8: at concurrency 1, send 3 warmup requests and measure 10; at concurrency 8, send 24 warmups and measure 80. During the run, you can see the activity and power draw of both GPUs change in sync, confirming they work together.

Citing reference results from the accompanying blog: as concurrency rises, per-user throughput decreases and average request latency increases; total output throughput peaks at about 608 tokens per second at concurrency 16 (token counts include reasoning content). Data for all four concurrency levels and reproduction instructions are in the original blog.

Wrapping Up and Reuse

At the end, stop the container on Station A first, then on B. Checkpoints and results remain on B's disk. The assets left by this setup are: a local large-model service, one API, two DGX Stations with verified GPU-to-GPU communication, and a performance baseline that can be extended further. For teams that need to run very large models locally, the key constraints of this path are clear—link quality, strictly identical software versions on both ends, and sufficient VRAM budget allocation.