NVIDIA L4
A data center GPU introduced by NVIDIA, based on the Ada Lovelace architecture and positioned for mid-range inference and lightweight training tasks, with support for AI acceleration.
Timeline (last 90 days)
The break-even point for per-million-document cost between L4 and cpu-repack is a 3.4% busy ratio: when the L4 is busy more than 3.4% of its uptime, its per-document cost is lower than repack's.
An L4 instance costs about $1.0465 per hour, roughly ten times that of a 1 vCPU instance.
On Google Cloud Run, the minimum instance specification required to attach a GPU to an L4 service is 4 vCPU plus 16 GiB of memory.
On NVIDIA L4, the bf16 model (6.32 GiB) has a theoretical memory-bandwidth ceiling of about 47 tokens/s, while the int4 model (2.85 GiB) has a theoretical ceiling of about 105 tokens/s, with measured peak reaching 131.9 tokens/s.
NVIDIA L4 (g6/SageMaker) natively supports bf16, fp8, and int8, and int4 is widened to 16 bits; NVIDIA T4G (g5g) supports bf16 (emulated) and int8, does not support fp8, and widens int4 to 16 bits; NeuronCore-v2 (inf2/trn1) supports bf16 and int8, fp8 has hardware support but the compiler has yet to catch up, and int4 is widened to bf16 and is 27 times slower per call.
Inferentia2 and Trainium have 2.0 to 2.7 times the memory bandwidth of NVIDIA L4, but their decode speed is only 0.28 times that of L4.
In Nodus's pilot on October 11, 2026, using a Qwen2.5-0.5B model running the same recipe on a single NVIDIA L4, supervised fine-tuning cost 67% less than Modal and continued pretraining cost 42.5% less; after counting one failed attempt, the fine-tuning savings dropped to 24.7%.
The training results of the 0.8B model are reportedly verified on an NVIDIA L4 GPU.
All Facts (4)
On Google Cloud Run, the minimum instance specification required to attach a GPU to an L4 service is 4 vCPU plus 16 GiB of memory.
50%UnverifiedNVIDIA L4 (g6/SageMaker) natively supports bf16, fp8, and int8, and int4 is widened to 16 bits; NVIDIA T4G (g5g) supports bf16 (emulated) and int8, does not support fp8, and widens int4 to 16 bits; NeuronCore-v2 (inf2/trn1) supports bf16 and int8, fp8 has hardware support but the compiler has yet to catch up, and int4 is widened to bf16 and is 27 times slower per call.
50%UnverifiedInferentia2 and Trainium have 2.0 to 2.7 times the memory bandwidth of NVIDIA L4, but their decode speed is only 0.28 times that of L4.
50%UnverifiedOn NVIDIA L4, the bf16 model (6.32 GiB) has a theoretical memory-bandwidth ceiling of about 47 tokens/s, while the int4 model (2.85 GiB) has a theoretical ceiling of about 105 tokens/s, with measured peak reaching 131.9 tokens/s.
50%