RTX 4090
A flagship consumer-grade graphics card released by NVIDIA in 2022, equipped with 24 GB of GDDR6X memory. It is among the consumer GPUs with the largest single-card VRAM capacity and is widely used for local AI model inference.
Core Facts
QLoRA introduces 4-bit quantization technology, making it possible to fine-tune models with 65 billion parameters on a single RTX 3090/4090 GPU with 24 GB of VRAM.
The consumer-grade GPU RTX 4090 has a VRAM ceiling of 24 GB.
NVIDIA claims that the weights of Nemotron 3.5 Bolt 74B-A9B can run in 24GB of memory, and that 24GB refers only to the weights themselves, excluding context, KV cache and runtime overhead.
Timeline (last 90 days)
Users can run local models with 7B to 70B parameters on consumer-grade GPUs (such as an RTX 4090 with 24GB of VRAM) or on Apple Silicon devices with unified memory.
The optimization measures for running Hy-Nomas on a single RTX 4090 include an INT8-quantized DiT model, an FP8 version of the video decoder, and a CPU offload strategy.
An open-source project rewrites GPU registers via a Linux kernel module to bypass Nvidia's compute limits on consumer graphics cards, pushing the RTX 4090's half-precision compute beyond 330 TFLOPS.
The open-source tool GPU Unlock removes some compute limits by modifying internal registers of Nvidia graphics cards without overclocking or modifying official drivers, and in tests some matrix operation throughput on the RTX 3090 improved by about 80%, with the relevant RTX 4090 tests nearly doubling.
Seven same-tier cross-version comparisons (with hardware differences excluded) showed: 5 got faster and 2 got slower; dual RX 6900XT was more than a third faster, RTX 4090 running Russian was actually slightly slower, and ARC Pro B65 dropped 11%; the median was only 1.08x.
The top submitter was RTX 4090 (12 submissions), second was RX 6900XT (11), and RTX 5090 had only 10.
For small- and medium-scale tasks, a single A100 or RTX 4090 may offer better cost-effectiveness than an H100; the H100's advantages truly show only in ultra-large-scale parallel training.
In single-user scenarios llama.cpp runs an 8B model at about 115 tok/s with about 12ms first-token latency on an RTX 4090, but at 32 concurrency throughput is only about 210 tok/s.
Developer Soto wrote his own inference engine and ran the 552B-parameter DeepSeek V4.1 Flash on a gaming PC with a single 24G RTX 4090, 90G of memory plus NVMe storage, measuring about 27 tokens/s decoding and 180-320 tokens/s prefill.
Magnitude's CUDA kernel autotuning activates the RTX 4090's Tensor Cores for INT4 operations, which is one of the keys to its doubled throughput.
40 more timeline events
All Facts (20)
QLoRA introduces 4-bit quantization technology, making it possible to fine-tune models with 65 billion parameters on a single RTX 3090/4090 GPU with 24 GB of VRAM.
90%VerifiedThe consumer-grade GPU RTX 4090 has a VRAM ceiling of 24 GB.
80%VerifiedNVIDIA claims that the weights of Nemotron 3.5 Bolt 74B-A9B can run in 24GB of memory, and that 24GB refers only to the weights themselves, excluding context, KV cache and runtime overhead.
80%VerifiedDeveloper Primitive reported a generation speed of 142.6 tokens/s for Qwen3 Flash Next running locally.
65%VerifiedThis workflow can generate continuous videos up to 60 seconds long on a 12 GB GPU.
65%VerifiedThe RTX 4090 has 24 GB of VRAM, and after loading the 27B Q4 model, there is typically enough space remaining to support a context of 30,000–40,000 tokens.
65%VerifiedThe RTX 4090 has more than 16,000 CUDA cores.
65%VerifiedThe RTX 4090 has 24GB of VRAM and can smoothly run most 13B–34B quantized models
65%VerifiedGenerating a single 1024×1024 image using Flux 2 on an RTX 4090 takes approximately 8–15 seconds.
65%UnverifiedRTX 4090作为消费级旗舰显卡，拥有24GB GDDR6X显存，售价约1,600美元
85%Unverified使用Ollama本地部署时，官方推荐Qwen 2.5视觉模型（32B参数），需要至少24GB显存的GPU（如RTX 4090）才能流畅运行
85%UnverifiedRTX 4090的功耗约为300W
85%UnverifiedUnsloth通常需要配合NVIDIA GPU使用，推荐RTX 3090/4090或更高
80%Unverified移动端RTX 5090采用完整的GB203核心，拥有超过10,000个CUDA核心和16GB GDDR7显存，理论算力接近上一代桌面端RTX 4090
80%Unverified单卡RTX 4090即可运行GPT-OSS 20B，单卡A100即可运行GPT-OSS 120B
80%Unverified将模型权重从FP16压缩到INT8或INT4的量化技术，可使消费级GPU如RTX 4090在本地部署时达到30-80 TPS的实用速度
75%UnverifiedWhen fine-tuning a Llama 7B model on an RTX 4090 using QLoRA, VRAM usage is approximately 5–8GB.
70%Unverified一个70亿参数的模型在RTX 4090上的推理延迟约为50-200ms
70%Unverified批量生成100张图片在RTX 4090上大约需要15-25分钟
65%UnverifiedRTX 4090的理论FP16算力为82.6 TFLOPS，显存带宽为1008 GB/s
60%