RTX 3090
NVIDIA's consumer-grade flagship graphics card featuring 24 GB of GDDR6X memory, suitable for AI training and inference workloads.
Core Facts
NVLink 4.0 has a bidirectional bandwidth of up to 900 GB/s, while the bandwidth of a traditional PCIe bus is approximately 64 GB/s.
QLoRA introduces 4-bit quantization technology, making it possible to fine-tune models with 65 billion parameters on a single RTX 3090/4090 GPU with 24 GB of VRAM.
Models with 32B parameters can typically be compressed to approximately 16–20 GB of VRAM usage after quantization, and can be run on a single NVIDIA RTX 4090 (24 GB VRAM).
Timeline (last 90 days)
The open-source tool GPU Unlock removes some compute limits by modifying internal registers of Nvidia graphics cards without overclocking or modifying official drivers, and in tests some matrix operation throughput on the RTX 3090 improved by about 80%, with the relevant RTX 4090 tests nearly doubling.
Paper evaluations show that, on RTX 3090 and Jetson Orin, BitMTP achieves up to 2.0x five-task average throughput improvement, with the MBPP+ single task reaching 3.3x.
Using LoRA, fine-tuning a 4B model can be completed on a single RTX 3090 (24GB VRAM) in just a few hours.
When fine-tuning Qwen on a single RTX 3090, the number of correct answers on the dev set rose from 7/28 to 13/28, but the average negative log-likelihood (NLL) worsened from 2.215 to 3.916.
At batch 128 with the 7B AWQ model, the L40S's cost per million output tokens is $0.036, lower than the 3090's $0.040.
At the prices the author actually paid ($0.414 per hour for the 3090, $0.826 per hour for the L40S) and counting only GPU busy time, at low batch sizes the 3090's cost per million output tokens is about half that of the L40S.
With the 7B AWQ model at batch 32 the 3090 already lags behind the L40S (0.71x), whereas the 7B fp16 is not overtaken until batch 128 (0.72x), showing that the quantized model makes the compute bottleneck appear earlier.
At batch 1, the RTX 3090's inference throughput is about 1.03~1.09 times that of the L40S (based on Qwen2.5 1.5B/7B models, measured on vLLM 0.27.1).
At small batch sizes the bottleneck is memory bandwidth, while at large batch sizes the rising compute density makes compute the bottleneck, which explains the performance crossover between the 3090 and the L40S.
For individual developers or small-scale services dominated by low concurrency, the 3090 provides performance close to the L40S at about half the cost; for high-throughput online services, the L40S is faster at batch 128 and cheaper with the AWQ model.
40 more timeline events
All Facts (20)
NVLink 4.0 has a bidirectional bandwidth of up to 900 GB/s, while the bandwidth of a traditional PCIe bus is approximately 64 GB/s.
90%VerifiedQLoRA introduces 4-bit quantization technology, making it possible to fine-tune models with 65 billion parameters on a single RTX 3090/4090 GPU with 24 GB of VRAM.
90%VerifiedModels with 32B parameters can typically be compressed to approximately 16–20 GB of VRAM usage after quantization, and can be run on a single NVIDIA RTX 4090 (24 GB VRAM).
80%VerifiedThe RTX 4060's internal VRAM bandwidth is approximately 288 GB/s.
65%VerifiedConsumer-grade graphics cards typically have only 8–24 GB of VRAM, which limits the scale of models that can be run.
65%VerifiedThe RTX 4090 has 24 GB of VRAM, and after loading the 27B Q4 model, there is typically enough space remaining to support a context of 30,000–40,000 tokens.
65%UnverifiedUnsloth通常需要配合NVIDIA GPU使用，推荐RTX 3090/4090或更高
80%UnverifiedMiniMind-V可以在普通消费级GPU（如RTX 3090）上完成全流程训练
80%UnverifiedGPT-OSS 20B在RTX 3090上可达约40 Token/s的生成速度，在RTX 5090上能达到200 Token/s
75%UnverifiedOfficial estimates suggest a 24GB GPU can run approximately 100–140 tokens/second.
60%UnverifiedLoRA can compress trainable parameters to 0.1%–1% of the original scale, enabling fine-tuning on a single consumer-grade GPU.
60%UnverifiedThe open-source tool GPU Unlock removes some compute limits by modifying internal registers of Nvidia graphics cards without overclocking or modifying official drivers, and in tests some matrix operation throughput on the RTX 3090 improved by about 80%, with the relevant RTX 4090 tests nearly doubling.
50%UnverifiedPaper evaluations show that, on RTX 3090 and Jetson Orin, BitMTP achieves up to 2.0x five-task average throughput improvement, with the MBPP+ single task reaching 3.3x.
50%UnverifiedUsing LoRA, fine-tuning a 4B model can be completed on a single RTX 3090 (24GB VRAM) in just a few hours.
50%UnverifiedWhen fine-tuning Qwen on a single RTX 3090, the number of correct answers on the dev set rose from 7/28 to 13/28, but the average negative log-likelihood (NLL) worsened from 2.215 to 3.916.
50%UnverifiedA single-card test does not represent all cards of the same model: another 3090 of the author's was markedly slower in all settings under the same script and driver, so it is best to test two cards when choosing hardware.
50%UnverifiedFor individual developers or small-scale services dominated by low concurrency, the 3090 provides performance close to the L40S at about half the cost; for high-throughput online services, the L40S is faster at batch 128 and cheaper with the AWQ model.
50%UnverifiedAt small batch sizes the bottleneck is memory bandwidth, while at large batch sizes the rising compute density makes compute the bottleneck, which explains the performance crossover between the 3090 and the L40S.
50%UnverifiedAt batch 1, the RTX 3090's inference throughput is about 1.03~1.09 times that of the L40S (based on Qwen2.5 1.5B/7B models, measured on vLLM 0.27.1).
50%UnverifiedAt the prices the author actually paid ($0.414 per hour for the 3090, $0.826 per hour for the L40S) and counting only GPU busy time, at low batch sizes the 3090's cost per million output tokens is about half that of the L40S.
50%