Running DeepSeek on a Small GPU? This GitHub Trending Project Offloads VRAM Pressure to CPU and SSD

When MOE Models Don't Fit in VRAM, Layered Computing Becomes the New Solution

Large models routinely reach hundreds of GB in size, putting them out of reach for ordinary gamers' GPUs. In the 136th issue of GitHub weekly trending, a local inference tool called Stratum offers its own answer: keep frequently used experts in VRAM, move the rest to system memory, let the CPU and GPU share the computation, and even place large lookup tables on the SSD.

This isn't simple offloading. Stratum primarily targets massive MOE models like Qianwen 3.8 Flash, Next, and DeepSeek^Ad V4.1 Flash. Its approach turns the CPU into a participant in expert computation—some experts on the CPU and GPU compute simultaneously, and the results are merged before entering the next layer. The list of experts held in VRAM also changes dynamically; the tool tracks invocation statistics and moves more frequently used experts into VRAM.

According to the data published by the project, the DeepSeek 2.0 BPW version runs at roughly 6 to 4 tokens per second on a 12G 5070 GPU with 64G of RAM. That speed isn't exactly smooth, but at least it turns "running large models on a small GPU" from impossible into something you can tinker with. The author also candidly notes that memory and SSD prices have risen dramatically this year, so anyone wanting to go down this path should first weigh the storage costs.

How Is It Different from Ollama and LM Studio

If Ollama and LM Studio already exist, why install Stratum too? This question is unavoidable. The author's hands-on experience is that Stratum has no unique advantage in basic capabilities like chat, file processing, and local APIs, and is actually less mature than LM Studio. Its differentiation comes down to two points: specialized adaptation for a handful of very large MOE models, and an interface design that blends running and workflows together.

In terms of usage, Stratum offers both a desktop app and a terminal version. The first launch includes a guided setup for choosing models and configuration, and it downloads models automatically—but the author describes the default download route as "basically unable to download anything." Another detail to note: DeepSeek isn't visible in the desktop app; to run DeepSeek you need the terminal version. For users who frequently switch between different models, there isn't much necessity to migrate to this tool.

MOE (Mixture of Experts) models are the key prerequisite for understanding this article. A traditional dense model activates all parameters for every token it processes, whereas MOE splits the feed-forward network into multiple "expert" sub-networks, with a gating network selecting only a few experts per token to participate in computation. This way the model's total parameter count can reach hundreds of GB or more, but the parameters actually used in a single inference pass are far smaller than the total. The problem is that all expert weights still need to be stored somewhere; if they don't fit in VRAM, they have to go to memory or SSD—and that's precisely the contradiction Stratum aims to solve. MOE's sparse activation property also explains why "scheduling experts on demand" is engineering-feasible: only experts selected by the gate need to participate in that round of computation.

BPW (Bits Per Weight) is a common metric in the quantization field, indicating how many bits each weight occupies on average. The "DeepSeek 2.0 BPW version" mentioned in the article refers to quantizing the model to about 2 bits per weight on average, which compresses the size to roughly one-eighth compared with FP16's 16 bits, at the cost of some loss in precision and output quality. Extremely low BPW is one of the prerequisites for running very large models on small VRAM, but it and layered scheduling are two independent means: quantization reduces overall size, while layering addresses peak occupancy.

DeepSeek Open-Sources Its Inference Engine, Ascend Becomes a First-Class Citizen

The second project, DeepInfer, comes from DeepSeek and is the inference engine behind DeepSeek V4.1 Flash. It includes core technologies such as MOE expert parallelism, asynchronous batch scheduling, custom kernels, and speculative decoding. Data disclosed by DeepSeek shows that a single cluster processes about 1.55 million tokens per day; at such massive traffic, even a little resource waste per GPU computation round adds up to something very significant.

Currently, DeepInfer has open-sourced the offline inference portion; storage and cross-part load balancing and other content have not yet been fully disclosed. The project connects previously familiar low-level optimizations like Flash LMA and DPEP. On hardware support, two paths have emerged: NVIDIA GPUs and Huawei Ascend 950 NPUs. Ascend becoming a first-class supported citizen this time shows that DeepSeek has increased its investment in domestic support.

For people in model serving, inference optimization, and compute platform directions, this project has practical reference value; ordinary users are mostly just watching the excitement. In the video, the author comments that DeepSeek's continued investment in open source makes it the true "OpenAI" of the AI world.

Speculative Decoding is one of the core technologies mentioned by DeepInfer. It first uses a small, fast "draft model" to generate several candidate tokens in succession, then has the large model verify these candidates in parallel all at once, accepting the correct ones. Since a single forward pass of the large model can verify multiple tokens, it can significantly improve decoding throughput without changing the output distribution—especially suited to scenarios where memory bandwidth is limited and per-token generation is expensive.

MOE expert parallelism refers to distributing different experts across different GPUs or NPUs, so that whichever expert the gate selects is computed by the device holding that expert, with results then aggregated through communication. It brings two major challenges—load imbalance and cross-device communication—which require scheduling strategies to avoid some devices idling. The "asynchronous batch scheduling" mentioned in the article is designed precisely to hide such communication and computation bubbles.

Ascend NPU is Huawei's AI accelerator chip, with the CANN software stack, independent of NVIDIA's CUDA ecosystem. For an inference engine to support Ascend, it needs to rewrite kernels and adapt to its operator library; therefore "first-class citizen" means Ascend is not an after-the-fact patch but a formal support path on par with GPUs.

OpenCode 2.0: From Coding Agent to Extensible Client Platform

OpenCode is an open-source coding agent that can connect to different models, and it has gained another wave of popularity with the 2.0 upgrade. Terminal users can upgrade by running OpenCode Upgrade-major; app users are advised to download the new version directly. 2.0 by default uses a shared backend service to manage sessions, configuration, permissions, and tool execution, while local clients connect to the various services; users can use the full terminal interface, a streamlined interface, or the app.

For developers, a unified server-side means they can develop their own clients. Improvements around the coding experience are also concrete: the terminal now has side-by-side diff viewing, and the desktop version can show changed files' line counts and differences by round, making it more intuitive to review after the agent finishes writing code. The latest 2.2 version also adds experimental content such as desktop plugins.

Another reason OpenCode attracts attention is OPUS 5.5. Among the various ways to use OPUS 5.5 in China, OpenCode is relatively easy to get started with, and payment even supports Alipay. Existing users upgrading need to watch for compatibility: old plugin implementations can't run directly in V2, and integrations calling old services and interfaces also need changes.

Coding agents refer to AI systems that can autonomously read and write codebases, execute commands, run tests, and iterate based on results; unlike code assistants that only do completions, they have multi-turn tool-calling capabilities. OpenCode 2.0 converges sessions, configuration, permissions, and tool execution into a shared backend service, effectively decoupling the agent runtime from the interface: clients are only responsible for display and interaction, while the server centrally manages state and tool execution permissions. This architecture lets the same set of agent capabilities be reused by the terminal, streamlined interface, and app, and leaves interfaces for third parties to develop their own clients. Note that centralized permission management on the server side means the security boundary for tool calls shifts from the client to the service process; the incompatibility of old plugins and old interfaces during upgrades is essentially the cost of this architectural migration.

Open-Source Humanoid Robot RPE: Making the Navigation Page Thicker

RPE comes from Robot Party and is an open-source robotics project. The core question worth paying attention to in robot open source is: how much is actually opened, and can others continue building along these materials? RPE organizes mechanical, electrical, joint, firmware, deployment, and learning-related content into a single entry point; the hardware repository contains original files, editable drawings, full-machine models, and PCB production materials, while the software side includes components such as robot descriptions, low-level interfaces, and deployment frameworks.

A reasonable onboarding path is to first look at the robot model and configuration, then study motions in simulation and training environments, and then connect to real hardware according to deployment instructions. The author also points out that this repository is more like a navigation page; which robot configurations are specifically supported and what the code looks like still need to be confirmed by going back to each sub-repository. For robotics teams and laboratories, this way of organizing things is quite practical. Previously, Hugging Face and some overseas robotics companies open-sourced several robots, and domestic open-source content has gradually emerged as well.

AI Pose Gallery and Two Industry Reports

The last project is AI Pose Gallery Chinese version, suitable for users doing AI painting, character design, and photography reference. It is based on a Japanese source site with Chinese localization and material mirroring, supports keyword search, allows filtering by action and by viewpoint such as front, side, and top-down, and can switch compositions by pose grouping. Related original images can be downloaded and taken into AI tools for creation. The author has deployed an online version, and the entire project can also be cloned locally for a static version browsable on an intranet. This kind of small material library is very practical, but copyright issues regarding materials and code need to be noted.

At the end of the video, two materials are shared. One is the End-Side Intelligent Industry Development Report 2026 from the China Academy of Information and Communications Technology and the China Artificial Intelligence Industry Development Alliance, which discusses how AI can run stably on resource-limited devices from chips, storage, and basic software to models and applications, and sorts out scenarios such as office, travel, home, and robotics. The other focuses on how AI is changing consumer shopping and e-commerce operations, observing the evolution of shopping entry points from search recommendations toward agents managing product filtering and executing transactions.

Three Signals from This Issue's Trending

The first signal is that local inference is shifting from "can it run" to "how to run it cost-effectively." Stratum's idea of using CPU, memory, and SSD to share VRAM pressure is essentially seeking an engineering compromise under the reality of high hardware costs. The second signal is the rising status of domestic compute in inference frameworks; Ascend becoming a first-class citizen of DeepSeek's inference engine is a clear statement. The third signal is the accumulation of open-source robots and AI creation material libraries—the former aimed at teams and laboratories, the latter at individual creators—both organizing scattered resources into reusable entry points.

If you're interested in these projects, Stratum suits tinkerers with large memory and ample SSD space, DeepInfer suits inference optimization practitioners for study, OpenCode suits developers who want to use OPUS 5.5 at low cost, and RPE suits robotics teams diving deep by subsystem.

One-line Summary

GitHub weekly trending: Stratum layered MOE running, DeepInfer open source, OpenCode 2.0 upgrade, and more

Paragraph Summary

This issue of GitHub weekly trending covers five projects. Stratum performs layered inference for very large MOE models like Qianwen and DeepSeek, keeping frequently used experts in VRAM and moving the rest to memory or even SSD, with CPU and GPU computing together; it runs the DeepSeek 2.0 BPW version at about 4 to 6 tokens per second on a 12G GPU with 64G RAM, but is less mature than LM Studio and constrained by rising storage prices. DeepSeek has open-sourced the offline inference portion of its DeepInfer inference engine, covering expert parallelism, asynchronous scheduling, custom kernels, and speculative decoding, supporting both NVIDIA GPUs and Huawei Ascend 950 NPUs. OpenCode 2.0 shifts to a shared backend service architecture, improves the diff viewing experience, and makes OPUS 5.5 easier to use in China. There are also the open-source humanoid robot RPE, an AI pose library, and two industry reports.