senn-techKTransformers vs FreeToken: Two Paths to Big MoE Models on Small GPUs
Frontier MoE models such as DeepSeek-R1/V3 or GLM-5.2 have hundreds of billions of parameters, but only a fraction is active per token. That sparsity is what makes them attackable with an old idea in new clothes: keep most of the model in CPU RAM and stream only the currently active weights to the GPU. KTransformers and FreeToken are the two newest public implementations of this approach.
Same goal, different origins
Both frameworks make the same promise: making a 671B DeepSeek or 753B GLM runnable on a single workstation GPU plus ordinary system memory. The price is the same too—PCIe bandwidth becomes the main bottleneck when experts have to be swapped between CPU RAM and GPU. So the difference is less in the architecture and more in the optimization strategy.
- KTransformers grew out of a research project at Tsinghua MADSys Lab together with Approaching.AI and 9#AISoft. It has been available longer, optimized around DeepSeek-R1/V3, and relies heavily on CPU-side AMX and AVX kernels.
- FreeToken comes from the FlashML ecosystem, is designed as an edge-native serving engine, and pursues the same goal for interactive, agentic use—with smoother prefill and faster warm-up.
KTransformers: the stable workhorse profile
KTransformers is the proven way to run DeepSeek-R1/V3 671B on a single 24 GB GPU. The modus operandi is simple: attention, MLP router, and KV cache live on the GPU, while experts are fully offloaded into host RAM. During prefill and decoding, the GPU fetches the experts it currently needs over the PCIe bus.
Strengths:
- CPU optimization: custom kernels for AMX/AVX speed up host-side expert management.
- Model depth: DeepSeek-V3/R1 671B is the primary target model and is correspondingly well tested.
- Ecosystem: integration into SGLang and other backends is growing.
Weaknesses:
- The focus is on maximum throughput in batch scenarios; interactive latency is secondary.
- Model startup can take a long time because standard HF checkpoints are read in full.
FreeToken: the interactive variant
FreeToken understands the same layout but optimizes for the case where a human user or agent expects fast responses. Instead of processing every prompt sequentially, it uses a global LRU expert cache and semantic anchor checkpoints to avoid rebuilding recurring contexts from scratch.
Strengths:
- Bandwidth-adaptive q policy*: dynamically decides whether a missing expert is fetched over PCIe or executed briefly on the CPU.
- Double-buffered full-layer prefill: allows the next layer to be prepared while the current one is still running.
- FTW format: a fast weight format that significantly reduces loading times.
- Agentic use: OpenAI- and Anthropic-compatible endpoints make it easier to deploy with coding and tool-use agents.
Weaknesses:
- As a newer project, long-term behavior and breadth of supported models are not yet as proven as with KTransformers.
Which framework when?
| Scenario | Recommendation | Reason |
|---|---|---|
| Single DeepSeek-R1/V3 671B workstation, batch inference | KTransformers | Mature, CPU-kernel optimized, battle-tested |
| Interactive agents on a consumer GPU | FreeToken | Lower latency, faster warm-up, tool-use endpoints |
| Mixed models, frequent switches | FreeToken | Bandwidth-adaptive q* policy and LRU cache |
| Maximum stability, low appetite for experimentation | KTransformers | Longer in the field, more community experience |
Our view
KTransformers and FreeToken prove that the line between "must run in a data center" and "runs under the desk" is blurring. For pure batch tasks with a large DeepSeek model, KTransformers is the safer harbor today. If you are building agents, chatbots, or coding assistants where every second counts, FreeToken deserves a closer look. Both are Apache 2.0—so testing them costs only hardware and time, not license fees.
Further reading
What do KTransformers and FreeToken have in common?+
Both frameworks share the same goal and the same basic trick: running frontier MoE models with hundreds of billions of parameters on a single consumer GPU plus system RAM. Only the active parts—attention, router, and KV cache—stay on the GPU, while most experts are parked in CPU RAM and streamed over PCIe when needed. Both are Apache 2.0 licensed, so they are unrestricted for commercial use.
When is KTransformers the better choice?+
KTransformers is the older, more stable option. It is heavily optimized for DeepSeek-R1/V3 671B, relies on CPU cores with AMX/AVX optimizations, and has proven itself for pure offline batch inference. If you need a specific large DeepSeek model running on a workstation with plenty of RAM, this is the safe bet. The focus is on maximum throughput, not smooth conversation.
Where does FreeToken set new standards?+
FreeToken is newer and designed specifically for interactive, agentic workloads. Its strengths include a bandwidth-adaptive q* policy, a double-buffered full-layer prefill, a global LRU expert cache, and semantic anchor checkpoints. The goal is low time-to-first-token and smooth streaming during long tool-use and reasoning chains. It also ships with a fast FTW weight format that noticeably speeds up model startup.