senn-techFreeToken: Frontier MoE Models on Gaming Hardware
FreeToken is a new take on edge inference: instead of offloading frontier MoE models to the cloud, the engine brings 290B+ parameters to a single gaming or workstation GPU backed by CPU/RAM. The project is released under Apache 2.0 and targets self-hosters directly.
Edge-native, not cloud-native
Most MoE serving stacks are data-center first: multiple GPUs, high node-to-node throughput, NVLink. FreeToken inverts that. The focus is on one host with one GPU and plenty of RAM — the hardware many developers and smaller teams already have on their desks.
Core building blocks:
- CPU-GPU heterogeneous execution: The model is not monolithically crammed into GPU VRAM; it is split by actual access frequency.
- q*-based bandwidth adaptation: Dynamically decides which experts must stay on the GPU for the current prompt.
- Semantic-aware caching: Recurring prompt patterns can skip the expert routing step.
- Elastic VRAM reallocation: Frees and reallocates GPU memory during the session instead of pinning a fixed layout.
Supported frontier models
FreeToken targets current open MoE architectures. The total parameter count is preserved; active compute per token is kept small.
FreeToken vs. KTransformers
Both projects make large language models usable on consumer hardware, but their architectural focus differs.
| Criterion | FreeToken | KTransformers |
|---|---|---|
| License | Apache 2.0 | Apache 2.0 |
| Architecture focus | Edge-native CPU/GPU MoE serving | CPU-GPU hybrid inference for long contexts |
| GPU count | 1 consumer GPU | 1 GPU, heavy CPU offload |
| Hot/cold experts | GPU-hot, CPU/DRAM-cold via q* | Non-active experts in CPU RAM |
| Best for | Frontier MoE on a workstation | Very large models on limited hardware |
| API style | OpenAI/Anthropic-compatible | OpenAI-like chat endpoint |
FreeToken is therefore not a KTransformers variant, but a distinct serving stack that also optimizes the MoE router logic.
Installation and operation
The fastest path is the Python package with uv:
uv pip install "freetoken[accel]"
freetoken serve --model DeepSeek-V4-Flash
Alternatively, a desktop app is available. Both expose an OpenAI- and Anthropic-compatible API, so existing clients can switch with a simple base-URL change.
Our take
FreeToken is another signal of the trend that decouples large MoE models from cloud dependencies. Anyone with a powerful desktop and a current GPU can now run frontier models locally — not as a demo, but as a productive endpoint. For our customers that means data sovereignty, control over latency, and no API rate limiting. The Apache 2.0 license also keeps the legal side straightforward.
Further reading
What does CPU-GPU heterogeneous execution mean in FreeToken?+
FreeToken distributes a single MoE model deliberately across multiple memory and compute tiers: active 'hot' experts live in GPU VRAM, less frequently used 'cold' experts stay in CPU-addressable DRAM. A q*-based bandwidth-adaptive policy decides per request which experts must remain on the GPU and which can be streamed from CPU on demand. This is not a generic quantization trick; it is an explicit memory hierarchy.
How does FreeToken save VRAM without cutting model quality?+
Semantic-aware caching and elastic VRAM reallocation prevent every expert from being loaded equally on every pass. Hot experts stay resident on the GPU, cold ones move to CPU/DRAM and are fetched over PCIe when needed. Combined with formats such as MXFP4, NVFP4, FP8, or BF16, a 290B+ frontier model fits onto a single consumer GPU with accompanying system RAM — without shrinking the model itself.
Which models and quantization formats does FreeToken support?+
The engine supports models including DeepSeek-V4-Flash, Qwen3.6-35B-A3B, and GLM-5.2. Quantization formats cover MXFP4, NVFP4, FP8, and BF16. For operation there is a desktop app and a CLI; install via `uv pip install freetoken[accel]`. The API is OpenAI- and Anthropic-compatible.