FreeToken: Run 290B+ MoE Models Locally on Consumer GPUs
FreeToken lets developers run 290B+ MoE models locally on consumer GPUs by treating CPU, GPU, and memory as an elastic inference platform.
GitHub FlashML-org/FreeToken Updated 2026-09-22 Branch main Stars 13.6K Forks 1.3K
Python/CUDA/C++ Edge inference DeepSeek-V4-Flash NVIDIA RTX 30/40/50

🧭 Decision Guide

Why trending now: 无法从材料判断

Try it if you

  • You have an NVIDIA RTX 30, 40, or 50 series GPU and want to run MoE models such as DeepSeek-V4-Flash locally.
    The README sections “Diverse Consumer Hardware” and “Broad MoE & Ecosystem Support” list RTX 30/40/50 and DeepSeek-V4-Flash.
  • Your agent uses Codex, Claude Code, or OpenCode and needs Anthropic/OpenAI-compatible APIs.
    The README’s “Broad MoE & Ecosystem Support” explicitly lists these agents and Anthropic/OpenAI-compatible APIs.
  • You need to reduce context recomputation after tool calls or thinking blocks.
    The README’s “Semantic-Aware Caching” says semantic anchor checkpoints avoid redundant recomputation after agentic context edits.
  • You want to adjust VRAM usage without restarting the engine or reloading weights.
    The README’s “Elastic Memory Management” explicitly supports runtime VRAM reallocation between expert caches and KV memory.

Skip it if you

  • Your GPU is not one of the NVIDIA RTX 30, 40, or 50 series listed by the README.
    The README’s “Diverse Consumer Hardware” explicitly names native support for NVIDIA RTX 30, RTX 40, and RTX 50.
  • You require a stable release rather than a Nightly rolling version.
    Project metadata labels the latest version “Nightly (rolling)” and reports 3 releases.
  • You need established VRAM, throughput, latency, or output-quality benchmarks.
    The README only states “290B+” and “blistering interactive speeds” without numerical benchmarks.
  • You need a non-MoE model or a quantization format outside the formats explicitly listed by the README.
    The README focuses on frontier open-weight MoE models and names MXFP4, NVFP4, FP8, and BF16; it does not define a wider scope.

Requirements

  • Hardware: The README states native support for “NVIDIA RTX 30, RTX 40, and RTX 50 series GPUs.”
  • Desktop: The README offers downloads for “Windows or Linux.”
  • CLI installation: The README recommends uv or pip and provides “uv pip install "freetoken[accel]".”
  • Models and formats: The README lists DeepSeek-V4-Flash, Qwen3.6-35B-A3B, GLM-5.2, and MXFP4, NVFP4, FP8, BF16.

First step (verbatim from README)

uv pip install "freetoken[accel]"

Watch out

  • Old FTW checkpoints may require an additional repair workflow.
    Getting Started lists a “Repairing old FTW checkpoints” document.
  • CLI installation includes the extra accel component, so the dependency scope cannot be inferred from the base package name alone.
    The README’s CLI command is “uv pip install "freetoken[accel]".”
  • Model, quantization, and CLI details are distributed across separate documents.
    The Getting Started “For More details” list includes install, quickstart, models, and CLI documentation.

Alternatives

  • SGLang：If your existing system is already built around SGLang, the README provides no migration or performance comparison, so no further conclusion is possible.
    README section “Acknowledgment”
  • vLLM：If the team already has a vLLM integration, the README only says FreeToken learned from and reused vLLM code; it does not state when vLLM is better.
    README section “Acknowledgment”
  • llama.cpp：If the goal is to use the llama.cpp ecosystem named by the README, the materials provide no feature or performance comparison with FreeToken.
    README section “Acknowledgment”

Not stated in the README

  • The README does not specify minimum VRAM, system RAM, CPU models, or disk space.
  • The README provides no throughput, time-to-first-token, or concurrency data for individual RTX 30/40/50 models.
  • The README does not explain CUDA, driver, or feature differences between Windows and Linux.
  • The README does not provide a complete model list, model acquisition procedure, or limitations for each quantization format.
  • The README does not describe compatibility policy, rollback procedures, or a stable-release plan for the Nightly rolling version.
  • The README provides no performance comparison with SGLang, vLLM, FlashInfer, LightLLM, or llama.cpp.

💡 Deep Analysis

6
Yes My coding agent repeatedly inserts tool results and thinking blocks, and its DeepSeek or Qwen MoE model grows the KV cache until it competes with expert caching. Can FreeToken handle this without interrupting the engine?
For: An application engineer maintaining long-context coding-agent and tool-calling workflows with DeepSeek or Qwen MoE models on a single machine with limited VRAM

Yes, it is a good fit: FreeToken explicitly targets agentic context edits and dynamic VRAM contention, although it cannot eliminate KV-memory growth from long contexts.

  • Semantic anchor checkpoints save and reuse recurrent state and KV caches; the README specifically cites tool calls and thinking blocks.
  • The engine can reallocate VRAM at runtime between expert caches and KV memory without restarting the engine or reloading weights.
  • Global LRU expert caching can reduce repeated expert-weight transfers when MoE requests continuously change their context.

The README does not specify cache-hit rates, reallocation triggers, maximum context length, or failure behavior when KV memory keeps growing, so target latency cannot be guaranteed for a particular agent workflow.

  • About: semantic anchor checkpoints for recurrent state and KV caches
  • About: agentic context edits (e.g., tool calls, thinking blocks)
  • About: dynamic, runtime VRAM re-allocation between expert caches and KV memory without engine restarts or weight reloading
  • About: global LRU expert caching
uv pip install "freetoken[accel]"
Not stated in the README：Creation, invalidation, and hit-detection rules for semantic anchors；Trigger thresholds for dynamic allocation between expert caches and KV memory；Maximum supported context length and error handling when KV memory is insufficient
Yes I use Claude Code and OpenCode and want to connect a local DeepSeek or GLM MoE model to a coding agent. I need Anthropic/OpenAI-compatible APIs and tool-calling workflows. Is FreeToken suitable?
For: An engineer integrating a local model with Codex, Claude Code, or OpenCode, requiring tool calling and context editing

Yes, it is a good fit: the README directly targets coding agents and tool-calling through compatible APIs, although compatibility is limited to behaviors documented by the project.

  • It provides Anthropic- and OpenAI-compatible APIs and explicitly names Codex, Claude Code, OpenCode, OpenClaw, and DeepSeek Harness.
  • Semantic anchor checkpoints reuse recurrent state and KV caches to reduce recomputation after tool calls, thinking blocks, and context edits.
  • Global LRU expert caching and runtime VRAM reallocation between expert caches and KV memory address changing context in agent workloads.

The README does not specify whether every tool-calling protocol, streaming event, stop condition, or error format exactly matches cloud services, and it provides no end-to-end agent latency figures.

  • About: Anthropic/OpenAI-compatible APIs ... Codex, Claude Code, OpenCode, OpenClaw, DeepSeek Harness
  • About: semantic anchor checkpoints for recurrent state and KV caches
  • About: agentic context edits (e.g., tool calls, thinking blocks)
uv pip install "freetoken[accel]"
Not stated in the README：Protocol-by-protocol compatibility with Codex, Claude Code, and OpenCode；Actual behavior for streaming, concurrent tool calls, and invalid function arguments
Yes I am researching CPU–GPU co-execution on RTX 30, 40, and 50-series devices and need to compare MoE inference paths for MXFP4, NVFP4, FP8, and BF16. Is FreeToken suitable as an experimental system?
For: A researcher studying CPU–GPU cooperative MoE inference on NVIDIA RTX 30, 40, and 50-series devices, comparing MXFP4, NVFP4, FP8, and BF16

Yes, it is a strong experimental fit because its architecture and supported matrix directly cover bandwidth, expert caching, and heterogeneous-resource scheduling.

  • The README treats GPUs, CPUs, host memory, and interconnects as a unified elastic inference platform.
  • The runtime uses a bandwidth-adaptive CPU–GPU co-execution q* policy, full-layer double-buffered prefill, global LRU expert caching, and graph-compatible execution.
  • It explicitly lists MXFP4, NVFP4, FP8, BF16, and NVIDIA RTX 30/40/50 hardware.
  • Apache License 2.0 permits research users to integrate, modify, and redistribute the project.

However, the README does not provide q* policy details, operator coverage for each format, benchmark data, or reproducible experiment scripts, so a rigorous quantitative comparison cannot be derived from the project description alone.

  • About: heterogeneous edge resources—GPUs, CPUs, host memory, and interconnects—as a unified, elastic inference platform
  • About: bandwidth-adaptive CPU–GPU co-execution (q* policy)
  • About: MXFP4, NVFP4, FP8, BF16; NVIDIA RTX 30, RTX 40, RTX 50
  • License: Apache License 2.0
git clone https://github.com/FlashML-org/FreeToken.git && cd FreeToken
uv venv && source .venv/bin/activate
uv pip install -e ".[accel]"
Not stated in the README：Inputs, decision logic, and tunable parameters of the q* policy；Operator coverage, model conversion workflow, and benchmarking method for each precision format；Reproducible measurements under different PCIe or interconnect bandwidths
It depends I only have a personal gaming PC with an NVIDIA RTX 40-series GPU, but I want to run DeepSeek, Qwen, or GLM MoE models above 290B locally. Is FreeToken suitable?
For: A local AI developer with an NVIDIA RTX 40-series GPU who wants to run 290B+ open-weight MoE models on a personal gaming PC

It depends: FreeToken is explicitly aimed at bringing 290B+ MoE models to consumer hardware, but actual feasibility depends on the whole machine, not only the GPU model.

  • The README explicitly supports NVIDIA RTX 30, 40, and 50 series, while treating GPUs, CPUs, host memory, and interconnects as one resource pool.
  • It lists DeepSeek-V4-Flash, Qwen3.6-35B-A3B, GLM-5.2, and MXFP4, NVFP4, FP8, and BF16 formats.
  • CPU–GPU co-execution, global LRU expert caching, and runtime VRAM reallocation target limited VRAM and expert-transfer overhead.

The README does not specify required VRAM, host memory, disk capacity, or PCIe bandwidth, nor does it guarantee interactive speed for every RTX 40 configuration.

  • About: Run 290B+ frontier MoE models locally on your gaming PC
  • About: native support for NVIDIA RTX 30, RTX 40, and RTX 50 series GPUs
  • About: DeepSeek-V4-Flash, Qwen3.6-35B-A3B, GLM-5.2; MXFP4, NVFP4, FP8, BF16
uv pip install "freetoken[accel]"
Not stated in the README：Minimum requirements for the target model on the specific RTX 40 model, host memory, and storage configuration；Actual time-to-first-token and generation speed for each quantization format
It depends I need to run FreeToken on a Windows or Linux workstation. My team wants to manage models through the desktop GUI first and automate with the CLI later. Is this deployment approach reliable given that the project has only 3 releases and the latest is marked nightly?
For: A developer deploying local model serving on a Windows or Linux workstation, with a team that prefers GUI and CLI workflows over building an inference engine

It depends: the startup path is friendly for Windows and Linux users, but the nightly status and small number of formal releases make long-term stability uncertain.

  • The README provides Windows and Linux desktop applications for engine setup, model execution, chat, and tuning.
  • The CLI supports installation through uv or pip and links to installation, Quick Start, model, and CLI reference documentation.
  • Project data shows only 3 releases, with the latest marked nightly; engine interfaces, weight formats, or runtime behavior may still change.
  • The README includes documentation for repairing old FTW checkpoints, indicating that weight-format compatibility requires attention.

The README does not define stability guarantees, version-pinning procedures, rollback methods, or Windows/Linux feature differences, so GUI convenience alone is insufficient to assess production risk.

  • Getting Started—Desktop app: Download FreeToken for Windows or Linux
  • Getting Started—CLI: Install FreeToken with uv (recommended) or pip
  • Project data: release_count is 3 and latest_release is nightly
  • Getting Started: Repairing old FTW checkpoints
uv pip install "freetoken[accel]"
Not stated in the README：Nightly release cadence, breaking-change policy, and rollback method；Feature differences between the desktop application and CLI for models, caching, and tuning；CUDA, driver, and hardware compatibility matrix for Windows versus Linux
Yes I plan to integrate FreeToken into a commercial local-AI product and may modify and redistribute its Python, CUDA, C++, and C code. Does Apache License 2.0 fit this licensing constraint?
For: An enterprise R&D engineer embedding a local MoE service into commercial software and needing to modify and redistribute the inference component

Yes, it is suitable: Apache License 2.0 generally permits enterprise integration, modification, and redistribution, but third-party dependencies and attribution obligations must be checked separately.

  • Project data explicitly identifies Apache License 2.0 as the project license.
  • The README License section links directly to the Apache License 2.0 text, so the licensing information is part of the repository documentation.
  • The project includes Python, CUDA, C++, C, and Shell, meaning integration may involve runtime code, native extensions, and build scripts rather than only a Python package.
  • The Acknowledgment section names SGLang, vLLM, FlashInfer, LightLLM, llama.cpp, and other projects whose designs or code were reused or consulted.

The README does not provide a complete license inventory for third-party code, NOTICE requirements, patent-scope details, or model-weight licenses, so Apache License 2.0 alone cannot establish full product compliance.

  • Project data: license is Apache License 2.0
  • License: Apache License 2.0
  • Project data: Python, CUDA, C++, C, Shell
  • Acknowledgment: SGLang, vLLM, FlashInfer, LightLLM, llama.cpp
Not stated in the README：Complete license inventory for all third-party dependencies and reused code；Separate licensing terms for model weights, quantized files, and FTW files；Whether the distribution package includes NOTICE or other attribution files

✨ Highlights

  • Runs 290B+ frontier MoE models on consumer hardware
  • The q★ policy enables bandwidth-adaptive CPU–GPU co-execution
  • Semantic anchor caching reduces recomputation after tool calls
  • Reallocates VRAM at runtime between expert cache and KV memory
  • The project has a Nightly rolling release and only 3 releases

🔧 Engineering

  • Provides an MoE serving runtime supporting DeepSeek-V4-Flash and GLM-5.2
  • Supports Anthropic/OpenAI-compatible APIs for Codex and Claude Code
  • The desktop app supports Windows and Linux, while the CLI supports uv installation
  • FTW format and global LRU expert caching support efficient MoE execution

⚠️ Risks

  • The latest version is Nightly rolling, with only 3 releases recorded
  • The materials list native support only for NVIDIA RTX 30, 40, and 50 series
  • The README provides no VRAM, speed, or quality figures for 290B+ models
  • The project has only 7 contributors; maintenance and response capacity are not established by the materials

👥 For who?

  • Developers who need to run DeepSeek-V4-Flash on RTX 30/40/50 GPUs
  • Teams building tool-calling agents with Codex or Claude Code
  • Users who want to run 290B+ MoE models on Windows or Linux desktops
  • Experimenters needing MXFP4, NVFP4, FP8, or BF16 quantization formats