colibri: Run 744B–2.8T MoE models on local hardware in pure C
A pure-C inference engine for running 744B–2.8T MoE models locally, streaming experts from disk instead of fitting everything in VRAM.
GitHub JustVugg/colibri Updated 2026-09-11 Branch main Stars 27.5K Forks 3.0K
C Local LLM inference MoE GLM-5.2 Consumer GPUs and heterogeneous hardware

🧭 Decision Guide

Why trending now: The README shows 744B GLM-5.2 running on 6× RTX 5090, about 1.8 tok/s on a 128 GB CPU-only desktop, and support for 744B–2.8T MoE models. Combined with the latest colibri 1.10.2 release and 98 stars added that day, these measured signals around running frontier models on consumer or heterogeneous hardware likely contributed to the attention, although the material cannot prove a specific causal link.

Try it if you

  • You have 372 GB of disk space and want to run GLM-5.2 int4 without fitting the entire model into VRAM.
    The README “Get started” section states that the program is a few hundred KB and the model is about 372 GB; “How it works” describes VRAM, RAM, and storage as one inference hierarchy.
  • You have a 128 GB RAM CPU-only desktop and can accept about 1.8 tok/s warm for GLM-5.2.
    The README “What it achieves” section reports about 1.8 tok/s warm on a 128 GB CPU-only desktop.
  • You have a single RTX 5070 Ti and want to test the GPU-resident pipeline for GLM-5.2.
    The README benchmark list reports 1.07 tok/s on a single RTX 5070 Ti laptop-class box.
  • You are researching MoE expert placement, storage I/O, or CPU/GPU overlap rather than requiring a fixed throughput SLA.
    The README lists model formats, memory hierarchy, storage I/O, placement, scheduling, kernels, and CPU/GPU overlap as research goals, and explicitly states that there is no speed SLA.

Skip it if you

  • Your deployment cannot hold the roughly 372 GB GLM-5.2 model, or cannot accept 0.05–0.1 tok/s cold on a 25 GB machine.
    The README “Get started” and “What it achieves” sections provide the 372 GB model size and the 25 GB dev-box benchmark.
  • You require a defined inference speed SLA instead of accepting that hardware changes where experts reside.
    The README explicitly says “no SLA on speed” before the “Faithful model, compressed state” discussion.
  • You only have an older per-row int4 GLM-5.2 mirror and do not want to obtain the gs64 container.
    The README warns that the older mirrors measure about 9pp worse and are associated with think-mode loops and never-terminating generations.
  • You plan to run Kimi K3 but do not have about 1.6 TB of disk space.
    The README says that Kimi K3 streams MXFP4 experts from the original checkpoint, but the snapshot is about 1.6 TB.

Requirements

  • You need “the program (a few hundred KB)” and “the model (372 GB)”.
  • For GLM-5.2, use the Hugging Face gs64 int4 container together with the int8 MTP head.
  • The runtime engine is pure C; Python is used only for the one-time converter and optional API gateway.
  • Inkling on a RAM-tight host needs the int4 dense container and a small expert cache; the default --cap 8 wants about 14 GB of additional cache.
  • Kimi K3 does not require conversion, but its original checkpoint snapshot is about 1.6 TB.

First step (verbatim from README)

./coli convert --model /nvme/glm52_i4     # download+convert shard by shard (python, one-time)

Watch out

  • Do not use the older per-row int4 mirror as the GLM-5.2 gs64 container.
    The README says the older mirrors are about 9pp worse and caused the original think-mode loops and never-terminating generations.
  • The MTP head must be int8 rather than int4; otherwise draft acceptance is 0%.
    The README “Get the model” section states int4 → 0% draft acceptance and provides an ls -l /out-mtp-* check.
  • Inkling’s default --cap 8 may require about 14 GB of additional cache on a RAM-tight host.
    The README model-specific note says the default --cap 8 wants about 14 GB of cache on top of the resident set.
  • Do not launch the .exe engine directly expecting it to load a model; it is not the launcher and exits immediately.
    The README Windows section states that the .exe files are the engines, not the launcher.

Alternatives

  • transformers：Use it when you primarily need the Python model stack described as the teacher-forcing oracle in the README, rather than pure-C disk-streamed expert inference.
    README “Faithful model, compressed state”; general domain knowledge

Not stated in the README

  • The README does not specify minimum operating-system, compiler, or CUDA versions for each platform.
  • The README does not provide a complete minimum VRAM, RAM, and disk table for every supported model.
  • The README does not describe authentication, concurrency control, or production security boundaries for coli serve or coli web.
  • The README does not provide normalized throughput, latency, or quality comparisons with common inference engines.
  • The README does not state the complete performance differences between Windows, Linux, and macOS.
  • The README does not explain additional restrictions on model weights, Hugging Face containers, or commercial use under Apache License 2.0.

💡 Deep Analysis

6
Yes I want to run the 975B Inkling model in int4 on a RAM-constrained machine. Should I choose Colibrì if the default expert cache adds about 14 GB?
For: A local inference user with tight RAM who wants to run the 975B Inkling model in int4

Yes, provided that you explicitly use a smaller expert cache; the README documents a dedicated Inkling path for RAM-constrained machines.

  • The project supports the 975B Inkling model through the same coli chat, coli serve, and coli web interfaces.
  • The README says RAM-tight hosts should use the int4 dense container and a small expert cache; the default --cap 8 requires about 14 GB of additional cache beyond the resident set.
  • It gives the direct command ./coli chat --model /nvme/inkling_i4 --cap 2, which matches this constraint better than the default cache.
  • A smaller cache may cause more expert reloads from disk; the project states that insufficient fast memory can reduce speed but must not silently change model semantics.
  • Supported models: “Inkling (975B)”
  • Per-model notes: “Inkling on a RAM-tight host needs the int4 dense container and a small expert cache”
  • Per-model notes: “the default --cap 8 wants ~14 GB of cache on top of the resident set”
  • Per-model command: “./coli chat --model /nvme/inkling_i4 --cap 2”
./coli chat --model /nvme/inkling_i4 --cap 2
Not stated in the README：The README does not give the exact RAM usage, cache hit rate, or tok/s for `--cap 2`.；It does not state the maximum context length supported by this configuration.
Yes I am studying GLM-5.2 int4, MLA-compressed KV, and DSA sparse attention, and require optimizations not to silently change routing or output semantics. Is Colibrì suitable as an experimental platform?
For: An inference-systems researcher who needs to verify that quantization, MLA, and DSA preserve model semantics

Yes, because Colibrì makes semantic fidelity and reproducible experimentation part of its stated boundary rather than optimizing only for tok/s.

  • The README says the default policy “never silently changes model precision or router semantics” and explicitly pairs this with no speed SLA.
  • The forward pass is checked against a transformers oracle using teacher forcing, typically reaching 30–32/32, which provides a comparison baseline for quantization and scheduling experiments.
  • MLA reduces the KV state from 32,768 floats per token to 576 and persists it across restarts; the README says this is byte-identical to an uninterrupted session.
  • DSA sparse attention can be forced into full-key selection and reproduce dense attention, making the project useful for studying end-to-end effects of model-specific optimizations.
  • README opening: “no SLA on speed, and a hard guarantee on semantics”
  • Faithful model, compressed state: “validated against a transformers oracle (teacher-forcing typically 30-32/32)”
  • Faithful model, compressed state: “576 floats/token instead of 32,768 (57× smaller)”
  • Faithful model, compressed state: “forcing full-key selection to reproduce dense attention exactly”
COLI_MODEL=/nvme/glm52_i4 ./coli doctor --deep  # strict tensors/shards/index/mirror preflight
Not stated in the README：The README does not provide oracle pass rates for every model, quantization granularity, and hardware target.；It does not explain how to automatically export a per-token router-difference report.
Yes I have six RTX 5090 GPUs and want GLM-5.2 to remain fully resident on the GPUs, accessed through a local Web or headless API. Is Colibrì suitable?
For: An administrator of a six-RTX-5090 workstation who wants to expose GLM-5.2 through a local Web/API interface

Yes for low-concurrency local Web/API inference, because the README reports full residency on six RTX 5090s; it is not a production service framework with an SLA.

  • “See it running” reports about 4 tok/s, 1.6-second TTFT, and zero disk activity for the 744B model on 6× RTX 5090.
  • coli web and coli serve are provided; serve starts the API and dashboard without opening a browser, which fits a headless workstation.
  • The unified launcher selects the model engine and chat template from config.json, so GLM-5.2 does not require a manually supplied template.
  • The README explicitly says “no SLA on speed”; authentication, throttling, auditing, and concurrency governance are not included guarantees.
  • See it running: “744B model at 4 tok/s, TTFT 1.6 s, disk 0 — full expert residency on 6× RTX 5090”
  • Get started: “./coli web --model /nvme/glm52_i4”
  • Get started: “./coli serve --model /nvme/glm52_i4”
  • README: “no SLA on speed”
COLI_MODEL=/nvme/glm52_i4 ./coli doctor --deep  # strict tensors/shards/index/mirror preflight
Not stated in the README：The README does not specify throughput, queue behavior, or context isolation under multiple concurrent users.；It does not provide authentication, authorization, TLS, or auditing features.
Yes I only have a 12-core CPU laptop, 25 GB RAM, and a sufficiently large SSD. Is Colibrì suitable for running GLM-5.2 int4 locally?
For: A developer with a 12-core CPU laptop and 25 GB RAM who wants to run the 744B GLM-5.2 locally

Yes, but only if you accept low throughput and substantial disk waiting; Colibrì is explicitly designed to make very large MoE models runnable on low-memory hardware.

  • The README says the project began on “a 12-core laptop with 25 GB of RAM” and treats storage, RAM, and VRAM as one inference hierarchy.
  • The GLM-5.2 int4 model is about 372 GB, so SSD capacity and sustained read performance are the main constraints rather than fitting all weights in RAM.
  • The README distinguishes cold and warm execution; the 25 GB baseline is only about 0.05–0.1 tok/s when cold.
  • The default policy does not silently change precision or router semantics, making this better for local feasibility testing, low concurrency, and privacy-sensitive use than for fast serving.
  • Why “colibrì”: “a 12-core laptop with 25 GB of RAM”
  • Get started: “the model (372 GB)”
  • Usage limitations / measured findings: approximately 0.05–0.1 tok/s cold on a 25 GB device
  • README: “no SLA on speed, and a hard guarantee on semantics”
COLI_MODEL=/nvme/glm52_i4 ./coli plan     # inspect the planned VRAM/RAM/disk placement
Not stated in the README：The README does not specify the laptop SSD's sequential read speed, filesystem, or actual warm decode rate.；It does not state the stable maximum context length for GLM-5.2 with 25 GB RAM.
It depends I plan to use the roughly 1.6 TB Kimi K3 snapshot directly and do not want to run a conversion pipeline. Can Colibrì meet that constraint?
For: A local-model researcher with about 1.6 TB of NVMe storage who wants to run the 2.8T Kimi K3 without model conversion

It depends: the model format fits Colibrì’s direct-streaming path, but roughly 1.6 TB alone does not prove that the machine can run it reliably; NVMe headroom, read bandwidth, and available memory still matter.

  • The README states that Kimi K3 streams MXFP4 experts from the original checkpoint, so conversion is not required.
  • The same section gives a snapshot size of about 1.6 TB; indexes, caches, logs, and runtime files need additional space.
  • Colibrì treats storage, RAM, and VRAM as a multitier hierarchy, so Kimi K3 does not need every parameter in fast memory, but disk performance becomes critical.
  • The project provides a dedicated build target, make -C c kimi_k3, yet gives no Kimi K3 tok/s or minimum RAM/VRAM figures.
  • Get started / per-model notes: “Kimi K3 streams its MXFP4 experts from the original checkpoint, so there is nothing to convert”
  • Get started / per-model notes: “the snapshot is ~1.6 TB”
  • README opening: “744B to 2.8T parameters”
  • Model build examples: “make -C c kimi_k3”
Not stated in the README：The README does not specify Kimi K3's minimum RAM, VRAM, SSD read speed, or startup time.；It does not state whether the original checkpoint requires a particular directory layout or additional index files.
It depends I am preparing to run the vision-capable 321B GLM-5.3-Flash on Windows and want to launch it directly from the command line. Should I use Colibrì?
For: A local application developer on Windows who wants to run the 321B vision-capable GLM-5.3-Flash

It depends: the Windows launch path and model-family support are clear, but the README does not sufficiently document GLM-5.3-Flash vision input or its hardware requirements.

  • The README lists GLM-5.3-Flash as a supported 321B vision model and says all models share coli chat, coli serve, and coli web entry points.
  • Windows release archives include coli.cmd; the README specifically warns not to launch the .exe directly because the engine does not automatically load a model.
  • The Windows example uses coli.cmd chat --model D:\glm52_i4, so both the launcher and model-path pattern are documented.
  • The runtime core is pure C, with Python used only for one-time conversion and the optional API gateway; however, the README does not say whether vision input is fully exposed through chat, web, or the API.
  • Supported models: “GLM-5.3-Flash (321B, with vision)”
  • Get started: “On Windows a release archive ships coli.cmd”
  • Get started: “The .exe files are the engines, not the launcher”
  • Get started: “coli.cmd chat --model D:\glm52_i4”
coli.cmd chat --model D:\glm52_i4
Not stated in the README：The README does not give GLM-5.3-Flash's download size, minimum RAM/VRAM, or Windows performance baseline.；It does not specify vision input formats, API fields, or currently supported image tasks.

✨ Highlights

  • Pure C with zero engine dependencies, supporting 744B–2.8T MoE models
  • The same coli chat command covers eight model families
  • 6× RTX 5090 achieves 5.8–6.8 tok/s in the reported benchmark
  • 744B GLM-5.2 uses VRAM, RAM, and disk as a multitier hierarchy
  • The GLM-5.2 int4 container is about 372 GB, creating a high storage barrier

🔧 Engineering

  • Run eight model families, including GLM-5.2, through coli chat, serve, and web
  • Build an inference hierarchy from VRAM, RAM, and disk while streaming MoE experts
  • MLA reduces KV state to 576 floats per token and persists it in .coli_kv
  • coli plan, doctor, and tune respectively inspect, plan, and measure the machine configuration

⚠️ Risks

  • GLM-5.2 is about 372 GB, while a 25 GB machine reaches only 0.05–0.1 tok/s cold
  • The project explicitly provides no speed SLA, and insufficient storage-tier memory reduces speed
  • Older per-row int4 GLM-5.2 mirrors measure about 9pp worse and may trigger loops
  • Kimi K3 needs no conversion but its snapshot is about 1.6 TB, requiring more disk

👥 For who?

  • Developers who want to run GLM-5.2 on consumer GPUs or CPU-only machines
  • Local inference users with 128 GB RAM or a single RTX 5070 Ti
  • Engineers researching storage, scheduling, memory tiers, and CPU/GPU overlap
  • C/CUDA developers who accept Python only for conversion or the optional API gateway