TensorFold: Run language models directly on Apple Silicon and NVIDIA GPUs
A Zig-based local inference engine for Qwen3.5-2B and other language models on Apple Silicon and NVIDIA GPUs, without Python or MLX.
GitHub ashhart/TensorFold Updated 2026-11-13 Branch main Stars 1.9K Forks 391
Zig LLM inference Metal/CUDA Qwen3.5-2B Apple Silicon NVIDIA GPU

🧭 Decision Guide

Why trending now: Cannot be determined from the provided material

Try it if you

  • You want to serve the Qwen3.5-2B 4-bit model with Metal on an Apple Silicon Mac.
    The Quick start / Mac with Apple Silicon section says Qwen3.5-2B was checked on Metal and GB10.
  • You need a local LLM server process without installing Python or MLX.
    The README explicitly says serving needs no Python or MLX installation.
  • You want to test the writable-model-weights experiment for Nemotron 3.5 Lightning.
    The Living Weights (experimental) section provides --slide, /v1/slide/learn, and slide rollback.
  • You care about deterministic concurrent draft inference and use Flash Next, Metal, or Qwen3.8-27B.
    The Exact decoding section defines consistency for accepted drafts, concurrent streams, and shared lane rounds.

Skip it if you

  • You need Living Weights to act as a stable long-term memory system, because the experimental section says recall drops past a few dozen facts.
    The Living Weights (experimental) section and Known issues.
  • You require identical outputs across different quantizations or Metal/CUDA backends.
    Exact decoding explicitly says different quantizations and different backends can produce different outputs.
  • You need production mobile support, because app/mobile/ is still marked preview software.
    The Mac app section.
  • You want /v1/models context_length alone to guarantee that every request fits in memory.
    The Model discovery section says advertised context is not a guarantee that every workload fits.

Requirements

  • An Apple Silicon Mac or NVIDIA GPU; the README covers Metal, CUDA, and Vulkan, while Quick start verifies Apple Silicon and GB10.
  • For Qwen3.5-2B, download the pinned revision first; Quick start provides the corresponding tensorfold pull command.
  • Living Weights requires Nemotron 3.5 Lightning, --slide, and a slide key; the README says --slide starts only with a key.
  • Model weights have separate licenses; Quick start says to review the model weights' own licenses.
  • Serving uses a Zig binary and does not require Python or MLX installation.

First step (verbatim from README)

brew install ashhart/tensorfold/tensorfold

Watch out

  • Living Weights must use a model copy, otherwise lessons rewrite checkpoint files.
    The Living Weights command comment says always teach a copy.
  • On APFS, cp -cRL shares the original data until lessons change it.
    The Living Weights section describes the APFS cp -cRL clone behavior.
  • Context compaction is off by default; enabling --compact-at turns older conversation into a memory note.
    The Context compaction section.
  • The Metal Flash Next concurrent path requires explicit --parallel N, and N sessions must fit the hardware.
    The Exact decoding section describes shared target rounds and --parallel N.

Not stated in the README

  • The README excerpt does not provide installation commands or support scope for Linux with an NVIDIA GPU.
  • The README excerpt does not provide throughput, latency, or memory figures for different models, quantizations, and GPUs.
  • The README excerpt does not describe Vulkan installation steps, supported GPU models, or validation results.
  • The README excerpt does not include the complete API request format; the Send a request section is omitted.
  • The README excerpt does not state the required Zig, TypeScript, Python, or CUDA versions.
  • The README excerpt does not describe the activity level or future maintenance plans of the 10 contributors.

💡 Deep Analysis

6
No I need a unified OpenAI/Anthropic-compatible endpoint across Macs, Linux GPU servers, and Windows machines, with OIDC or SAML, SCIM, model authorization, budgets, and token accounting. Is deploying only the TensorFold core repository sufficient?
For: An enterprise platform owner who requires OIDC/SAML, SCIM, node scheduling, budgets, and auditing

No, not with the TensorFold core repository alone. The README explicitly places enterprise identity, node orchestration, and budget governance in the separate TensorFold Starlight project.

  • Starlight provides the governed OpenAI/Anthropic-compatible endpoint and OIDC, SAML, and SCIM integration.
  • It decides who may use which model and where it runs, and accounts for every token.
  • It manages Macs, Linux GPU servers, and Windows machines, but lives in a separate repository and pins the TensorFold release it tests.
  • TensorFold can serve as the controlled inference node, but it cannot alone satisfy the stated governance boundary. Starlight is also described as new, with some qualification gates requiring real GPU clusters or identity-provider tenants.
  • TensorFold Starlight (enterprise): "one governed OpenAI- and Anthropic-compatible endpoint"
  • TensorFold Starlight: OIDC, SAML, SCIM, model authorization, node placement, and token accounting
  • TensorFold Starlight: supports Macs, Linux GPU servers, and Windows machines
  • TensorFold Starlight: "Starlight lives in its own repository"
Not stated in the README：The excerpt does not specify Starlight's current version, production SLA, tenant-isolation details, or audit-log retention policy.；It does not state whether the TensorFold core itself provides authentication, TLS, rate limiting, or multi-tenancy isolation.
Yes I use an Apple Silicon Mac, only need the README-verified Qwen3.5-2B 4-bit model, and do not want to maintain a Python or MLX environment. Is TensorFold suitable?
For: An individual developer who wants to run a quantized model offline on an Apple Silicon Mac without installing Python or MLX

Yes. This is exactly the path validated in the README Quick start, and serving does not require installing Python or MLX.

  • TensorFold is a Zig binary that reads checkpoints and runs Metal kernels directly on Apple Silicon.
  • The README identifies Qwen3.5-2B-MLX-4bit as a small 4-bit checkpoint checked on Metal and NVIDIA GB10.
  • pull uses a pinned revision, while serve can set an 8192 context and disable thinking.
  • The model weights have their own license, so TensorFold’s Apache License 2.0 does not determine whether the weights may be used or redistributed.
  • Quick start: "Start with Qwen3.5-2B, the small 4-bit checkpoint checked on Metal and a GB10."
  • Project description: "Serving needs no Python or MLX installation."
  • Mac with Apple Silicon: the pinned `pull` command and `tensorfold serve ... --context 8192 --no-thinking` command
  • Project data: main_language is Zig; license is Apache License 2.0
brew install ashhart/tensorfold/tensorfold
Not stated in the README：The provided README excerpt does not give actual speed or peak memory for each Apple Silicon chip and memory size.；It does not state whether all Apple Silicon models provide the same Metal performance.
No I run Nemotron 3.5 Lightning 30B-A3B 4-bit on Apple Silicon or an NVIDIA GB10 and want to write a small set of facts into the model with rollback. Is Living Weights suitable for a production knowledge base?
For: An Apple Silicon or NVIDIA GB10 user who wants Nemotron 3.5 Lightning to learn dozens of facts through experimental checkpoint writes

No. Living Weights should not be treated as a production knowledge base because the README labels it experimental and reports fact interference, recall degradation after a few dozen facts, and additional effects when thinking is enabled.

  • --slide writes facts that pass lesson checks into checkpoint files rather than into a separate, standard knowledge store.
  • The README says related facts can disturb one another, recall drops past a few dozen facts, and learned facts reduce decode speed on a Mac.
  • Users must copy the model directory and create a slide key; the server keeps a hidden snapshot and slide rollback can undo lessons.
  • It runs on Apple-silicon Macs and NVIDIA GB10, and a learned folder can serve on either, but portability does not remove the reliability limitations of this experimental feature.
  • Living Weights: the section is explicitly marked experimental
  • Living Weights: "related facts can disturb each other, recall drops past a few dozen facts"
  • Living Weights: learned facts cost some decode speed on a Mac
  • Living Weights: it runs on Apple-silicon Macs and NVIDIA GB10 and provides rollback
tensorfold pull TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit
Not stated in the README：The excerpt gives no exact recall rates by fact type, acceptable fact count, or rollback duration.；It does not specify recovery guarantees for failed checkpoint writes, disk corruption, or process interruption.
It depends I integrate multiple Hugging Face checkpoints and do not want to start a model merely because its repository name looks compatible. I need to inspect model_type, family, format, context, and memory floor first. Does TensorFold fit this workflow?
For: A model integration engineer who needs to load Hugging Face checkpoints but wants to verify family, format, context, and memory floor first

It depends. TensorFold provides a useful inspection and discovery workflow for supported models, but it is not presented as a universal executor for arbitrary Hugging Face checkpoints.

  • pull verifies downloaded files and refuses a checkpoint whose model_type has no native family, exposing some compatibility problems before loading.
  • info shows a checkpoint’s family, format, context, and memory floor, while models --json emits repository, family, and model type for programs.
  • serve REPO relies on a known checkpoint in the cache, so a repository name alone does not prove compatibility.
  • The workflow fits models belonging to a supported native family; the README does not promise automatic support for new or unadapted models.
  • Models on disk: `pull` refuses a `model_type` with no native family
  • Models on disk: `info` shows family, format, context, and memory floor
  • Models on disk: `models --json` outputs repository, family, and model type
  • Models on disk: `tensorfold serve REPO` serves a cached checkpoint
tensorfold info TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit
Not stated in the README：The excerpt does not list all supported model families, model types, or quantization formats.；It does not specify whether a compatible checkpoint is fully rejected, partially usable, or merely missing specific features.
Yes I have an NVIDIA GB10 (DGX Spark), want to run the README-verified Qwen3.5-2B 4-bit model, and expose it to a local application over HTTP. Is TensorFold suitable?
For: A GPU engineer with an NVIDIA GB10 (DGX Spark) who wants to run models as a native local service

Yes. The README explicitly lists NVIDIA GB10 alongside Apple Silicon as a checked platform for the Qwen3.5-2B 4-bit model, and serve starts an HTTP model service.

  • TensorFold is a native Zig binary that calls CUDA kernels directly and does not require Python or MLX.
  • The same checkpoint can be downloaded with pull and served by repository name; the service exposes discovery endpoints such as /v1/models.
  • Being checked on GB10 does not imply identical compatibility or performance on every NVIDIA GPU, and the excerpt gives no CUDA version, peak-memory, or throughput figures.
  • For a fleet spanning Macs, Linux GPU servers, and Windows machines, governance belongs to the separate Starlight project rather than the core engine.
  • Quick start: Qwen3.5-2B is a 4-bit checkpoint checked on Metal and GB10
  • Project description: "runs Metal or CUDA kernels directly"
  • Model discovery: `GET /v1/models` is supported, along with `/models`
  • TensorFold Starlight: a separate repository governs Macs, Linux GPU servers, and Windows machines
tensorfold pull mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f69842a46bc13dbdc0f19e291392a3
Not stated in the README：The excerpt provides no GB10 throughput, time-to-first-token, memory usage, or concurrency limit.；It does not specify the required NVIDIA driver or CUDA version for TensorFold v1.2.7.
It depends I am building an OpenAI-style coding assistant, and the client must read `context_length`, `max_model_len`, and `reasoning` from `/v1/models`. Can TensorFold meet this discovery requirement?
For: An application developer building an OpenAI-compatible coding assistant that needs model context-window and reasoning metadata discovery

It depends. TensorFold can expose these discovery fields, but the client must still handle per-request memory refusals; an advertised context length is not a guarantee that every request will fit.

  • /v1/models advertises served names and aliases, and finite context windows include context_length and max_model_len.
  • Thinking models may include reasoning.default_enabled, default_effort, and supported_efforts, which is useful for exposing reasoning controls.
  • The README defines the context value as the effective prompt-plus-reply limit after startup memory fitting, while individual workloads may still be refused.
  • The fit is therefore strong for OpenAI-style discovery, but not for an assumption that every model has complete reasoning metadata.
  • Model discovery: `GET /v1/models` is supported, along with `/models`
  • Model discovery: "each entry includes equal `context_length` and `max_model_len` integer extensions"
  • Model discovery: thinking models carry `reasoning` metadata
  • Model discovery: "clients should still handle per-request memory refusals"
tensorfold models --json
Not stated in the README：The excerpt does not provide a complete compatibility matrix for OpenAI chat, Responses, and streaming request fields.；It does not specify the exact trigger conditions or error format for context compaction in OpenAI-compatible requests.

✨ Highlights

  • A Zig binary runs inference directly with Metal or CUDA
  • Serving requires no Python or MLX installation
  • Qwen3.5-2B 4-bit provides a Metal-checked starting path
  • Exact decoding makes accepted drafts equal native engine results
  • Experimental Living Weights can reduce recall and Mac decode speed

🔧 Engineering

  • tensorfold serve serves models by Hugging Face repository name
  • tensorfold pull checks model files with sha256 and git blob sha1
  • Living Weights uses --slide to write facts into checkpoint files
  • Context compaction uses --compact-at for conversations nearing overflow

⚠️ Risks

  • Living Weights rewrites the model folder and relies on slide rollback for recovery
  • Living Weights facts can interfere, with recall dropping past a few dozen
  • Different quantizations and backends can produce different outputs
  • The advertised model context does not guarantee memory for every request

👥 For who?

  • Developers who need local LLM serving on Apple Silicon Macs
  • Teams testing Metal/CUDA inference on NVIDIA GPUs or GB10
  • Engineers serving Qwen3.5-2B or Nemotron locally