DeepSeek-V4 Flash: A 1M-context vLLM recipe for two DGX Sparks
A vLLM recipe for DeepSeek-V4 Flash on two DGX Sparks, combining 1M context with DSpark speculative decoding.
GitHub MiaAI-Lab/DeepSeek-v4-Flash-DSpark-2x-DGX-Spark Updated 2026-09-09 Branch main Stars 1.3K Forks 179
Python Shell vLLM DeepSeek-V4-Flash-Vision-Exp DGX Spark RoCE/NCCL Docker

🧭 Decision Guide

Why trending now: Cannot be determined from the provided material

Try it if you

  • You have two DGX Sparks and need a 1M-token service for DeepSeek-V4-Flash-Vision-Exp.
    The Quick start requires two DGX Sparks, and the README states that max_model_len is 1,048,576.
  • Your cluster already has working RoCE/NCCL, a ConnectX network, and passwordless SSH.
    The Quick start requires working RoCE/NCCL; the Optional: three Sparks section requires passwordless SSH and a ConnectX link.
  • You need OpenAI image_url/path input rather than video encoding.
    The README says Anemll 0.1.1 supports OpenAI image_url/path and explicitly has no video encoder.
  • You want 16 slots and about 200 tok/s aggregate on three DGX Sparks.
    The Optional: three Sparks (TP=3) section specifies TP3_MAX_NUM_SEQS=16 and about 200 tok/s aggregate at 16 streams.

Skip it if you

  • You do not have two DGX Sparks or cannot provide a working RoCE/NCCL fabric.
    The Quick start explicitly requires two DGX Sparks and working RoCE/NCCL.
  • You need the official weights to process video or full animated GIFs.
    The README states that the official weights have no video encoder and that GIF is a still frame.
  • Your main workload is single-user 128K–256K long context and you plan to use TP=3.
    The TP=3 section reports about 22% prefilling overhead in this range and says single-user long-context work is better served by the 2-node lane.
  • You can run only in a CPU environment but need to validate real tok/s.
    The Quick start marks scripts/ci-validate.sh as no GPU and unable to measure tok/s; live tok/s requires a 2× Spark pair.

Requirements

  • Two DGX Sparks; the README Quick start explicitly requires “two DGX Sparks”.
  • Working RoCE/NCCL; the README Quick start lists it as a startup requirement.
  • The same image on both nodes: ghcr.io/anemll/dspark-vllm-gx10:0.1.1.
  • Passwordless SSH from the head to the worker; this is explicitly listed in the Optional: three Sparks section.
  • The DSPARK_VLLM_IMAGE must be pulled on the worker, with the README citing approximately 19 GB.
  • Configure WORKER_HOST, MASTER_ADDR, NCCL_IB_HCA, NCCL_SOCKET_IFNAME, VLLM_HOST_IP, WORKER_VLLM_HOST_IP, and HF_CACHE.
  • The default runtime uses Anemll 0.1.1 with /usr/local/bin/vllm serve, TP=2, mp, and nnodes 2.

First step (verbatim from README)

cp .env.dspark.example .env.dspark

Watch out

  • Start the worker before the head; the README explicitly specifies this order.
    Step 5 of Quick start says “Start (worker first, then head)”.
  • The default two-node setup downloads a checkpoint to the worker; set DSPARK_WORKER_HF_NFS=1 to use NFS instead.
    The Weights on the head and NFS notes in Quick start.
  • After a reboot, if dockerd restored the ranks, start exits with code 3 and you should not immediately run stop.
    The Start paragraph in Quick start describes restart: unless-stopped and exit 3.
  • Six 1M requests cannot all remain resident; the KV cache table shows 6.0M exceeds the shared pool and extra requests queue.
    The 6 × 1M = 6.0M impossible example in How the KV cache works.
  • Do not manually set DSPARK_MODEL or GPU_MEMORY_UTILIZATION.
    The .env.dspark switches section explicitly says not to set them by hand.
  • At 6×6, max-cudagraph-capture-size should be 48; a plain 42 truncates to 40 and costs 12%.
    The Runtime flags section, measured on 2026-09-02.
  • The first TP=3 startup compiles for several minutes, and every rank must show DSv4 TP pad: heads 64 -> 72.
    The first-boot notes in Optional: three Sparks (TP=3).

Alternatives

  • Two-node TP=2 lane：It is better for single-user long-context workloads, especially 128K–256K prefilling.
    Optional: three Sparks (TP=3)
  • Three-node TP=3 lane：It is better when you need 16 slots, about 5M cached tokens, or about 200 tok/s aggregate at 16 streams.
    Optional: three Sparks (TP=3)

Not stated in the README

  • The README does not provide the exact DGX Spark hardware model, memory capacity, or CUDA driver version.
  • The README does not provide complete TP=2 tok/s, time-to-first-token, or per-context-length benchmarks; it only links results/RESULTS-2026-08-14.md.
  • The README does not state the release date or image digest for ghcr.io/anemll/dspark-vllm-gx10:0.1.1.
  • The README does not describe production authentication, TLS, access control, or multi-tenant isolation.
  • The README does not state the full checkpoint download time, disk-space requirement, or required network bandwidth.
  • The project has no published releases: metadata shows 0 releases and “No releases”; formal compatibility boundaries are unknown.
  • The README does not establish a causal link between the 2026-09-09 Trending appearance and specific user growth, commit activity, or an external event.

💡 Deep Analysis

6
No I mainly work with single-machine Python and ordinary Docker deployments and have not maintained NCCL, RoCE, head/worker roles, or multi-node vLLM. Is this project suitable for me to take over directly?
For: An application developer familiar mainly with single-machine Python or ordinary Docker who wants to deploy DeepSeek-V4-Flash-Vision-Exp across two DGX Spark systems

No, not as a direct handoff: the project treats networking, containers, model caches, and distributed parallelism as prerequisites, so its learning curve is substantially higher than a normal single-node service.

  • The Quick start requires two DGX Spark systems, working RoCE/NCCL, the same image on both machines, and coordination from the head node.
  • .env.dspark must configure at least WORKER_HOST, MASTER_ADDR, NCCL_IB_HCA, NCCL_SOCKET_IFNAME, matching TP/Gloo interface names, service IPs, and Hugging Face cache paths.
  • The project insights list common failures including interface or GID mismatches, inconsistent worker images, incorrect cache paths, premature restarts during JIT compilation, and earlyoom killing vLLM.
  • CPU validation only checks configuration; it does not prove that the multi-node GPU service works. Real startup also involves weight preparation, Docker rank recovery, and potentially lengthy JIT compilation.
  • README Quick start: `You need two DGX Sparks, RoCE/NCCL working, and the same image on both`
  • README Env: `WORKER_HOST`, `MASTER_ADDR`, `NCCL_IB_HCA`, `NCCL_SOCKET_IFNAME`, and `TP_` / `GLOO_` interface names
  • Project insight: the learning curve is high and the project suits users experienced with Linux, Docker, SSH, multi-node GPUs, NCCL/RoCE, and vLLM
  • Project insight: common failures include network mismatches, inconsistent images or cache paths, premature JIT restarts, and earlyoom
bash scripts/ci-validate.sh
Not stated in the README：The README does not say whether an automated installer, graphical configuration interface, or managed deployment path exists for single-machine Python or ordinary Docker users.；It does not describe a locally equivalent alternative when DGX Spark and RoCE/NCCL are unavailable.
Yes I have two DGX Spark systems and plan to deploy DeepSeek-V4-Flash-Vision-Exp with vLLM TP=2 over RoCE/NCCL. Can this project serve directly as my private 1M-token inference deployment?
For: An AI infrastructure engineer maintaining two NVIDIA DGX Spark systems with RoCE/NCCL configured, who wants to privately serve DeepSeek-V4-Flash-Vision-Exp with a 1M-token context ceiling

Yes, it is a good fit because it is specifically a two-DGX-Spark TP=2 deployment recipe combining a 1M-token ceiling, NVFP4 MLA KV cache, and multi-node startup procedures.

  • The Quick start requires two DGX Spark systems, working RoCE/NCCL, and the same image on both nodes; the default image is ghcr.io/anemll/dspark-vllm-gx10:0.1.1.
  • The default runtime uses TP=2, nnodes 2, --kv-cache-dtype nvfp4_ds_mla, and --max-model-len 1048576.
  • The startup baseline reports 17.04 GiB of KV cache, 2,331,430 cached tokens, and about 2.22x concurrency for full 1M-token requests. This does not mean unlimited simultaneous 1M requests.
  • The service exposes an OpenAI-style endpoint at http://HEAD_NODE_IP:8888/v1, while authentication, TLS, and rate limiting are outside the project.
  • README: Two-node DGX Spark recipe for `deepseek-ai/DeepSeek-V4-Flash-Vision-Exp`
  • README: `vLLM TP=2`, `1M-token ceiling`, and `nvfp4_ds_mla` KV
  • README: `Available KV cache memory: 17.04 GiB`; `Maximum concurrency ... 2.22x`
  • Project insight: TP=2 distributed inference startup for two DGX Spark systems
cp .env.dspark.example .env.dspark
Not stated in the README：The README does not establish whether your exact RoCE interface, GID, driver version, and network throughput meet your latency target.；It does not promise time-to-first-token or sustained throughput for 1M-token requests.
It depends I plan to let multiple users submit 200K- to 1M-token documents concurrently while keeping the default `max_num_seqs=6`. Can this project meet that concurrency requirement?
For: A research platform engineer using the default `max_num_seqs=6` who needs to handle multiple long-document requests through an internal OpenAI API service

It depends: the project can handle limited long-context concurrency, but the default configuration should not be interpreted as six simultaneous full 1M-token requests.

  • The README reports a KV pool of 2,331,430 tokens. max_model_len is a per-request ceiling of 1,048,576, while max_num_seqs=6 is the maximum number of active sequences, not a guarantee that six full-length requests fit.
  • Its examples show 6x200K = 1.2M fitting in the pool, while 6x500K = 3.0M is near or beyond capacity; 6x1M = 6.0M is impossible and extra requests queue.
  • The startup log reports 2.22x concurrency for full 1M requests, indicating a small number of very long requests rather than high-volume concurrency.
  • The project insights also state that the default max_num_seqs is 6 and excessive concurrency can queue or exhaust the KV pool. The HTTP service does not include production-grade rate limiting or failover.
  • README How the KV cache works: `KV pool ... 2,331,430 tokens`; `max_num_seqs ... 6`
  • README examples: `6 × 200k = 1.2M fits`; `6 × 1M = 6.0M impossible — extras queue`
  • README: `Maximum concurrency for 1,048,576 tokens per request: 2.22x`
  • Project insight: the default `max_num_seqs` is 6, and excessive concurrency can queue or exhaust the KV pool
curl -fsS http://127.0.0.1:8888/v1/models
Not stated in the README：The README does not provide queueing latency for your actual input-length distribution, output lengths, or mixed multi-user load.；It does not define the server queue depth, request-timeout policy, or client-side rate-limiting mechanism.
Yes I need to submit long documents and images through an OpenAI-compatible interface using `image_url` or a local `path`, but I do not need video understanding. Is this project suitable as an internal vision inference backend?
For: An enterprise or research team integrating long documents, codebases, and research material through a private OpenAI-compatible API, with single-image input but no video-understanding requirement

Yes, provided that your vision requirement is limited to single-image input; the project explicitly supports native image calls but does not include an official video encoder.

  • The README says Vision-Exp image support is connected through a startup hotfix using the checkpoint’s ViT and Aligner, with OpenAI-compatible image_url and path forms.
  • The former Qwen3-VL sidecar and MCP path have been removed, so the deployment does not require an additional vision service.
  • The README explicitly states that the official weights have no video encoder and that GIFs are treated as still frames. Multi-frame or complex video analysis should not be assumed.
  • The API is exposed at http://HEAD_NODE_IP:8888/v1, with a default max_model_len of 1,048,576; production authentication, TLS, auditing, and rate limiting must be added separately.
  • README: `Native image support ... OpenAI image_url / path`
  • README: `There is no video encoder in the official weights; GIF is a still frame`
  • README: `The old Qwen3-VL sidecar / MCP path is removed`
  • README: `API: http://HEAD_NODE_IP:8888/v1`
curl -fsS http://127.0.0.1:8888/v1/models
Not stated in the README：The README does not specify image resolution limits, image-size limits, or the additional memory and latency cost of visual inputs.；It does not provide a complete compatibility matrix for multi-image requests or image-plus-very-long-text requests.
It depends I do not want to store the 157 GiB checkpoint on both DGX Spark systems. Can I use this project's `DSPARK_WORKER_HF_NFS=1` so the worker reuses the head node's model cache over NFS?
For: An infrastructure engineer with two DGX Spark systems who must manage very large model weights locally or through NFS, while maintaining Docker, SSH, and cache consistency across nodes

It depends: the project explicitly supports shared weights over NFS, but the worker then depends on the head node’s NFS service, ConnectX link, and correct mount configuration, making it less self-contained than keeping a local copy on each node.

  • The Quick start states that DSPARK_WORKER_HF_NFS=0 downloads the checkpoint to the worker; with 1, the worker mounts the head cache through NFSv4 over the ConnectX link.
  • NFS mode still requires WORKER_HF_CACHE, which is described as the worker’s local JIT overlay rather than a complete removal of worker-side cache configuration.
  • Both nodes must have the same Docker image. If checkout paths differ, the README requires WORKER_DIR, WORKER_SCRIPT_DIR, and cache paths to be configured in .env.dspark.
  • Preparation forces online Hugging Face access; only after the cache is complete can serving switch to HF_HUB_OFFLINE=1. NFS instability, permissions, or mount failures can prevent worker loading.
  • README Quick start: `Set DSPARK_WORKER_HF_NFS=1 to keep weights only on the head`
  • README Quick start: `the worker mounts that cache over NFSv4 on the ConnectX link`
  • README Quick start: `Default DSPARK_WORKER_HF_NFS=0 also downloads onto the worker`
  • Project insight: weights may be stored independently or shared from the head node over NFS
cp .env.dspark.example .env.dspark
Not stated in the README：The README does not provide your NFS throughput, mount options, permission setup, or tolerance to network jitter.；It does not quantify how NFS mode changes startup time or inference performance compared with local copies.
Yes I want to evaluate `nvfp4_ds_mla`, the default six speculative tokens, and the 1M context on two DGX Spark systems, then compare TP=3. Is this repository suitable as an experimental baseline?
For: A model engineer evaluating DeepSeek MLA NVFP4 KV caching and DSpark speculative decoding while comparing two-node TP=2 with three-node TP=3

Yes, it is suitable as a hardware- and runtime-specific experimental baseline, but not as evidence for generic vLLM or generic GPU behavior.

  • The default runtime explicitly uses --kv-cache-dtype nvfp4_ds_mla, paged KV cache, chunked prefill, asynchronous scheduling, and the FlashInfer MoE backend.
  • DSpark speculative decoding is configured with num_speculative_tokens: 6, giving experiments a defined starting point.
  • TP=3 uses a separate ./start-tp3.sh launcher and requires WORKER2_HOST. The README says the third node can increase total KV capacity and multi-stream throughput, while adding communication, padding, and prefill overhead.
  • CI is CPU-only through scripts/ci-validate.sh; it cannot validate real GPU tok/s, cross-node communication, vision input, or long-context stability. Those measurements require the actual two- or three-Spark setup.
  • README Runtime flags: `--kv-cache-dtype nvfp4_ds_mla`, `--enable-chunked-prefill`, and `--async-scheduling`
  • README Runtime flags: `num_speculative_tokens: 6`
  • README: `Optional three Sparks (TP=3)` and `./start-tp3.sh`
  • README: CI is `CPU-only`; live tok/s still needs the 2x Spark pair
  • Project insight: TP=3 increases KV capacity and multi-stream concurrency but adds communication and padding overhead
bash scripts/ci-validate.sh
Not stated in the README：The supplied README excerpt does not provide a complete TP=2 versus TP=3 table for time-to-first-token, prefill throughput, and decode tok/s.；It does not state speculative-decoding acceptance rates across output lengths, thinking modes, and vision requests.

✨ Highlights

  • vLLM TP=2 supports a 1M-token context for DeepSeek-V4-Flash
  • The nvfp4_ds_mla KV cache pool reaches 2,331,430 tokens
  • DSpark provides 6-token speculative decoding
  • Anemll 0.1.1 startup hotfix supports image_url and path
  • TP=3 reaches about 200 tok/s but slows long-context prefilling

🔧 Engineering

  • start-deepseek-v4-flash-dspark.sh launches the two-node vLLM service
  • It provides an OpenAI image_url interface for DeepSeek-V4-Flash-Vision-Exp
  • The defaults are max_model_len 1048576 and max_num_seqs 6
  • start-tp3.sh can extend the deployment to three DGX Sparks

⚠️ Risks

  • It requires two DGX Sparks, working RoCE/NCCL, and the same image on both nodes
  • The default two-node setup prepares a worker checkpoint, potentially creating a second weight copy
  • TP=3 is not preferred for single-user long context; 128K–256K prefilling is about 22% slower
  • The official weights have no video encoder, so GIF is handled as a still frame
  • scripts/ci-validate.sh is CPU-only and cannot measure tok/s

👥 For who?

  • vLLM engineers with two DGX Sparks and the ability to configure RoCE/NCCL
  • Deployment teams needing DeepSeek-V4-Flash 1M context and image_url
  • DGX Spark users wanting TP=3 and 16 slots for multi-stream requests