MiniMind: Train a 64M Language Model from Scratch in 2 Hours
A native-PyTorch training project for LLM learners that reproduces an approximately 64M model in 2 hours instead of only doing LoRA fine-tuning.
GitHub jingyaogong/minimind Updated 2026-09-04 Branch master Stars 58.8K Forks 7.6K
PyTorch LLM training MiniMind-3 NVIDIA RTX 3090

🧭 Decision Guide

Try it if you

  • You want to understand low-level LLM training code from Pretrain and SFT through DPO.
    The README “项目介绍” and overview list native PyTorch implementation and the full Pretrain, SFT, and DPO process.
  • You have one NVIDIA RTX 3090 and want to reproduce the 64M-scale MiniMind-3 experiment.
    The README note says “2 hours” is the measured time for 1 SFT epoch on one NVIDIA 3090.
  • You need to connect your own model to FastGPT, OpenWebUI, or Dify.
    The README section “基于 MiniMind 的 API 服务接口” states that scripts/serve_openai_api.py is OpenAI API-compatible.

Skip it if you

  • You need a production model validated by strict benchmarks rather than a 64M experimental model.
    The README explicitly states that the model comparison is a “non-strict benchmark” with limited samples and subjectivity.
  • You depend on stably maintained standalone weights for DPO, PPO, GRPO, or LoRA.
    The README says weights for DPO, PPO, GRPO, Agent, and LoRA will not all be continuously maintained and separately published.
  • Your environment does not match the README-listed combination of Python 3.10.16, CUDA 12.2, and Ubuntu 20.04.
    The README “快速开始” lists that software and hardware configuration for reference but does not document compatibility with other environments.

Requirements

  • README reference environment: Ubuntu==20.04, CUDA==12.2, and Python==3.10.16.
  • README hardware: NVIDIA GeForce RTX 3090 (24GB) * 8; the “2 hours” figure corresponds to 1 SFT epoch on one RTX 3090.
  • Dependency installation uses requirements.txt.

First step (verbatim from README)

git clone --depth 1 https://github.com/jingyaogong/minimind

Watch out

  • All training scripts must be executed from the ./trainer directory.
    This is an explicit note under README “主要训练(必须)”.
  • The API service startup command is run in the scripts directory, with 8998 used in the example port.
    The README section “基于 MiniMind 的 API 服务接口” gives the cd scripts command and a localhost:8998 request example.
  • Model weights depend on actual releases, and some experimental weights may be for learning use only.
    This comes from the weight-maintenance note under README “训练结果开源”.

Alternatives

  • baby-llama2-chinese:When you want to compare MiniMind-3 (0.06B) with the 0.2B Chinese model listed in the README.
    README “与其他模型对比”
  • chatlm-mini-chinese:When your evaluation needs the other 0.2B Chinese comparison model listed in the README.
    README “与其他模型对比”

Not stated in the README

  • The README does not provide current maintainer information, the latest update time, or actual commit records.
  • The README does not provide a compatibility matrix beyond Python 3.10.16 and CUDA 12.2.
  • The README does not provide objective scores or complete benchmark configurations for the 64M model.
  • The README does not specify the exact dependency versions in requirements.txt or memory differences across training stages.

💡 Deep Analysis

7
No I plan to use a self-trained 64M MiniMind model for high-accuracy QA or production customer service and require factuality, instruction following, and service reliability. Is this project suitable for direct adoption?
For: An application owner planning to use a self-trained 64M model for high-accuracy QA or production customer service, requiring reliable factual output and sustained service

No, not for direct adoption. The README positions MiniMind as a learning, reproduction, and experimentation project, while a roughly 64M-parameter model is not aligned with production requirements for accuracy, robustness, and service reliability.

  • Project insights state that a 64M model has limited knowledge capacity, reasoning, instruction following, and context modeling, and is unsuitable for high-accuracy QA, complex reasoning, or production customer service.
  • The README labels its model comparison as experience reference rather than a strict benchmark.
  • In the ToolUse math test, the agent checkpoint achieved 17/20, or 85%, and still made three errors; full_sft achieved 60%.
  • Project insights list hallucinations, repetition, factual errors, semantic loops, and unstable output formats as limitations.

It can support training education or a local prototype, but the README does not provide production safety evaluation, SLA, concurrency, authentication, privacy, or recovery design.

  • Project insight `usage_limitations`: the roughly 64M model is unsuitable for high-accuracy QA, complex reasoning, professional decisions, or production customer service
  • README “与其他模型对比” note: the comparison is experience reference, not a strict benchmark, with limited and subjective samples
  • README “测试2:轻 Agent 任务对比”: `agent: 17/20 = 85.00%`, `full_sft: 12/20 = 60.00%`
  • Project insight `common_pitfalls`: displayed results contain factual errors, repeated phrasing, semantic drift, and obvious hallucinations
Not stated in the README:The README does not provide production metrics for concurrency, latency, availability, authentication, auditing, or recovery;The README does not provide an independent customer-service test set or factuality and safety evaluations
Yes I am studying Transformers and want to understand the underlying Tokenizer, pretraining, SFT, DPO, and PPO workflows instead of only calling wrappers from transformers, trl, and peft. Is this project suitable as learning material?
For: A university student learning Transformer internals who wants to avoid relying directly on the high-level interfaces of transformers, trl, and peft

Yes, because the README explicitly positions the project as an from-scratch LLM tutorial and states that core algorithms are implemented with native PyTorch rather than high-level abstractions from third-party libraries.

  • It covers Tokenizer, data cleaning, Pretrain, SFT, LoRA, DPO, PPO, GRPO, CISPO, Tool Use, and Agentic RL.
  • The README contrasts MiniMind with the high-level interfaces of transformers, trl, and peft, emphasizing understanding the core code line by line.
  • The minimind-3 model is roughly 64M parameters with dim=768, n_layers=8, making experiments cheaper than with billion-parameter models.

The tradeoff is a medium-to-high learning curve. The README assumes familiarity with Python, PyTorch, Linux, CUDA, data formats, and training stability. It is better suited to staged study than to readers with no neural-network background.

  • README “大道至简”: all core algorithms are implemented from scratch with native PyTorch
  • README “项目介绍”: libraries such as `transformers`, `trl`, and `peft` expose highly abstract interfaces
  • README “大道至简”: covers MoE, data cleaning, pretraining, SFT, LoRA, DPO, PPO, GRPO, CISPO, Tool Use, and Agentic RL
  • Project insight `user_experience.learning_curve`: the overall learning curve is medium to high
cd ./trainer
Not stated in the README:The README does not map every training stage to source files or provide a complete recommended reading order;The README does not provide prerequisite tutorials for PyTorch, CUDA, or Transformers beginners
Yes I already use FastGPT, OpenWebUI, or Dify and want to expose my own `minimind-3` weights through an OpenAI-compatible API while retaining Tool Calling and Thinking fields. Is this project suitable?
For: An application developer building a local prototype with FastGPT, OpenWebUI, or Dify who wants to expose a self-trained `minimind-3` model through an OpenAI-compatible service

Yes, for local or experimental prototypes. The project directly provides a lightweight OpenAI-compatible service and names FastGPT, OpenWebUI, and Dify as integration targets, but it is not presented as a production service stack.

  • The README identifies scripts/serve_openai_api.py as an OpenAI-compatible chat server.
  • It additionally supports reasoning_content, tool_calls, and open_thinking for Tool Calling and Thinking scenarios.
  • The README provides server startup, API testing, and curl examples using localhost:8998/v1/chat/completions.
  • The exported model directory requires files such as config.json, Tokenizer files, and pytorch_model.bin or model.safetensors.

Concurrency, authentication, rate limiting, monitoring, and compatibility of extension fields with every third-party client remain unspecified, so the best fit is local validation and application prototyping.

  • README “基于 MiniMind 的 API 服务接口”: provides a lightweight OpenAI-compatible chat service for FastGPT, OpenWebUI, and Dify
  • README “基于 MiniMind 的 API 服务接口”: additionally supports `reasoning_content`, `tool_calls`, and `open_thinking`
  • README command: `cd scripts && python serve_openai_api.py`
  • README API example: `curl http://localhost:8998/v1/chat/completions`
cd scripts && python serve_openai_api.py
Not stated in the README:The README does not specify API concurrency limits, authentication, rate limiting, logging, or monitoring;The README does not guarantee that all FastGPT, OpenWebUI, and Dify versions support the `reasoning_content`, `tool_calls`, and `open_thinking` extensions
Yes I only have one NVIDIA RTX 3090 running Ubuntu 20.04 with Python 3.10.16 and CUDA 12.2. Is this project suitable if I want to pretrain and SFT a roughly 64M-parameter MiniMind model from scratch?
For: An individual developer using Ubuntu 20.04, Python 3.10.16, CUDA 12.2, and a single NVIDIA RTX 3090 who wants to train a roughly 64M-parameter Chinese language model from scratch

Yes, because the project explicitly targets low-cost reproduction of a roughly 64M-parameter model on personal GPUs. However, the “2 hours and 3 RMB” claim only covers one SFT epoch on a single 3090, not pretraining or RL.

  • The reference environment is Ubuntu 20.04, Python 3.10.16, CUDA 12.2, and an NVIDIA 3090 with 24GB VRAM.
  • The minimind-3 mainline uses dim=768, n_layers=8, keeping the model small enough for experiments.
  • The project includes data cleaning, Tokenizer, Pretrain, SFT, LoRA, and preference optimization.

The README does not guarantee that every RL, long-context, or multi-run experiment fits on one 3090, and it does not provide complete pretraining duration. Start by entering the training directory and checking requirements.txt.

  • README “大道至简”: train a roughly 64M-parameter model with 3 RMB and 2 hours
  • README note: 2 hours refers to one SFT epoch on a single NVIDIA 3090
  • README “快速开始”: Ubuntu 20.04, CUDA 12.2, Python 3.10.16, NVIDIA GeForce RTX 3090 (24GB)
  • README “模型配置”: the `minimind-3` mainline uses `dim=768, n_layers=8`
cd ./trainer
Not stated in the README:The README does not provide runtime or peak VRAM figures for complete Pretrain, DPO, PPO, GRPO, or Agentic RL runs on one 3090;The README does not specify minimum batch size, gradient accumulation, or training steps for different dataset sizes
It depends I use PyTorch for small-model research and plan to compare `d_model` and `n_layers` at a fixed parameter budget, then test MoE, LoRA, DPO, GRPO, and CISPO. Can MiniMind support this type of experiment?
For: A PyTorch experimenter researching small-model architectures who wants to compare `d_model` and `n_layers` at a fixed parameter budget, then test Dense, MoE, LoRA, and RL methods

It depends. The project is well suited for studying small-model mechanisms and training workflows, but results from a 64M model should not be treated as conclusions for large models.

  • The README directly discusses the tradeoff between d_model and n_layers at a fixed parameter count, with dim=768, n_layers=8 as the minimind-3 baseline.
  • It includes MoE, LoRA, DPO, PPO, GRPO, CISPO, knowledge distillation, and Agentic RL.
  • The README states that deeper and narrower structures often outperform wider and shallower ones in the small-model range, while d_model < 512 introduces embedding and attention-head limitations.

However, small-model results have strong scale dependence, and datasets, seeds, hyperparameters, and hardware affect outcomes. Some RL, Agent, and LoRA weights are mainly experimental and may require retraining.

  • README “模型配置”: the `d_model` and `n_layers` tradeoff affects training stability and final quality
  • README “模型配置”: the `minimind-3` mainline uses `dim=768, n_layers=8`
  • README “大道至简”: covers MoE, LoRA, DPO, PPO, GRPO, CISPO, knowledge distillation, and Agentic RL
  • Project insight `usage_limitations`: small-model findings cannot be directly extrapolated to billion- or tens-of-billions-parameter models
cd ./trainer
Not stated in the README:The README does not provide complete configurations, evaluation metrics, or statistical significance for fixed-parameter architecture comparisons;The README does not state whether every algorithm branch runs unchanged in the same environment version
Yes I want to modify MiniMind's core PyTorch code under Apache License 2.0, connect the model to OpenWebUI or Dify, and redistribute it in a research prototype. Is the license and project form suitable?
For: A research-prototype maintainer working under Apache License 2.0 who wants to modify MiniMind's core PyTorch code, connect it to OpenWebUI or Dify, and redistribute it

Yes, for code modification and research-prototype redistribution. The project uses Apache License 2.0 and openly provides core PyTorch implementations and API integration paths. However, the project license alone does not determine the rights for downstream datasets, weights, or application content.

  • The README’s “开源协议” section explicitly states Apache License 2.0.
  • The project says its core algorithms are implemented from scratch in native PyTorch, making model, objective, and data-flow modification practical.
  • The API section explicitly names OpenWebUI and Dify as integration targets and provides an OpenAI-compatible server script.
  • Project insights state that user-provided datasets, model weights, and downstream applications require separate license, privacy, and content-compliance review.

Thus, it is a suitable extensible code base. If redistribution includes external data or third-party weights, each component’s authorization scope and conditions must be checked separately.

  • README “开源协议”: the project uses the Apache License 2.0
  • README “大道至简”: all core algorithms are implemented from scratch with native PyTorch
  • README “基于 MiniMind 的 API 服务接口”: enables integration with FastGPT, OpenWebUI, and Dify
  • Project insight `usage_limitations`: datasets, model weights, and downstream applications require separate license, privacy, and compliance review
Not stated in the README:The README does not provide an itemized license list for external datasets and released model weights;The README does not specify the exact copyright notices, NOTICE files, or modification notices required for redistribution
It depends I need to convert the roughly 64M-parameter `minimind-3` from PyTorch or Transformers format to vLLM, SGLang, llama.cpp, Ollama, or MNN. Is MiniMind a suitable deployment starting point?
For: A local deployment engineer who wants to convert a roughly 64M-parameter model to vLLM, SGLang, llama.cpp, Ollama, or MNN

It depends. MiniMind provides model formats and conversion paths for several inference backends, making it suitable for local deployment validation. However, its custom structure and special-token configuration may require additional backend adaptation.

  • The README provides both PyTorch and Transformers model sources and shows directories containing config.json, Tokenizer files, and model weights.
  • The “模型转换” section lists SGLang, vLLM, llama.cpp, Ollama, and MNN, showing that deployment integration is covered.
  • At roughly 64M parameters, the model is small and should impose less local resource pressure than large models.
  • Project insights identify possible issues involving custom model structure, Tokenizer, special tokens, context length, and operator compatibility.

It is therefore a reasonable starting point for validating conversion paths. The README does not guarantee production-grade performance or complete Tool Calling compatibility on any specific backend.

  • README “训练结果开源”: provides PyTorch and Transformers models
  • README “其他”: the model directory includes `config.json`, `tokenizer.json`, and `pytorch_model.bin or model.safetensors`
  • README “模型转换”: lists SGLang, vLLM, llama.cpp, Ollama, and MNN
  • Project insight `common_pitfalls`: conversion may encounter custom model structure, Tokenizer, special-token, context-length, and operator-compatibility issues
Not stated in the README:The provided README excerpt does not include complete conversion commands or a tested version matrix for each backend;The README does not provide throughput, latency, or VRAM comparisons for vLLM, SGLang, llama.cpp, Ollama, or MNN

✨ Highlights

  • MiniMind-3 can train an approximately 64M-parameter model from scratch
  • The README reports about 2 hours for 1 SFT epoch on one RTX 3090
  • Native PyTorch covers Pretrain, SFT, and DPO
  • The training pipeline also includes PPO, GRPO, and Tool Use
  • It provides a lightweight OpenAI API-compatible serving interface

🔧 Engineering

  • Uses native PyTorch code to show the full LLM path from Pretrain to RLHF
  • scripts/serve_openai_api.py integrates with FastGPT, Dify, and OpenWebUI
  • Model extensions include MiniMind-V, MiniMind-O, and MiniMind-dLM

⚠️ Risks

  • The README explicitly says the comparisons are limited, non-strict benchmarks with subjectivity
  • Weights for DPO, PPO, and GRPO are not guaranteed to remain separately available
  • Project metadata reports 0 contributors and No releases
  • The “2 hours” figure only refers to 1 measured SFT epoch on a single RTX 3090

👥 For who?

  • Developers who want to learn LLM training pipelines with native PyTorch code
  • Individual developers with an NVIDIA RTX 3090 who want to reproduce a 64M model
  • FastGPT or Dify users needing Tool Calling or Thinking interfaces