Forge: A reliability layer for self-hosted LLM tool-calling
Forge delivers a reliability infrastructure for self-hosted LLMs focused on tool-calling: combining guardrails middleware, an OpenAI-compatible proxy, and context compaction to markedly improve usability and stability of ~8B models on multi-step, tool-heavy tasks — suitable for teams requiring local inference and strong privacy control.
GitHub antoinezambelli/forge Updated 2026-05-22 Branch main Stars 1.5K Forks 75
Python LLM orchestration Tool-calling Proxy / Guardrails Context management Self-hosted models

💡 Deep Analysis

5
As an engineer, when should I choose WorkflowRunner, Guardrails middleware, or the Proxy server? What are the trade‑offs?

Core Analysis

Question Core: How should you weigh ease of use, control granularity, and customization when choosing between WorkflowRunner, Guardrails middleware, and the Proxy server?

Technical Comparison & Tradeoffs

  • WorkflowRunner:
  • Pros: Turnkey orchestration of prompts, tools, context compaction, guardrails, and SlotWorker for multi‑agent scheduling.

  • Cons: Higher coupling to Forge’s runtime model and moderate learning curve.

  • Guardrails middleware:

  • Pros: Maximum flexibility—embed only the validation/repair layer into your existing orchestrator.

  • Cons: You must manage the loop, tool execution, and state yourself—higher integration effort.

  • Proxy server:

  • Pros: Minimal client changes—drop in an OpenAI‑compatible proxy to get guardrails quickly.
  • Cons: Less control over complex workflow semantics; synthetic respond tool injection can conflict with custom loops.

Practical Recommendations

  1. Quick PoC / Minimal changes: Use Proxy server and point clients to python -m forge.proxy.
  2. Existing orchestrator: Integrate Guardrails middleware for fine control without replacing your loop.
  3. New agent platform: Use WorkflowRunner with SlotWorker and ContextManager for full lifecycle management.

Important Notice: Whichever path you choose, run Forge’s eval suite on your target hardware to validate guardrails and context strategies in an end‑to‑end manner.

Summary: Proxy is best for low‑invasion PoCs; middleware suits highly customized existing systems; Runner fits teams building a new agent runtime that want Forge to manage execution and scheduling.

87.0%
How do Forge's Guardrails (e.g., Rescue Parsing, Retry Nudge, StepEnforcer) practically correct or prevent malformed tool calls?

Core Analysis

Question Core: Models frequently produce malformed tool calls, missing fields, or plain text instead of function invocations. How can we automatically detect, correct, and ensure step completion at the engineering layer?

Technical Analysis

  • Schema validation: ResponseValidator checks model outputs against ToolSpec (Pydantic-based) schemas to reduce parsing failures.
  • Rescue Parsing: When outputs are malformed, Forge attempts robust parsing of free text to extract function names and parameters and synthesize a compliant call.
  • Retry Nudge: If parsing/step completion fails, Forge issues a targeted retry prompt (nudge) guiding the model to emit a properly formatted tool call.
  • StepEnforcer: Ensures mandatory workflow steps are completed, looping or escalating to failure if retries are exhausted.
  • Proxy synthetic respond tool: The proxy can inject an invisible tool to force the model into a function‑call response path rather than plain text, addressing the text‑vs‑tool decision instability.

Practical Recommendations

  1. Define precise ToolSpec (Pydantic) for each tool to enable accurate validation and rescue logic.
  2. Enable Rescue Parsing logs during debugging to refine prompt/tool definitions against common failure modes.
  3. Configure reasonable retry limits and fallback behavior to avoid infinite loops and resource waste.

Important Notice: Guardrails’ effectiveness depends heavily on accurate tool schemas and backend support (e.g., native function‑calling). Misconfigured backends can weaken validation and repair steps.

Summary: Forge converts malformed outputs into repairable or retryable events through schema validation, parsing, nudges, and step enforcement, significantly improving tool‑call success—but requires careful schema and backend configuration.

86.0%
What are common pitfalls and best practices when using Forge in practice? How to avoid configuration errors that degrade performance?

Core Analysis

Question Core: When rolling Forge into experiments or production, what misconfigurations or misunderstandings commonly degrade performance, and how can you avoid them?

Technical Analysis (Common Pitfalls)

  • Backend/model misconfiguration: Missing backend flags or layers that alter function‑calling semantics (prompt injection, jinja toggles) can break Guardrails’ validation/repair flow.
  • Incorrect context budgets: Poorly set budget_tokens or keep_recent can compress away critical information, causing multi‑step failures.
  • Unclear tool schemas: Without strict Pydantic ToolSpec, ResponseValidator and Rescue Parsing become less effective, increasing parse failures.
  • Proxy behavior misunderstandings: The proxy injects a synthetic respond tool to enforce tool‑call behavior; if your external loop also wraps calls, you may end up with double‑wrapping or conflicts.

Best Practices (Actionable)

  1. Use README‑recommended backend/model combos (e.g., llama‑server + Ministral‑3 8B) and verify backend flags (e.g., --jinja, streaming).
  2. Define strict ToolSpec (Pydantic) for each tool to enable robust validation and rescue.
  3. Enable TieredCompact and tune budget_tokens / keep_recent via hardware‑specific benchmarks.
  4. Run the 26‑scenario eval harness before production and stress test retry/step‑enforcer policies.
  5. Document and test proxy usage to ensure synthetic tool injection doesn’t conflict with existing loops.

Important Notice: The repository metadata shows an unknown license—confirm licensing and release stability before commercial deployment.

Summary: Following README recommendations, defining strict tool schemas, and performing hardware‑specific end‑to‑end benchmarks are the most effective steps to avoid configuration‑induced regressions.

86.0%
How does Forge's ContextManager (specifically TieredCompact) improve multi-step reasoning coherence under limited VRAM and long-context scenarios?

Core Analysis

Question Core: Under limited VRAM, how can a long conversation or multi‑step agentic flow retain the most decision‑critical information and avoid failures due to discarded context?

Technical Analysis

  • Tiered retention logic: TieredCompact divides context into a short‑term high‑fidelity layer and a long‑term compressed layer. The keep_recent parameter ensures the most recent interactions are preserved with minimal compression so intermediate outputs (e.g., tool results) are retained.
  • Budget‑driven decisions: budget_tokens represents the VRAM/context window budget and triggers compression/summarization instead of blindly discarding the oldest tokens.
  • Optimized for multi‑step reasoning: Compared to a simple sliding window, tiered strategies better balance short‑term memory and long‑term background, preserving context that materially affects subsequent steps (tool outputs, step state).

Practical Recommendations

  1. Initial config: Set budget_tokens based on model and GPU memory (e.g., README’s example 8192) and choose keep_recent to hold the number of recent interactions needed (commonly 2–5).
  2. Task‑driven tuning: Prioritize retention of tool outputs and step artifacts (mark them high priority). Use higher compression for chatty, low‑utility history.
  3. Validation: Run the provided 26‑scenario eval harness on your target hardware to measure success rates against different budget_tokens / keep_recent settings.

Important Notice: Excessive compression can drop essential long‑term context—misconfigured budget_tokens is a frequent cause of reduced task success.

Summary: TieredCompact implements an engineered balance of retention vs compression under VRAM limits, but requires task‑ and hardware‑aware tuning to realize consistent multi‑step coherence improvements.

84.0%
In multi‑agent/multi‑workflow scenarios sharing a single GPU, how does Forge's SlotWorker manage priorities and preemption to reduce latency and failures?

Core Analysis

Question Core: When multiple agents or workflows share a single GPU, how can we ensure low latency and high success for critical tasks during contention while minimizing errors for preempted tasks?

Technical Analysis

  • Priority queueing: SlotWorker enqueues inference requests by priority so that critical requests obtain the GPU first.
  • Automatic preemption: When a high‑priority task arrives and the GPU is occupied, SlotWorker can preempt the running lower‑priority job. Preemption must respect backend constraints (interruptibility, resumability).
  • Consistency and interrupt boundaries: To avoid leaving inconsistent state mid‑step, decompose workflows into resumable atomic units (e.g., single tool calls) and preempt at those boundaries.

Practical Recommendations

  1. Task decomposition: Break long jobs into resumable subtasks to allow safe preemption and retry.
  2. Priority policy: Mark interactive/low‑latency workflows as high priority; batch/evaluation jobs as low priority.
  3. Backend capability check: Verify if your backend (llama‑server, Ollama, etc.) supports streaming/interrupt and configure preemption thresholds accordingly.
  4. Monitoring & circuit breakers: Implement timeouts and retry caps to prevent runaway preemption and resource exhaustion.

Important Notice: Preemption reduces critical path latency, but improper decomposition or lack of backend interrupt support can introduce inconsistent state or extra failures—stress test in staging first.

Summary: SlotWorker provides practical priority and preemption semantics for single‑GPU multi‑agent scenarios, but its safety and effectiveness require resumable task design and alignment with backend interrupt capabilities.

82.0%

✨ Highlights

  • Elevates 8B self-hosted models' reliability for tool-calling workflows
  • Provides an OpenAI-compatible proxy that transparently applies guardrails
  • Missing license information — compliance considerations required
  • Recorded contributor/release activity is limited — community health unclear

🔧 Engineering

  • Comprehensive guardrails and context management to improve small-model stability on multi-step workflows
  • Includes WorkflowRunner, guardrails middleware, and an OpenAI-compatible proxy as three integration modes
  • Supports Ollama, llama-server, Llamafile and Anthropic backends
  • Context strategies include VRAM-aware budgets and tiered compaction to reduce long-session costs

⚠️ Risks

  • Hardware and backend configuration sensitive — deployment is complex and requires extra ops effort
  • No explicit license — commercial use carries legal and compliance uncertainty
  • Repository metadata shows limited contributors/releases — assess long-term maintenance risk
  • Depends on specific backends/models (e.g., Ministral-3) — migration costs are non-trivial

👥 For who?

  • Suited for ML engineers and platform teams requiring local deployment and privacy
  • Also for architects building multi-agent systems or shared-GPU slot orchestration
  • More valuable to enterprise users who prefer control and are willing to invest in ops