Core Analysis¶

Project Positioning: workweave/router implements per-action routing using a local ONNX embedder and a cluster scorer to score each request and pick the best upstream model, aiming to optimize quality/latency/cost while preserving privacy.

Technical Features¶

• On-box vectorization (ONNX): Runs locally to reduce decision latency and data exposure.
• Cluster scorer + optional HMM sidecar: Default lightweight scorer for fast decisions; HMM provides frozen, reproducible policies for better explainability.
• Multi-protocol support: Works with Anthropic, OpenAI, Gemini, and OpenRouter-compatible endpoints enabling both commercial and OSS models.

Usage Recommendations¶

1. Pin or force-model for high-risk requests to avoid unpredictable automatic choices.
2. Validate scorer outputs with /v1/route peek before production and calibrate the embedder.

Caution¶

• Auto-selection depends on embedder and scoring quality — validate and log decisions for auditing.
• HMM sidecar improves stability but adds deployment complexity.

Important Notice: Per-action routing increases flexibility but also cost/result variance; use explicit constraints for critical flows.

Summary: The router uses local vectorization and pluggable strategies to enable fine-grained cross-provider model selection, suitable for teams that prioritize privacy and cost-performance trade-offs.

Core Analysis¶

Core Question: Localizing routing decisions vs. relying on a cloud decision service — the trade-offs are privacy/compliance, decision latency, and operational complexity.

Technical Analysis¶

• Privacy & Compliance: README highlights BYOK and local key storage; an on-box embedder ensures routing signals never leave your environment, reducing exposure.
• Latency: Local ONNX inference avoids network round trips to a remote policy service, benefitting latency-sensitive requests.
• Operations: You must manage embedder versions and calibration, but ONNX models are lightweight and runnable on common environments.

Practical Recommendations¶

1. Use local embedder + BYOK if you have strict data sovereignty or compliance needs.
2. Run baseline performance tests on target hardware to confirm embedder latency is acceptable.

Caution¶

• Local deployment incurs calibration and maintenance overhead; plan versioning and rollback.
• If you cannot sustain local maintenance, consider OpenRouter as a trade-off, keeping in mind increased exposure risks.

Important Notice: On-box embedder prioritizes privacy and latency but requires robust versioning and monitoring.

Summary: On-box ONNX offers clear privacy and latency benefits, suitable for teams with strict data sovereignty and real-time needs.

Core Analysis¶

Core Question: Ensuring automatic routing decisions are explainable and stable requires observability, replayability, and governance.

Technical Analysis¶

• Observability: Use built-in OTLP traces to record input vectors, scores, final choice, and upstream responses for post-hoc analysis.
• Replay & peek: Use the /v1/route peek endpoint to replay historical or synthetic requests and validate scorer behavior across contexts.
• Policy smoothing: If the in-process scorer exhibits jitter, the frozen HMM sidecar can provide history-dependent smoothing and more predictable choices.

Practical Recommendations¶

1. Create a representative test set (covering high-cost/high-risk scenarios) and replay with /v1/route peek, logging decision distributions.
2. Correlate routing decisions with business KPIs (response quality, latency, cost) and set thresholds/alerts for high-cost mis-selections.
3. For high-jitter categories, adopt an HMM sidecar and version policies for traceable behavior.

Caution¶

• Scores alone don’t guarantee result quality — close the loop with upstream quality metrics.
• Persist decision logs and audit trails for compliance needs.

Important Notice: Router explainability depends on full traces and replay capability — enable OTLP and store decision samples in production.

Summary: Using traces + /v1/route replays + KPI correlation allows systematic scorer calibration; use HMM sidecar when you need greater stability and explainability.

Core Analysis¶

Core Question: Suitability depends on team priorities: privacy/compliance, operational capacity, and cross-model optimization needs.

Suitable Scenarios¶

• High compliance / data sovereignty: BYOK and local key storage fit regulated environments.
• Multi-model / multi-vendor strategies: Platforms needing per-request trade-offs between cost, latency, and capability.
• Unified API for heterogeneous clients: Teams integrating Claude Code, Codex, Cursor, etc., through one adapter.

Not Suitable / Limitations¶

• Zero-ops preference: If teams cannot maintain Postgres, sidecars, and monitoring, self-hosting is a poor fit.
• Single-provider, no compliance need: Direct upstream calls are simpler and cheaper.
• Release & license sensitivity: Repo has no releases and license is unclear — assess compliance before production adoption.

Practical Recommendations¶

1. If compliance is critical, run a self-hosted PoC and quantify operational costs.
2. Low-ops teams should trial via npx and evaluate benefits before migrating to full self-host.

Important Notice: Router does not replace upstream models — it still depends on upstream availability and rate limits; validate upstream SLAs and fallback strategies.

Summary: Best for teams needing privacy and per-request multi-model optimization; not for zero-ops or single-provider scenarios.

Core Analysis¶

Core Question: The npx installer offers a quick start, but production self-hosting involves database, key management, observability, and multi-instance synchronization — increasing learning and operational load.

Technical Analysis¶

• Onboarding path: npx @workweave/router enables quick configuration for common tools — ideal for dev and validation.
• Production needs: Deploy Postgres (rk_ keys), set BYOK env variables, configure OTLP collector, and ensure Pub/Sub for cross-instance cache coherence.
• Common pitfalls: Key permissions or unencrypted storage, misunderstanding scorer vs HMM behavior, misconfigured Pub/Sub causing stale routing decisions.

Practical Recommendations¶

1. Rapidly validate routing with npx; use Hosted/OpenRouter keys for functional checks.
2. Prepare infra templates (Postgres init, env templates, OTLP settings) and validate multi-instance cache invalidation under canary traffic.
3. Enforce strategy audits and use /v1/route peek for scoring replay tests.

Caution¶

• Never skip key encryption and permission hardening in production.
• Roll out strategy changes in phases and retain rollbacks.

Important Notice: There’s a large gap between quick start and production hardening — move in stages.

Summary: npx lowers trial barriers, but production self-hosting requires automation, auditing, and operational discipline to avoid security and consistency pitfalls.

Core Analysis¶

Core Question: In horizontal scaling, the router must synchronize strategies and caches across instances to avoid stale routing decisions and inconsistent behavior.

Technical Analysis¶

• Shared state & events: Store configuration and metadata in Postgres, and broadcast change/cache-invalidate events via Pub/Sub so all instances are notified.
• Failure & retry: Lost or failed message delivery leads to inconsistencies; ensure idempotent handlers and retry policies.
• Observability: Use OTLP traces to compare decisions across instances and trigger alerts on divergence.

Practical Recommendations¶

1. Validate Pub/Sub reliability in a staging environment (message loss and delay scenarios) before multi-node rollout.
2. Make change/invalidation handlers idempotent to tolerate duplicates and retries.
3. Use short cache TTLs and perform controlled rollouts (drain/roll-update) for critical policy changes.

Caution¶

• If a reliable Pub/Sub is not available, consider a single-node control plane or external message services (Redis Streams, cloud pub/sub).
• Monitor inter-instance divergence and keep audit logs for traceability.

Important Notice: Cross-instance consistency is not automatic — design message reliability and idempotency at the infra layer.

Summary: Postgres + reliable Pub/Sub, idempotent handling, and observability enable consistent routing under horizontal scaling.