Core Analysis¶

Project Positioning: whichllm’s primary value is expanding the simple “fits into VRAM” check into a multi-dimensional selection problem: it considers weights, KV cache, activation peaks, quantization efficiency, MoE active parameters, and backend differences rather than just model size.

Technical Features¶

• Evidence merging: Aggregates scores from LiveBench, Aider, Open LLM Leaderboard and discounts by direct/variant/base/self-reported types to reduce misleading influence.
• Architecture-aware estimation: Models VRAM usage as weights + KV cache + activations + overhead; speed is modeled as bandwidth-bound with per-quant efficiency, MoE active-vs-total splits, and unified-memory vs PCIe partial offload costs.
• Quantization/backend penalties: Supports GGUF, AWQ/GPTQ, FP16/BF16 and incorporates penalties for expected speed/quality impact into ranking.

Practical Recommendations¶

1. Run whichllm on your machine to obtain an evidence-scored ranking rather than selecting by model size alone.
2. For latency-sensitive use, inspect speed_confidence and speed_range to set realistic expectations.
3. Use --gpu simulation and the plan command before hardware purchases to compare real-world run feasibility across cards.

Important Notice: Recommendations are estimation-driven and rely on public benchmarks; unusual drivers, CUDA versions, or proprietary backends can cause deviations.

Summary: whichllm integrates multi-source benchmarks and runtime cost modeling to reduce the engineering risk of “it should fit but doesn’t run well” or buying inappropriate hardware.

Core Analysis¶

Issue: whichllm offers an automated flow from recommendation to launch, enabling quick local trials, but it depends on local drivers, compiled dependencies, and network resources—these remain common failure points.

Technical Analysis¶

• One-click flow: whichllm run will create an isolated env (uv), install dependencies, download the model and launch an interactive session; whichllm snippet provides runnable Python snippets.
• Common failure points:
• Mismatched GPU drivers or CUDA versions cause compile/run errors.
• auto-gptq/autoawq or llama-cpp-python require local compilation and may fail if system libs/build tools are missing.
• Large model downloads are time-consuming and disk-heavy, susceptible to rate limits or offline failures.

Best Practices¶

1. Prepare drivers/deps: Ensure CUDA, NVIDIA drivers, and build toolchains are ready; verify hardware detection with whichllm before heavy ops.
2. Validate at small scale: Use whichllm snippet or --top 1 --json to fetch model info and run a small interactive test in an isolated env first.
3. Plan network/storage: Reserve disk space before large downloads and use caching/frozen fallbacks if bandwidth is limited.
4. Conservative quantization path: If hitting quant/backend issues, start with mainstream stacks (llama-cpp-python + GGUF/FP16) before experimenting with AWQ/GPTQ.

Important Notice: whichllm shortens the path to trying models but cannot eliminate the complexity of local environments and compilation chains.

Summary: Treat whichllm as a rapid trial facilitator, but perform driver/disk/build-tool checks and small-scale verifications before a full run.

Core Analysis¶

Issue: whichllm provides GPU simulation, plan, and upgrade to visualize what models will run and how well on candidate cards. It’s a valuable decision-support tool but not the sole basis for purchase.

Technical Analysis¶

• Decision-support features:
• --gpu simulation: Emulates VRAM usage and inference speed across different GPUs.
• plan: Reverse-plans the minimum hardware or recommends cards for a given model.
• upgrade: Compares model coverage and performance across multiple cards for cost/benefit analysis.
• Modeling boundaries: Estimates rely on generic bandwidth/VRAM modeling and public benchmarks; they cannot fully capture unified-memory behavior, PCIe offload dynamics, or vendor-specific accelerations.

Practical Advice¶

1. Use whichllm’s lower-bound estimates as conservative inputs; reserve 10–30% headroom for driver/system variances.
2. For critical workloads, perform a target-machine empirical test—borrow/try a representative card and compare whichllm run results against estimates.
3. Include cost/power/cooling and licensing/compliance considerations in purchase decisions—these are outside whichllm’s scope.

Important Notice: whichllm is not the final arbiter for hardware purchases but significantly narrows candidates and quantifies runability and expected performance.

Summary: Use whichllm for evidence-driven preselection and capacity planning; combine with empirical tests and engineering/cost constraints for the final buy decision.

Core Analysis¶

Issue: whichllm’s VRAM/speed estimations provide meaningful expectations for common hardware/backend combinations, but they are not absolute guarantees. speed_confidence and speed_range explicitly express uncertainty.

Technical Analysis¶

• Estimation components: VRAM = weights + GQA KV cache + activation peaks + overhead; speed depends on bandwidth, quantization efficiency, backend implementation differences, and MoE active ratios.
• Sources of uncertainty: Input sequence length (activation peaks), driver/CUDA versions, backend (llama-cpp vs transformers), use of unified memory or PCIe offload, and quantization implementation efficiency.
• Meaning of speed_confidence: Reflects benchmark coverage and historical consistency of a model/backend combo. High values indicate multiple similar benchmarks support the estimate; low values indicate sparse or divergent data.
• Meaning of speed_range: Presents a throughput interval across different system loads, quant backends, and sequence lengths.

Practical Recommendations¶

1. If speed_confidence is high and speed_range is tight, treat the estimate as a reliable capacity-planning input; otherwise run whichllm run or a snippet to validate on target hardware.
2. For purchasing, simulate multiple GPUs with --gpu and use the lower bound of estimates as a conservative metric, leaving ~10–30% headroom.
3. With MoE, unified memory, or non-standard backends, treat whichllm results as guidance and perform an end-to-end test.

Important Notice: whichllm gives expected ranges and confidence, not production SLAs. Critical production choices should be validated empirically.

Summary: Estimations are valuable for mainstream setups; speed_confidence and speed_range are the signals telling you whether to trust the estimate or to run empirical checks.

Core Analysis¶

Issue: whichllm provides automation-friendly outputs (--json, snippet) and an isolated install flow (uv), making it suitable for embedding selection and trial steps into CI, but network, drivers, and native build dependencies limit full automation on generic runners.

Technical Analysis¶

• Reusable outputs:
• --json: Structured list with scores, speed_confidence, and speed_range—ideal for programmatic filtering (e.g., score>80 and speed_confidence>0.7).
• snippet: Executable Python snippets usable as CI smoke tests to validate model load/inference on runners.
• uv/uvx: Isolated install helps reproduce environment setup.
• Limits & mitigations:
• Network dependency: HF API and model downloads fail in air-gapped CI—use cache/frozen fallbacks or private mirrors.
• Native builds: auto-gptq/llama-cpp-python require compilation—solve with containers or prebuilt binary caches.
• Hardware limits: GPU-less CI can only perform static selection; place runtime validation on GPU-equipped staging runners.

Practical Recommendations¶

1. Use --json as the selection engine input in CI; apply threshold filters and then dispatch snippet to test runners.
2. Containerize or prebuild artifacts to eliminate flaky native builds; run smoke tests on dedicated GPU runners.
3. Maintain frozen fallbacks and internal model mirrors for offline or air-gapped CI.

Important Notice: Automation speeds decision-making, but final compatibility checks must run on runners that mirror production hardware and drivers.

Summary: whichllm has strong automation touchpoints (JSON, snippet, uv) but requires network, driver, and build reproducibility practices to be fully reliable in CI.

Core Analysis¶

Issue: whichllm not only aggregates multiple benchmarks but discounts each entry based on whether it is direct, inherited/variant, or self-reported, and on recency—this reduces distortion from inflated or inherited scores.

Technical Analysis¶

• Evidence stratification: Scores are labeled as direct / variant / base / interpolated / self-reported and weighted down by category, preventing small forks from inheriting high ranks via large-base scores.
• Recency weighting: Older benchmarks are demoted along model lineages; README prints snapshot dates so users can detect stale information.
• Multi-source cross-check: Aggregating LiveBench, Aider, Open LLM Leaderboard reduces individual test bias but relies on sufficient overlap or calibration between leaderboards.

Advantages¶

• More robust recommendations: Resists manipulation by self-reported or inherited scores.
• Transparent: Provides benchmark dates and confidence tags for manual verification.

Pitfalls & Recommendations¶

1. Data sparsity: Models with few benchmarks may be over-penalized; in such cases use conservative filters (--evidence strict or only direct entries).
2. Benchmark semantics: Different leaderboards test differing tasks—inspect which metrics are being combined.
3. Offline/cached lag: Frozen fallbacks in offline mode may be stale.

Important Notice: Treat the evidence weights as decision parameters, not as the single authority.

Summary: Evidence aggregation plus confidence discounting increases recommendation quality but requires user interpretation of confidence tags and benchmark dates for final selection.