Honest benchmarks
Azure run mid_20260708_111539 — 100 tasks/scenario, real Gemini, fair paired comparison. Benchmark-only fixes July 2026 (prompt + reporting); no library version bump.
Task success (LangGraph → ChorusGraph)
| Scenario | LangGraph | ChorusGraph |
|---|---|---|
| Finance single (FL1/FC1) | 87.0% | 98.0% |
| Finance multi (FL2/FC2) | 87.0% | 94.0% |
| Healthcare single (HL1/HC1) | 74.0% | 79.0% |
| Healthcare multi (HL2/HC2) | 59.0% | 85.0% |
LLM calls & latency (mean per task)
| Scenario | LLM calls (L → C) | Mean latency ms (L → C) | Cache hit (C) |
|---|---|---|---|
| FL1 / FC1 | 3.24 → 0.77 | 4760 → 1348 | 52% |
| FL2 / FC2 | 2.03 → 0.69 | 3269 → 1085 | 40% |
| HL1 / HC1 | 3.00 → 1.56 | 7036 → 3990 | 60% |
| HL2 / HC2 | 3.82 → 3.15 | 10296 → 10753 | 51% |
Reproduce
pip install -e ".[benchmark,gemini]" python -m benchmark.run_scenarios --tier mid --scenarios all --seed 42 # Azure: .\benchmark\azure\deploy_and_run.ps1 -Tier mid -Wait -Cleanup
Full report: COMPARISON_REPORT.md · latency summary · methodology
H10 repeat band (FX, 40% repeat)
Median latency ~8× lower on repeat FX band (575 ms vs 4498 ms). Sliced metric — see FAIRNESS_H9.md.