01
A capable harness beat a raw request.
The largest effect appeared when the task required current information. Yet harnesses also beat Raw on historical-control cases with search disabled, so retrieval alone does not explain the gain.
← Benchmarks
CoffeeBench V1 · Research report
The harness mattered more than the specialist tools.
Across 20 matched coffee-research cases, every agent harness was preferred to a raw model request. We did not find a clear winner among the harnesses, and an always-visible catalog tool added work without improving answer quality.
- Cases
- 20
- System trials
- 400
- Pairwise votes
- 1,800
- Judge families
- 3
Abstract
Abstract
CoffeeBench V1 asked how much the system around a fixed model changes its ability to answer real coffee-industry research questions.
We ran DeepSeek V4 Flash as a raw request and inside three agent harnesses. Harnessed systems were preferred over Raw in 70.3–74.5% of overall pairwise judgments and 88.3–90.8% on live-web cases. Purveyors Search ranked first, but its uncertainty overlapped the other harnesses. Adding a frozen Parchment catalog did not improve pairwise quality. Trace analysis suggests a useful design principle for the next round: expose specialized capabilities only when their evidence is relevant.
Results
Three findings changed what we want to test next.
02
Purveyors’ directional lead remained inconclusive.
Purveyors Search received 55.5% against Pi Search overall, but the quality intervals overlap and the live-web head-to-head narrowed to 53.3–46.7. V1 does not establish a harness winner.
03
An irrelevant tool became a distraction.
The Parchment arm called the catalog 28 times even though no case asked the model to choose a coffee for sale. Fifteen calls returned nothing, and four answers carried an empty result into unrelated market reasoning.
Figure 1
No harness separated cleanly.
Purveyors Search ranked first on Bradley–Terry preference. Every interval overlaps, so V1 does not establish a winner.
Purveyors Search
0.582 · 0.547–0.619
Purveyors, Parchment, and Search
0.561 · 0.520–0.609
Pi Search
0.537 · 0.487–0.592
Raw
0.327 · 0.249–0.400
0.249 Relative pairwise quality 0.619
Figure 2
Current-information cases widened the gap.
Preference share against Raw, with ties split evenly. Search was disabled on historical control cases and enabled on live-web cases.
Purveyors Search
Historical
61.4%
Live web
88.3%
Purveyors, Parchment, and Search
Historical
63.6%
Live web
90.8%
Pi Search
Historical
58.3%
Live web
88.3%
A 50% share is even. The harnesses retained an advantage over Raw on historical cases, where web retrieval was disabled.
Figure 3
Raw was faster, used fewer reported input/output tokens, and produced weaker answers.
Reported input/output tokens only; reasoning-token usage was unavailable for every treatment. Successful-task median latency and jury-marked unacceptable answers are also shown. Bar lengths are scaled within each metric.
Purveyors Search
Reported input/output tokens / task
6,785
Median time
9.8 s
Unacceptable
30%
Purveyors, Parchment, and Search
Reported input/output tokens / task
7,624
Median time
9.3 s
Unacceptable
30%
Pi Search
Reported input/output tokens / task
7,045
Median time
8.8 s
Unacceptable
32%
Raw
Reported input/output tokens / task
1,801
Median time
5.8 s
Unacceptable
41%
Experiment
One model, four systems around it.
Every treatment used the same DeepSeek V4 Flash 0731 FP8 endpoint, case input, evidence, temperature, output limit, and provider route. “Purveyors Search” and Pi used the same Brave search backend. Their comparison changes the agent runtime and system prompt; Purveyors versus Parchment changes only whether the catalog tool is exposed.
Historical-control cases disabled public-web tools. Live-web cases enabled the same search and fetch tools for all three harnesses. The Parchment treatment retained its frozen catalog snapshot in both cohorts.
Fixed model settings
- Model
- DeepSeek V4 Flash 0731
- Route
- OpenRouter → DeepInfra FP8
- Temperature
- 0.4
- Top-p
- 1.0
- Output cap
- 4,096 tokens
- Fallbacks
- Disabled
Raw
One direct model request, without a system prompt or agent loop.
Tools: No tools in either cohort.
Pi Search
Pi agent loop with a general research prompt and up to five steps.
Tools: Brave search and page fetch on live-web cases only.
Purveyors Search
Purveyors AI SDK loop with a green-coffee decision prompt and up to five steps.
Tools: The same Brave search and page fetch contract as Pi.
Purveyors + Parchment + Search
The same Purveyors loop, prompt, model, and step budget.
Tools: The shared web tools plus a frozen Parchment catalog snapshot.
Judge-model voting
Every matchup, broken down by model family.
OpenAI, Google, and Anthropic judge agents each voted on every response pair. The bars below show their preference shares with ties split evenly; the headline row combines all three families. Switch cohorts to inspect all 1,800 pairwise ballots.
Pi Search vs Purveyors, Parchment, and Search
107 wins · 78 ties · 115 losses
48.7% · 51.3%
OpenAI
46% · 54%
47% · 53%
Anthropic
53% · 47%
Pi Search vs Purveyors Search
95 wins · 77 ties · 128 losses
44.5% · 55.5%
OpenAI
42% · 58%
45.5% · 54.5%
Anthropic
46% · 54%
Pi Search vs Raw
182 wins · 58 ties · 60 losses
70.3% · 29.7%
OpenAI
69.5% · 30.5%
74.5% · 25.5%
Anthropic
67% · 33%
Purveyors, Parchment, and Search vs Purveyors Search
102 wins · 77 ties · 121 losses
46.8% · 53.2%
OpenAI
52% · 48%
44.5% · 55.5%
Anthropic
44% · 56%
Purveyors, Parchment, and Search vs Raw
201 wins · 45 ties · 54 losses
74.5% · 25.5%
OpenAI
74% · 26%
77.5% · 22.5%
Anthropic
72% · 28%
Purveyors Search vs Raw
190 wins · 53 ties · 57 losses
72.2% · 27.8%
OpenAI
71.5% · 28.5%
78% · 22%
Anthropic
67% · 33%
Discussion
What we think this means, and what we will do with it.
Model capability, orchestration, and evidence access interacted to shape answer quality. The raw model answered quickly while omitting important content. The catalog tool supplied legitimate data, yet the harness exposed it when that data was unrelated to the decision. The next system should make relevant capability easy to reach and irrelevant capability easy to ignore.
Route specialized tools by intent.
The next Parchment arm will expose catalog search for inventory, supplier, price, and purchase decisions, then keep it out of unrelated research tasks.
Isolate the Cherry model.
Cherry should face the current model inside the same prompt, tools, evidence, and output contract. That is the clean way to measure specialist-model lift.
Design output for progressive disclosure.
We want Cherry to lead with the decision, evidence, and uncertainty, while making deeper structured detail available when the calling harness needs it.
Methods & data
A frozen suite with three independent views of quality.
CoffeeBench used 12 historical-control cases and eight live-web cases, with five trials per treatment and case. Three judge-model families produced 1,200 absolute rubric evaluations and 1,800 pairwise ballots. We report Bradley–Terry preference, absolute rubric outcomes, and operational measurements separately rather than combining them into one score.
Why total cost is only available for Raw
Model-call cost was captured for every treatment. The published cost field, however, requires a complete model-plus-tool total. Brave search and Parchment tool calls did not have pinned marginal prices, so tool-using trials cannot support a complete end-to-end cost. Raw used no tools and is therefore the only fully priced treatment. Reported input/output token usage, excluding unavailable reasoning tokens, and latency remain comparable in Figure 3; future runs will capture tool-inclusive cost directly.
Interpretation limits
V1 studies one model on 20 cases with an agent jury and no human calibration. It identifies useful system patterns and concrete follow-up experiments; it does not establish a universal model or harness winner, isolate retrieval as the sole cause of the Raw gap, or prove that Parchment exposure caused the observed quality result.
Full aggregate metrics
| Treatment | Quality · 95% interval | Strict pass | Must-miss | Critical | Unacceptable | Reported input/output tokens / task | Median latency | Complete cost / task |
|---|---|---|---|---|---|---|---|---|
| Purveyors Search | #1 · 0.582 · 0.547–0.619 | 69% | 17% | 17% | 30% | 6,785 | 9.8 s | Unavailable |
| Purveyors, Parchment, and Search | #2 · 0.561 · 0.520–0.609 | 68% | 22% | 14% | 30% | 7,624 | 9.3 s | Unavailable |
| Pi Search | #3 · 0.537 · 0.487–0.592 | 64% | 24% | 18% | 32% | 7,045 | 8.8 s | Unavailable |
| Raw | #4 · 0.327 · 0.249–0.400 | 59% | 41% | 8% | 41% | 1,801 | 5.8 s | $0.0001419654 |
Result identity and provenance
- Result version
- 1.0.0.coffeebench-v0-deepseek-v4-reliable-official.published.50ca8fbd22a8523e
- Status
- published
- Result content SHA-256
- 50ca8fbd22a8523eacb13bc21c5eb890a2fd60f3e55db8a386a00bc8b94bb087
- Methodology SHA-256
- c5374bb3015860e87d8d8ae35a3893682ccf8c53284f84b6b92538d4b1f11900
- Subject cards SHA-256
- ebecb71d35f71eecb7673ae404cb7aa17ccc3138927eba5f3b2b5395e33d5803
- Scoring contract
- independent_tracks_v1
Purveyors
Coffee intelligence platform. Daily-normalized data from 40+ US specialty importers, turned into procurement-ready analytics for green coffee buyers and roasting teams.
© 2026 Purveyors.