Extend prompt caching across Research Copilot
Mark the shared prefix with cache_control on the 60.2% of Research Copilot calls not yet caching it
Verify before implementing — this recommendation rests on evidence the engine does not have:
- No proof the uncovered cohort's prefix is byte-identical to the covered cohort's — same template is inferred, not measured.
- No independent quality evaluation — the quality impact is unmeasured.
Implement only if these checks pass. The full self-critique is in the reasoning trail below.
Payback divides by an implementation cost of ≈€220 under the agent-assisted-v1-2026-08 cost model: 2h of human review at €100/h plus a €20 agent-run allowance — an AI agent implements, your team reviews and verifies. Implemented by hand instead, the same move is estimated at 2 engineer-days (≈€1,600). Every figure is a prior, not a measurement of your team — and the model is versioned so a changed assumption can never move a figure silently.
Context completeness
Confidence 67% is how sure the engine is on the evidence it has. This is how much of the evidence that could change the conclusion the engine could establish for this application — a different question. The declare-context page counts what you have answered workspace-wide, which can legitimately read differently: an “I don't know” is an answer there and establishes nothing here.
- Telemetry
- Provider bill— calibrated from the monthly spend declared for this workspace, not a reconciled invoice
- Per-app criticality
- Prior experiments
- Codebase
- Quality evaluation
We cannot detect Codebase, Quality evaluation — you can tell us, and confidence is re-derived on the fuller evidence.
Observation → Evidence → Inference → Recommendation
— every fact, judgment, self-critique and failure condition behind this decision
Observations (facts)
- [cache_hit_rate] 39.8% of Research Copilot requests hit a cache; 0% were cacheable. (N=24,960)
- [avg_input_tokens] Research Copilot sends 30774 input tokens on average vs 900 output. (N=24,960)
Evidence (objective-relevant)
- Research Copilot caches a prompt prefix on only 39.8% of requests — the rest re-pay the full input rate on the SAME 7003-token prefix. Extending the proven cache to the uncovered cohort is quality-neutral. (strength 0.5606)
Inference (judgment)
Prompt caching is partially adopted; extending it to the uncovered cohort is a quality-neutral cost cut.
Recommendation (move)
Mark the shared prefix with cache_control on the 60.2% of Research Copilot calls not yet caching it
Critical Review (self-critique)
Assumptions
- Assumes the uncovered requests share the SAME prefix the covered ones already cache.
Alternative explanations
- The prompt prefix may rotate per request (dynamic context first) — then there is nothing stable to cache.
Missing evidence
- No proof the uncovered cohort's prefix is byte-identical to the covered cohort's — same template is inferred, not measured.
- No independent quality evaluation — the quality impact is unmeasured.
This is wrong if…
- Wrong if request cadence exceeds the cache TTL (the prefix expires before reuse), or if the prefix genuinely varies per call.
Download the canonical decision record (JSON) — the deterministic decision block (byte-pinned by CI), the run context (engine + catalog versions, measured-window boundaries, exact pricing rows, fixed FX with effective date, formula inputs, the overlap group, and the ledger version), and the LLM-phrased narrative as a visibly separate section.
Business case & technical detail
— the phrased explanation — the decision above is computed without it
Business case
Research Copilot already caches on some traffic but pays full input rate on the rest of the same prefix; closing that gap is quality-neutral savings — provided the shared prefix is verified byte-identical across the uncovered cohort. Estimated €2,606/yr at the current run-rate, payback 31 days, confidence 67%.
Technical detail
Place the stable prefix first and mark it with the provider's cache_control; verify cache-read tokens climb in the next export. Caches are per-model, so re-do this after any model change. Affects ~31,200 req/mo; expected cost -10.3%, latency -5%, quality unchanged by construction.
Benchmark context & Optimization Memory
— your value against the cohort, and what prior outcomes taught the engine
Benchmark context
Share of requests reading a warm prompt-cache prefix on cache-priced models. (illustrative)
Optimization Memory
No prior implementations yet — confidence derived from evidence only.
Why NOT the alternatives?
— each rejected route, with the constraint that rejected it
- Compress the prompt insteadcost-15%latency-5%qualityunmeasured (low risk)Cuts the same tokens but risks quality and has a learned under-delivery penalty. Caching KEEPS the tokens and just stops paying full rate for them.
- Right-size the model insteadcost-30%latency+0%qualityunmeasured (medium risk)A bigger cut, but it carries an UNMEASURED quality risk — do the model move first if warranted, then cache on the new model (caches are per-model). Not Pareto here because caching is quality-neutral and this isn't.
- Semantic response cachecost-20%latency-15%quality+0%Serves repeated ANSWERS, not shared prefixes — a different mechanism needing a similarity threshold and staleness policy. Orthogonal; both can be right.
Implementation packet
executablerisk: configReviewed by Bayeto engineering on 2026-07-26 · rule version v1 · 94% of the €56,350/yr open on this workspace is carried by a packet an engineer can execute (5 of 8 open moves); 3 of 14 specs meet that standard.
Review attests that these steps are sound and reversible as written. It does not guarantee an outcome: whether the change saves money is measured by the validation loop afterwards, never promised by the reviewer.
Complete packet, fictional system
Shown in full on the public demo because this move is reviewed to the executable standard — the one worked example a buyer can evaluate. Every figure derives from the fictional Thornbury dataset and its pinned scenario catalog: synthetic, for judging the packet's form, not a customer result.Enable PROVIDER-MANAGED PROMPT-PREFIX CACHING — never response caching and never semantic caching. Where the serialized tools, system instructions and message prefix are unchanged and only supported cache metadata is added, task semantics are expected to remain unchanged. Quality impact is expected to be zero and is EMPIRICALLY VALIDATED, never inferred from output identity: prefix caching reuses the processed prefix, it does not replay the generated answer, and inference is not bit-deterministic — so 'byte-identical output' is not available as a safety argument and is not made here.
Preconditions
The serving provider and model support prompt-prefix caching, and the pinned catalog records a minimum cacheable prefix length for them.
directcatalog.promptCacheMinTokens
Cache tariffs used, per model — claude-haiku: an ESTIMATED cache tariff was used; claude-sonnet: an ESTIMATED cache tariff was used. Because at least one tariff is estimated rather than published, this packet supports a CONDITIONAL hypothesis and its validation plan only — it cannot establish verified savings.
directcatalog.cacheTariffFor(family)
Cache usage fields are observable in this workspace's telemetry, so the effect can be measured after the change rather than assumed.
directtelemetry.cacheReadTokens / cacheWriteTokens
The observed average input length plausibly exceeds the provider's minimum cacheable prefix, so a cache breakpoint can exist at all.
calculatedrollup.avgInputTokens vs catalog.promptCacheMinTokens
CACHE TTL vs CADENCE: 31,200 requests/month is a MEAN gap of about 1.4 minutes between requests. The provider's cache TTL must exceed that gap or the cache expires between calls and pays the write every time. This is a mean, not a percentile — a workload that arrives in bursts has quiet stretches far longer than it, so size the TTL against your own quiet period rather than this figure alone.
calculatedsnapshot.apps[].monthlyRequests (mean gap = window / requests)
FAN-OUT ESTIMATE: 31,200 requests/month at a p95 of 2,519 ms — mean arrival rate x p95 response time (Little's Law). About 0.03 requests are in flight at once on average, so a fan-out is not expected at this workload's MEAN rate. That is an average and it cannot see bursts — a scheduled job or a retry storm can put many requests in flight while averaging to this figure, so the pre-check below still stands.
calculatedsnapshot.apps[].monthlyRequests x latencyP95 (Little's Law)
CHECK BEFORE ENABLING — naive prompt caching can LOSE money on a parallel fan-out. A provider cache generally becomes available only once the FIRST response begins; a burst of concurrent calls sharing a prefix therefore produces several cache WRITES instead of one useful write, and writes cost more than plain input tokens. Measure how many requests sharing this prefix start before the first one completes. If that number is routinely greater than one, either warm the cache with a single request before the fan-out, or do not enable caching for this workload.
human-authoredreviewer analysis of provider cache semantics + measured concurrency
VERIFY BEFORE IMPLEMENTING; Bayeto does not observe prompt content. The static prefix (tools, system instructions, leading messages) must be BYTE-STABLE across requests — a timestamp, request id or user name anywhere in it defeats the cache entirely while still paying every write.
human-authoredreviewer instruction — prompt content is never sent to Bayeto
VERIFY: dynamic or customer-specific content sits AFTER the cache breakpoint, so one customer's data can never be served from another's cached prefix.
human-authoredreviewer instruction — prompt content is never sent to Bayeto
VERIFY: this enables provider-managed prefix caching only. Do not introduce response caching or semantic caching as part of this change — those alter what the model returns and are a different decision with a different risk class.
human-authoredreviewer instruction
VERIFY: deployment topology and routing are unchanged by this edit. A cache that is warm on one endpoint is cold on another, so a concurrent routing change would confound the measurement.
human-authoredreviewer instruction
Steps
Add the provider's cache-control metadata at the end of the static prefix, leaving the serialized prefix content itself byte-identical.
human-authoredprovider API — cache_control breakpoint
Choose the cache TTL against the mean gap between requests stated in the preconditions, not the provider default.
calculatedsnapshot.apps[].monthlyRequests (mean gap = window / requests)
If concurrent requests share the prefix, warm the cache with a single request before the fan-out.
human-authoredreviewer instruction — see the concurrency pre-check
Roll out to one application first, not the whole workload.
human-authoredreviewer instruction
Guardrails — watch these
Request error rate, before vs after.
directtelemetry.status
The workload's own quality metric, before vs after.
human-authoredcustomer-supplied quality metric
Cache-read rate — the share of input tokens served from cache.
directtelemetry.cacheReadTokens
CACHE WRITE/READ ECONOMICS: cost of writes incurred vs cost saved by reads. This is the number that decides whether the change paid, and it is the one a naive rollout gets wrong.
calculatedtelemetry.cacheWriteTokens × catalog.cacheWritePer1M vs cacheReadTokens × cacheReadPer1M
End-to-end latency distribution.
directtelemetry.latencyMs
Provider usage reconciliation: the provider's own reported cache usage against what our telemetry recorded.
human-authoredprovider usage export vs telemetry
No change to the serialized semantic prompt content — only cache metadata was added.
human-authoredreviewer instruction — diff the serialized request
Stop if
Cache writes cost more than reads save over a full cadence cycle.
calculatedcache write/read economics guardrail
Request error rate rises.
directtelemetry.status
The prefix turns out not to be stable — cache-read rate stays near zero while writes accrue.
calculatedtelemetry.cacheReadTokens vs cacheWriteTokens
The workload's quality metric falls.
human-authoredcustomer-supplied quality metric
Quality validation
Run the workload's existing quality metric over a held-out sample before and after, and compare. Expected impact is zero — but it is CHECKED, not assumed, because inference is not bit-deterministic and identical output is therefore not evidence of anything.
human-authoredcustomer-supplied quality metric
Do NOT accept output identity as proof of unchanged quality. Prefix caching reuses the processed prefix; it does not replay the generated answer.
human-authoredreviewer instruction
Rollback
Remove the cache-control metadata — a one-line configuration revert. No data migration, no model change, no routing change.
human-authoredprovider API — cache_control breakpoint
Simulate — Before You Build
The cost delta derives from this workspace’s telemetry. The latency and quality deltas are heuristic priors carried by the rule: no quality evaluation is connected, and no rule measures the latency effect of its own move — the figures are per-rule constants, so nothing here observes either outcome.
- · Applies to ~31,200 requests/month at current mix.
- · Scenario realization assumption (sim-v1, not a statistical expected value): 0.7 + 0.3 × confidence = 0.901. The 0.7 floor encodes implementation shortfall — even a certain move rarely captures 100% at rollout; rationale published at bayeto.ai/methodology.
- · Per-request cost/latency/quality deltas hold at current traffic shape.