Engineering notes

On 30 July at 12:29 we promoted our Content-Security-Policy from report-only to enforcing. That is the change that turns a policy from advisory into a browser actually refusing to load things. It is a good change and we had spent two days observing the live site to make it safely.

The policy did not permit challenges.cloudflare.com. That is the origin of the CAPTCHA widget our authentication provider renders when its bot protection is switched on. Bot protection was switched on. So from that moment, on every sign-up attempt, the browser refused to load the widget, the widget never mounted, and the form could not be submitted.

Not slow. Not degraded. Sign-up did not work, for everyone, for about thirty-six hours.

The error a prospect sees in that situation is: 'The CAPTCHA failed to load. This may be due to an unsupported browser or a browser extension.' It is a generic string and it points at the client. The person who reported it had already tried Chrome, Edge, incognito, and with extensions disabled — four eliminations, all of them in the wrong place, because the message sent him there. It was never his browser.

We found out because he was on a call with us when it happened. Everybody else who tried in that window is unrecorded and unrecoverable: they never got an account, so there is no row, no session, and no support ticket. We do not know how many there were and we never will.

The part we find hardest to write down is that we had predicted it. The file that builds the policy contains a comment, written before the incident, identifying this exact widget, naming the two directives it would need, and observing that the origin was absent only because a dashboard setting was off. It then left the origin out and pointed at a live check that would catch the widget appearing.

That check could never have fired first. It compares the origins a page references against the origins the policy permits — and the widget is not referenced until bot protection is on, at which point sign-up is already broken. The first thing it could possibly observe was a customer failing to create an account.

So the rule we took from this is about detectors rather than about CSP: a foreseen failure deferred to a detector is only mitigated if you can state the observation the detector fires on. If that observation is the harm, it is not a mitigation. It is a record of the incident, written in advance.

The remedy was one origin in two directives, and it cost nothing while the feature was off. The general form, which is what we actually changed: when a dashboard we cannot read controls what gets rendered, the policy has to cover the superset of what that dashboard may render. The safe direction is to permit. The unsafe direction is to discover.

Bayeto keeps a catalog of what each model can do: whether it caches prompts, whether it bills reasoning tokens, how large its context window is, whether it supports tool calls. That catalog is detailed and we are proud of it.

It answers the wrong question on its own. A catalog capability answers 'is this possible'. A recommendation needs 'did it happen, how much, and on which of your applications'. Those are different, and we had never checked how far apart they were.

So we checked. At the time of writing the catalog modelled eighteen such capabilities. Three had a matching observation in the telemetry we ingest: input tokens, output tokens, and prompt caching. One was partial. Two were mis-specified — we model a rate-limit ceiling where a rule would need the event of hitting it. Twelve had nothing at all. That is fourteen capabilities with no observed counterpart, and the count is a snapshot: a row moves as a rule lands that needs it.

Every one of those is a recommendation we cannot make, however good the catalog is.

The sharpest is reasoning tokens. A five-word test prompt — we asked a model to reply with a single word — billed sixty-four completion tokens, fifty-two of which were reasoning. Eighty-one percent of the charged output was the model thinking about a task that required no thinking. That is exactly the kind of finding this product exists to surface, and we cannot surface it, because we ingest one opaque output-token count and the split arrives in a field we discard.

Context occupancy is next. We know each model's window; we do not record how much of it a call used. So 'you are paying for a 200k context to send 3k' is not something we can tell you yet, and it is a migration recommendation of exactly the sort we sell.

We are publishing the list rather than the three good rows because a gap analysis that only reports coverage is the one nobody re-runs. Our own first pass got a row wrong in the flattering direction — it counted our API's rate limiter as evidence that we observe provider rate limits, which it is not. The reassuring answer is the one that survives unexamined, so the false positive stays in the write-up.

None of this is a plan to add fourteen fields. Each earns its way in behind a rule that needs it, or the schema grows to match a vendor's API rather than a decision — which is the same mistake as a catalog that models what is possible instead of what happened.

We spent this week encoding a new rate card into the engine. The work was supposed to be mechanical: new bands, new rates, move the default. It was not. Auditing the path from telemetry to a euro figure turned up six defects in arithmetic we had written ourselves, and the pattern in them is more useful than any individual fix.

The first was a wrong independent variable. The savings fee banded over the savings amount when the rate card blends it from the account's spend. At a €250,000 basis with €40,000 of verified savings that is €5,625 against €5,100 — a 10.3% overcharge. It was found by writing out the published worked example by hand and comparing, not by a test, because every test we had asserted the code matched itself. It is fixed, and the fix is versioned rather than global: the model is a property of the rate card, so a figure computed under the old terms still recomputes the old way.

The second was a ceiling that did not exist. The euro ceiling was implemented, unit-tested, and called by nothing: the engine only ever consulted the old tier table, so under the new terms it returned nothing at every spend level and the cap silently did not apply. The pricing page said the ceiling always applies. It would have doubled maximum exposure at the top end on the day the default moved — so it was wired before that day, not after.

The third and fourth are still open, and we are saying so rather than waiting until they are closed. A settlement calculation and a pilot refund calculation both exist, are both tested, and are both reachable from nothing. They compute money owed back to a customer. There is no statement path yet to apply them on, so nothing is wrong today — and the closure condition is written into the build rather than into anyone's memory: a check fails if a statement-issuing path lands while either is still unreachable. They cannot quietly stay open past the moment they would start to matter.

The fifth was a published promise that had gone false. The page stated a retention floor that the newer Operator rate ladder does not clear. The test protecting that claim named the previous ladder explicitly, so it stayed green while the ladder it was written to watch had been superseded. Nobody re-examines a box that is already ticked. The claim now names the product it is true for, and a check fails the build if that scope is removed while the other ladder still misses the floor — or if that product ships without its own figure published.

The sixth was a name. A parameter called verified received a quantity that was explicitly not verified — everything identified, including work a customer had rejected. It reached only a preview screen that disclaims itself, so it was never a live overcharge. The type now makes the mistake impossible to compile.

The direction is the finding. Four of these moved money, and all four moved it toward us. That is not because anyone intended it; it is construction order. What produces a visible figure gets wired, because something renders it. What constrains a figure does not, because a constraint that never binds renders as nothing at all. A fee engine built in the obvious order therefore grows a complete revenue path and a partial protection path, and the gap is invisible to every test that asks whether the code does what it says.

So the response is not care. It is a sweep that fails the build when an exported function is reachable only from tests, because a passing test over unreachable code is a green check on an absence. And a second that fails when a cap is offered as a candidate but can never win under any input the engine accepts. Both now run on every commit.

At the time of writing, no fee had ever been billed — because no code path existed to issue one. That is why the audit was worth doing now rather than after a first invoice: every one of these was cheap to fix precisely because nothing downstream depended on the wrong answer yet.