The system moved from monitoring to autonomous enforcement in October 2026, with expiring actions, post-action checks and automatic return to shadow mode. Calibration exposed a confident model error on real browser activity; a deterministic veto keeps that activity out of the acting set.
I built an hourly service that detects automated traffic and challenges it at the edge. Models advise; code limits and verifies each action.
From the analytics read to an action with an expiry, or a record of what would have been done.
One worker owns the edge credentials and reads analytics under a database lock. Code sets each actor’s maximum response before any model is asked. The admin panel queues requests for that worker and can revoke its actions.
In the October calibration recordings, Jev classified an organic browser cohort as an automated fleet with probabilities of 0.89–0.93. Shared browser strings looked persuasive to the model. The detector’s asset-fetch veto excludes that cohort before it can be proposed for action.
That failure shaped the trust boundary: a model score cannot authorize a challenge. The labeled replay checks the detector, guardrails and renderer together, including organic activity that must never act. These recordings test decision safety; they do not establish a production false-positive rate or quantify traffic reduction.
| Recorded cohort | Fleet probability | Independent decision |
|---|---|---|
| Three automated fetcher cohorts | 0.93–0.99 | Must still pass detection, action limits and replay gates |
| Organic desktop browsers | 0.89–0.93 | Excluded by the asset-fetch veto; labeled never-act |
Source: recorded calibration runs and the system’s decision log. The overlapping scores show why confidence alone cannot authorize action.
Unit and property tests, a labeled replay, a live drill, and a day in shadow before enforcement was first allowed.
The replay sends each labeled case through the actual detector, guardrails and renderer. A live drill applies and removes a one-hour challenge to a made-up browser string, then checks that the rule matches its before-state byte for byte. Sixteen cases cover known failure classes; they are not a population-wide safety estimate.
The edge logs a challenge interstitial as an HTML response, so a count of “HTML requests” took challenged bots for page readers, and every HTML measure read a fleet’s challenge pages as its own traffic. I moved every page measure to served pages: status below 400.
I first set the browser-reach cap by intuition, and it would have refused the very cohorts the system exists to catch. A week of measurement put it well above any bot cohort’s share of real browsers and far below a real desktop cohort’s.
The selector proposes a refused candidate again every tick, so counting every refusal let one standing veto fill the demotion count by itself and drop enforcement back to shadow. A safety veto now counts once a day per target and reason; three distinct ones still demote.
One action a tick and an expiry on every action mean a fleet with several cohorts is challenged over hours, not minutes, and a cohort that never leaves is proposed again rather than held, which is why the hand rules stay as the first layer.
The detector’s ceiling tops out at a managed challenge today; the block surfaces are built and tested and wait for replay evidence that would raise a ceiling. In return, nothing waits for me, and any action can be revoked from the panel or left to expire.
Policy limits, not measured traffic outcomes.
The edge provider’s plan has no dry run for custom rules, and nobody watches an edge dashboard around the clock. The goal was mitigation that runs itself and never touches real users, verified crawlers or me. Code decides; models advise. Once an hour, under a database advisory lock, a worker reads the edge network’s analytics and the application’s own page-view data, scores every actor on hard numbers, asks a judgment model about the ones worth acting on, lets a selector pick at most one, and hands the pick to guardrails that decide whether anything is sent.
Two processes share the application’s PostgreSQL database. The worker alone holds the edge token; it runs the hourly tick, a check every minute for the operator’s revoke and run requests, and a check every fifteen minutes for a stalled tick, keeps its mode, its ledger of actions and shadow records, the queues and the notices in rows of the application’s settings table, and publishes its report to Redis. The admin panel holds no token and only writes requests for the worker. The panel shows the mode switch (off, shadow, enforce), the live actions with their evidence and rationale, a revoke button, the shadow records and “run now”. Every proposal, action, expiry, revoke, post-check and mode change lands in an audit trail, a daily digest and immediate alerts go out by email, and two kill switches exist: the mode row, and a hard-off variable on the worker that wins over everything.
Actors are networks, single addresses, /24 prefixes and user-agent cohorts: strings grouped by their shape, each string admitted only on its own evidence (enough served pages a day, almost no asset fetches and at most one engaged API call per served page). Every actor gets hard-number features: request rates, address spread, served pages, assets per page, path mix (a catalog walk of long-tail pages against a refresh loop on one page), status mix, protocol, verified-bot share and origin cost. From those numbers the detector sets a ceiling per actor, the strongest response it may receive: none, observe (watched, with nothing sent to the edge) or managed challenge. Vetoes run in a fixed order before any bot pattern is scored: my own networks, verified bots, actors an existing rule already contains, self-declared crawlers, actors below volume, and actors that fetch assets the way a browser does. Unsupported targets and protected platform networks stay at observe, so the detector does not repeatedly propose an action the executor must refuse.
For each actor whose ceiling is a managed challenge, code words the evidence into sentences, with no addresses in them, and sends TypeSafe Jev one request at a pinned model version. The request holds a multiple-choice question (which kind of traffic source does this evidence describe: a scraper on a cloud network, one client spread over many home addresses, a verified crawler, a crawler that names itself, real people, or none of these), four yes-or-no questions, such as whether the strings look like one automated client repeated, and a three-level cost score. The answers are probabilities. Two of them gate: the class the detector claims must be Jev’s answer at a calibrated probability, and the strings must not read as a crawler that names itself; the rest are recorded for the selector and the digest. The judgment may lower the actor’s tier below its ceiling, never raise it, and a missing or failed answer lands the actor on observe. The thresholds are not set by hand: they are derived from a labeled replay, and a policy test pins them to the recording that produced them. Each has a review band of ±0.03, about how far identical requests move: an answer inside the band never passes, and every recorded answer of every labeled case lands outside it on its correct side. The option order of each question is pinned too, because reordering a question’s options changes the request and can change the answer.
An OpenAI model makes the call an operator would otherwise make: whether any candidate warrants acting this tick, which one costs the site the most, the least severe action that would stop it, and for how long, with a short rationale for the panel. It receives a code-built list of candidates and returns an index into it, an action from a typed set, a time to live and the rationale, under a strict JSON schema whose index bounds are the list’s own. Code validates the answer again, copies the target from the candidate it indexed, and clamps the action to that candidate’s judged tier. The model authors nothing: no test can cover every expression a model might write, but every index into a list can be checked. The rationale is stored as untrusted text and escaped where it is shown. A failed answer proposes nothing that tick, and a persistent failure, such as a rejected request, raises an alert.
A proposal above the detector’s ceiling is refused whatever its confidence. Caps: one action a tick, three a day, six a week. Blast radius is measured live over a week: a network, address or prefix challenge may cover at most 5% of the site’s real traffic, counting requests not already refused and leaving verified crawlers out (2% for a block), a user-agent cohort at most 5% of the site’s real browsers, and the shares of all live actions add up against each cap. The automation never creates a rule. It acts only on four existing surfaces (a network appended to the set inside an existing challenge rule, an owned clause appended to an existing rule, an owned IP list, a key-value deny with a native TTL), never on a protected rule, never on a country, and every action carries an expiry of one hour to seven days. Every owned clause exempts verified bots and my own networks and addresses; the user-agent clause, which matches a browser string from any network, also exempts the major platform networks, the API paths and the well-known paths, and a network, address or prefix target with any platform traffic is refused outright. A refused proposal becomes an audit row, with no retry that tick. A refusal that stops something the layers upstream should never have proposed, such as a target on my own network or a verified crawler, is a trip; a refusal on an allowlist, a cap or the blast radius also sends an alert.
A prefix-wide action affects every address in that prefix, so its profile reads the whole prefix rather than just the suspicious members that first drew attention. Before an address, prefix or network action, a separate count checks for the application’s own API clients, including clients absent from the most common user-agent rows. Any such traffic vetoes the action: those clients cannot solve a browser challenge. An unreadable protected-address range blocks the decision instead of being counted as zero protected traffic.
Ten and twenty-one minutes after an action, the intervals the hand procedure it replaced used, the worker reads the action’s own firewall events. If they show that the action caught a request from my networks, protected addresses or a verified crawler, the action is revoked at once, enforcement drops to shadow, and I am alerted. Actions whose enforcement surface cannot encode every exemption continue to be checked while active, with another check due after forty-five minutes. An address that first appears after the initial checks is still covered. Real-user collateral has no automatic trip of its own: the blast-radius cap bounds it in advance, and after each change I read challenges issued against challenges solved, by browser version. Every write can be undone byte for byte: the clause the worker appends ends in a marker it can find again, removing it twice is harmless, no addition may pass the provider’s expression limit, and a live entry can always be removed from the rule or list it was written to, even after the configured identifier changes. The admin panel holds no edge token: it writes a mode change to the mode row, which the next tick reads, and queues revokes and run requests, which the worker polls every minute.
In shadow the whole pipeline runs and the exact calls are rendered and stored as “would have done”. Nothing is sent. Enforcement switches itself on, with no step from me, when the labeled replay is false-positive-free and a shadow window completes clean: at least a day, at least twenty-two counted ticks, no gap over three hours, the judgment present on every tick, and no safety veto. It drops back to shadow on a dirty replay, three trips inside a day, or collateral found by a post-action check. A tick whose detection could not run is a gap in the window, not evidence. Once enforcing, a veto repeated on the same target and reason counts as one trip toward demotion, not one a tick.
Hand-written edge rules hold known, frozen signatures permanently and cost nothing. The automation covers whatever moves, and its “already contained” veto keeps it from duplicating a hand action. It is not a permanent hold: re-proposal waits for the next tick, and the caps and the automatic drop back to shadow bound it. Neither layer is widened to match current browser versions, which real users run, until a measurement shows how many real browsers the change would reach.
The unit tests include failure injection (a sink that throws once, timeouts, 4xx and 5xx answers), ordering invariants (a trip is remembered only after its audit row lands) and boundary values on the caps (5% passes, a hair over is refused). Property-based tests cover the pure pieces, among them the renderer’s refusal to ever write a country into an expression. Regression cases also cover a protected address appearing after the first checks, prefix-wide profiling, an API client outside the top user-agent rows, and an invalid allowlist range.
The labeled replay is a set of real observations from the analytics API, with addresses summarized and none stored, each labeled by hand. A gate test runs every case through the live detector, then tries the worst case against it: every action the actor’s kind has a surface for, each put through the same guardrails and renderer the tick uses. The gate is green only if no false positive would ever execute. Because the judgment layer can only lower a tier, code alone has to refuse every false positive before any model answer is trusted. Sixteen cases is a small set: enough to rule out the known false-positive classes, not to trust a threshold’s exact value, which is why the detector’s ceilings and the guardrails carry the safety, and the set grows with each new labeled wave.
A drill command first checks, with no write, that one action would clear every guardrail, the renderer’s refusals and a length margin. It can then apply a one-hour challenge on a made-up browser string that nobody sends, through the worker’s own executor, revoke it, and confirm the rule is back to its before-state byte for byte. Before enforcement was first allowed, the service ran a full day in shadow in production and I read its “would have done” records; the evidence gate above is what first turned enforcement on. After every change I read the audit rows, the edge’s firewall events (challenges issued against challenges solved, by browser version, network and country, with my own networks read separately) and the human baseline from web analytics, hour by hour across the change.