Your user-agent check believes anyone

The header everyone forges

Every link that treats machines differently has to decide, on every request, who is asking. Almost every link layer makes that decision on the User-Agent header, and anyone can type GPTBot into curl. IP lists are not much better for agents: they run in the same clouds as everyone else.

At a redirect this fails in two directions. A script that copies an agent's user agent gets whatever you reserved for agents, or drops out of your click counts. A real agent driving a full browser sends an ordinary Chrome user agent and gets counted as a person. Our Links for LLMs piece routes the machine answer on a query parameter and uses the user agent only to label traffic; this article is about replacing that label with proof.

Web Bot Auth is the proof. The agent signs each request with HTTP Message Signatures (RFC 9421) and publishes its public keys on a domain it controls. On 1 September 2026 the IETF's webbotauth working group published draft-ietf-webbotauth-httpsig-protocol-00, its first adopted document, by Thibault Meunier (Cloudflare) and Sandor Major (Google). It is an Internet-Draft, not an RFC: everything below is "as of draft-00". The charter's first milestones were dated 30 April 2026 and passed before anything was adopted, so treat the timeline as soft.

What a signed request carries

Three headers:

• Signature-Agent: an https URL that says where the agent's keys live.
• Signature-Input: which parts of the request are signed, plus the parameters.
• Signature: the signature bytes.

The draft sets the rules. The tag parameter must be web-bot-auth. The signature must cover at least @authority (the host the request was sent to) or @target-uri. It must carry created, expires and keyid, and expiry is recommended to be no more than 24 hours. The keyid is the base64url SHA-256 thumbprint of the public key as a JWK. The Signature-Agent header must itself be covered, so nobody can swap the key location without breaking the signature. The keys sit at /.well-known/http-message-signatures-directory on the agent's origin, as a JSON Web Key Set served over HTTPS with the media type application/http-message-signatures-directory+json.

Here is a request signed and then verified with Cloudflare's web-bot-auth package (0.2.0), using the public Ed25519 test key from RFC 9421. Never use that key for anything real.

$ node test.mjs
keyid: poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U
Signature-Agent: agent1="https://signature-agent.test"
Signature-Input: sig1=("@authority" "signature-agent";key="agent1");created=1791781395;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U";alg="ed25519";expires=1791781455;nonce="NV0TJzqaeT9hyI8bhmz7C6LWB68sOEJwUXawzQMFVOv4Ldkek3XBOW+UPOcr1mNgYX3ziTzGbjE1GbZsJ7cQnA==";tag="web-bot-auth"
Signature: sig1=:EmZkw1WQP/b4bYU60xyoZ/XsDSRcYWDZDl...
verify go.example.com -> OK https://signature-agent.test
verify www.example.com -> FAIL: Signature verification failed
{"kind":"legacy","entries":[{"label":"","uri":"https://chatgpt.com","type":"directory"}]}
{"kind":"current","entries":[{"label":"g","uri":"https://agent.bot.goog","type":"directory"}]}

Three things to read from it. The keyid matches the one in the draft's own Appendix E example, because both are derived from the same RFC 9421 test key: a cheap check that your thumbprint code is right. The identical headers verify on go.example.com and fail on www.example.com, because @authority is part of what was signed. And the last two lines show the two formats of Signature-Agent that are in use right now.

One header, two formats

The draft defines Signature-Agent as a Structured Fields Dictionary, like g="https://agent.bot.goog". Signers must send that form; verifiers may also accept the older bare string. Google and OpenAI split down that line. Google sends the dictionary form for its Google-Agent fetcher. OpenAI documents its cloud browser's header as "https://chatgpt.com", "including the quotation marks", which is the legacy string.

Cloudflare's verifier sides with the legacy form. Its verification reference, updated on 8 October 2026, accepts only the quoted string, says the dictionary form fails, and lists the key component parameter as unsupported. If you verify yourself, accept both forms.

Check your library, too. Covering one member of a dictionary header uses the key parameter from RFC 9421 section 2.1.2. In our test, the Python package http-message-signatures 2.0.1 verified a signature over @authority alone, but rejected a valid signature once it also covered "signature-agent";key="agent1": it fed the whole header into the signature base instead of the one member. A verifier that says "invalid" may be the thing that is wrong.

Verifying at the edge

The order of operations at a redirect:

1. No Signature header: the request is unsigned. Fall back to whatever you did before.
2. Find the right signature. The draft lets you discard signatures whose tag is not web-bot-auth.
3. Check the agent before you fetch anything. Signature-Agent is attacker-controlled. If your edge fetches whatever URL it names, your redirect becomes a fetcher for strangers. Keep an allowlist of agent origins you have decided to recognise.
4. Fetch the directory carefully. Accept only a 200, and never follow redirects: the draft says verifiers "MUST NOT automatically follow HTTP redirects" during discovery. Cache it under normal HTTP caching rules, and do not throw away a good cached entry because a refresh failed.
5. Look keys up by the pair of agent URL and keyid, never by keyid alone, then verify and check the created to expires window.

The same steps as a sketch on the web-bot-auth package. We ran it against requests signed in both header formats, using a stubbed directory that held the test key:

import { verify } from "web-bot-auth";
import { verifierFromJWK } from "web-bot-auth/crypto";

// Only fetch key directories for agents you have decided to recognise.
const KNOWN_AGENTS = new Set([
  "https://chatgpt.com",
  "https://agent.bot.goog",
]);
const cache = new Map(); // agent URL -> { verifiers, until }

async function verifiersFor(agentUrl) {
  const hit = cache.get(agentUrl);
  if (hit && hit.until > Date.now()) return hit.verifiers;
  const url = new URL("/.well-known/http-message-signatures-directory", agentUrl);
  const res = await fetch(url, { redirect: "manual" }); // never follow redirects here
  if (res.status !== 200) throw new Error(`directory answered ${res.status}`);
  const { keys } = await res.json();
  const settled = await Promise.allSettled(keys.map((k) => verifierFromJWK(k)));
  const verifiers = settled.filter((s) => s.status === "fulfilled").map((s) => s.value);
  cache.set(agentUrl, { verifiers, until: Date.now() + 60 * 60 * 1000 });
  return verifiers;
}

export async function classifyAgent(request) {
  if (!request.headers.has("signature")) return { tier: "unsigned" };
  try {
    const sig = await verify(request, {
      resolver: async ({ keyid, signatureAgent }) => {
        const agent = signatureAgent?.uri;
        if (!KNOWN_AGENTS.has(agent)) throw new Error(`unknown agent ${agent}`);
        const match = (await verifiersFor(agent)).find((v) => v.keyid === keyid);
        if (!match) throw new Error("unknown key");
        return match;
      },
    });
    return { tier: "verified", agent: sig.signatureAgent?.uri };
  } catch (err) {
    return { tier: "claimed", reason: err.message };
  }
}

The fixed one-hour cache keeps the sketch short; production code should honour the directory's Cache-Control. The package's own README notes it has not been audited.

The signature stops at the redirect

A short link is a redirect, and that is where Web Bot Auth gets interesting for us. A signature over @authority covers the host the agent sent that one request to. When the short link answers 302, the agent's next request goes to a different authority, and the old signature is worthless there, exactly as the www.example.com line above shows. The destination only sees a verified agent if the agent signs the second request afresh.

Whether today's agents do that on every hop, we could not establish. Neither OpenAI's nor Google's documentation says, and Google states that it signs only "a subset of requests" from Google-Agent. We did not have a signing agent to test against for this article. Until someone publishes a test, assume nothing.

Two consequences follow. A link that verified an agent cannot hand that verdict to the destination in the URL: a query parameter like verified=1 is something anyone can add. The draft does allow verifying at a proxy and passing the result over a trusted internal channel, which works inside your own infrastructure and nowhere else. And everything between the agent and your verifier must leave the headers alone. The draft says a proxy "SHOULD NOT strip" Signature or Signature-Agent, OpenAI's troubleshooting asks you to confirm proxies keep all three, and a signature over @target-uri breaks if anything reorders the query string. If your short domain sits behind a CDN or an edge worker, check that it forwards them.

What changes for routing and analytics

With signatures, agent traffic falls into three tiers instead of two:

• Verified: the signature checks out against the directory of an agent you recognise.
• Claimed: an agent-looking user agent, an unknown Signature-Agent, or a signature that fails.
• Unsigned: everything else, including every agent that does not sign yet.

For routing, nothing public needs to change: a machine-readable answer behind a query parameter can serve anyone who asks. Anything with a cost or a privilege should change: higher rate limits, gated content, skipping a challenge. Those belong to the verified tier only.

For analytics, report verified agents as their own series rather than folding them into bots. A verified ChatGPT agent is often fetching because a person asked it about you, which is closer to a human visit than to a crawler. Claimed agents stay in the bot bucket, and our piece on security scanners clicking email links covers why that bucket needs to exist in the first place.

The verification side is already in production at the big edges. Vercel added Web Bot Auth to its bot verification on 12 August 2025, and AWS WAF followed on 21 November 2025, for CloudFront distributions only. OpenAI's page lists how its agent is identified at Cloudflare, Akamai, HUMAN and Vercel.

Where rii.link stands

rii.link flags known crawlers and AI agents by their user agent and filters bots out of its analytics by default, so your click charts are not inflated by them. That is a classification for cleaner reporting, with every caveat in the first section of this article. Nothing in rii.link's published product pages describes Web Bot Auth verification, so do not read the agent flag as proof of identity.

What to do this quarter

1. Log Signature, Signature-Input and Signature-Agent on your redirect now, even before you verify anything. You will want the history.
2. Check every hop in front of your verifier (CDN, worker, load balancer) passes all three headers through unchanged.
3. Stop granting anything that costs you money on the user agent alone.
4. Accept both Signature-Agent formats, and test your library on both.
5. Allowlist agent origins before your edge fetches a single key directory.
6. Watch the draft. The working group's document list already holds related work on hosted key directories and anonymous bot authentication, and draft-00 expires on 5 March 2027, so a revision is due before then.