What is content negotiation for AI, and why does it cut costs by 90%?

How does an AI ask for markdown?

With a standard HTTP header. The client sends Accept: text/markdown and a server that supports negotiation returns markdown with a matching content-type. A server that does not simply returns HTML and the client makes do.

$ curl -H "Accept: text/markdown" https://code4x.com/registry -I

HTTP/2 200
content-type: text/markdown; charset=utf-8
vary: accept
link: </llms.txt>; rel="alternate"; type="text/plain"

The Vary: accept header matters and is the most commonly omitted piece. Without it, caches and CDNs will happily serve your markdown to a browser or your HTML to an agent.

How much does it actually save?

On a typical marketing page, between eighty and ninety-nine per cent of what an AI reads is markup it does not need.

The second row is the one that matters commercially. An agent working within a context budget will read more of a cheap source and less of an expensive one, and when it is choosing between two sites that say roughly the same thing, cost is a real tiebreaker.

It also helps you directly: a page that fits comfortably in context gets summarised accurately, while a page that has to be truncated gets summarised from whatever fragment survived.

How do I implement content negotiation?

Next.js

export function middleware(req: NextRequest) {
  const accept = req.headers.get('accept') || ''
  if (accept.includes('text/markdown')) {
    const url = req.nextUrl.clone()
    url.pathname = `${req.nextUrl.pathname}.md`
    return NextResponse.rewrite(url)
  }
}

nginx

location / {
  if ($http_accept ~* "text/markdown") {
    rewrite ^(.*)$ $1.md break;
  }
  add_header Vary Accept;
}

Cloudflare Workers

export default {
  async fetch(request) {
    const accept = request.headers.get('Accept') || ''
    const url = new URL(request.url)
    if (accept.includes('text/markdown')) url.pathname += '.md'
    const res = await fetch(url, request)
    const out = new Response(res.body, res)
    out.headers.set('Vary', 'Accept')
    return out
  }
}

All three assume a markdown version exists at the same path with a .md suffix. If you write in markdown already, that is close to free. If you write in a CMS, generate the markdown at build time from the same source your HTML comes from — never by scraping your own HTML, which reintroduces every problem you are solving.

Which tools send this header today?

Coding assistants led here. Claude Code, Cursor and OpenCode send it natively, which makes documentation sites the highest-value early adopters. Several crawlers and fetch libraries have followed.

Adoption on the serving side remains around four per cent of sites, so the competitive window is unusually wide for something this cheap to implement.

Where this is contested

The savings are real and easy to verify yourself; whether they translate into more citations is not established. The mechanism is plausible — cheaper sources get read more completely — but we are not aware of a controlled study, and we would rather say so.

The honest case for implementing it is that it is a few hours of work, it makes your content demonstrably easier to consume, and it costs you nothing if the citation effect turns out to be small. That is a different argument from a guaranteed return, and it is the one we make.