How AI Bots Like GPTBot Find Your Markdown Pages

Waggle publishes a Markdown (.md) version of your content so AI crawlers and
assistants can read your pages as clean text — roughly an 80% token reduction
compared to parsing the HTML, which directly improves how well an AI system
understands and cites your content. A natural question follows: can Waggle make
AI bots read the .md version instead of the HTML page?

The honest answer: you can invite, you can't compel. Here's how the invitation
works, how to see whether bots accept it, and why the one "forcing" technique
that exists is a trap Waggle deliberately avoids.

The signals Waggle sends

Real-world crawler-log research shows AI bots discover Markdown variants through
advertised links — when the signals are present, GPTBot makes up to a third of
its requests to .md URLs. Waggle sends every signal that works:

1. An alternate link in each page's HTML —
   <link rel="alternate" type="text/markdown" href="…/your-page.md">, the same
   mechanism feed readers have used for decades. This is the discovery channel
   crawler logs show actually gets used.
2. The same link as an HTTP response header — Link: <…/your-page.md>; rel="alternate"; type="text/markdown" on the HTML page, for bots that read
   headers without parsing HTML.
3. llms.txt links straight to .md — every entry in your /llms.txt hands
   agents the Markdown URL, not the HTML permalink, and the preamble tells them
   any page's Markdown is one .md suffix away.
4. Content negotiation — a client sending Accept: text/markdown to any
   page URL gets Markdown back. No AI crawler observed in the wild uses this
   yet, but MCP clients and agent tooling do.

How to see whether bots take the invitation

The Bot traffic screen shows it directly: under the fetch count you'll see
what share of AI-bot reads took the Markdown channel (e.g. "18% of those reads
(342) took the Markdown (.md) channel"). The same split is available to agents
through the waggle/get-ai-bot-traffic ability as md_share. Expect the share
to differ by bot — OpenAI's crawlers adopted Markdown fastest; others still
mostly read HTML. Both work: the HTML page and the .md page carry the same
content, so either way the bot gets your message.

Why Waggle will never redirect bots to .md

The one technique that would literally make bots "visit .md instead" is
user-agent sniffing: detect GPTBot or ClaudeBot by its user-agent string and
serve or redirect it to the Markdown. Waggle deliberately does not do this,
ever. Serving different responses to identified crawlers than to users is
cloaking — the practice Google's spam policies explicitly penalize, and a
ranking risk no visibility gain justifies. Every response Waggle serves depends
only on what was requested (the URL, the Accept header), never on who
requested it.

Two more non-fixes we're often asked about, and skip on purpose:

• Blocking .md in robots.txt ("so Google only sees the HTML") — Google
  may still index a robots-blocked URL from links alone, contentless, and the
  block would hide the noindex signal that actually keeps .md out of
  search. The .md pages are already noindex with a canonical header
  pointing at the HTML page; that is Google's own documented pattern for
  alternate-format files, and it's what keeps them out of search results while
  staying fully readable by AI crawlers.
• Hidden "AI, read this" text in the page — parsers strip or ignore it
  inconsistently, and llms.txt already carries the same instruction in a place
  agents actually look.