--- title: "How AI Bots Like GPTBot Find Your Markdown Pages" url: "https://plugpress.co/docs/waggle-how-ai-bots-find-markdown-pages.md" canonical: "https://plugpress.co/docs/waggle-how-ai-bots-find-markdown-pages/" published: "2026-07-28" modified: "2026-07-28" author: "Fahim" tags: - "Waggle" --- # How AI Bots Like GPTBot Find Your Markdown Pages Waggle publishes a Markdown (`.md`) version of your content so AI crawlers and assistants can read your pages as clean text — roughly an 80% token reduction compared to parsing the HTML, which directly improves how well an AI system understands and cites your content. A natural question follows: *can Waggle make AI bots read the `.md` version instead of the HTML page?* The honest answer: you can invite, you can't compel. Here's how the invitation works, how to see whether bots accept it, and why the one "forcing" technique that exists is a trap Waggle deliberately avoids. ## The signals Waggle sends Real-world crawler-log research shows AI bots discover Markdown variants through advertised links — when the signals are present, GPTBot makes up to a third of its requests to `.md` URLs. Waggle sends every signal that works: 1. **An alternate link in each page's HTML** — ``, the same mechanism feed readers have used for decades. This is the discovery channel crawler logs show actually gets used. 2. **The same link as an HTTP response header** — `Link: <…/your-page.md>; rel="alternate"; type="text/markdown"` on the HTML page, for bots that read headers without parsing HTML. 3. **llms.txt links straight to `.md`** — every entry in your `/llms.txt` hands agents the Markdown URL, not the HTML permalink, and the preamble tells them any page's Markdown is one `.md` suffix away. 4. **Content negotiation** — a client sending `Accept: text/markdown` to any page URL gets Markdown back. No AI crawler observed in the wild uses this yet, but MCP clients and agent tooling do. ## How to see whether bots take the invitation The AI Visibility screen shows it directly: under the fetch count you'll see what share of AI-bot reads took the Markdown channel (e.g. "18% of those reads (342) took the Markdown (.md) channel"). The same split is available to agents through the `waggle/get-ai-bot-traffic` ability as `md_share`. Expect the share to differ by bot — OpenAI's crawlers adopted Markdown fastest; others still mostly read HTML. Both work: the HTML page and the `.md` page carry the same content, so either way the bot gets your message. ## Why Waggle will never redirect bots to .md The one technique that would literally make bots "visit .md instead" is user-agent sniffing: detect GPTBot or ClaudeBot by its user-agent string and serve or redirect it to the Markdown. **Waggle deliberately does not do this, ever.** Serving different responses to identified crawlers than to users is cloaking — the practice Google's spam policies explicitly penalize, and a ranking risk no visibility gain justifies. Every response Waggle serves depends only on what was *requested* (the URL, the Accept header), never on *who* requested it. Two more non-fixes we're often asked about, and skip on purpose: - **Blocking `.md` in robots.txt** ("so Google only sees the HTML") — Google may still index a robots-blocked URL from links alone, contentless, and the block would hide the `noindex` signal that actually keeps `.md` out of search. The `.md` pages are already `noindex` with a canonical header pointing at the HTML page; that is Google's own documented pattern for alternate-format files, and it's what keeps them out of search results while staying fully readable by AI crawlers. - **Hidden "AI, read this" text in the page** — parsers strip or ignore it inconsistently, and llms.txt already carries the same instruction in a place agents actually look. ## Related guides - [Machine files: llms.txt, Markdown, and pricing.md](/docs/waggle-machine-files/) - [Troubleshooting: Google is indexing your .md pages](/docs/waggle-troubleshooting-google-indexing-md-pages/) - [Troubleshooting: a page has a .md version you didn't expect](/docs/waggle-troubleshooting-unexpected-md-pages/)