PlugPress
Howdy, visitor Get PlugPress One

How AI Bots Like GPTBot Find Your Markdown Pages

Waggle publishes a Markdown (.md) version of your content so AI crawlers and assistants can read your pages as clean text — roughly an 80% token reduction compared to parsing the HTML, which directly improves how well an AI system understands and cites your content. A natural question follows: can Waggle make AI bots read the .md version instead of the HTML page?

The honest answer: you can invite, you can’t compel. Here’s how the invitation works, how to see whether bots accept it, and why the one “forcing” technique that exists is a trap Waggle deliberately avoids.

The signals Waggle sends

Real-world crawler-log research shows AI bots discover Markdown variants through advertised links — when the signals are present, GPTBot makes up to a third of its requests to .md URLs. Waggle sends every signal that works:

  1. An alternate link in each page’s HTML — <link rel="alternate" type="text/markdown" href="…/your-page.md">, the same mechanism feed readers have used for decades. This is the discovery channel crawler logs show actually gets used.
  2. The same link as an HTTP response header — Link: <…/your-page.md>; rel="alternate"; type="text/markdown" on the HTML page, for bots that read headers without parsing HTML.
  3. llms.txt links straight to .md — every entry in your /llms.txt hands agents the Markdown URL, not the HTML permalink, and the preamble tells them any page’s Markdown is one .md suffix away.
  4. Content negotiation — a client sending Accept: text/markdown to any page URL gets Markdown back. No AI crawler observed in the wild uses this yet, but MCP clients and agent tooling do.

How to see whether bots take the invitation

The AI Visibility screen shows it directly: under the fetch count you’ll see what share of AI-bot reads took the Markdown channel (e.g. “18% of those reads (342) took the Markdown (.md) channel”). The same split is available to agents through the waggle/get-ai-bot-traffic ability as md_share. Expect the share to differ by bot — OpenAI’s crawlers adopted Markdown fastest; others still mostly read HTML. Both work: the HTML page and the .md page carry the same content, so either way the bot gets your message.

Why Waggle will never redirect bots to .md

The one technique that would literally make bots “visit .md instead” is user-agent sniffing: detect GPTBot or ClaudeBot by its user-agent string and serve or redirect it to the Markdown. Waggle deliberately does not do this, ever. Serving different responses to identified crawlers than to users is cloaking — the practice Google’s spam policies explicitly penalize, and a ranking risk no visibility gain justifies. Every response Waggle serves depends only on what was requested (the URL, the Accept header), never on who requested it.

Two more non-fixes we’re often asked about, and skip on purpose:

  • Blocking .md in robots.txt (“so Google only sees the HTML”) — Google may still index a robots-blocked URL from links alone, contentless, and the block would hide the noindex signal that actually keeps .md out of search. The .md pages are already noindex with a canonical header pointing at the HTML page; that is Google’s own documented pattern for alternate-format files, and it’s what keeps them out of search results while staying fully readable by AI crawlers.
  • Hidden “AI, read this” text in the page — parsers strip or ignore it inconsistently, and llms.txt already carries the same instruction in a place agents actually look.