Why AI Bots Read Your Normal Pages Instead of Your .md Files

Symptom. Your AI bot log shows GPTBot, ClaudeBot, PerplexityBot and friends
crawling the site — but they always fetch your normal pages, never the .md
versions. The Markdown share on the Bot traffic screen reads 0%. It looks like
the whole Markdown feature is being ignored.

Why this happens. Nothing is broken — this is how the two kinds of AI
visitors actually behave, and Waggle is built around the difference.

AI crawlers work like Google. The big crawlers (GPTBot, ClaudeBot,
PerplexityBot, Google-Extended, CCBot) discover pages the classic way: they read
robots.txt, follow the Sitemap: line, and fetch the pages the sitemap lists.
Waggle's sitemap lists only your normal pages — deliberately. When the .md
versions were listed there too, Google treated that as an endorsement and
indexed the .md pages right beside the real ones, creating duplicate-content
damage visible in Search Console. Keeping them out of the sitemap is what
protects your search results.

There's a second reason crawlers stick to your normal pages: trust. A .md
file is self-declared — a site could put anything in it. Crawlers read the
rendered page because that's the version a human sees; an alternate copy they
can't verify would be an easy way to feed them different content (search
engines call this cloaking, and they penalize it). So no major AI company's
crawler fetches .md alternates or llms.txt today.

And that costs you nothing. AI models read the text of your rendered pages
— that's what gets tokenized, learned from, and cited. A crawler fetching your
normal page gets everything it needs. The .md version is not something
crawler traffic was ever going to switch to.

So who uses the .md files? The other kind of visitor: AI agents — tools
that fetch a page on demand because a user or workflow asked for it. Assistants
doing live lookups (the -User fetchers in your bot log, like ChatGPT-User and
Claude-User), coding agents, MCP-connected tools, and anything that reads
llms.txt. For them, a clean Markdown version with no theme markup is genuinely
better — fewer tokens, no navigation noise, front matter with your title and
description. That audience is smaller and newer than the crawlers, which is why
the Markdown share starts low. Waggle advertises the channel everywhere that
audience looks: a link tag on every page, an HTTP header, an entry in
llms.txt, and an llms.txt pointer in your robots.txt.

How to confirm. In Waggle, open AI Visibility → Bot traffic: the line
under the fetch count shows what share of bot reads took the Markdown channel,
and the crawler list below shows which companies are reading the site. To see
the channel working, open any page's .md version yourself — AI Visibility →
LLMs.txt & Markdown → See what AI sees — or fetch it from a terminal:

curl -sI https://example.com/your-page.md

A 200 with Content-Type: text/plain means the channel is live and serving;
agents that ask for it get it.

One caveat while checking robots.txt: if another SEO plugin is active alongside
Waggle (All in One SEO, Yoast, Rank Math), it may take over robots.txt output
entirely and drop Waggle's Sitemap: and # llms.txt: lines. The .md pages
and llms.txt itself still serve either way — only the robots.txt pointer is
affected.

Fix. There's nothing to fix — but there are two things not to do, because
they look like fixes and backfire:

1. Don't add .md URLs to your sitemap (with Waggle or any other tool).
   That's the configuration that got .md pages indexed next to their real
   pages in search results.
2. Don't serve Markdown to bots at your normal URLs based on their
   user-agent. That's cloaking; it damages trust with every crawler and can get
   a site penalized.

The honest picture: your normal pages serve the crawlers, your Markdown serves
the agents, and Waggle keeps the two channels from stepping on each other.