---
title: "Why AI Bots Read Your Normal Pages Instead of Your .md Files"
url: "https://plugpress.co/docs/waggle-why-ai-bots-read-html-not-markdown.md"
canonical: "https://plugpress.co/docs/waggle-why-ai-bots-read-html-not-markdown/"
published: "2026-08-03"
modified: "2026-08-03"
author: "Fahim"
tags:
  - "Waggle"
---

# Why AI Bots Read Your Normal Pages Instead of Your .md Files

**Symptom.** Your AI bot log shows GPTBot, ClaudeBot, PerplexityBot and friends
 crawling the site — but they always fetch your normal pages, never the `.md`
 versions. The Markdown share on the Bot traffic screen reads 0%. It looks like
 the whole Markdown feature is being ignored.

**Why this happens.** Nothing is broken — this is how the two kinds of AI
 visitors actually behave, and Waggle is built around the difference.

**AI crawlers work like Google.** The big crawlers (GPTBot, ClaudeBot,
 PerplexityBot, Google-Extended, CCBot) discover pages the classic way: they read
 `robots.txt`, follow the `Sitemap:` line, and fetch the pages the sitemap lists.
 Waggle's sitemap lists only your normal pages — **deliberately**. When the `.md`
 versions were listed there too, Google treated that as an endorsement and
 indexed the `.md` pages right beside the real ones, creating duplicate-content
 damage visible in Search Console. Keeping them out of the sitemap is what
 protects your search results.

There's a second reason crawlers stick to your normal pages: trust. A `.md`
 file is self-declared — a site could put anything in it. Crawlers read the
 rendered page because that's the version a human sees; an alternate copy they
 can't verify would be an easy way to feed them different content (search
 engines call this *cloaking*, and they penalize it). So no major AI company's
 crawler fetches `.md` alternates or `llms.txt` today.

**And that costs you nothing.** AI models read the text of your rendered pages
 — that's what gets tokenized, learned from, and cited. A crawler fetching your
 normal page gets everything it needs. The `.md` version is not something
 crawler traffic was ever going to switch to.

**So who uses the .md files?** The other kind of visitor: AI *agents* — tools
 that fetch a page on demand because a user or workflow asked for it. Assistants
 doing live lookups (the `-User` fetchers in your bot log, like ChatGPT-User and
 Claude-User), coding agents, MCP-connected tools, and anything that reads
 `llms.txt`. For them, a clean Markdown version with no theme markup is genuinely
 better — fewer tokens, no navigation noise, front matter with your title and
 description. That audience is smaller and newer than the crawlers, which is why
 the Markdown share starts low. Waggle advertises the channel everywhere that
 audience looks: a link tag on every page, an HTTP header, an entry in
 `llms.txt`, and an `llms.txt` pointer in your `robots.txt`.

**How to confirm.** In Waggle, open **AI Visibility → Bot traffic**: the line
 under the fetch count shows what share of bot reads took the Markdown channel,
 and the crawler list below shows which companies are reading the site. To see
 the channel working, open any page's `.md` version yourself — **AI Visibility →
 LLMs.txt & Markdown → See what AI sees** — or fetch it from a terminal:

```
curl -sI https://example.com/your-page.md
```

A `200` with `Content-Type: text/plain` means the channel is live and serving;
 agents that ask for it get it.

One caveat while checking `robots.txt`: if another SEO plugin is active alongside
 Waggle (All in One SEO, Yoast, Rank Math), it may take over `robots.txt` output
 entirely and drop Waggle's `Sitemap:` and `# llms.txt:` lines. The `.md` pages
 and `llms.txt` itself still serve either way — only the robots.txt pointer is
 affected.

**Fix.** There's nothing to fix — but there are two things *not* to do, because
 they look like fixes and backfire:

1. **Don't add `.md` URLs to your sitemap** (with Waggle or any other tool). That's the configuration that got `.md` pages indexed next to their real pages in search results.
2. **Don't serve Markdown to bots at your normal URLs** based on their user-agent. That's cloaking; it damages trust with every crawler and can get a site penalized.

The honest picture: your normal pages serve the crawlers, your Markdown serves
 the agents, and Waggle keeps the two channels from stepping on each other.