Google Is Indexing Your .md Pages: Why It Happens and How to Fix It

What to do when Google Search Console shows your pages indexed twice — once as the
normal page, once as its .md Markdown version.

Google indexes both /my-post/ and /my-post.md

Symptom. Search Console's Pages report (or a site:example.com search) shows the
same content indexed at two URLs: the normal page and a plain-text .md variant.
Sometimes the .md version even appears in search results, showing raw Markdown to
visitors.

Why this happens. Waggle publishes a Markdown version of every page (append .md
to any URL) so AI assistants can read your content cleanly. In Waggle versions up to
1.0.0, those .md URLs were also listed in your XML sitemap, and the .md response
relied on a Link: rel="canonical" header alone to tell search engines "the real page
is the HTML one." Google treats a canonical header as a hint, not a rule — and treats
a sitemap listing as a vote for indexing. With the two versions' content differing
(Markdown vs. HTML), Google sided with the sitemap and indexed both.

How to confirm.

1. In Search Console, open URL Inspection and inspect any .md URL — if it says
   "URL is on Google," you're affected.

2. Or search Google for site:example.com inurl:.md and look for plain-text results.

3. Check what your site currently serves:

   curl -sI https://example.com/some-post.md | grep -i 'x-robots-tag|link'
   

   On a fixed site you'll see both X-Robots-Tag: noindex and the
   Link: <…>; rel="canonical" header. If X-Robots-Tag is missing, you're on the
   affected version.

Fix.

1. Update Waggle past 1.0.0. The update does two things at once: .md URLs leave
   your sitemap (it goes back to normal pages only), and every .md response sends
   X-Robots-Tag: noindex. AI assistants still find and read the Markdown — it stays
   listed in your llms.txt, and every page keeps its <link rel="alternate">
   pointer — but search engines are now explicitly told to drop it.
2. Let Google clean itself up. As Google recrawls each .md URL it sees the
   noindex and removes it from the index. This takes days to a few weeks depending on
   crawl rate. In Search Console the Pages report's "Excluded by 'noindex' tag" count
   will grow as the duplicates drain out — that's the fix working.
3. For anything embarrassing right now, use Search Console → Removals →
   New request with the .md URL (or the .md prefix) to hide it from results
   immediately while the recrawl catches up.
4. Do not block .md in robots.txt. It feels like the stronger move, but a
   blocked URL can stay in the index ("indexed, though blocked by robots.txt")
   because Google can no longer see the noindex. Let the noindex do its job — and
   blocking would also cut off the AI crawlers the Markdown exists for.