Check robots.txt before AI search can find your blog

A maker on X warns that one wrong line in robots.txt, or a Cloudflare setting, can block the bots ChatGPT, Claude and Perplexity use to show sites in their answers. The shared prompt has Claude write a robots.txt that allows AI search bots, makes AI training a separate choice, and pairs it with an llms.txt file listing your key pages.

Done with CClaude

"one wrong line in robots.txt (or a cloudflare setting) can block their bots, and most makers never check"x.com

How do you do it, step by step?

A maker on X warns that one wrong line in robots.txt, or a Cloudflare setting, can block the bots ChatGPT, Claude and Perplexity use to show sites in their answers.
The shared prompt has Claude write a robots.txt that allows AI search bots, makes AI training a separate choice, and pairs it with an llms.txt file listing your key pages.
Copy the prompt on this page into Claude and fill in your own details.
Edit the output in your own voice and check any facts before you publish.

Which tool does this use?

Claude. See all Claude use cases or more in WordPress.

What does it cost?

Free; Pro $20/mo

What should you watch out for?

Features and prices change. Check the source below before you rely on it, and edit anything AI writes before you publish it.

What prompt should you use with Claude?

Rebuild any prompt with the prompt helper

You are a blog editor who knows WordPress and search.

Context:
- I write for [WHO YOU WRITE FOR] on [YOUR PLATFORM].
- My voice: [YOUR VOICE]
- My goal: [YOUR GOAL]

Task:
Here is my current robots.txt [PASTE YOUR TEXT HERE] and my main pages [PASTE URLS]. My blog is about [TOPIC]. Tell me whether it blocks any AI search bots such as OAI-SearchBot, Claude-SearchBot or PerplexityBot. Then write a corrected robots.txt that allows search bots, asks me separately whether to allow training bots like GPTBot and ClaudeBot, blocks private paths, and lists my sitemap. Also draft an llms.txt that lists my key pages with one-line descriptions.

Constraints:
- Avoid [WHAT TO AVOID].
- Use only facts I give you. Don't invent stats, quotes, names or examples. Write [NEEDS DETAIL] wherever something is missing.
- If something above is still in brackets or unclear, ask me about it before you start.

Output format:
Follow the counts and format given in the task. Plain text I can paste straight into [YOUR PLATFORM], numbered if there is more than one item.

Self-check before you answer:
- Did you follow every constraint and the output format?
- Does it sound like my voice and not generic AI writing? Cut filler openers and buzzwords.
- Is every claim backed by something I gave you, or marked [NEEDS DETAIL]?
Fix anything that fails, then give me the result.

Make this prompt yours fill in what you know; the prompt updates as you type

Anything left in [BRACKETS] is for you to fill in, or the AI will ask you. Nothing you type leaves your browser. Want to rebuild a prompt you found elsewhere? Use the prompt helper.

Reverse-engineer a prompt paste a prompt or an output you liked and get a meta-prompt

Your meta-prompt

Paste this into ChatGPT or Claude. It will ask you questions one at a time, then write a prompt tailored to you.

Open Claude ↗

Frequently asked questions

How do I check robots.txt before AI search can find your blog?

Which tool do I need for this, and what does it cost?

This use case uses Claude. Pricing listed for it: Free; Pro $20/mo. Check claude.com for current prices.

What prompt can I use?

You are a blog editor who knows WordPress and search. Context: - I write for [WHO YOU WRITE FOR] on [YOUR PLATFORM]. - My voice: [YOUR VOICE] - My goal: [YOUR GOAL] Task: Here is my current robots.txt [PASTE YOUR TEXT HERE] and my main pages [PASTE URLS]. My blog is about [TOPIC]. Tell me whether it blocks any AI search bots such as OAI-SearchBot, Claude-SearchBot or PerplexityBot. Then write a corrected robots.txt that allows search bots, asks me separately whether to allow training bots like GPTBot and ClaudeBot, blocks private paths, and lists my sitemap. Also draft an llms.txt that lists my key pages with one-line descriptions. Constraints: - Avoid [WHAT TO AVOID]. - Use only facts I give you. Don't invent stats, quotes, names or examples. Write [NEEDS DETAIL] wherever something is missing. - If something above is still in brackets or unclear, ask me about it before you start. Output format: Follow the counts and format given in the task. Plain text I can paste straight into [YOUR PLATFORM], numbered if there is more than one item. Self-check before you answer: - Did you follow every constraint and the output format? - Does it sound like my voice and not generic AI writing? Cut filler openers and buzzwords. - Is every claim backed by something I gave you, or marked [NEEDS DETAIL]? Fix anything that fails, then give me the result.

What are related use cases?

Map the sub-queries AI search fires for your topic (ChatGPT)
When Google traffic falls off a cliff, check Search Console first (Google Search Console)
Write what AI Overviews can't summarize (Claude)
Don't burn your domain on autoblogging (ChatGPT)
Have Claude take apart another writer's Substack post (Claude)
You can't judge your own LinkedIn hook (Claude)

Where does this come from?

Checked Oct 18, 2026.

5 use cases every week

Get 5 like these in your inbox

Each one comes with a prompt you can paste. One short email a week.

Unsubscribe any time. Privacy

Last updated Oct 18, 2026