robots.txt
This is the robots.txt tag archive on Will's Journal: every published post that shares this tag, listed in one place.
It is a collection page, not a topic essay — scan the cluster here instead of filtering the full journal index.
Which posts are tagged robots.txt?
AI Visibility Why do ~30% of sites block AI crawlers without knowing it
Top-1,000 GPTBot Disallows hit 35.7% in Originality.AI’s Aug 2024 cut. Accidental blocks come from CDN defaults, plugins, leftover robots, and Googlebot mixups.
AI Visibility Can blocking AI crawlers remove me from ChatGPT answers
No. Blocking GPTBot stops future training crawls. ChatGPT search uses OAI-SearchBot. Trained weights do not rewind, and other sources can still name you.
AI Visibility Block Training Bots if You Want — Don’t Accidentally Block Being Cited
Block GPTBot for training if you want — keep OAI-SearchBot allowed for ChatGPT search citations. Google-Extended does not control Google Search indexing.
What should you know about this tag archive?
What is this tag archive?
This is the robots.txt tag archive on Will's Journal: every published post that shares this tag, listed in one place.
Is this a guide to the topic, or a list of posts?
A list of posts. This page is a collection, not a topic essay — the 3 posts below are the robots.txt cluster on Will's Journal.