Is your website telling ChatGPT to go away?
More and more people looking for a tradesman now ask their phone instead of scrolling Google. We checked 259 trade websites, and nineteen of them have a file sat on the server telling those assistants not to read the site at all.
Not one of those owners, as far as we can tell, decided that. It arrived on its own.
How much does this actually matter?
A survey this year found 45% of people have used an AI tool to find a local business, up from 6% the year before. That is a big jump in twelve months.
Being straight with you though: this is not where your work comes from today. Google and the map on your phone still bring in most of the calls, and anybody telling you to drop everything and chase AI is selling something. What has changed is that it is now worth two minutes of your time, and two minutes is genuinely all this takes.
The two-minute check
Type your website address into your phone, then add /robots.txt on the end. So if your site is smithfencing.co.uk you would type:
smithfencing.co.uk/robots.txt
You will get a short bit of plain text. Nothing will break. Now look for any of these words:
- GPTBot — that is ChatGPT
- ClaudeBot — that is Claude
- Google-Extended — Google’s AI answers
- CCBot — feeds a lot of the others
If you find one of those with the words Disallow: / underneath it, that is an instruction saying do not read this website.
Where does it come from?
Usually a security setting. Cloudflare, which protects a huge number of small business websites, has a single switch labelled something like “block AI scrapers”. Plenty of website plugins have the same. Somebody ticks it while setting the site up, often with the best of intentions, and the file gets written.
We could tell it was a switch rather than a decision because the list of blocked names was near enough identical on all nineteen sites. No nineteen separate people independently type the same eight names in the same order.
What to do if you find it
Ring whoever built or hosts your site and say: “my robots.txt is blocking the AI crawlers, can you turn that off?” It is a two-minute job for them. That sentence is all you need — you do not have to understand the file to have it changed.
And if you would rather stay blocked, that is a fair choice. It is your website and your content. Just make it your choice rather than a checkbox somebody else ticked.
One thing worth saying
While we were checking, twelve other sites refused our own checking tool. It would have been easy to write those up as blocking AI too. They were not — we went back and asked again pretending to be Google, then ChatGPT, then Claude, and all three were let straight in. The only thing being turned away was us.
We mention it because it would have made a better headline and it would not have been true.