Vector Index AI AI Search Intelligence

Free AI answer snapshot

Is your website blocking AI search?

Check your website and leave with an improvement plan. You'll see whether anything on it blocks ChatGPT search, Claude or Perplexity, and whether your robots.txt lets Google and Bing in, get one next step to hand to whoever runs your site, and get business information written from your own details. Add your details and we also email what ChatGPT and Google AI Overviews say when a customer asks. Free, no sign-up.

What you leave with

  1. Website access, one line per engine
  2. Your next step, with how to check it's done
  3. Business information, written from your details
  4. A plan you can print or save
  5. What ChatGPT and Google AI Overviews say, by email, if you add your details

We reviewed 29 free AI crawler checkers. 11 of them treat GPTBot as ChatGPT's search crawler. Read the study.

Part two

What AI says when a customer asks

Access is the first gate, not the whole answer. A crawler that can read your site still has to choose to cite it. Add your details and we ask ChatGPT and Google AI Overviews three questions a customer would ask, then email you every answer and whether your business was named. Only the first three fields are required. The AI answers need your email, city and customer question.

We email you the snapshot because you asked for it, plus one follow-up if you do not reply, and then we stop. We do not ask for your phone number and we do not sell your details to anyone.

For developers

Two tools that use the same rules as the check above. Nothing you paste is saved.

Test a robots.txt before you publish it

Paste a new or edited robots.txt and see what each crawler would be allowed to read, using the same rules as the check above. If you've just checked a site, your draft is compared with its live file so you can see exactly what would change. Your draft isn't fetched from anywhere or saved.

Where is my robots.txt, and how do I change it?

To see it, type your web address followed by /robots.txt into your browser, like example.com/robots.txt. Or run the check at the top of this page and click "Start from the live file".

Where you change it depends on what your site is built with. Menu names change from time to time, so each line links to that platform's own instructions.

  • WordPress with Yoast SEO: Yoast SEO, then Tools, then File Editor. Yoast help
  • WordPress with Rank Math: turn on Advanced Mode, then Rank Math SEO, General Settings, Edit robots.txt. Rank Math help
  • Shopify: Online Store, your theme's "..." menu, Edit code, Add a new template, then robots. Shopify says its support team can't help with this file and that mistakes can cost you traffic, so a Shopify Partner or developer is the safer route. Shopify help
  • Wix: your site dashboard, SEO & GEO, Tools and settings, Robots.txt Editor. Wix help
  • Squarespace: you can't edit the file, but check Settings, then Crawlers. If "Block known artificial intelligence crawlers" is ticked, Squarespace blocks 26 AI crawlers, GPTBot and ClaudeBot among them. It's unticked unless someone ticked it. Squarespace help
  • Webflow: Site settings, SEO, Indexing. The traffic control switches for search crawlers and AI bots are on by default; publish the site after any change. Adding your own rules needs a Site plan or paid Workspace plan. Webflow help
  • HubSpot: Settings, Content, Pages, choose your domain, then the SEO & Crawlers tab. HubSpot help
  • Anything else, or not sure: send the "message for whoever runs your site" from your results to your web developer or host.

Using Cloudflare? It can turn AI crawlers away before your robots.txt is ever read, so a clean robots.txt doesn't prove they can get in. Check Security Settings, then Configure AI bot policies, and AI Crawl Control, then Crawlers. Cloudflare's docs say domains added from September 15, 2026 block training and agent bots on pages that show ads unless you change it. Cloudflare help

Confirm it with your server logs

The checks above can only see what your site serves to a request that uses a crawler's name. Your server log shows what the real crawlers actually got. Paste lines from your access log, or choose the file, and every request that claims to be an AI crawler is checked against the IP addresses that crawler's maker publishes. You'll see which visits were real, which came from look-alikes borrowing the name, and whether the real crawler was turned away.

Your log stays on your computer. It's read in your browser and never uploaded. The one exception is Bingbot: Bing publishes no address list, so the addresses of requests claiming to be Bingbot are sent to us for a DNS lookup, the check Bing itself recommends.

What's an access log, and how do I get mine?

An access log is a file your web host keeps automatically: one line for every visit, with the visitor's address, the page, what your site sent back, and the name of the browser or crawler. A day or a week of it is plenty. Upload the file just as your host gives it to you; compressed .gz files are fine.

On Wix, Squarespace or Webflow, you can't download this file, and we found no way to in Shopify's help either. If that's you, the robots.txt and firewall checks above are the ones to use.

  • Bluehost, Namecheap and other hosts with cPanel: log in to cPanel, then Metrics, Raw Access, and click your domain under "Download Current Raw Access Logs". cPanel help
  • SiteGround: Site Tools, Statistics, Access Log shows the latest 300 visits. The full daily files are in File Manager, in your site's logs folder, kept for up to 30 days. SiteGround help
  • WP Engine: User Portal, Sites, your environment, Advanced, Logs, the Access tab, then Edge, and download. It covers the last 72 hours. WP Engine help
  • Kinsta: MyKinsta, Sites, your site, then its file manager: open the logs folder and download access.log. Kinsta keeps only 4 days. Kinsta help
  • DreamHost: with an FTP app like FileZilla, open logs, your site, https, and download access.log. DreamHost keeps 7 days. DreamHost help
  • Cloudways: with an FTP app and your Master Credentials, open applications, your app's folder, then logs. Cloudways help
  • Cloudflare: its live log stream is on the Business and Enterprise plans, and stored logs need a paid add-on. Export them as JSON, which this check reads. Cloudflare help
  • Host not listed? Ask them: "Can I download my raw access log for my website?"

Reads standard Apache and nginx log lines, CSV exports with address, user-agent and status columns, and JSON lines, including Cloudflare's. Compressed .gz files are fine.

What the website check reads

It fetches your site's robots.txt and the page you entered, then applies the robots.txt standard, RFC 9309, to each crawler:

  • A crawler follows the section that names it exactly. Lines listed one after another share one section, so a site that names GPTBot and ClaudeBot together gives both the same rules.
  • If no section names it, the crawler uses the * section.
  • Inside a section, the longest matching path wins, and a tie goes to Allow.
  • A robots.txt that returns 404 means no limits. One that returns a server error means crawlers treat the whole site as off limits until it comes back.

It also checks four things robots.txt alone doesn't show:

  • The page itself: noindex, nosnippet and max-snippet, the settings Google names for AI Overviews and AI Mode.
  • What arrives without JavaScript: our fetch runs no scripts, so we see what a crawler that doesn't run them gets. If the page is an empty app shell, we say so.
  • www and the bare domain: each host has its own robots.txt, and we flag it when the two say different things.
  • Content Signals: lines from Cloudflare's Content Signals Policy that state whether your pages may be used for search, AI answers or training. Cloudflare says a signal that isn't stated conveys no meaning, and we report it that way.

When something is blocked, you get the exact line to change and a message you can send to whoever runs your site.

Firewalls and CDNs. After reading robots.txt, the checker asks for your page as a normal browser, as a bot your site has never heard of, and once under each crawler's name, using the user-agent string the crawler's maker publishes. If a crawler is turned away while the browser gets in, we flag a possible block. We call it unconfirmed because our requests don't come from the crawlers' own IP addresses, and some firewalls treat a look-alike differently from the real crawler. If the browser request is refused too, or your site starts refusing us partway through, we say the result is inconclusive rather than guess.

Your server logs, to confirm it. Paste your access log and every request claiming to be an AI crawler is checked against the IP addresses its maker publishes (OpenAI, Anthropic, Perplexity, Google, Apple, DuckDuckGo, Mistral and Common Crawl), or against Bing's own DNS check. That separates the real crawler from look-alikes and shows what the real one was served. The log is read in your browser and never uploaded.

Before you publish a change, paste your draft robots.txt into the tester below the results. It shows what each crawler could read under the draft, what would change from your live file, and any lines crawlers would skip, like a misspelt Disallow.

Names that miss. A section for an old or invented name, like claude-web or anthropic-ai, doesn't reach the crawler you meant, because crawlers follow only the exact names their makers use. We flag those for Anthropic, OpenAI and Perplexity and show the current names. Every result also gives the line number of the rule that decided it, and you can download the results as CSV or JSON.

The three kinds of AI crawler

Search crawlers decide whether you can appear in answers. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers. Anthropic says blocking Claude-SearchBot may reduce your visibility in Claude's search results. PerplexityBot surfaces sites in Perplexity's answers. For Google's AI Overviews and AI Mode, Google says the control is robots.txt rules for Googlebot, plus the page-level noindex, nosnippet and max-snippet settings.

Training crawlers decide whether your pages may train models. GPTBot and ClaudeBot collect content for training. OpenAI says each of its settings is independent of the others, so you can block GPTBot and still appear in ChatGPT search. Google-Extended isn't a crawler at all: it's a robots.txt token for Gemini training and grounding, and Google says it does not affect inclusion in Google Search.

User fetchers visit a page because a person asked. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt. Anthropic says blocking Claude-User stops Claude retrieving your page for that user.

How we measure AI citations · What ChatGPT looks for on your website · Does llms.txt work?

Questions about AI crawlers

Does blocking GPTBot remove me from ChatGPT?

No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, and OpenAI says the two settings are independent. Block GPTBot if you don't want your pages used for training, and leave OAI-SearchBot allowed if you want to appear in ChatGPT search.

Does Google-Extended control AI Overviews?

No. Google says Google-Extended covers Gemini model training and grounding in Gemini Apps and Vertex AI, and doesn't affect Google Search. AI Overviews and AI Mode are part of Search, so Googlebot access and the noindex, nosnippet and max-snippet settings are what count there.

My robots.txt allows everything. Why am I still not mentioned?

Access only means a crawler can read the page. Whether an engine names you depends on what it finds when it searches: your own pages, directories, reviews and articles about your field. Our free AI answer snapshot shows what the engines currently say about your business.

Can it tell whether my firewall blocks AI crawlers?

Yes, if you have your server logs. Without them we can show a likely block: we ask for your page under each crawler's name and as a normal browser, and flag any crawler that's refused while the browser gets in. That stays unconfirmed, because our requests don't come from the crawler's own network. With your logs, the check confirms it: it picks out the visits that really came from each crawler's published addresses and shows what your site served them.

Are the GPTBot visits in my logs really from OpenAI?

Not always. Anyone can put a crawler's name in a request. OpenAI, Anthropic, Perplexity, Google, Apple, DuckDuckGo, Mistral and Common Crawl publish the IP addresses their crawlers use, and the log check tests every visit against them. Requests from other addresses are reported separately, so you can block look-alikes without blocking the real crawler.

Does this tool store the sites I check?

It keeps each result for ten minutes so repeat checks of the same page load quickly. It doesn't ask for an email or save a list of the sites people check.