Agent templates · Search agent
AI crawler access check
Confirm which AI systems can read your site. It tests fourteen AI crawlers against your robots.txt, page tags and server headers, and pairs any block with the exact fix.
Free · reviewed by a person
Run it with Flow AI
Flow AI runs this agent on your site and Amber reviews the result, which reaches your inbox within two working days.
Open source · MIT
Install it yourself
Runs in Claude Code on your own machine.
npx skills add Amberlanqinyun/geo-seo-claude --skill geo-crawlers
How it runs
Step by step, then a person checks it.
- 01Read robots.txtIt tests every rule against fourteen AI crawlers in three tiers: the answer engines, the wider AI ecosystem, and training-only crawlers.
- 02Check page tagsMeta robots tags that can quietly block AI systems page by page.
- 03Check server headersX-Robots-Tag headers, which override everything else.
- 04Look for AI filesWhether llms.txt and a sitemap are in place to point AI systems at your best pages.
- 05Check renderingWhether your content needs JavaScript to appear, which many AI crawlers do not run.
- 06A marketing engineer reviews itAmber checks the result on every Flow AI run before it reaches you.
In and out
What it needs, and what it returns.
What it needs
- Your site address
What comes back
- An access table for all fourteen crawlers
- An AI visibility score out of 100
- Any critical blocks, named
- A ready-to-paste robots.txt recommendation
A real run
flowai.co.nz, checked 2 October 2026
All fourteen crawlers can read the site. Nine are named in robots.txt and five are allowed by the default rule. No page tags or headers block them. The sitemap and llms.txt are both in place.
| Crawler | Operator | Tier | Status | How |
|---|---|---|---|---|
| GPTBot | OpenAI | 1 | Allowed | Named in robots.txt |
| OAI-SearchBot | OpenAI | 1 | Allowed | Named in robots.txt |
| ChatGPT-User | OpenAI | 1 | Allowed | Named in robots.txt |
| ClaudeBot | Anthropic | 1 | Allowed | Named in robots.txt |
| PerplexityBot | Perplexity | 1 | Allowed | Named in robots.txt |
| Google-Extended | 2 | Allowed | Named in robots.txt | |
| GoogleOther | 2 | Allowed | Default rule | |
| Applebot-Extended | Apple | 2 | Allowed | Default rule |
| Amazonbot | Amazon | 2 | Allowed | Default rule |
| FacebookBot | Meta | 2 | Allowed | Default rule |
| CCBot | Common Crawl | 3 | Allowed | Named in robots.txt |
| anthropic-ai | Anthropic | 3 | Allowed | Named in robots.txt |
| Bytespider | ByteDance | 3 | Allowed | Named in robots.txt |
| cohere-ai | Cohere | 3 | Allowed | Default rule |
Keep the search agent running every month.
A template is one run. A sprint installs the agent on your business in a week, with documentation and a handover to your team. Then the Flow Engine runs it every month.