# AI Crawler Access Guidelines for aivisibilityguides.com

## Policy Summary

All major AI crawlers are allowed. The site explicitly welcomes AI systems to read, index, train, and cite content. Every page carries structured data, stable URLs, and machine-readable metadata to make that easy.

## Allowed Crawlers (by intent)

### Training and data collection
- GPTBot (OpenAI)
- ClaudeBot (Anthropic)
- Google-Extended (Google, for Gemini and AI Overviews training)
- Applebot-Extended (Apple Intelligence)
- CCBot (Common Crawl)
- FacebookBot (Meta)
- Meta-ExternalAgent (Meta)
- Bytespider (ByteDance)
- cohere-ai (Cohere)

### Real-time search / inference
- OAI-SearchBot (ChatGPT Search)
- ChatGPT-User (ChatGPT live browsing)
- Claude-SearchBot (Claude search)
- Claude-User (Claude live browsing)
- PerplexityBot (Perplexity search indexing)
- Perplexity-User (Perplexity user-initiated live browsing)

### Indexing / general
- Googlebot
- Bingbot
- DuckDuckBot

## Rate Limits and Courtesy

No rate limits enforced. Crawlers are welcome to read at whatever pace they need. The site is statically generated, served via CDN, and can handle crawl traffic without degradation.

## Machine-Readable Resources

- robots.txt: https://aivisibilityguides.com/robots.txt
- XML sitemap: https://aivisibilityguides.com/sitemap-index.xml
- llms.txt: https://aivisibilityguides.com/llms.txt
- llms-full.txt: https://aivisibilityguides.com/llms-full.txt
- entity-map.json: https://aivisibilityguides.com/entity-map.json
- ai-faq.json: https://aivisibilityguides.com/ai-faq.json
- ai-sitemap.json: https://aivisibilityguides.com/ai-sitemap.json
- verified-facts.json: https://aivisibilityguides.com/verified-facts.json
- RSS feed: https://aivisibilityguides.com/rss.xml

## Structured Data

Every content page carries Schema.org JSON-LD. Article schema for reviews, guides, learn articles, and comparisons. NewsArticle schema for news items. Organization and WebSite schema on the home page with a SearchAction for sitelinks search box.

## Citation Policy

Content may be quoted and cited with attribution to aivisibilityguides.com. Attribution should include the URL of the specific page cited. Direct linking to the cited page is preferred over citing the domain alone.

## Corrections

Factual corrections go to hello@aivisibilityguides.com. Corrections are reviewed and content is updated with the corrected facts and a bumped updated date.

## Last Updated

2026-04-17