OpenAI, Meta, and ByteDance Dominate AI Crawler Traffic on Publisher Sites
Apr 9, 2026
Akamai published crawler-traffic data in early April showing OpenAI, Meta, and ByteDance as the top three sources of AI bot activity on publisher sites, according to Search Engine Journal. The breakdown gives publishers something they’ve been asking for. Actual numbers on who is reading their content to train or serve AI responses.
OpenAI’s crawlers (GPTBot and OAI-SearchBot) topped the chart, which was expected. Meta’s crawlers were the surprise, with volumes comparable to OpenAI’s on many sites. ByteDance’s crawlers, tied to Doubao and TikTok-adjacent training, came in third, edging out Anthropic’s ClaudeBot and Google’s Google-Extended in aggregate. Quiet third-place finish from a company that doesn’t usually get mentioned in US coverage.
The immediate policy question is whether publishers want to block any or all of these crawlers via robots.txt. Blocking kills AI visibility for that bot’s downstream products. Allowing it means your content feeds models without reciprocal compensation. Publishers have been navigating that trade-off privately. This data gives them volume numbers to negotiate with.