Wharton/Rutgers Study: Blocking AI Crawlers Costs Publishers 7% Traffic

A Wharton-and-Rutgers research paper revisited in late April puts a hard number on what publishers risk by blocking AI crawlers. About 7% of weekly traffic, lost within six weeks of adding the block to robots.txt. The drop shows up in human browsing panel data, not just bot logs, which makes the loss real.

Hangcheng Zhao at Rutgers Business School and Ron Berman at Wharton ran the analysis across 30 major newspaper domains and extended the methodology to the top 500 news publishers, using SimilarWeb, Semrush, and Comscore. The first version of the paper went up on December 31, 2025. The latest revision was posted April 21, 2026. PPC Land’s writeup carried it forward on April 26.

The proposed mechanism is the part site readers should sit with. When a publisher blocks AI crawlers, the AI systems stop citing that publisher. Stopped citations weaken brand recall over time. Weaker brand recall produces fewer direct visits. The 7% isn’t AI traffic disappearing. It’s the upstream brand-mention effect drying up downstream human visits.

A couple of implications for any business making the same call. The cost of blocking is bigger than expected because the loss isn’t visible in your AI referral logs. It shows up in branded search and direct traffic, which most teams aren’t tracking against AI bot policy. The study covered news publishers with already-strong brand recognition. A small business or B2B site has a thinner brand-recall buffer to draw on, which suggests 7% might be a floor rather than a ceiling for less recognized sites.

Worth pinning the takeaway. Blocking AI crawlers is cheap to do and expensive to undo. The cost shows up in places most analytics setups don’t watch.

Sources: PPC Land, April 26, 2026.