Blog
Notes on AI crawler access, parse failures, and what an evidence-first audit can — and cannot — tell you about AI system readability.
September 16, 2026
We probed 500 popular domains for AI access signals. Nearly half did not answer one request.
One bounded observation of 500 domains sampled from the Tranco top-1M list (every 2000th rank, list N2PYW): 264 answered within the 3.2-second page bound, 169 served a parseable robots.txt, 55 served llms.txt, and 5 sent an X-Robots-Tag header — with the method, the non-response accounting, and the limits stated.
September 16, 2026
We reviewed how our own site presents to AI systems. Here is what we fixed.
We ran our own audit checks against generativemetrics.com, found six gaps, fixed them in two commits on September 16, 2026, and re-ran the checks — with an honest account of what that does and does not prove.
September 15, 2026
On September 15, Cloudflare changes the default. Will you know whether AI crawlers still see your site?
The blocking that exists is concentrated where organizations have paid attention. Everyone else is running an unexamined default — and most site owners cannot answer the question the moment forces.
New posts are added as we publish audits and findings.
Subscribe via RSS