{"summary":"theguardian.com: blocks 9 of 16 AI crawlers. It still allows 4 that most of 24 major news sites block: Google-Extended (blocked by 58%), Perplexity-User (blocked by 54%), GPTBot (blocked by 50%), meta-externalfetcher (blocked by 50%). Meta: meta-externalagent blocked (63% of peers block), meta-externalfetcher allowed (50% of peers block), meta-webindexer allowed (29% of peers block).","domain":"theguardian.com","path":"/","robots_txt":"https://theguardian.com/robots.txt","robots_found":true,"bots":[{"bot":"GPTBot","owner":"OpenAI","purpose":"model training","allowed":true,"decided_by":"no rule for this path in the * group","peers_blocking":"12/24"},{"bot":"OAI-SearchBot","owner":"OpenAI","purpose":"ChatGPT search index","allowed":true,"decided_by":"no rule for this path in the * group","peers_blocking":"6/24"},{"bot":"ChatGPT-User","owner":"OpenAI","purpose":"fetches pages for a ChatGPT user","allowed":true,"decided_by":"no rule for this path in the * group","peers_blocking":"8/24"},{"bot":"ClaudeBot","owner":"Anthropic","purpose":"model training","allowed":false,"decided_by":"Disallow: /","peers_blocking":"21/24"},{"bot":"Claude-SearchBot","owner":"Anthropic","purpose":"Claude search index","allowed":false,"decided_by":"Disallow: /","peers_blocking":"14/24"},{"bot":"Claude-User","owner":"Anthropic","purpose":"fetches pages for a Claude user","allowed":false,"decided_by":"Disallow: /","peers_blocking":"13/24"},{"bot":"PerplexityBot","owner":"Perplexity","purpose":"Perplexity search index","allowed":false,"decided_by":"Disallow: /","peers_blocking":"18/24"},{"bot":"Perplexity-User","owner":"Perplexity","purpose":"fetches pages for a Perplexity user","allowed":true,"decided_by":"no rule for this path in the * group","peers_blocking":"13/24"},{"bot":"Google-Extended","owner":"Google","purpose":"control token for Gemini training and grounding","allowed":true,"decided_by":"no rule for this path in the * group","peers_blocking":"14/24"},{"bot":"Applebot-Extended","owner":"Apple","purpose":"control token for Apple AI training","allowed":false,"decided_by":"Disallow: /","peers_blocking":"18/24"},{"bot":"meta-externalagent","owner":"Meta","purpose":"training and AI products","allowed":false,"decided_by":"Disallow: /","peers_blocking":"15/24"},{"bot":"meta-externalfetcher","owner":"Meta","purpose":"fetches pages for a Meta AI user","allowed":true,"decided_by":"no rule for this path in the * group","peers_blocking":"12/24"},{"bot":"meta-webindexer","owner":"Meta","purpose":"Meta AI search index and Muse page reads","allowed":true,"decided_by":"no rule for this path in the * group","peers_blocking":"7/24"},{"bot":"CCBot","owner":"Common Crawl","purpose":"open web crawl widely used for training","allowed":false,"decided_by":"Disallow: /","peers_blocking":"21/24"},{"bot":"Bytespider","owner":"ByteDance","purpose":"training","allowed":false,"decided_by":"Disallow: /","peers_blocking":"21/24"},{"bot":"Amazonbot","owner":"Amazon","purpose":"Alexa and Amazon AI","allowed":false,"decided_by":"Disallow: /","peers_blocking":"15/24"}],"gaps_vs_major_news_sites":[{"bot":"Google-Extended","owner":"Google","purpose":"control token for Gemini training and grounding","peers_blocking":"14/24","e_g":["nytimes.com","wsj.com","bbc.co.uk","reuters.com"]},{"bot":"Perplexity-User","owner":"Perplexity","purpose":"fetches pages for a Perplexity user","peers_blocking":"13/24","e_g":["nytimes.com","wsj.com","bbc.co.uk","cnn.com"]},{"bot":"GPTBot","owner":"OpenAI","purpose":"model training","peers_blocking":"12/24","e_g":["nytimes.com","bbc.co.uk","cnn.com","reuters.com"]},{"bot":"meta-externalfetcher","owner":"Meta","purpose":"fetches pages for a Meta AI user","peers_blocking":"12/24","e_g":["nytimes.com","washingtonpost.com","wsj.com","reuters.com"]}],"lines_to_close_the_gaps":"User-agent: Google-Extended\nDisallow: /\n\nUser-agent: Perplexity-User\nDisallow: /\n\nUser-agent: GPTBot\nDisallow: /\n\nUser-agent: meta-externalfetcher\nDisallow: /","peer_sites":["nytimes.com","washingtonpost.com","wsj.com","theguardian.com","bbc.co.uk","cnn.com","reuters.com","apnews.com","bloomberg.com","ft.com","economist.com","theatlantic.com","theverge.com","wired.com","axios.com","politico.com","usatoday.com","nbcnews.com","foxnews.com","latimes.com","businessinsider.com","techcrunch.com","people.com","forbes.com"],"peer_sites_checked":24,"note_on_blocking":"robots.txt is a request, not an enforcement: the reporting below describes crawlers that ignore it.","context":[{"fact":"Meta said the update does not override website owners’ preferences expressed through robots.txt.","quote":"Meta also said the update does not override website owners’ preferences expressed through robots.txt.","source":"https://mediacopilot.ai/meta-ai-scraping-terms-publishers/","reported":"2026-10-19"},{"fact":"The article does not provide any deadline related to Meta’s terms.","quote":"OpenAI’s ChatGPT-User bot leads the pack, scraping at a rate five times higher than the second-most-active scraper (Meta) and 16 times higher than Perplexity’s agent.","source":"https://mediacopilot.ai/ai-scrapers-bypassing-publisher-protections/","reported":"2026-02-04"},{"fact":"Meta’s updated commercial terms take effect on Oct. 30 for businesses that continue using Meta’s products commercially.","quote":"The new terms take effect Oct. 30 and cover businesses that continue using Meta’s products commercially.","source":"https://mediacopilot.ai/meta-commercial-terms-scraping-news/","reported":"2026-10-16"},{"fact":"Meta declined to say whether its access could bypass publisher paywalls.","quote":"Meta declined to tell Nieman Lab whether they could bypass publisher paywalls.","source":"https://mediacopilot.ai/meta-commercial-terms-scraping-news/","reported":"2026-10-16"},{"fact":"Meta said websites can still manage crawler access through robots.txt.","quote":"The spokesperson said Meta does not plan to change how it crawls publishers and said websites can still manage access through robots.txt.","source":"https://mediacopilot.ai/meta-commercial-terms-scraping-news/","reported":"2026-10-16"},{"fact":"Legal experts questioned how much protection robots.txt instructions would provide.","quote":"The legal experts questioned how much protection those instructions would provide.","source":"https://mediacopilot.ai/meta-commercial-terms-scraping-news/","reported":"2026-10-16"}],"sources":[{"title":"Meta’s new business terms could allow news-site scraping for AI training","url":"https://mediacopilot.ai/meta-commercial-terms-scraping-news/","date":"2026-10-16"},{"title":"Publishers face Oct. 30 deadline over Meta’s updated crawling terms","url":"https://mediacopilot.ai/meta-ai-scraping-terms-publishers/","date":"2026-10-19"}],"disclosure":"Sponsored tool by oasy.ai (test brand in an Oasy ad experiment). Facts come from The Media Copilot's reporting; each one links its article and quotes it.","docs":"https://www.oasy.ai/api/crawlguard/zhachy48"}