Bytespider — ByteDance's ai training crawler

Multiple independent measurements have placed Bytespider among the most aggressive AI crawlers by request volume, and publisher logs routinely show it crawling paths explicitly disallowed in robots.txt.

With no published ranges and a user-agent string that any scraper can wear, Bytespider is best handled as hostile traffic: match the user-agent at the web server or WAF and block outright, alongside behavioral bot detection for unlabeled ByteDance crawling.

Bytespider identifies itself with the following user-agent string:

Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)

Never trust the user-agent string alone. Scrapers routinely impersonate well-known crawlers to inherit their access. ByteDance publishes no IP ranges and no crawler documentation. There is no sanctioned way to verify Bytespider traffic — which tells you most of what you need to know about it.

Note that most AI crawlers and fetchers, Bytespider included, do not execute JavaScript — so this traffic is invisible to GA4 and every script-based analytics tool. Server logs, CDN analytics, or a dedicated bot-analytics layer are the only places you will see it. For the full picture of measuring AI-driven visits, see our guide on how to track AI traffic.

To refuse Bytespider access to your entire site, add this to your robots.txt:

User-agent: Bytespider
Disallow: /

To restrict it from specific sections only (for example, premium content) while leaving the rest open:

User-agent: Bytespider
Disallow: /premium/
Disallow: /members/

The case for blocking: It takes at scale, gives nothing back, ignores the web's consent mechanism, and cannot be verified. This is the clearest block in the directory.

The case for allowing or monetizing: You cannot monetize an actor that won't identify itself or honor access rules. Enforceable blocking is the precondition for any commercial conversation.