What is Bytespider?
Bytespider is ByteDance's web crawler. DevTune counts it as a training crawler.
- Operator
- ByteDance
- Class
- Crawls pages that can be used to train AI
- Verification
- User agent onlyDevTune has no published network or signing key to check this agent against, so it identifies requests by user agent alone. Anyone can send that user agent, so these requests can't be authenticated.
On your site
It requests pages to collect content, not to answer a question someone is asking right now. What it collects can be used to train AI models. AI Traffic counts its requests as training crawls.
AI Traffic setup
Connect a crawler sensor (Cloudflare pull, edge middleware or a Vercel log drain) to see which pages Bytespider requests on your site.
Set up AI Traffic