category, plus a kind with more detail.
Agent or AI crawler?
Ask whether a person is waiting on this request. If someone is, it’s anagent. If it would have happened anyway on a schedule, it’s an ai-crawler.
Search engines whose index also feeds AI answers, like Googlebot and Bingbot, stay crawler. Their main job is search, and their operators document them that way.
Kinds
Edge cases
A script fetching my markdown
A script fetching my markdown
Scripts, docs mirrors and integrations fetch
.md and llms.txt files too. A request counts as an agent only if it identifies itself as one or asks for markdown in its Accept header. A bare curl or axios request is a bot with kind http-client.Native apps
Native apps
An app’s own HTTP stack (
CFNetwork, Dalvik, okhttp) fetches pages programmatically, even when a person is using the app, so it’s a bot. Pages opened in an app’s built-in browser, like the Twitter or LinkedIn apps or an iOS web view, use a browser user agent and count as human.Platforms that fetch for other products
Platforms that fetch for other products
Some agents fetch through another company’s infrastructure. Cursor’s web fetches have been reported to go through Exa as
ExaUser, and Codex through OpenAI as ChatGPT-User. The result names the identity it can prove, and the entry’s notes record known cases.