Dataset

The dataset is plain JSON, so you can use it from any language or SQL engine. Import it from the package or read it from the repository.

import { agents, httpClients, headers } from '@mintlify/agent-traffic'
import agentsJson from '@mintlify/agent-traffic/data/agents.json' with { type: 'json' }

File	Contents
`agents.json`	Known agents, AI crawlers, search crawlers and named bots
`http-clients.json`	Generic HTTP libraries, native-app HTTP stacks and headless browsers
`headers.json`	What each request header tells you and who controls it

Agent entries

{
  "id": "gptbot",
  "name": "GPTBot",
  "operator": "OpenAI",
  "category": "ai-crawler",
  "kind": "training",
  "runsOn": "operator",
  "userAgents": ["\\bGPTBot/"],
  "verify": { "ipRanges": ["https://openai.com/gptbot.json"] },
  "docs": "https://platform.openai.com/docs/bots",
  "examples": ["Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot)"],
  "lastVerified": "2026-10-23"
}

string

required

Stable identifier, returned as id by classify.

string

required

agent, ai-crawler, crawler or bot.

string

required

Detail within the category.

'operator' | 'user'

required

Where requests come from. user entries are never IP-checked.

string[]

required

Case-insensitive regular expressions anchored on the product token. A (?<version>…) group fills version.

string[]

Exact Accept strings the product sends.

string[]

Hosts named in the product’s Signature-Agent header.

object

Operator-published sources: ipRanges URLs, reverseDns domains and a webBotAuth key directory.

string[]

required

Real user agents. The tests require each to classify as its own entry and match no other.

string

Caveats, such as shared identities.

string

required

The date the entry was last confirmed.

Get started

Concepts

Guides

Reference

Project

Agent entries

​Agent entries

Agent entries