We looked at 179 million web sessions for AI agents. Here's what we found.

We wanted to know how much of our customers' web traffic comes from AI today. So we measured it.

For five days in October, we ran a small, read-only service on our production data plane. It randomly sampled web sessions: 179 million sessions and 2.85 billion events in all.

The short answer is that AI traffic is still small. The more useful answer is that AI crawlers and AI agents are two different visitors, and they show up in different places.

AI traffic is still small: 29 in every 10,000 sessions

AI agents were 21 in every 10,000 sessions. AI crawlers were 8.

By crawlers, we mean AI companies collecting content (Meta, Amazon, ByteDance, and OpenAI). By agents, we mean an assistant browsing a site for someone.

Crawlers and agents show up in different verticals

Developer tools had about 5x the average agent traffic, and many more agents than crawlers.

Retail was the only vertical where crawlers were ahead of agents. It also had about twice the crawler share of automotive, and one retailer alone received half of all the AI crawler sessions we saw.

Our read is that crawlers go after product catalogs, and agents show up where people already build with AI. We aren't fully sure of this yet.

The same site can draw very different AI visitors

On one AI app builder, agents outnumbered crawlers almost 7 to 1. Same site, same pages, two very different kinds of visitors.

An online marketplace had a good amount of agent traffic and almost no crawlers. We don't have a clear explanation. It may be geography, or it may be that agents are there to buy tickets and crawlers have nothing to index.

Almost no agents say who they are

Only 182 sessions in five days had a user agent naming an assistant. Most crawlers in our sample named themselves. The other ~380,000 agent sessions were flagged from behavior alone.

How we decided what counts as an agent

We ran a deterministic rule engine and later compared it against Jev for confidence. On high-confidence scores, the two came out very close.

The behavior score uses a few simple signals:

• A Linux Chrome user agent coming from a cloud or hosting network (the strongest signal)
• A device fingerprint already seen in an agent session
• A VM-like screen shared by 10 or more live sessions
• A window larger than the screen, which a real display can't produce
• Page views less than a second apart
• A deep first page with no referrer

Long pauses between events, like someone reading, bring the score down.

The service only looks at device and timing information on the live stream. It doesn't read event content, and nothing is cached beyond aggregate counts.

What this data can't tell us

This isn't perfect. We can't prove that every flagged session is a large language model (LLM) driving a browser. What we can say is that it behaves like automation, not like a person.

We also only see sessions where a browser ran our SDK. The major AI crawlers don't run JavaScript, so this undercounts them.

Some of our inputs are thin too. Many sites send no page views. And we work out hosting networks ourselves from network owner data, so we have no VPN or proxy flag on IPs.

Count crawlers and agents as two numbers, not one

If you’re reporting on AI traffic, crawlers and agents are worth tracking separately, rather than as one "AI traffic" number. Use user agents to catch the crawlers, then add behavior signals like the ones above to catch the agents that don't announce themselves.

Soumyadeb shares what we learn from our own data on X. If you're detecting bots or agents on your own traffic, he'd like to know which signals worked for you. Follow him at @soumyadeb_mitra.