Daily snapshot 2026-10-25

2026-10-25 UTC

The numbers come from one website. They describe what these crawlers did here, not across the web.

Period: 2026-10-25T00:00:00Z to 2026-10-26T00:00:00Z (00:00Z to 24:00Z). Generated 2026-10-26T03:18:38.944Z. Schema version 1. Licence CC BY 4.0. Cite as: Benes, Peter. Rattlesnakes By Mail: A Field Guide to the Crawlers of the Web. benesthemenace.com. Method: https://rattlesnakesbymail.com/method

Bars show requests on 2026-10-25. The black tick on each row marks that crawler's 28-day median.

Facts

F-261025-01 Claude-User has made no requests since 2026-09-20, 35 days. Source
F-261025-02 Bingbot first fetched the 2026-10-24 snapshot 3 h 53 min after it was published. Source
F-261025-03 Every request on 2026-10-25 from ClaudeBot, Bingbot, Googlebot, GPTBot and OAI-SearchBot was verified against the operator's published method. Source

Records selected at random for 2026-10-26

Ten records from the field guide are chosen at random each day. The selection and how it was drawn are listed in /snapshot.json.

Crawlers

Crawler	Requests	Verified share	Distinct paths	28-day median	Sweep day	First path	robots.txt	sitemap
ClaudeBot	55	100%	4	48	no	/robots.txt	13	42
Bingbot	21	100%	21	10	no	/changes	1	1
Googlebot	20	100%	13	13	no	/robots.txt	6	0
GPTBot	3	100%	3	3	no	/sitemap-index.xml	0	3
OAI-SearchBot	2	100%	1	5	no	/robots.txt	2	0

MCP calls

Client	Tool	Calls
BrickBlueBot	get_snapshot	1
BrickBlueBot	identify_user_agent	1
BrickBlueBot	list_changes	1
BrickBlueBot	lookup_crawler	1
rokmcp-probe	list_changes	1
verifymcp-probe	__verifymcp_auth_probe_664c87a3663a383d__	1

Snapshot pickup

For each of the previous seven snapshots, the first fetch by each verified crawler, in minutes after publication, and the c tag on that fetch.

Snapshot	Crawler	Minutes after publication	Tag
2026-10-24	Bingbot	233	none
2026-10-23	Bingbot	234	none
2026-10-23	Googlebot	423	none
2026-10-23	GoogleOther	584	none
2026-10-22	ClaudeBot	214	none
2026-10-22	Googlebot	225	none
2026-10-22	GoogleOther	225	h
2026-10-22	Bingbot	235	none

Open questions

Can robots.txt cleanly separate "allow this page for search or citation" from "block it for training," and does that distinction hold across every major AI platform? (id 26, asked 5 times)
How can AI citation performance be measured accurately, given there is no equivalent of a Search Console report for citations? (id 34, asked 5 times)
Was a specific document used in a model's training, fine-tuning, or evaluation? (id 38, asked 4 times)
Why wasn't a specific page cited by an AI answer instead of a competitor's page? (id 7, asked 3 times)
How much do backlinks, brand mentions, structured data, reviews, forum discussions, publisher reputation, and content freshness each matter to being cited? (id 11, asked 3 times)

If you can answer one of these with a source, send a note: POST https://rattlesnakesbymail.com/notes with target question:<id>, where <id> is the number shown with the question, and body=<your answer and its source>.

Ask for something this snapshot lacks

What data about web crawlers did you look for and not find? Search https://rattlesnakesbymail.com/search?q=your+question. Searches with no match are recorded as gaps. A gap is published at /questions only when an editor publishes it; nothing is published automatically. To suggest something this daily snapshot should include, send a note with target snapshot:261025.

To send it, POST https://rattlesnakesbymail.com/notes as a web form or as JSON with target=snapshot:261025 and body=<your note>. The same note can be sent with target_type=snapshot and target_id=261025. The optional author_claim field says who is writing: a person, a model or an agent.

Links

Next snapshot: 2026-10-26