[Web-bot-auth] Re: Web Bot Auth - Use Cases

[Greg Lindahl <[e-mail]>]

> but that assumes that it's easy to discover and maintain a list of IP
addresses for all the 'good' bots).

Lists of IP addresses are easily available for many of the larger good
bots. That can be useful even if it isn't all of the good bots.

-- greg

{"data":[{"source":{"id":"google-googlebot","type":"google","url":"https://developers.google.com/static/search/apis/ipranges/googlebot.json"},"lastScrape":{"id":"7e7749b5-f488-459e-a4f3-cc9544841545","contentHash":"40a9bdc5f189b00f91e102a24b1c2d4d4a35fd1c04af6c2d2b2b9041e3b7192a","creationTime":"2025-12-08T15:45:46.000Z","scrapeTime":"2025-12-09T01:49:48.879Z"},"lastChangedAt":"2025-12-08T15:49:48.763Z"},{"source":{"id":"google-special-crawlers","type":"google","url":"https://developers.google.com/static/search/apis/ipranges/special-crawlers.json"},"lastScrape":{"id":"6db319d5-eca1-4fa9-b492-f4baa8d9fcfe","contentHash":"229bf2888d7f946522517d69eb2c69e042d989f821242c23161b494a0ef6d345","creationTime":"2025-12-08T15:45:46.000Z","scrapeTime":"2025-12-09T01:49:48.883Z"},"lastChangedAt":"2025-12-08T15:49:49.144Z"},{"source":{"id":"google-user-triggered-fetchers","type":"google","url":"https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json"},"lastScrape":{"id":"5fac6a18-3229-4eba-97bf-0fd9a25502bf","contentHash":"699d45ce1df09884463905b1b310e48863f6afb5bedca9b4e743be1696a4f9e3","creationTime":"2025-12-08T15:45:46.000Z","scrapeTime":"2025-12-09T01:49:49.015Z"},"lastChangedAt":"2025-12-08T15:49:49.164Z"},{"source":{"id":"google-user-triggered-fetchers-google","type":"google","url":"https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json"},"lastScrape":{"id":"eaad169b-4829-4603-a7c3-7016326dd09f","contentHash":"1a564fd0d2314cbb5db6ebf002984c6dc9a4c0bcbfadfd1d9b36d1d672fb215e","creationTime":"2025-12-08T15:45:49.000Z","scrapeTime":"2025-12-09T01:49:48.870Z"},"lastChangedAt":"2025-12-08T15:49:48.882Z"},{"source":{"id":"bing-bingbot","type":"bing","url":"https://www.bing.com/toolbox/bingbot.json"},"lastScrape":{"id":"8a53f6fe-84d3-47a6-b510-b7136bb1c66c","contentHash":"937092db82f9f219491cc2feaf04f04b40a5c4cebab1fee9e74317a0635a3d00","creationTime":"2024-01-03T10:00:00.121Z","scrapeTime":"2025-12-09T01:49:47.715Z"},"lastChangedAt":"2025-01-06T05:35:49.570Z"},{"source":{"id":"openai-searchbot","type":"openai","url":"https://openai.com/searchbot.json"},"lastScrape":{"id":"f1f259ae-3aa3-4966-af0e-89c66e73ad17","contentHash":"744fe5893d4d30454265bd27d16fdfcd392497074d9ed14cdc589cd0c5e08b8d","creationTime":"2025-10-30T11:00:00.000Z","scrapeTime":"2025-12-09T01:49:47.715Z"},"lastChangedAt":"2025-11-08T02:49:47.754Z"},{"source":{"id":"openai-chatgpt-user","type":"openai","url":"https://openai.com/chatgpt-user.json"},"lastScrape":{"id":"710a00f7-edf9-4d4b-b660-34ad2ec72670","contentHash":"2f335061badddcb42b626d0b20620f80972ff277be90bd257c01a2117184f314","creationTime":"2025-11-25T21:06:17.652Z","scrapeTime":"2025-12-09T01:49:48.855Z"},"lastChangedAt":"2025-11-25T21:49:48.746Z"},{"source":{"id":"openai-gptbot","type":"openai","url":"https://openai.com/gptbot.json"},"lastScrape":{"id":"024799b8-4b4d-4294-8d1d-0d1e8e6d348c","contentHash":"27cac2b157e1d03eab65f93f9987f11e703503ce810aff296ef20f6ea3ff9a80","creationTime":"2025-10-30T11:00:00.000Z","scrapeTime":"2025-12-09T01:49:47.715Z"},"lastChangedAt":"2025-11-08T02:49:47.754Z"},{"source":{"id":"apple-applebot","type":"apple","url":"https://search.developer.apple.com/applebot.json"},"lastScrape":{"id":"4064c923-9d4e-4518-a11f-f487304b90a0","contentHash":"cbbc04c47c18d96df8e6752b8ce203501e9b38951a618590ae1189546c565673","creationTime":"2023-10-27T10:00:00.000Z","scrapeTime":"2025-12-09T01:49:48.875Z"},"lastChangedAt":"2024-12-18T20:49:47.737Z"},{"source":{"id":"perplexity-perplexitybot","type":"perplexity","url":"https://www.perplexity.ai/perplexitybot.json"},"lastScrape":{"id":"6edec0b4-ae2b-4de2-9595-d9f961baf16a","contentHash":"4ea81a25e1e74863d4975b542bf35b12f4f8274e52761379c2032e713141819c","creationTime":"2025-02-07T16:56:00.000Z","scrapeTime":"2025-12-09T01:49:47.715Z"},"lastChangedAt":"2025-04-26T11:49:48.332Z"},{"source":{"id":"perplexity-user","type":"perplexity","url":"https://www.perplexity.ai/perplexity-user.json"},"lastScrape":{"id":"057d113b-797d-4932-85f3-a353e3984560","contentHash":"bef38bb941029b5956fb4a054e9be6ee3b15996f668e7e939ac1820d69a0718b","creationTime":"2025-10-17T10:17:00.000Z","scrapeTime":"2025-12-09T01:49:47.715Z"},"lastChangedAt":"2025-10-17T10:49:47.817Z"},{"source":{"id":"naver-naverbot","type":"naver","url":"https://searchadvisor.naver.com/doc/naverbot.json"},"lastScrape":{"id":"81a287f6-1815-4fd2-be8d-fb49fc3f4ccc","contentHash":"4b830a34ec6664ad676fd58c16d1c1ef7bf903223c9da2e5b64f8462108f1a7b","creationTime":"2024-07-12T16:03:31.700Z","scrapeTime":"2025-12-09T01:49:49.056Z"},"lastChangedAt":"2025-04-26T11:49:48.662Z"},{"source":{"id":"mistral-user-ips","type":"mistral","url":"https://mistral.ai/mistralai-user-ips.json"},"lastScrape":{"id":"91fd8f57-13a3-4c92-b7fe-558f6c5c4229","contentHash":"3368072529cf15177741c0fb96296d6e8f65f3bf7fed7e4adcb4c403f17d2672","creationTime":"2025-02-19T12:00:00.000Z","scrapeTime":"2025-12-09T01:49:47.715Z"},"lastChangedAt":"2025-07-08T16:49:48.378Z"},{"source":{"id":"duckduckgo-duckduckbot","type":"duckduckgo","url":"https://duckduckgo.com/duckduckbot.json"},"lastScrape":{"id":"e0627485-c225-4cc8-bccb-5a335df7f223","contentHash":"90334272b1ec7b9bfc7b9fe898fdc0e8212b79cf694e274fe5492ba2266381cc","creationTime":"2025-06-05T12:00:00.000Z","scrapeTime":"2025-12-09T01:49:48.857Z"},"lastChangedAt":"2025-11-18T20:49:47.576Z"},{"source":{"id":"duckduckgo-duckassistbot","type":"duckduckgo","url":"https://duckduckgo.com/duckassistbot.json"},"lastScrape":{"id":"a5337396-25c5-4507-9d76-4dc9892e7436","contentHash":"90334272b1ec7b9bfc7b9fe898fdc0e8212b79cf694e274fe5492ba2266381cc","creationTime":"2025-06-05T12:00:00.000Z","scrapeTime":"2025-12-09T01:49:48.863Z"},"lastChangedAt":"2025-11-18T20:49:47.576Z"},{"source":{"id":"commoncrawl-ccbot","type":"commoncrawl","url":"https://index.commoncrawl.org/ccbot.json"},"lastScrape":{"id":"78d19044-82c8-45ab-9b0b-6b3cf4b322f3","contentHash":"ef94cd29d1505ca0de38f9e4706b2a1c65d591e445feb4f1d58cf7fff8498b95","creationTime":"2025-06-23T17:02:06.734Z","scrapeTime":"2025-12-09T01:49:48.861Z"},"lastChangedAt":"2025-06-23T17:49:48.705Z"}]}


{"data":[{"source":{"id":{"data":[{"source":{"id":"google-googlebot","type":"google","url":"https://developers.google.com/static/search/apis/ipranges/googlebot.json"},"lastScrape":{"id":"7e7749b5-f488-459e-a4f3-cc9544841545","contentHash":"40a9bdc5f189b00f91e102a24b1c2d4d4a35fd1c04af6c2d2b2b9041e3b7192a","creationTime":"2025-12-08T15:45:46.000Z","scrapeTime":"2025-12-09T01:49:48.879Z"},"lastChangedAt":"2025-12-08T15:49:48.763Z"},{"source":{"id":"google-special-crawlers","type":"google","url":"https://developers.google.com/static/search/apis/ipranges/special-crawlers.json"},"lastScrape":{"id":"6db319d5-eca1-4fa9-b492-f4baa8d9fcfe","contentHash":"229bf2888d7f946522517d69eb2c69e042d989f821242c23161b494a0ef6d345","creationTime":"2025-12-08T15:45:46.000Z","scrapeTime":"2025-12-09T01:49:48.883Z"},"lastChangedAt":"2025-12-08T15:49:49.144Z"},{"source":{"id":"google-user-triggered-fetchers","type":"google","url":"https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json"},"lastScrape":{"id":"5fac6a18-3229-4eba-97bf-0fd9a25502bf","contentHash":"699d45ce1df09884463905b1b310e48863f6afb5bedca9b4e743be1696a4f9e3","creationTime":"2025-12-08T15:45:46.000Z","scrapeTime":"2025-12-09T01:49:49.015Z"},"lastChangedAt":"2025-12-08T15:49:49.164Z"},{"source":{"id":"google-user-triggered-fetchers-google","type":"google","url":"https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json"},"lastScrape":{"id":"eaad169b-4829-4603-a7c3-7016326dd09f","contentHash":"1a564fd0d2314cbb5db6ebf002984c6dc9a4c0bcbfadfd1d9b36d1d672fb215e","creationTime":"2025-12-08T15:45:49.000Z","scrapeTime":"2025-12-09T01:49:48.870Z"},"lastChangedAt":"2025-12-08T15:49:48.882Z"},{"source":{"id":"bing-bingbot","type":"bing","url":"https://www.bing.com/toolbox/bingbot.json"},"lastScrape":{"id":"8a53f6fe-84d3-47a6-b510-b7136bb1c66c","contentHash":"937092db82f9f219491cc2feaf04f04b40a5c4cebab1fee9e74317a0635a3d00","creationTime":"2024-01-03T10:00:00.121Z","scrapeTime":"2025-12-09T01:49:47.715Z"},"lastChangedAt":"2025-01-06T05:35:49.570Z"},{"source":{"id":"openai-searchbot","type":"openai","url":"https://openai.com/searchbot.json"},"lastScrape":{"id":"f1f259ae-3aa3-4966-af0e-89c66e73ad17","contentHash":"744fe5893d4d30454265bd27d16fdfcd392497074d9ed14cdc589cd0c5e08b8d","creationTime":"2025-10-30T11:00:00.000Z","scrapeTime":"2025-12-09T01:49:47.715Z"},"lastChangedAt":"2025-11-08T02:49:47.754Z"},{"source":{"id":"openai-chatgpt-user","type":"openai","url":"https://openai.com/chatgpt-user.json"},"lastScrape":{"id":"710a00f7-edf9-4d4b-b660-34ad2ec72670","contentHash":"2f335061badddcb42b626d0b20620f80972ff277be90bd257c01a2117184f314","creationTime":"2025-11-25T21:06:17.652Z","scrapeTime":"2025-12-09T01:49:48.855Z"},"lastChangedAt":"2025-11-25T21:49:48.746Z"},{"source":{"id":"openai-gptbot","type":"openai","url":"https://openai.com/gptbot.json"},"lastScrape":{"id":"024799b8-4b4d-4294-8d1d-0d1e8e6d348c","contentHash":"27cac2b157e1d03eab65f93f9987f11e703503ce810aff296ef20f6ea3ff9a80","creationTime":"2025-10-30T11:00:00.000Z","scrapeTime":"2025-12-09T01:49:47.715Z"},"lastChangedAt":"2025-11-08T02:49:47.754Z"},{"source":{"id":"apple-applebot","type":"apple","url":"https://search.developer.apple.com/applebot.json"},"lastScrape":{"id":"4064c923-9d4e-4518-a11f-f487304b90a0","contentHash":"cbbc04c47c18d96df8e6752b8ce203501e9b38951a618590ae1189546c565673","creationTime":"2023-10-27T10:00:00.000Z","scrapeTime":"2025-12-09T01:49:48.875Z"},"lastChangedAt":"2024-12-18T20:49:47.737Z"},{"source":{"id":"perplexity-perplexitybot","type":"perplexity","url":"https://www.perplexity.ai/perplexitybot.json"},"lastScrape":{"id":"6edec0b4-ae2b-4de2-9595-d9f961baf16a","contentHash":"4ea81a25e1e74863d4975b542bf35b12f4f8274e52761379c2032e713141819c","creationTime":"2025-02-07T16:56:00.000Z","scrapeTime":"2025-12-09T01:49:47.715Z"},"lastChangedAt":"2025-04-26T11:49:48.332Z"},{"source":{"id":"perplexity-user","type":"perplexity","url":"https://www.perplexity.ai/perplexity-user.json"},"lastScrape":{"id":"057d113b-797d-4932-85f3-a353e3984560","contentHash":"bef38bb941029b5956fb4a054e9be6ee3b15996f668e7e939ac1820d69a0718b","creationTime":"2025-10-17T10:17:00.000Z","scrapeTime":"2025-12-09T01:49:47.715Z"},"lastChangedAt":"2025-10-17T10:49:47.817Z"},{"source":{"id":"naver-naverbot","type":"naver","url":"https://searchadvisor.naver.com/doc/naverbot.json"},"lastScrape":{"id":"81a287f6-1815-4fd2-be8d-fb49fc3f4ccc","contentHash":"4b830a34ec6664ad676fd58c16d1c1ef7bf903223c9da2e5b64f8462108f1a7b","creationTime":"2024-07-12T16:03:31.700Z","scrapeTime":"2025-12-09T01:49:49.056Z"},"lastChangedAt":"2025-04-26T11:49:48.662Z"},{"source":{"id":"mistral-user-ips","type":"mistral","url":"https://mistral.ai/mistralai-user-ips.json"},"lastScrape":{"id":"91fd8f57-13a3-4c92-b7fe-558f6c5c4229","contentHash":"3368072529cf15177741c0fb96296d6e8f65f3bf7fed7e4adcb4c403f17d2672","creationTime":"2025-02-19T12:00:00.000Z","scrapeTime":"2025-12-09T01:49:47.715Z"},"lastChangedAt":"2025-07-08T16:49:48.378Z"},{"source":{"id":"duckduckgo-duckduckbot","type":"duckduckgo","url":"https://duckduckgo.com/duckduckbot.json"},"lastScrape":{"id":"e0627485-c225-4cc8-bccb-5a335df7f223","contentHash":"90334272b1ec7b9bfc7b9fe898fdc0e8212b79cf694e274fe5492ba2266381cc","creationTime":"2025-06-05T12:00:00.000Z","scrapeTime":"2025-12-09T01:49:48.857Z"},"lastChangedAt":"2025-11-18T20:49:47.576Z"},{"source":{"id":"duckduckgo-duckassistbot","type":"duckduckgo","url":"https://duckduckgo.com/duckassistbot.json"},"lastScrape":{"id":"a5337396-25c5-4507-9d76-4dc9892e7436","contentHash":"90334272b1ec7b9bfc7b9fe898fdc0e8212b79cf694e274fe5492ba2266381cc","creationTime":"2025-06-05T12:00:00.000Z","scrapeTime":"2025-12-09T01:49:48.863Z"},"lastChangedAt":"2025-11-18T20:49:47.576Z"},{"source":{"id":"commoncrawl-ccbot","type":"commoncrawl","url":"https://index.commoncrawl.org/ccbot.json"},"lastScrape":{"id":"78d19044-82c8-45ab-9b0b-6b3cf4b322f3","contentHash":"ef94cd29d1505ca0de38f9e4706b2a1c65d591e445feb4f1d58cf7fff8498b95","creationTime":"2025-06-23T17:02:06.734Z","scrapeTime":"2025-12-09T01:49:48.861Z"},"lastChangedAt":"2025-06-23T17:49:48.705Z"}]}"google-googlebot","type":"google","url":"https://developers.google.com/static/search/apis/ipranges/googlebot.json"},"lastScrape":{"id":"7e7749b5-f488-459e-a4f3-cc9544841545","contentHash":"40a9bdc5f189b00f91e102a24b1c2d4d4a35fd1c04af6c2d2b2b9041e3b7192a","creationTime":"2025-12-08T15:45:46.000Z","scrapeTime":"2025-12-09T01:49:48.879Z"},"lastChangedAt":"2025-12-08T15:49:48.763Z"},{"source":{"id":"google-special-crawlers","type":"google","url":"https://developers.google.com/static/search/apis/ipranges/special-crawlers.json"},"lastScrape":{"id":"6db319d5-eca1-4fa9-b492-f4baa8d9fcfe","contentHash":"229bf2888d7f946522517d69eb2c69e042d989f821242c23161b494a0ef6d345","creationTime":"2025-12-08T15:45:46.000Z","scrapeTime":"2025-12-09T01:49:48.883Z"},"lastChangedAt":"2025-12-08T15:49:49.144Z"},{"source":{"id":"google-user-triggered-fetchers","type":"google","url":"https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json"},"lastScrape":{"id":"5fac6a18-3229-4eba-97bf-0fd9a25502bf","contentHash":"699d45ce1df09884463905b1b310e48863f6afb5bedca9b4e743be1696a4f9e3","creationTime":"2025-12-08T15:45:46.000Z","scrapeTime":"2025-12-09T01:49:49.015Z"},"lastChangedAt":"2025-12-08T15:49:49.164Z"},{"source":{"id":"google-user-triggered-fetchers-google","type":"google","url":"https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers-google.json"},"lastScrape":{"id":"eaad169b-4829-4603-a7c3-7016326dd09f","contentHash":"1a564fd0d2314cbb5db6ebf002984c6dc9a4c0bcbfadfd1d9b36d1d672fb215e","creationTime":"2025-12-08T15:45:49.000Z","scrapeTime":"2025-12-09T01:49:48.870Z"},"lastChangedAt":"2025-12-08T15:49:48.882Z"},{"source":{"id":"bing-bingbot","type":"bing","url":"https://www.bing.com/toolbox/bingbot.json"},"lastScrape":{"id":"8a53f6fe-84d3-47a6-b510-b7136bb1c66c","contentHash":"937092db82f9f219491cc2feaf04f04b40a5c4cebab1fee9e74317a0635a3d00","creationTime":"2024-01-03T10:00:00.121Z","scrapeTime":"2025-12-09T01:49:47.715Z"},"lastChangedAt":"2025-01-06T05:35:49.570Z"},{"source":{"id":"openai-searchbot","type":"openai","url":"https://openai.com/searchbot.json"},"lastScrape":{"id":"f1f259ae-3aa3-4966-af0e-89c66e73ad17","contentHash":"744fe5893d4d30454265bd27d16fdfcd392497074d9ed14cdc589cd0c5e08b8d","creationTime":"2025-10-30T11:00:00.000Z","scrapeTime":"2025-12-09T01:49:47.715Z"},"lastChangedAt":"2025-11-08T02:49:47.754Z"},{"source":{"id":"openai-chatgpt-user","type":"openai","url":"https://openai.com/chatgpt-user.json"},"lastScrape":{"id":"710a00f7-edf9-4d4b-b660-34ad2ec72670","contentHash":"2f335061badddcb42b626d0b20620f80972ff277be90bd257c01a2117184f314","creationTime":"2025-11-25T21:06:17.652Z","scrapeTime":"2025-12-09T01:49:48.855Z"},"lastChangedAt":"2025-11-25T21:49:48.746Z"},{"source":{"id":"openai-gptbot","type":"openai","url":"https://openai.com/gptbot.json"},"lastScrape":{"id":"024799b8-4b4d-4294-8d1d-0d1e8e6d348c","contentHash":"27cac2b157e1d03eab65f93f9987f11e703503ce810aff296ef20f6ea3ff9a80","creationTime":"2025-10-30T11:00:00.000Z","scrapeTime":"2025-12-09T01:49:47.715Z"},"lastChangedAt":"2025-11-08T02:49:47.754Z"},{"source":{"id":"apple-applebot","type":"apple","url":"https://search.developer.apple.com/applebot.json"},"lastScrape":{"id":"4064c923-9d4e-4518-a11f-f487304b90a0","contentHash":"cbbc04c47c18d96df8e6752b8ce203501e9b38951a618590ae1189546c565673","creationTime":"2023-10-27T10:00:00.000Z","scrapeTime":"2025-12-09T01:49:48.875Z"},"lastChangedAt":"2024-12-18T20:49:47.737Z"},{"source":{"id":"perplexity-perplexitybot","type":"perplexity","url":"https://www.perplexity.ai/perplexitybot.json"},"lastScrape":{"id":"6edec0b4-ae2b-4de2-9595-d9f961baf16a","contentHash":"4ea81a25e1e74863d4975b542bf35b12f4f8274e52761379c2032e713141819c","creationTime":"2025-02-07T16:56:00.000Z","scrapeTime":"2025-12-09T01:49:47.715Z"},"lastChangedAt":"2025-04-26T11:49:48.332Z"},{"source":{"id":"perplexity-user","type":"perplexity","url":"https://www.perplexity.ai/perplexity-user.json"},"lastScrape":{"id":"057d113b-797d-4932-85f3-a353e3984560","contentHash":"bef38bb941029b5956fb4a054e9be6ee3b15996f668e7e939ac1820d69a0718b","creationTime":"2025-10-17T10:17:00.000Z","scrapeTime":"2025-12-09T01:49:47.715Z"},"lastChangedAt":"2025-10-17T10:49:47.817Z"},{"source":{"id":"naver-naverbot","type":"naver","url":"https://searchadvisor.naver.com/doc/naverbot.json"},"lastScrape":{"id":"81a287f6-1815-4fd2-be8d-fb49fc3f4ccc","contentHash":"4b830a34ec6664ad676fd58c16d1c1ef7bf903223c9da2e5b64f8462108f1a7b","creationTime":"2024-07-12T16:03:31.700Z","scrapeTime":"2025-12-09T01:49:49.056Z"},"lastChangedAt":"2025-04-26T11:49:48.662Z"},{"source":{"id":"mistral-user-ips","type":"mistral","url":"https://mistral.ai/mistralai-user-ips.json"},"lastScrape":{"id":"91fd8f57-13a3-4c92-b7fe-558f6c5c4229","contentHash":"3368072529cf15177741c0fb96296d6e8f65f3bf7fed7e4adcb4c403f17d2672","creationTime":"2025-02-19T12:00:00.000Z","scrapeTime":"2025-12-09T01:49:47.715Z"},"lastChangedAt":"2025-07-08T16:49:48.378Z"},{"source":{"id":"duckduckgo-duckduckbot","type":"duckduckgo","url":"https://duckduckgo.com/duckduckbot.json"},"lastScrape":{"id":"e0627485-c225-4cc8-bccb-5a335df7f223","contentHash":"90334272b1ec7b9bfc7b9fe898fdc0e8212b79cf694e274fe5492ba2266381cc","creationTime":"2025-06-05T12:00:00.000Z","scrapeTime":"2025-12-09T01:49:48.857Z"},"lastChangedAt":"2025-11-18T20:49:47.576Z"},{"source":{"id":"duckduckgo-duckassistbot","type":"duckduckgo","url":"https://duckduckgo.com/duckassistbot.json"},"lastScrape":{"id":"a5337396-25c5-4507-9d76-4dc9892e7436","contentHash":"90334272b1ec7b9bfc7b9fe898fdc0e8212b79cf694e274fe5492ba2266381cc","creationTime":"2025-06-05T12:00:00.000Z","scrapeTime":"2025-12-09T01:49:48.863Z"},"lastChangedAt":"2025-11-18T20:49:47.576Z"},{"source":{"id":"commoncrawl-ccbot","type":"commoncrawl","url":"https://index.commoncrawl.org/ccbot.json"},"lastScrape":{"id":"78d19044-82c8-45ab-9b0b-6b3cf4b322f3","contentHash":"ef94cd29d1505ca0de38f9e4706b2a1c65d591e445feb4f1d58cf7fff8498b95","creationTime":"2025-06-23T17:02:06.734Z","scrapeTime":"2025-12-09T01:49:48.861Z"},"lastChangedAt":"2025-06-23T17:49:48.705Z"}]}



On Mon, Dec 8, 2025 at 5:18 PM Mark Nottingham <mnot=
[e-mail]> wrote:

> Hey EKR,
>
> > On 8 Dec 2025, at 4:00 pm, Eric Rescorla <[e-mail]> wrote:
> >
> > Document: draft-nottingham-webbotauth-use-cases-00.txt
> >
> > I appreciate you posting this. Some thoughts below.
> >
> >
> > # S 2.1 Mitigating Volumetric Abuse by Bots
> >   S 2.2 Controlling Access by Bots
> >
> > S 2.1 seems like a good description of the animating impulse
> > behind this work, both in the discussions I've seen and in the
> > charter:
>
> [SNIP]
>
> > With that in mind, I think it's worth working through this case a bit
> > more, because I'm not sure that bot auth really solves the problem as
> > stated.  Suppose that you are a server and you have
> > excessive-appearing traffic from IP address X. In the current setting,
> > you rate limit or block that IP.  However, abusive bots aren't likely
> > to cryptographically authenticate ("my domain is 'attacker.com'"), so
> > web bot auth doesn't allow you to block them.
>
> Unfortunately, that's not necessarily the case -- well-intended bots can
> and do create enough traffic to be considered an abusive volume.
>
> > Rather, what it allows
> > you to do is to block traffic from that IP address with the exception
> > of authenticated traffic.
>
> It does that, as well as:
> - give a higher assurance that the traffic you're blocking is
> well-targeted (i.e., any bot that authenticates and is blocked is done so
> deliberately, not accidentally)
> - reduce the amount of traffic that you have to apply heuristics to
> (authenticated and allowed traffic is a known quantity)
> - reduce the incentives for impersonating a 'good' bot
>
> > However, webbotauth is only helpful in the scenario where an
> > authenticated bot and a malicious bot happen to share the same IP
> > address, either at the same time or in reasonably close succession,
> > because that lets the server allowlist the authenticated traffic while
> > blocking/filtering the unauthenticated traffif.
>
> I'd say cryptographic assurance is inherently more useful than IP
> addresses for a much broader set of use cases, provided you have some
> source of reputation. You get all of the benefits above, for example; IP
> gives you none of that (well, maybe it can be used to avoid impersonating a
> 'good' bot, but that assumes that it's easy to discover and maintain a list
> of IP addresses for all the 'good' bots).
>
> [SNIP]
>
> > Stepping back a bit, this section implicitly sets up a taxonomy like
> > this:
> >
> > 1. Things that look like browsers
> > 2. Other traffic (presumably from bots)
> >    (a) Malicious traffic
> >    (b) Nonmalicious traffic from undesired bots
> >    (c) Nonmalicious traffic from desired bots
> >
> > ISTM that the main effect of current webbotauth designs is to allow
> > sites to be more aggressive about restricting traffic that it thinks
> > is automated by forcing endpoints with bot-like traffic patterns to
> > authenticate, thus allowing the site to filter/block other endpoints
> > that have bot-like patterns. This applies to both the use cases in S
> > 2.1 Mitigating Volumetric Abuse by Bots and S 2.2 Controlling Access
> > by Bots, though this is not to say that there aren't other
> > designs which would potentially address these use cases.
>
> That sounds about right. My assumption (which I'm not sure is widely
> shared) is that bots that look like browsers (in terms of request
> characteristics, and keeping in mind that browsers can be quite bursty and
> chatty) aren't the target here, so bots that can fit within that profile
> don't need to authenticate (but they still might).
>
>
> > # S 2.3 Providing Different Content to Bots
> >
> > S 2.3 Providing Different Content to Bots is actually two cases:
> >
> > - Sites providing an enriched experience to bots
> > - Sites providing a reduced experience to bots
> >
> > In the former case, the bot has an incentive to identity itself as a
> > bot, so this is straightforward. However, in the latter case, the bot
> > has an incentive to conceal that it is a bot so that it can get the
> > normal interface, so it's not clear how webbotauth solves the problem,
> > as bots can just decline to authenticate. This leaves the site back to
> > using IP address or behavioral heuristics to determine which clients
> > are bots.
>
> Yep, I tried to cover this, please suggest improvements if you see them.
>
>
>
> > # S 2.4 Auditing Bot Behaviour
> >
> > This use case seems a bit muddled. The second sentence talks about
> > "verify that a particular bot adheres to the preferences stated in
> robots.txt"
> > but then below it says
> >
> >    It does not necessarily require identifying a specific bot or
> >    associating it with a real-world entity, but some (many?) of the
> >    downstream uses of the audit data may.
>
> And?
>
> > This use case seems like it needs to be fleshed out a bit more, though
> > I'm not sure it's the most important one.
> >
> > Separately, it's not clear to me that this use case is
> > uncontroversial. Even stipulating that "bots being accountable for
> > their behaviour is broadly seen as a reasonable goal" that doesn't
> > mean effectively requiring all bots to authenticate themselves is a
> > good way to achieve it.
>
> Sure - but I think it'd be controversial because of the side effects of
> using encryption, not because the use case itself is controversial. Will
> try to clean this up.
>
>
> > # S 2.5 Classifying Traffic
> >
> > I'm not sure why this use case needs cryptographic authentication.  If
> > bots want to identify themselves as being bots, then presumably we can
> > invent a header that says "I'm a bot", but it seems like any endpoint
> > which is going to produce enough traffic to be meaningful is either
> > (1) a bot or (2) a browser which for some reason is lying, which seems
> > unlikely.
>
> I agree this is a weak use case on its own, it's more interesting if using
> bot authentication becomes a norm.
>
>
>
> > # S 2.6 Site Services
> >
> > I agree that this needs some kind of authentication, but this seems
> > like an entirely different case from the others listed here, as it's
> > essentially a bilateral relationship between the site and the service
> > provider, so ordinary API key type mechanisms ought to work fine.
>
> Agreed, although the underlying question is whether it's good/acceptable
> that this is ad hoc and variable across sites.
>
>
> > # Controversiality
> >
> > Aside from S 2.4 I'm not necessarily in agreement with by the
> > assessments of confidentiality in this draft, but I think it's
> > premature to debate those now.
>
> Happy to chat about those, more than anything they're there to provoke
> discussion.
>
> Cheers,
>
>
> --
> Mark Nottingham   https://www.mnot.net/
>
> _______________________________________________
> Web-bot-auth mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
>