[Web-bot-auth] Re: Thank you and follow up

[Sarah McKenna <[e-mail]>]

It might be helpful for those that missed the in person session this week
to watch the recording of my talk on our customer use cases.  This shows
why a bot won't self identify in many legitimate cases.
https://youtu.be/yAwxJMxT9z4

If there is a published policy or signal (i.e., "I am not healthy now"),
bots could follow it.  Robots.txt only addresses crawlers and doesn't
address scrapers and ai agents that navigate and interact with a site like
a human does.  A typical robots.txt will tell crawlers not to access the
/search directory, for example,  when that is the entry point for ai agents
and scrapers.

Alot of folks are judgemental but likely just don't understand the universe
of legitimate workloads and use cases for web data.  Scraping is the
interoperability layer of the internet and this is what ai agents use too.



On Thu, Nov 6, 2025 at 10:00 PM Maynard, Brent <bmaynard=
[e-mail]> wrote:

> I agree that “misbehave” is too broad to define in any useful way. It
> might be more productive to describe how automated agents show consent,
> identity, and backoff. Those are the things operators can actually observe
> and build controls around.
>
> In practice, the question is simple. Does the agent respect published
> policy, identify itself consistently, and react appropriately when a site
> signals that it is overloaded or asks it to stop? Focusing on those
> mechanics keeps us neutral on intent while setting clear expectations.
>
> A few small standards could help, such as a well-known policy path for
> permissions and contact details, a lightweight signal for health or
> overload, and a defined way for agents to slow down when needed. Even if
> only the cooperative agents follow them, reducing that load often
> stabilizes systems enough to handle the rest.
>
> Instead of trying to define misbehavior, it may be better to share
> contrasting examples. Ignoring policy versus honoring it. Rotating IPs
> versus using a stable identity. Fixed-rate scraping versus adaptive
> backoff. Continuing during an outage versus pausing when signaled. That
> kind of framing is simple, testable, and flexible enough for both
> traditional crawlers and newer AI-driven agents.
>
>
> Best,
>
>
> Brent
>
> *From: *Michael Richardson <[e-mail]>
> *Date: *Thursday, November 6, 2025 at 3:34 PM
> *To: *[e-mail] <[e-mail]>
> *Subject: *[Web-bot-auth] Re: Thank you and follow up
>
>
> Farzaneh Badiei <[e-mail]> wrote:
>     > what can be done at the IETF if anything. The term “misbehave” is
>     > ambiguous. Misbehaving implies so many things. Is that a technical
>     > term?
>
> You are right.
> There are a hundred ways to misbehave.
>
> There are only a few ways to behave.
> (One of them is to obey robots.txt)
> I don't think we should make any normative statements about good behaviour,
> but we probably should have informative examples.
>
>     > On Thu, Nov 6, 2025 at 5:27 PM Sarah McKenna <sarah.mckenna=
>     > [e-mail]> wrote:
>
>     >> I imagine in the cat and mouse game, the bots persisting after an
>     >> unhealthy signal is sent would be much easier to identify.  Would be
>     >> interesting to see if a little signalling dropped traffic
> effectively,
>     >> even traffic considered to be "human".
>     >>
>     >>
>     >>
>     >> On Thu, Nov 6, 2025 at 5:19 PM Michael Richardson
>     >> <[e-mail]> wrote:
>     >>
>     >>>
>     >>> Sarah McKenna <sarah.mckenna=[e-mail]>
> wrote: >
>     >>> 1.  Simple idea -- what if we gave websites a simple way to
> express >
>     >>> their health.  Not a signal that privileged any one bot but which >
>     >>> answered a yes / no question "Are you healthy?"  When a site is
>     >>> getting > hit too hard this gives them a way to alert all bots that
>     >>> they need > them to slow down or halt.  Could this be a quick win
> for
>     >>> overloaded > sites while we work out all the nuances we have
> started
>     >>> discussing?  I > imagine this would be simple for scrapers to
>     >>> implement if the signal > were in effect.
>     >>>
>     >>> It sounds simple, but it will be only the well-behaved bots that
> look
>     >>> for and listen to this signal.  The site will still have to defend
>     >>> against poorly behaved bots...
>     >>>
>     >>> That's why the group wants to annoit well-behaved bots with a
>     >>> blessing that allows them to be excluded from the heavy defenses.
>     >>>
>     >>> > 2.
>     >>> >
>     >>>
> https://www.conserver.io/deep-dives/vcon-lifecycle-management-using-scitt
>     >>> > -- I had never heard of SCITT and it was suggested by two
>     >>> independent > people as a good way to record datasets, their
>     >>> provenance and > potentially as a basis for an accreditation /
>     >>> standards effort.  At > ARDC
>     >>> <https://responsibledatacollection.org/>we have been looking at >
>     >>> croissant as a dataset standard for web data and have done a bit
> of >
>     >>> work to this end for a combined dataset documentation standard for
>     >>> both > crawlers and scrapers but that's in draft still (we may need
>     >>> two, one > for crawling and a separate one for scraping) I will
> also
>     >>> try to wrap > my mind around SCITT to see how it might fit as a
>     >>> solution.
>     >>>
>     >>> Does not seem to fit within web-bot-auth itself to me.  Obviously,
>     >>> vCon and SCITT *are* existing WGs producing specifications, which
>     >>> maybe some crawlers would want to leverage in some way.
>     >>>
>     >>>
>     >>> --
>     >>> Michael Richardson <[e-mail]>, Sandelman Software
> Works
>     >>> -= IPv6 IoT consulting =- *I*LIKE*TRAINS*
>     >>>
>     >>>
>     >>>
>     >>> _______________________________________________
>     >> Web-bot-auth mailing list -- [e-mail] To unsubscribe
> send
>     >> an email to [e-mail]
>     >>
>
>     > ----------------------------------------------------
>     > Alternatives:
>
>     > ----------------------------------------------------
> _______________________________________________
> Web-bot-auth mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
>