[ai-control] Re: AI Processing - a more focused top category?

[Alissa Cooper <[e-mail]>]

> On Sep 13, 2025, at 11:08 AM, Eric Rescorla <[e-mail]> wrote:
> 
> Hi Alissa,
> 
> I feel like even "obtained" is kind of fuzzy. To take the example Greg Lindahl gave, in a typical crawling scenario, the crawler will retrieve an asset, then parse it for links, and follow those linke. Is that parsing part of access or is it part of processing?

My interpretation of the current version of draft-ietf-aipref-attach is that parsing is part of usage (processing). This is where the underlying motivation of Krishna’s vocabulary proposal seems worth exploring. Before gen AI, preferences about usage could not be expressed in robots.txt, but a bunch of detailed preferences could (and still are) conveyed via meta tags and X-Robots-Tag headers, including nofollow, which I think covers Greg’s use case. If we add automated processing to the vocabulary (which allows sites to set it =n in a Content-Usage rule), then it seems like the vocabulary has to say something explicit about these other usage preferences. Otherwise, at a minimum we’ll have situations where the interpretation of the preference is ambiguous, or worse yet we’ll have sites conveying blanket preferences to disallow automated processing without realizing the consequences.

Alissa

> 
> -Ekr
> 
> 
> 
> 
> On Sat, Sep 13, 2025 at 3:33 AM Alissa Cooper <[e-mail] <mailto:[e-mail]>> wrote:
>> I think an underlying issue here is that the vocabulary spec has no stated limitation about restricting the subset of automated activities to which the definitions may apply, but the one attachment mechanism we’ve been discussing in this group does express such a restriction. I.e., draft-ietf-aipref-attach says that for assets made available via HTTP, the allow/disallow rules say whether an asset can be accessed, and the content-usage rules say whether it can be used for various purposes:
>> 
>> The Allow and
>>    Disallow rules determine what resources can be crawled, using the
>>    rule that has the longest matching path prefix, as defined in
>>    Section 2.2.2 of [ROBOTS].
>> 
>>    This creates a two-stage arrangement that distinguishes acquisition
>>    and usage.  Acquisition relies on Allow/Disallow rules; usage
>>    preference relies on Content-Usage rules.
>> 
>>   ...
>> 
>>    Usage preferences apply only to those resources that can be crawled
>>    according to Allow/Disallow rules; no preferences are implied for
>>    resources that are disallowed.
>> 
>> In other words, in the context of this specific attachment mechanism, implementers and users are being told that one set of directives proscribes access/crawling, and the other set proscribes what happens after access/crawling. So even if one reads the definition of “automated processing” and it looks on its face as if setting it to =n would tell a crawler not to crawl, the attachment spec explains that content-usage preferences are only meant to apply to what is done with an asset after crawling. (I appreciate Greg’s separate point that it is hard for habituated human behavior to adapt to new rules and people don’t read the spec and therefore "Content-Usage: bots=n” will be misinterpreted by many.)
>> 
>> This raises the question of whether the vocabulary is only meant to be used to express preferences about usage and not initial access regardless of the attachment mechanism. The very first sentence of the vocab doc uses the ambiguous term “processing” while most of the rest of the doc uses “usage.” If the intent is to define a vocabulary that only expresses preferences about an asset after the asset has been obtained (by whatever means), it needs an explicit scoping limitation to that effect. And then every attachment mechanism that gets defined in the IETF or elsewhere needs to reflect that.
>> 
>> Making this distinction explicit might be a simple resolution to some of the concerns that have been raised about the automated processing category, because it would make clear that it’s unnecessary to resolve the question of whether crawling/accessing a site is itself considered automated processing. No preferences are being expressed about access, other than insofar as sites would expect crawlers not to crawl their site if the site wishes to disallow all the uses of interest to the crawler. It would not resolve the concerns about behavior habituation and misinterpretation, though.
>> 
>> Even with this distinction made explicit, there would still be the possibility of basically nonsensical use cases, e.g.:
>> 
>> User-Agent: *
>> Content-Usage: /foo bots=n
>> 
>> Like what is the point of allowing crawling/access but then disallowing all usage? Maybe there is a use case I’m missing.
>> 
>> Note also that it may be entirely unrealistic to say that everywhere an attachment mechanism is specified, it has to preserve this distinction between access and usage. It’s not like we can constrain mechanisms that get defined elsewhere.
>> 
>> I’ve never really been convinced of the need for the top-level category, but if we are going to have one we need to be clear about when it applies, and clarifying that it only applies after an asset has been obtained might help mitigate concerns about its breadth.
>> 
>> Alissa
>> 
>>> On Sep 3, 2025, at 6:23 PM, Greg Lindahl <[e-mail] <mailto:[e-mail]>> wrote:
>>> 
>>> For "Automated processing", I imagine that running code on html to extract outgoing links is "automated processing".
>>> 
>>> Therefore no crawler, even crawlers totally unrelated to AI, could function in such a world.
>>> 
>>> It would be smart for us to be careful to keep use cases in mind, including non-AI use cases. Thank you Timid Robot for bringing up this one.
>>> 
>>> On Tue, Sep 2, 2025 at 8:43 AM Timid Robot Zehta <[e-mail] <mailto:[e-mail]>> wrote:
>>>> Eric-
>>>> 
>>>> Regarding the end of your email, I should have said "Automated or AI processing" instead of just "processing". I think the answer may change depending on whether the proposed changes are accepted.
>>>> 
>>>> I can accept "there's not enough difference between 'automated' and 'AI' for this change to be meaningful" as a criticism, but I would prefer that you open a new thread to discuss your concerns with the definitions, or lack thereof, and whether there is a meaningful difference between retrieving and using data.
>>>> 
>>>> Sincerely,
>>>> 
>>>> -Timid Robot
>>>> 
>>>> On Tue, Sep 2, 2025 at 5:16 PM Eric Rescorla <[e-mail] <mailto:[e-mail]>> wrote:
>>>>> 
>>>>> On Tue, Sep 2, 2025 at 8:01 AM Timid Robot Zehta <[e-mail] <mailto:[e-mail]>> wrote:
>>>>>> I expect there are many scripts that download and/or scrape web pages that are not AI.
>>>>> 
>>>>> Well, I agree that there are many scripts that download and/or scrape Web pages that aren't what we would colloquially call AI. However, the relevant question is whether they meet the definition of AI in this specification, which recall, is (in the editor's copy):
>>>>> 
>>>>> "An engineered system of sufficient complexity that, for a given set of human-defined objectives, learns from data to generate outputs such as content, predictions, recommendations, or decisions"
>>>>> 
>>>>> And when combined with processing, we have, 
>>>>> "The act of using automated processing on one or more assets to analyze text and data in order to generate information which includes but is not limited to patterns, trends and correlations"
>>>>> 
>>>>> If, as you suggest, we replace "automated" with "AI", we end up with something like:
>>>>> 
>>>>> "The act of using an engineered system of sufficient complexity that, for a given set of human-defined objectives, learns from data to generate outputs such as content, predictions, recommendations, or decisions on one or more assets to analyze text and data in order to generate information which includes but is not limited to patterns, trends and correlations"
>>>>> 
>>>>> But this is largely redundant in that it basically says a system of "sufficient complexity" that "analyzes text and data in order to generate information", but, as I noted above, "sufficient complexity" is never defined. I'm still left asking what system would meet the definition with "automated" that doesn't meet this definition.
>>>>> 
>>>>>> 
>>>>>> To answer your second question, the first example disallows crawling of /foo and the second example disallows processing of the /foo data that has been crawled.
>>>>> 
>>>>> Hmm... What scenario are you envisioning where the automated client "crawls" /foo but doesn't "process" the data. I note that this draft doesn't seem to provide a definition of "processing" or even a link to one, so this is ambiguous. I suppose you could argue that if it just stored it without examining it at all that wasn't processing (though of course as a practical matter storing data on disk requires quite a bit of processing throughout the system). But if you're doing almost anything else (indexing it, counting the number of characters, etc.), I think you're almost certainly processing it according to a colloquial definition.
>>>>> 
>>>>> -Ekr
>>>>> 
>>>>>> 
>>>>>> On Tue, Sep 2, 2025 at 4:47 PM Eric Rescorla <[e-mail] <mailto:[e-mail]>> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> On Tue, Sep 2, 2025 at 7:31 AM Timid Robot Zehta <[e-mail] <mailto:[e-mail]>> wrote:
>>>>>>>> Greetings,
>>>>>>>> 
>>>>>>>> I wonder if replacing "Automated Processing" (all) with "AI Processing" (all-ai) would help to increase the accuracy of the vocabulary without hindering its effectiveness. I think this change might reduce the chance of unintended consequences while keeping the essential function.
>>>>>>>> 
>>>>>>>> Current text:
>>>>>>>> 
>>>>>>>> 4.1. Automated Processing Category
>>>>>>>> 
>>>>>>>> The act of using automated processing on one or more assets to analyze text and data in order to generate information which includes but is not limited to patterns, trends and correlations.
>>>>>>>> 
>>>>>>>> The use of assets for automated processing encompasses all the subsequent categories.
>>>>>>>> 
>>>>>>>> Proposed text:
>>>>>>>> 
>>>>>>>> 4.1. AI Processing Category
>>>>>>>> 
>>>>>>>> The act of using AI processing on one or more assets to analyze text and data in order to generate information which includes but is not limited to patterns, trends and correlations.
>>>>>>>> 
>>>>>>>> The use of assets for AI processing encompasses all the subsequent categories.
>>>>>>> 
>>>>>>> Given the very broad definition of AI, as noted in my previous email [0], I'm not sure what the practical difference would be of this change. Can you provide an example of some form of processing which you think would be "automated processing" but not "AI processing" using the definition of AI in Section 2? I see the addition of the term "sufficient complexity" but it's entirely unclear to me what that means. How am I to know whether a system is of sufficient complexity?
>>>>>>> 
>>>>>>> With that said, I'm not sure I understand the purpose of this category at all. Specifically, how is:
>>>>>>> 
>>>>>>> User-Agent: *
>>>>>>> disallow: /foo
>>>>>>> 
>>>>>>> different from:
>>>>>>> 
>>>>>>> User-Agent: *
>>>>>>> Content-Usage: /foo all=n
>>>>>>> 
>>>>>>> -Ekr
>>>>>>> 
>>>>>>> [0] https://mailarchive.ietf.org/arch/msg/ai-control/AmFQ2GTFAJwqNKZvtJIMfuRt0jI/
>>>>>>>> 
>>>>>>>> Sincerely,
>>>>>>>> 
>>>>>>>> -Timid Robot
>>>>>>>> 
>>>>>>>> --
>>>>>>>> Timid Robot Zehta (they/she) | Director, Technology | Creative Commons
>>>>>>>> [e-mail] <mailto:[e-mail]> | https://calendly.com/cc-timidrobot
>>>>>>>> 
>>>>>>>> Support CC with a donation:
>>>>>>>> https://e-donate.creativecommons.org/
>>>>>>>> 
>>>>>>>> Get CC news & updates:
>>>>>>>> https://mail.creativecommons.org/subscribe
>>>>>>>> -- 
>>>>>>>> ai-control mailing list -- [e-mail] <mailto:[e-mail]>
>>>>>>>> To unsubscribe send an email to [e-mail] <mailto:[e-mail]>
>>>> -- 
>>>> ai-control mailing list -- [e-mail] <mailto:[e-mail]>
>>>> To unsubscribe send an email to [e-mail] <mailto:[e-mail]>
>>> -- 
>>> ai-control mailing list -- [e-mail] <mailto:[e-mail]>
>>> To unsubscribe send an email to [e-mail] <mailto:[e-mail]>
>> 
>> -- 
>> ai-control mailing list -- [e-mail] <mailto:[e-mail]>
>> To unsubscribe send an email to [e-mail] <mailto:[e-mail]>