Re: [Last-Call] Last Call: <draft-koster-rep-06.txt> (Robots Exclusion Protocol) to Informational RFC

[Mark Nottingham <[e-mail]>]

Hi Ted,

Thanks for the response, especially since this is not strictly your problem any more. Responses below.


> On 8 Mar 2022, at 8:18 pm, Ted Hardie <[e-mail]> wrote:
> 
> Hi Mark,
> 
> On Mon, Feb 28, 2022 at 10:56 PM Mark Nottingham <[e-mail]> wrote:
>> 
>> 
>> > On 1 Mar 2022, at 9:29 am, John Levine <[e-mail]> wrote:
>> > 
>> > Most importantly, the copyright license is broken. At the top it has
>> > the "no derivatives" license, which is fine,
>> 
>> Ah - I missed, that, thanks for pointing it out. 
>> 
>> I'm uncomfortable leaving change control for a key interoperability mechanism in the search market in the hands of one competitor, yet blessing it as part of the IETF stream. I think the IETF as a whole should be uncomfortable with that too, given current competition enforcement trends.
> 
> Having the original author of the spec be the principal author here is a bit of a bulwark against that, as I don't believe he is or would be interested in handing change control over to Google. I also believe Gary and the other authors have reached out to the rest of the relevant community (though my change in employer means I know longer have the relevant e-mails to cite).

Hmm. Because Google employees are co-authors in a joint work, my understanding is that they have the ability to publish derivative works in the future, at least in the US. If that's true, it does give them a form of change control -- or at least significant privilege (regarding the future of the spec) over other members of the community.

 (Obviously, one would need to talk to a copyright lawyer -- my understanding here is informed by a similar situation in a different venue)


> On the more general topic of why this has the "no derivatives" clause,  I understand your reluctance, but I think this is a case where the combination is valid.  First, it's important to note that the specification was brought to the IETF for substantive review, to make sure that the elements it uses (like ABNF) were being used in the right way and to eliminate any possibility of ambiguity.  From my perspective, that's been very useful and it would not have occurred to the same extent had this gone directly to the ISE. 

I find this a bit surprising -- surely it's possible to get adequate review for ISE documents without putting them into the IETF stream? Otherwise, the Independent stream would jeopardise the quality of the RFC Series overall... Was the ISE involved in this discussion?


> However, this spec reflects operations which have been stable/backwards compatible for a very long time.  Given that, it is important to the community which deploys this that it be fairly difficult to amend.  One way to achieve that would have been to make this standards track; that would require standards action to update or obsolete it later.  When we discussed that back at the beginning of this process, though, it was pretty clear  that some folks would use the working group discussion around that to try to insert functionality that would result in breaking changes.  While it would have been kind of unlikely for any of those to win out against the need for maintaining interoperability, the result would have been a pretty big increase in the amount of effort needed to get this published.

This is the rub -- depending on your definitions of "the community" and "some folks" in the statement above, the outcome might be completely reasonable and justified, or blatantly illegitimate. 

It's also notably out of step with the direction that pretty much all other Internet and Web standards are taking. HTML, DOM, and many other aspects of the platform have considerable requirements for stability and backwards compatibility, and yet they are not locked behind a no-derivatives clause. 


> Another option for getting an archival spec with a high bar for change was this one:  an IETF informational with a no-derivatives clause. That gave the full benefit of IETF review and made the bar for amendment high  enough to allay the concerns of the original author and the relevant community.  It had this clause when Adam agreed to sponsor it and it has had it in every iteration since, so I thought this was well understood.  As shepherd, my apologies if it was not.
> 
> There is another option that gets the full set of characteristics needed:  AD sponsored on the standards track.  At the time this went through the first set of discussions that was something folks had become very reluctant to do.  If it is on the table, I personally believe that a standards track document with the usual clauses would work as well.  Those can't be superseded or amended without serious work and plenty of time for the relevant community to chip in.  
> 
> But, absent that, I think this kind of document is why BCP78 permits this combination: documents which need and have received significant IETF review but which also have a significant external community for whom the usual clauses result in a risk of inappropriate later amendments. To put this slightly differently, I think you'll see that this falls under the logic in RFC 5378, Section 3, in the penultimate paragraph. 

Assuming that you're referring to this paragraph in s 3.3:

~~~
   The IETF has historically encouraged organizations to publish details
   of their technologies, even when the technologies are proprietary,
   because understanding how existing technology is being used helps
   when developing new technology.  But organizations that publish
   information about proprietary technologies are frequently not willing
   to have the IETF produce revisions of the technologies and then
   possibly claim that the IETF version is the "new version" of the
   organization's technology.  Organizations that feel this way can
   specify that a Contribution be published with the other rights
   granted under this document but may withhold the right to produce
   derivative works other than translations.
~~~

... then the question is whether the robots.txt format is really 'proprietary' technology, or whether it's a public good. Given its wide deployment and use as what amounts to an API for search engines to interoperate with Web sites, I struggle to see it as the former.


> If you and the broader community prefer the standards track approach, now would be a good time to let the sponsoring AD know.

To be clear, I think that this document is almost certainly a reasonable record of how the robots.txt format works today, and that its authors are acting in good faith. However, given the circumstances I'm concerned that from the 'outside', publishing the document in this manner won't look legitimate -- and therefore call into question the legitimacy of the IETF itself, in some eyes.

I think there are a few different ways we could address that while meeting the authors' goals.

0) I still believe that removing the no-derivatives clause is the most straightforward to do so. TCP, QUIC, HTTP, and many other Internet specifications remain stable and backwards compatible without the benefit of a no-derivatives clause; I don't see how robots.txt is different.

1) Alternatively, statements from at least some other search engines that they are aware of this work and do not object to it being published would change how this action is perceived considerably. Ideally, this would be represented by adding authors from other search engines to the document.

2) Or, since the document has now been reviewed for ABNF, nothing stops it from being switched to the Independent Stream with a title like "The Google Robots Exclusion Protocol" (to reflect its 'proprietary' nature).

Cheers,


--
Mark Nottingham   https://www.mnot.net/