Re: [auth48] AUTH48: RFC-to-be 9309 <draft-koster-rep-12> for your review

[Lynne Bartholomew <[e-mail]>]

Hi, Martijn.

We are preparing this document for publication and could not find a reply regarding the following:

> Martijn, your contact information in the Authors' Addresses section now appears as follows.  Please confirm that this is as desired:
> 
>   Martijn Koster (editor)
>   Suton Lane
>   Wymondham, Norfolk
>   NR18 9JG
>   United Kingdom
>   Email: [e-mail]

Please let us know whether this listing is OK "as is" or should be updated.

Thank you!

RFC Editor/lb


> On Sep 2, 2022, at 9:30 AM, Lynne Bartholomew <[e-mail]> wrote:
> 
> Hi, Gary.
> 
> Thank you, and glad to hear that the figures look good!
> 
> Also, thank you as always for the updated XML file!  Removing the quotes in Figure 5 looks good to us as well.
> 
> Thanks also for the clarification re. "more" and "valid".
> 
> The latest files are posted here:
> 
>   https://www.rfc-editor.org/authors/rfc9309.txt
>   https://www.rfc-editor.org/authors/rfc9309.pdf
>   https://www.rfc-editor.org/authors/rfc9309.html
>   https://www.rfc-editor.org/authors/rfc9309.xml
>   https://www.rfc-editor.org/authors/rfc9309-diff.html
>   https://www.rfc-editor.org/authors/rfc9309-rfcdiff.html
>   https://www.rfc-editor.org/authors/rfc9309-auth48diff.html
>   https://www.rfc-editor.org/authors/rfc9309-lastdiff.html
>   https://www.rfc-editor.org/authors/rfc9309-lastrfcdiff.html
> 
>   https://www.rfc-editor.org/authors/rfc9309-xmldiff1.html
>   https://www.rfc-editor.org/authors/rfc9309-xmldiff2.html
> 
> We have confirmed your approval on the AUTH48 status page and will move this document forward for publication shortly:
> 
>   https://www.rfc-editor.org/auth48/rfc9309
> 
> Again, many thanks for your help!
> 
> RFC Editor/lb
> 
> 
>> On Sep 2, 2022, at 4:11 AM, Gary Illyes <[e-mail]> wrote:
>> 
>> Hi Lynne,
>> 
>> The conversion of the tables to artwork is ingenious. The tables now look good. 
>> I made one modification in figure 5 in section 2.2.3. Specifically, removed the double quotes from the 1st and 3rd columns as they seemed superfluous. Feel free to change it back if the double quotes are an RFC style requirement. The modified XML is attached and also available at https://github.com/google/robotstxt/blob/master/protocol-draft/rfc9309.xml
>> 
>> The "a more valid" was changed to "valid"; the "more" qualifier could've suggested that robots.txt is in fact a valid security measure, just not as valid as some other, which is just wrong. 
>> 
>> As for the merged group in figure 2/table 2, I staged the draft using author-tools.ietf.org, both as HTML and TXT, and it looks good to me. 
>> 
>> Overall, pending those double quotes in figure 5, this looks good to me and is ready for shipping. 
>> 
>> Thanks for working with us on this Lynne!
>> 
>> 
>> 
>> On Thu, Sep 1, 2022 at 11:27 PM Lynne Bartholomew <[e-mail]> wrote:
>> Resending, with inline email headers pared down, as we received a bounce due to the many full headers that I'd included earlier.  (I'd included the full headers for tracking purposes.)  Hoping that this reaches you all.
>> 
>>> On Sep 1, 2022, at 1:06 PM, Lynne Bartholomew <[e-mail]> wrote:
>>> 
>>> Hi, Gary**, Henner, and Murray.  Thank you for the emails.  Gary, thanks for the updated XML files as well!
>>> 
>>> ** Gary, although we now have all approvals for publication, due to the number of updates made since 31 August, please let us know if further changes are needed; we have a couple questions below.
>>> 
>>> After everything is resolved and we get confirmation re. same, we will be able to move this document forward for publication.
>>> 
>>> Gary, regarding this item:
>>> 
>>>> Line 495 changed: removed "a more valid", which may have suggested that robots.txt is, in fact, a valid security measure—which most certainly isn't.
>>> 
>>> We still see the word "valid" in this sentence.  Please confirm that this is as desired.
>>> 
>>> Regarding the table-format issues:  We have converted the tables to figures; it was the only way to fix the text output while conforming to current formatting guidelines.  Please note that due to font and spacing differences between the different outputs, we could not always get the same layout in the .txt output as what was previously seen in the tables in the .html output.  (For example, Figure 1 maxes out the margins in the .txt output.)
>>> 
>>> Please review all of the figures carefully.  For example, in the previous iteration, the position of "disallow: /baz" in what was Table 2 (now Figure 2) was different in the .txt output than in the .html output (where "disallow: /baz" was on its own line); which is correct?* It isn't clear from the original file what is correct.  Please let us know if further updates are needed.
>>> 
>>> * We kept "disallow: /baz" on its own line for now, as it appears to be part of the single "merged" group.
>>> 
>>> The latest files are posted here (please refresh your browser to see the latest):
>>> 
>>>   https://www.rfc-editor.org/authors/rfc9309.txt
>>>   https://www.rfc-editor.org/authors/rfc9309.pdf
>>>   https://www.rfc-editor.org/authors/rfc9309.html
>>>   https://www.rfc-editor.org/authors/rfc9309.xml
>>>   https://www.rfc-editor.org/authors/rfc9309-diff.html
>>>   https://www.rfc-editor.org/authors/rfc9309-rfcdiff.html
>>>   https://www.rfc-editor.org/authors/rfc9309-auth48diff.html
>>>   https://www.rfc-editor.org/authors/rfc9309-lastdiff.html
>>>   https://www.rfc-editor.org/authors/rfc9309-lastrfcdiff.html
>>> 
>>>   https://www.rfc-editor.org/authors/rfc9309-xmldiff1.html
>>>   https://www.rfc-editor.org/authors/rfc9309-xmldiff2.html
>>> 
>>> Henner and Murray, we have noted your approvals on the AUTH48 status page:
>>> 
>>>   https://www.rfc-editor.org/auth48/rfc9309
>>> 
>>> As noted above, we will move this document forward for publication after we receive confirmation that everything is correct.
>>> 
>>> Thanks again!
>>> 
>>> RFC Editor/lb
>>> 
>>> 
>>>> From: Gary Illyes 
>>>> Subject: Re: *[AD] Re: AUTH48: RFC-to-be 9309 <draft-koster-rep-12> for your review
>>>> Date: August 31, 2022 at 12:54:32 PM PDT
>>>> 
>>>> 
>>>> Hi Lynne,
>>>> 
>>>> We just noticed an oddity and we're wondering if we're doing something wrong or if it's a bug in the XML to text converter. Specifically, some of our tables get line breaks in the text representation when they really shouldn't. For example, table 1 renders as such on author-tools.ietf.org:
>>>> | User-Agent: Mozilla/5.0               | user-agent:     |
>>>> | (compatible; ExampleBot/0.1;      | ExampleBot      |
>>>> 
>>>> In the XML version (and HTML) "ExampleBot" in the second column is on the same line as "user-agent:". However, as illustrated above, in the converted text version it's on a new line. This renders the reference plain wrong. 
>>>> 
>>>> Do you know if we can fix that somehow? Perhaps adding &nbps; to prevent the converter from breaking the line?
>>>> 
>>>> Attached a screenshot for reference.
>>>> 
>>>> <image.png>
>>> 
>>>> From: Gary Illyes 
>>>> Subject: Re: *[AD] Re: AUTH48: RFC-to-be 9309 <draft-koster-rep-12> for your review
>>>> Date: August 31, 2022 at 11:43:31 AM PDT
>>>> 
>>>> 
>>>> I put an xref to RFC 9110 at the first appearance of HTTP in the prose: https://github.com/google/robotstxt/blob/master/protocol-draft/rfc9309.xml#L180
>>>> 
>>>> Updated XML also attached. 
>>> 
>>>> From: Gary Illyes 
>>>> Subject: Re: *[AD] Re: AUTH48: RFC-to-be 9309 <draft-koster-rep-12> for your review
>>>> Date: August 31, 2022 at 11:13:41 AM PDT
>>>> 
>>>> 
>>>> Sure, makes sense; we can do that. In that case I'll use the latest HTTP RFC instead of 1945
>>> 
>>>> From: "Murray S. Kucherawy" <[e-mail]>
>>>> Subject: Re: *[AD] Re: AUTH48: RFC-to-be 9309 <draft-koster-rep-12> for your review
>>>> Date: August 31, 2022 at 11:11:30 AM PDT
>>>> To: Gary Illyes 
>>>> 
>>>> 
>>>> Fair enough.  I wonder though if we should have some kind of link to what HTTP is anyway, perhaps attached to its first appearance in the prose after the introduction.  It really is a normative dependency here, even if it is ubiquitous.
>>>> 
>>>> -MSK
>>> 
>>>> From: Gary Illyes 
>>>> Subject: Re: *[AD] Re: AUTH48: RFC-to-be 9309 <draft-koster-rep-12> for your review
>>>> Date: August 31, 2022 at 11:02:18 AM PDT
>>>> 
>>>> 
>>>> We could, if there was an RFC that says that clients must follow *at least* 5 hops. Unfortunately all I can find is the opposite, follow *up to* 5 hops (because more is likely a loop).
>>> 
>>>> From: "Murray S. Kucherawy" <[e-mail]>
>>>> Subject: Re: *[AD] Re: AUTH48: RFC-to-be 9309 <draft-koster-rep-12> for your review
>>>> Date: August 31, 2022 at 10:58:54 AM PDT
>>>> To: Lynne Bartholomew <[e-mail]>
>>>> 
>>>> 
>>>> The new "MUST NOT" is fine.  Rather than deleting it, should we replace the reference to RFC 1945 with a reference to one of the newer HTTP RFCs?
>>>> 
>>>> -MSK, ART AD
>>> 
>>>> From: Gary Illyes 
>>>> Subject: Re: *[AD] Re: AUTH48: RFC-to-be 9309 <draft-koster-rep-12> for your review
>>>> Date: August 31, 2022 at 10:52:12 AM PDT
>>>> To: Lynne Bartholomew <[e-mail]>, "Murray S. Kucherawy" <[e-mail]>
>>>> 
>>>> 
>>>> With Henner's approval, I believe we're only missing the AD approval for the minute changes we made and Lynne highlighted at the beginning of this fork of the thread.
>>>> 
>>>> https://www.rfc-editor.org/auth48/rfc9309
>>>> 
>>>> 
>>>> On Mon, 29 Aug 2022 at 20:36, Lynne Bartholomew <[e-mail]> wrote:
>>>> Dear Gary,
>>>> 
>>>> No worries, and thank you for the clarification.
>>>> 
>>>> RFC Editor/lb
>>>> 
>>>>> On Aug 29, 2022, at 10:49 AM, Gary Illyes wrote:
>>>>> 
>>>>> Apologies, should've noted the new sentence:
>>>>> 
>>>>> "For example, a "Sitemaps" record MUST NOT terminate a group."
>>>>> 
>>>>> The normative reference to RFC 1945 was an erroneous reference; we implied that that RFC defines how many hops to follow in a redirect chain, which is simply not the case. 
>>>>> 
>>>>> On Mon, 29 Aug 2022 at 19:39, Lynne Bartholomew <[e-mail]> wrote:
>>>>> Dear Gary, Lizzi, Martijn, and *AD (Murray),
>>>>> 
>>>>> * Murray, please let us know if you approve (1) the removal of Normative Ref. RFC 1945 and (2) a new sentence containing "MUST NOT" in Section 2.2.4.
>>>>> 
>>>>> Gary, thank you very much for the updated XML files!
>>>>> 
>>>>> Martijn, your contact information in the Authors' Addresses section now appears as follows.  Please confirm that this is as desired:
>>>>> 
>>>>>   Martijn Koster (editor)
>>>>>   Suton Lane
>>>>>   Wymondham, Norfolk
>>>>>   NR18 9JG
>>>>>   United Kingdom
>>>>>   Email: [e-mail]
>>>>> 
>>>>> The latest files are posted here:
>>>>> 
>>>>>   https://www.rfc-editor.org/authors/rfc9309.txt
>>>>>   https://www.rfc-editor.org/authors/rfc9309.pdf
>>>>>   https://www.rfc-editor.org/authors/rfc9309.html
>>>>>   https://www.rfc-editor.org/authors/rfc9309.xml
>>>>>   https://www.rfc-editor.org/authors/rfc9309-diff.html
>>>>>   https://www.rfc-editor.org/authors/rfc9309-rfcdiff.html
>>>>>   https://www.rfc-editor.org/authors/rfc9309-auth48diff.html
>>>>> 
>>>>>   https://www.rfc-editor.org/authors/rfc9309-xmldiff1.html
>>>>>   https://www.rfc-editor.org/authors/rfc9309-xmldiff2.html
>>>>> 
>>>>> Gary, Lizzi, and Martijn, we have noted your approvals on the AUTH48 status page:
>>>>> 
>>>>>   https://www.rfc-editor.org/auth48/rfc9309
>>>>> 
>>>>> Thanks again!
>>>>> 
>>>>> RFC Editor/lb
>>>>> 
>>>>> 
>>>>>> From: Gary Illyes 
>>>>>> Subject: Re: AUTH48: RFC-to-be 9309 <draft-koster-rep-12> for your review
>>>>>> Date: August 29, 2022 at 2:41:50 AM PDT
>>>> 
>>>>>> 
>>>>>> Thank you. Done on github (https://github.com/google/robotstxt/blob/master/protocol-draft/rfc9309.xml#L18) and attached to this email again.
>>>>>> 
>>>>>> Henner, we need the last LGTM from you. 
>>>>> 
>>> 
>>> <snipped>
>>> 
>>>> On Aug 31, 2022, at 10:08 AM, Gary Illyes wrote:
>>>> 
>>>> New version up addressing Henner's last comment (also attached to this email): https://github.com/google/robotstxt/blob/master/protocol-draft/rfc9309.xml#L495
>>>> 
>>>> Line 495 changed: removed "a more valid", which may have suggested that robots.txt is, in fact, a valid security measure—which most certainly isn't.
>>>> 
>>>> I think that was the last of the approvals from our side. Passing baton to AD 
>>>> 
>>>> On Wed, Aug 31, 2022 at 6:59 PM Henner Zeller <[e-mail]> wrote:
>>>> Looks good to me. Approved!
>>>> 
>>>> On Mon, 29 Aug 2022 at 02:42, Gary Illyes wrote:
>>>> Thank you. Done on github (https://github.com/google/robotstxt/blob/master/protocol-draft/rfc9309.xml#L18) and attached to this email again.
>>>> 
>>>> Henner, we need the last LGTM from you. 
>>>> 
>>>> On Mon, Aug 29, 2022 at 11:01 AM Martijn Koster <[e-mail]> wrote:
>>>> Hi Gary,
>>>> 
>>>> Please remove my organisation Stalworthy Computing, Ltd  from
>>>> https://github.com/google/robotstxt/blob/master/protocol-draft/rfc9309.xml#L18
>>>> 
>>>> The rest LGTM
>>>> 
>>>> Thanks,
>>>> 
>>>> — Martijn
>>>> 
>>>>> On 26 Aug 2022, at 15:08, Gary Illyes wrote:
>>>>> 
>>>>> Thank you for your edits and review. Lizzi and I addressed the comments received and attached the updated XML to this email (also at https://github.com/google/robotstxt/blob/master/protocol-draft/rfc9309.xml)
>>>>> 
>>>>> The draft attached looks good to me and from my perspective approved for publication. 
>>>>> 
>>>>> Martijn, Lizzi, Henner, please review this draft and provide feedback (probably on GitHub) and/or approval for publication.
>>>>> 
>>>>> 
>>>>> On Fri, Aug 26, 2022 at 8:24 AM <[e-mail]> wrote:
>>>>> *****IMPORTANT*****
>>>>> 
>>>>> Updated 2022/08/25
>>>>> 
>>>>> RFC Author(s):
>>>>> --------------
>>>>> 
>>>>> Instructions for Completing AUTH48
>>>>> 
>>>>> Your document has now entered AUTH48.  Once it has been reviewed and 
>>>>> approved by you and all coauthors, it will be published as an RFC.  
>>>>> If an author is no longer available, there are several remedies 
>>>>> available as listed in the FAQ (https://www.rfc-editor.org/faq/).
>>>>> 
>>>>> You and you coauthors are responsible for engaging other parties 
>>>>> (e.g., Contributors or Working Group) as necessary before providing 
>>>>> your approval.
>>>>> 
>>>>> Planning your review 
>>>>> ---------------------
>>>>> 
>>>>> Please review the following aspects of your document:
>>>>> 
>>>>> *  RFC Editor questions
>>>>> 
>>>>>   Please review and resolve any questions raised by the RFC Editor 
>>>>>   that have been included in the XML file as comments marked as 
>>>>>   follows:
>>>>> 
>>>>>   <!-- [rfced] ... -->
>>>>> 
>>>>>   These questions will also be sent in a subsequent email.
>>>>> 
>>>>> *  Changes submitted by coauthors 
>>>>> 
>>>>>   Please ensure that you review any changes submitted by your 
>>>>>   coauthors.  We assume that if you do not speak up that you 
>>>>>   agree to changes submitted by your coauthors.
>>>>> 
>>>>> *  Content 
>>>>> 
>>>>>   Please review the full content of the document, as this cannot 
>>>>>   change once the RFC is published.  Please pay particular attention to:
>>>>>   - IANA considerations updates (if applicable)
>>>>>   - contact information
>>>>>   - references
>>>>> 
>>>>> *  Copyright notices and legends
>>>>> 
>>>>>   Please review the copyright notice and legends as defined in
>>>>>   RFC 5378 and the Trust Legal Provisions 
>>>>>   (TLP – https://trustee.ietf.org/license-info/).
>>>>> 
>>>>> *  Semantic markup
>>>>> 
>>>>>   Please review the markup in the XML file to ensure that elements of  
>>>>>   content are correctly tagged.  For example, ensure that <sourcecode> 
>>>>>   and <artwork> are set correctly.  See details at 
>>>>>   <https://authors.ietf.org/rfcxml-vocabulary>.
>>>>> 
>>>>> *  Formatted output
>>>>> 
>>>>>   Please review the PDF, HTML, and TXT files to ensure that the 
>>>>>   formatted output, as generated from the markup in the XML file, is 
>>>>>   reasonable.  Please note that the TXT will have formatting 
>>>>>   limitations compared to the PDF and HTML.
>>>>> 
>>>>> 
>>>>> Submitting changes
>>>>> ------------------
>>>>> 
>>>>> To submit changes, please reply to this email using ‘REPLY ALL’ as all 
>>>>> the parties CCed on this message need to see your changes. The parties 
>>>>> include:
>>>>> 
>>>>>   *  your coauthors
>>>>> 
>>>>>   *  [e-mail] (the RPC team)
>>>>> 
>>>>>   *  other document participants, depending on the stream (e.g., 
>>>>>      IETF Stream participants are your working group chairs, the 
>>>>>      responsible ADs, and the document shepherd).
>>>>> 
>>>>>   *  [e-mail], which is a new archival mailing list 
>>>>>      to preserve AUTH48 conversations; it is not an active discussion 
>>>>>      list:
>>>>> 
>>>>>     *  More info:
>>>>>        https://mailarchive.ietf.org/arch/msg/ietf-announce/yb6lpIGh-4Q9l2USxIAe6P8O4Zc
>>>>> 
>>>>>     *  The archive itself:
>>>>>        https://mailarchive.ietf.org/arch/browse/auth48archive/
>>>>> 
>>>>>     *  Note: If only absolutely necessary, you may temporarily opt out 
>>>>>        of the archiving of messages (e.g., to discuss a sensitive matter).
>>>>>        If needed, please add a note at the top of the message that you 
>>>>>        have dropped the address. When the discussion is concluded, 
>>>>>        [e-mail] will be re-added to the CC list and 
>>>>>        its addition will be noted at the top of the message. 
>>>>> 
>>>>> You may submit your changes in one of two ways:
>>>>> 
>>>>> An update to the provided XML file
>>>>> — OR —
>>>>> An explicit list of changes in this format
>>>>> 
>>>>> Section # (or indicate Global)
>>>>> 
>>>>> OLD:
>>>>> old text
>>>>> 
>>>>> NEW:
>>>>> new text
>>>>> 
>>>>> You do not need to reply with both an updated XML file and an explicit 
>>>>> list of changes, as either form is sufficient.
>>>>> 
>>>>> We will ask a stream manager to review and approve any changes that seem
>>>>> beyond editorial in nature, e.g., addition of new text, deletion of text, 
>>>>> and technical changes.  Information about stream managers can be found in 
>>>>> the FAQ.  Editorial changes do not require approval from a stream manager.
>>>>> 
>>>>> 
>>>>> Approving for publication
>>>>> --------------------------
>>>>> 
>>>>> To approve your RFC for publication, please reply to this email stating
>>>>> that you approve this RFC for publication.  Please use ‘REPLY ALL’,
>>>>> as all the parties CCed on this message need to see your approval.
>>>>> 
>>>>> 
>>>>> Files 
>>>>> -----
>>>>> 
>>>>> The files are available here:
>>>>>   https://www.rfc-editor.org/authors/rfc9309.xml
>>>>>   https://www.rfc-editor.org/authors/rfc9309.html
>>>>>   https://www.rfc-editor.org/authors/rfc9309.pdf
>>>>>   https://www.rfc-editor.org/authors/rfc9309.txt
>>>>> 
>>>>> Diff file of the text:
>>>>>   https://www.rfc-editor.org/authors/rfc9309-diff.html
>>>>>   https://www.rfc-editor.org/authors/rfc9309-rfcdiff.html (side by side)
>>>>> 
>>>>> Diff of the XML: 
>>>>>   https://www.rfc-editor.org/authors/rfc9309-xmldiff1.html
>>>>> 
>>>>> The following files are provided to facilitate creation of your own 
>>>>> diff files of the XML.  
>>>>> 
>>>>> Initial XMLv3 created using XMLv2 as input:
>>>>>   https://www.rfc-editor.org/authors/rfc9309.original.v2v3.xml 
>>>>> 
>>>>> XMLv3 file that is a best effort to capture v3-related format updates 
>>>>> only: 
>>>>>   https://www.rfc-editor.org/authors/rfc9309.form.xml
>>>>> 
>>>>> 
>>>>> Tracking progress
>>>>> -----------------
>>>>> 
>>>>> The details of the AUTH48 status of your document are here:
>>>>>   https://www.rfc-editor.org/auth48/rfc9309
>>>>> 
>>>>> Please let us know if you have any questions.  
>>>>> 
>>>>> Thank you for your cooperation,
>>>>> 
>>>>> RFC Editor
>>>>> 
>>>>> --------------------------------------
>>>>> RFC9309 (draft-koster-rep-12)
>>>>> 
>>>>> Title            : Robots Exclusion Protocol
>>>>> Author(s)        : M. Koster, Ed., G. Illyes, Ed., H. Zeller, Ed., L. Sassman, Ed.
>>>>> WG Chair(s)      : 
>>>>> Area Director(s) : 
>>>>> 
>>>>> 
>>>>> <rfc9309.xml>
>>>> 
>>>> <rfc9309.xml>
>>> 
>> 
>> <rfc9309.xml>
>