Re: [secdir] SECDIR Review draft-koster-rep

[Gary Illyes <[e-mail]>]

And uploaded: https://datatracker.ietf.org/doc/draft-koster-rep/11/

On Thu, Jun 23, 2022 at 12:38 PM Gary Illyes <[e-mail]> wrote:

> Thanks! Updated:
> https://github.com/google/robotstxt/commit/d69b3e4fdfd2c2a767c2cda7c27370a47b1fb575
>
> On Thu, Jun 23, 2022 at 12:09 PM tirumal reddy <[e-mail]> wrote:
>
>> On Thu, 23 Jun 2022 at 13:25, Gary Illyes <[e-mail]> wrote:
>>
>>>
>>>
>>> On Thu, 23 Jun 2022 at 09:42, tirumal reddy <[e-mail]> wrote:
>>>
>>>> On Wed, 22 Jun 2022 at 21:06, Gary Illyes <[e-mail]>
>>>> wrote:
>>>>
>>>>> Thanks Tiru!
>>>>>
>>>>> I updated our public repository with your suggestions and a diff of
>>>>> the changes can be seen at
>>>>> https://github.com/google/robotstxt/commit/a048272f9091570db556cf3656b6d33250797bba
>>>>>
>>>>> Specifically on point c) we added a new paragraph with a list of
>>>>> vectors related to implementors based on a conversation we had with our
>>>>> security team. On point a) and b) we restated that robots.txt is not a
>>>>> security measure whatsoever and folks should employ a valid security
>>>>> measure such as IP based ACL
>>>>>
>>>>
>>>> Thanks Gary, changes look good to me. Are there better security
>>>> measures to refer to than the IP based ACL (IP reputation is challenging
>>>> with IPv6 addresses and the IP address can possibly be spoofed) ?
>>>>
>>>
>>> Ah, fair. How about the authentication framework of the used protocol?
>>> i.e. http auth for http, I think RFC 7235
>>>
>>
>> Sounds good to me. You may want to refer to
>> https://datatracker.ietf.org/doc/html/rfc9110#section-11.
>>
>> -Tiru
>>
>>
>>>
>>>
>>>> -Tiru
>>>>
>>>>
>>>>>
>>>>> On Mon, Jun 20, 2022 at 1:51 PM tirumal reddy <[e-mail]>
>>>>> wrote:
>>>>>
>>>>>> SECDIR Review draft-koster-rep
>>>>>>
>>>>>>
>>>>>> Reviewer: Tirumaleswar Reddy
>>>>>> Review result: Ready with Issues
>>>>>>
>>>>>>
>>>>>>
>>>>>> I have reviewed this document as part of the security directorate's
>>>>>>
>>>>>> ongoing effort to review all IETF documents being processed by the
>>>>>>
>>>>>> IESG..  Document editors and WG chairs should treat these comments
>>>>>>
>>>>>> just like any other last call comments.
>>>>>>
>>>>>>
>>>>>>
>>>>>> You may want to discuss the following security threats:
>>>>>>
>>>>>>
>>>>>>
>>>>>> a) Revealing disallowed URIs will make its paths easily discoverable.
>>>>>> However, security by obscurity will not maintain or increase the security
>>>>>> of the content provider (you can refer to
>>>>>> https://datatracker.ietf.org/doc/html/rfc4949).
>>>>>>
>>>>>> b) A malicious crawler will not honor the disallow rules and can try
>>>>>> to access the disallowed URIs, it should be mitigated by access control
>>>>>> restrictions. Discuss any other count-measures used to block such malicious
>>>>>> crawlers (like blocking the IP address).
>>>>>>
>>>>>> c) Attacks possible on crawlers because of a malicious robots.txt
>>>>>> file.
>>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> -Tiru
>>>>>>
>>>>> --
>>> Thanks,
>>> Gary
>>>
>>