Re: [secdir] SECDIR Review draft-koster-rep

[Gary Illyes <[e-mail]>]

Thanks! Updated:
https://github.com/google/robotstxt/commit/d69b3e4fdfd2c2a767c2cda7c27370a47b1fb575

On Thu, Jun 23, 2022 at 12:09 PM tirumal reddy <[e-mail]> wrote:

> On Thu, 23 Jun 2022 at 13:25, Gary Illyes <[e-mail]> wrote:
>
>>
>>
>> On Thu, 23 Jun 2022 at 09:42, tirumal reddy <[e-mail]> wrote:
>>
>>> On Wed, 22 Jun 2022 at 21:06, Gary Illyes <[e-mail]> wrote:
>>>
>>>> Thanks Tiru!
>>>>
>>>> I updated our public repository with your suggestions and a diff of the
>>>> changes can be seen at
>>>> https://github.com/google/robotstxt/commit/a048272f9091570db556cf3656b6d33250797bba
>>>>
>>>> Specifically on point c) we added a new paragraph with a list of
>>>> vectors related to implementors based on a conversation we had with our
>>>> security team. On point a) and b) we restated that robots.txt is not a
>>>> security measure whatsoever and folks should employ a valid security
>>>> measure such as IP based ACL
>>>>
>>>
>>> Thanks Gary, changes look good to me. Are there better security measures
>>> to refer to than the IP based ACL (IP reputation is challenging with IPv6
>>> addresses and the IP address can possibly be spoofed) ?
>>>
>>
>> Ah, fair. How about the authentication framework of the used protocol?
>> i.e. http auth for http, I think RFC 7235
>>
>
> Sounds good to me. You may want to refer to
> https://datatracker.ietf.org/doc/html/rfc9110#section-11.
>
> -Tiru
>
>
>>
>>
>>> -Tiru
>>>
>>>
>>>>
>>>> On Mon, Jun 20, 2022 at 1:51 PM tirumal reddy <[e-mail]>
>>>> wrote:
>>>>
>>>>> SECDIR Review draft-koster-rep
>>>>>
>>>>>
>>>>> Reviewer: Tirumaleswar Reddy
>>>>> Review result: Ready with Issues
>>>>>
>>>>>
>>>>>
>>>>> I have reviewed this document as part of the security directorate's
>>>>>
>>>>> ongoing effort to review all IETF documents being processed by the
>>>>>
>>>>> IESG..  Document editors and WG chairs should treat these comments
>>>>>
>>>>> just like any other last call comments.
>>>>>
>>>>>
>>>>>
>>>>> You may want to discuss the following security threats:
>>>>>
>>>>>
>>>>>
>>>>> a) Revealing disallowed URIs will make its paths easily discoverable.
>>>>> However, security by obscurity will not maintain or increase the security
>>>>> of the content provider (you can refer to
>>>>> https://datatracker.ietf.org/doc/html/rfc4949).
>>>>>
>>>>> b) A malicious crawler will not honor the disallow rules and can try
>>>>> to access the disallowed URIs, it should be mitigated by access control
>>>>> restrictions. Discuss any other count-measures used to block such malicious
>>>>> crawlers (like blocking the IP address).
>>>>>
>>>>> c) Attacks possible on crawlers because of a malicious robots.txt file.
>>>>>
>>>>>
>>>>> Cheers,
>>>>>
>>>>> -Tiru
>>>>>
>>>> --
>> Thanks,
>> Gary
>>
>