Re: [secdir] SECDIR Review draft-koster-rep

[Gary Illyes <[e-mail]>]

On Thu, 23 Jun 2022 at 09:42, tirumal reddy <[e-mail]> wrote:

> On Wed, 22 Jun 2022 at 21:06, Gary Illyes <[e-mail]> wrote:
>
>> Thanks Tiru!
>>
>> I updated our public repository with your suggestions and a diff of the
>> changes can be seen at
>> https://github.com/google/robotstxt/commit/a048272f9091570db556cf3656b6d33250797bba
>>
>> Specifically on point c) we added a new paragraph with a list of vectors
>> related to implementors based on a conversation we had with our security
>> team. On point a) and b) we restated that robots.txt is not a security
>> measure whatsoever and folks should employ a valid security measure such as
>> IP based ACL
>>
>
> Thanks Gary, changes look good to me. Are there better security measures
> to refer to than the IP based ACL (IP reputation is challenging with IPv6
> addresses and the IP address can possibly be spoofed) ?
>

Ah, fair. How about the authentication framework of the used protocol? i.e.
http auth for http, I think RFC 7235



> -Tiru
>
>
>>
>> On Mon, Jun 20, 2022 at 1:51 PM tirumal reddy <[e-mail]> wrote:
>>
>>> SECDIR Review draft-koster-rep
>>>
>>>
>>> Reviewer: Tirumaleswar Reddy
>>> Review result: Ready with Issues
>>>
>>>
>>>
>>> I have reviewed this document as part of the security directorate's
>>>
>>> ongoing effort to review all IETF documents being processed by the
>>>
>>> IESG..  Document editors and WG chairs should treat these comments
>>>
>>> just like any other last call comments.
>>>
>>>
>>>
>>> You may want to discuss the following security threats:
>>>
>>>
>>>
>>> a) Revealing disallowed URIs will make its paths easily discoverable.
>>> However, security by obscurity will not maintain or increase the security
>>> of the content provider (you can refer to
>>> https://datatracker.ietf.org/doc/html/rfc4949).
>>>
>>> b) A malicious crawler will not honor the disallow rules and can try to
>>> access the disallowed URIs, it should be mitigated by access control
>>> restrictions. Discuss any other count-measures used to block such malicious
>>> crawlers (like blocking the IP address).
>>>
>>> c) Attacks possible on crawlers because of a malicious robots.txt file.
>>>
>>>
>>> Cheers,
>>>
>>> -Tiru
>>>
>> --
Thanks,
Gary