Re: [secdir] SECDIR Review draft-koster-rep

[tirumal reddy <[e-mail]>]

On Thu, 23 Jun 2022 at 13:25, Gary Illyes <[e-mail]> wrote:

>
>
> On Thu, 23 Jun 2022 at 09:42, tirumal reddy <[e-mail]> wrote:
>
>> On Wed, 22 Jun 2022 at 21:06, Gary Illyes <[e-mail]> wrote:
>>
>>> Thanks Tiru!
>>>
>>> I updated our public repository with your suggestions and a diff of the
>>> changes can be seen at
>>> https://github.com/google/robotstxt/commit/a048272f9091570db556cf3656b6d33250797bba
>>>
>>> Specifically on point c) we added a new paragraph with a list of vectors
>>> related to implementors based on a conversation we had with our security
>>> team. On point a) and b) we restated that robots.txt is not a security
>>> measure whatsoever and folks should employ a valid security measure such as
>>> IP based ACL
>>>
>>
>> Thanks Gary, changes look good to me. Are there better security measures
>> to refer to than the IP based ACL (IP reputation is challenging with IPv6
>> addresses and the IP address can possibly be spoofed) ?
>>
>
> Ah, fair. How about the authentication framework of the used protocol?
> i.e. http auth for http, I think RFC 7235
>

Sounds good to me. You may want to refer to
https://datatracker.ietf.org/doc/html/rfc9110#section-11.

-Tiru


>
>
>> -Tiru
>>
>>
>>>
>>> On Mon, Jun 20, 2022 at 1:51 PM tirumal reddy <[e-mail]> wrote:
>>>
>>>> SECDIR Review draft-koster-rep
>>>>
>>>>
>>>> Reviewer: Tirumaleswar Reddy
>>>> Review result: Ready with Issues
>>>>
>>>>
>>>>
>>>> I have reviewed this document as part of the security directorate's
>>>>
>>>> ongoing effort to review all IETF documents being processed by the
>>>>
>>>> IESG..  Document editors and WG chairs should treat these comments
>>>>
>>>> just like any other last call comments.
>>>>
>>>>
>>>>
>>>> You may want to discuss the following security threats:
>>>>
>>>>
>>>>
>>>> a) Revealing disallowed URIs will make its paths easily discoverable.
>>>> However, security by obscurity will not maintain or increase the security
>>>> of the content provider (you can refer to
>>>> https://datatracker.ietf.org/doc/html/rfc4949).
>>>>
>>>> b) A malicious crawler will not honor the disallow rules and can try to
>>>> access the disallowed URIs, it should be mitigated by access control
>>>> restrictions. Discuss any other count-measures used to block such malicious
>>>> crawlers (like blocking the IP address).
>>>>
>>>> c) Attacks possible on crawlers because of a malicious robots.txt file.
>>>>
>>>>
>>>> Cheers,
>>>>
>>>> -Tiru
>>>>
>>> --
> Thanks,
> Gary
>