Re: [Int-area] Adam Roach's Discuss on draft-ietf-intarea-provisioning-domains-10: (with DISCUSS and COMMENT)

Tommy Pauly <tpauly@apple.com> Thu, 23 January 2020 00:18 UTC

Return-Path: <tpauly@apple.com>
X-Original-To: int-area@ietfa.amsl.com
Delivered-To: int-area@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 29F6312003F; Wed, 22 Jan 2020 16:18:17 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2
X-Spam-Level:
X-Spam-Status: No, score=-2 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=apple.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id r3z6uovqOPSk; Wed, 22 Jan 2020 16:18:14 -0800 (PST)
Received: from ma1-aaemail-dr-lapp03.apple.com (ma1-aaemail-dr-lapp03.apple.com [17.171.2.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 4107812002E; Wed, 22 Jan 2020 16:18:14 -0800 (PST)
Received: from pps.filterd (ma1-aaemail-dr-lapp03.apple.com [127.0.0.1]) by ma1-aaemail-dr-lapp03.apple.com (8.16.0.27/8.16.0.27) with SMTP id 00N0BvHI010707; Wed, 22 Jan 2020 16:18:12 -0800
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=sender : from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=k8ZUO9mSWQzpVrGP/2GeiY2qtMVkJ0fuYASq9dtrtmQ=; b=wrGWBUi12yoHup3qQrXl0oEZ7SQfFv+F0wuI/rV3OEVjWH3fU2HcQ8e227Br26XWDY52 D08GIQp1j1hfXWvq4zsMSzCuihzrh7EZGO6NiamfgUwfxpWY1/MR7/qW4HV9e948nN2a /D9Q+C3Vrq2jrt0t7/ntHPvKDxNRLjN6qDhlyQa/fQ8c44f5Hxgz7jUCjlCpxPS77lO4 0Rb8Nu+ErUmMvc9wS8BnEzU8/3IE4KgXSmhSCXx/53UPIx47eP489hUSPSRBGL1vZkld amhoenXHtzZdTpt0vPrj9socg7xrXSuhByHEEP/QVsHt6KH3cxbGzUXeUROFVh6t9RUz Ug==
Received: from ma1-mtap-s03.corp.apple.com (ma1-mtap-s03.corp.apple.com [17.40.76.7]) by ma1-aaemail-dr-lapp03.apple.com with ESMTP id 2xm1w1agmr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Wed, 22 Jan 2020 16:18:12 -0800
Received: from nwk-mmpp-sz10.apple.com (nwk-mmpp-sz10.apple.com [17.128.115.122]) by ma1-mtap-s03.corp.apple.com (Oracle Communications Messaging Server 8.0.2.4.20190507 64bit (built May 7 2019)) with ESMTPS id <0Q4J00EOWA69H920@ma1-mtap-s03.corp.apple.com>; Wed, 22 Jan 2020 16:18:12 -0800 (PST)
Received: from process_milters-daemon.nwk-mmpp-sz10.apple.com by nwk-mmpp-sz10.apple.com (Oracle Communications Messaging Server 8.0.2.4.20190507 64bit (built May 7 2019)) id <0Q4J0090099C6I00@nwk-mmpp-sz10.apple.com>; Wed, 22 Jan 2020 16:18:11 -0800 (PST)
X-Va-A:
X-Va-T-CD: f3fe944e6c679f2c3d972f4bd0691c8b
X-Va-E-CD: d60a7396dd592520d25d54063e76f07d
X-Va-R-CD: bd63ceb45b8fd24fe4c278576743557a
X-Va-CD: 0
X-Va-ID: 453e6f0f-fd75-475a-8ebc-4596b6cdfe6e
X-V-A:
X-V-T-CD: f3fe944e6c679f2c3d972f4bd0691c8b
X-V-E-CD: d60a7396dd592520d25d54063e76f07d
X-V-R-CD: bd63ceb45b8fd24fe4c278576743557a
X-V-CD: 0
X-V-ID: ea6e9353-cbab-4dbc-b1e1-73fdea1317b2
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2020-01-22_08:,, signatures=0
Received: from [17.230.170.238] by nwk-mmpp-sz10.apple.com (Oracle Communications Messaging Server 8.0.2.4.20190507 64bit (built May 7 2019)) with ESMTPSA id <0Q4J00AIXA6AGN00@nwk-mmpp-sz10.apple.com>; Wed, 22 Jan 2020 16:18:10 -0800 (PST)
Sender: tpauly@apple.com
From: Tommy Pauly <tpauly@apple.com>
Message-id: <B873983A-1327-4388-8B9E-BEC8D2008450@apple.com>
Content-type: multipart/alternative; boundary="Apple-Mail=_0D50D5D0-3D1B-4A3A-9E91-CB7AAE3AB6EF"
MIME-version: 1.0 (Mac OS X Mail 13.0 \(3594.4.17\))
Date: Wed, 22 Jan 2020 16:18:07 -0800
In-reply-to: <4b2b529f-9e67-b6d0-1a9c-b6ad5cd96f01@nostrum.com>
Cc: The IESG <iesg@ietf.org>, ek@loon.com, draft-ietf-intarea-provisioning-domains@ietf.org, int-area@ietf.org, intarea-chairs@ietf.org
To: Adam Roach <adam@nostrum.com>
References: <157967080772.28909.16443816599872682093.idtracker@ietfa.amsl.com> <6AFB6A09-59BF-411D-816F-914BAAF86A9B@apple.com> <a1daf959-3331-e86d-2734-1f63a98d7625@nostrum.com> <BF4953C0-2502-4E08-B8B3-B55D04475416@apple.com> <3c3fb029-be06-02a2-1ac2-d23a3183d09a@nostrum.com> <7BBB92DD-7C0D-4A30-AE5D-3DB6A8424B9A@apple.com> <4b2b529f-9e67-b6d0-1a9c-b6ad5cd96f01@nostrum.com>
X-Mailer: Apple Mail (2.3594.4.17)
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2020-01-22_08:, , signatures=0
Archived-At: <https://mailarchive.ietf.org/arch/msg/int-area/2-J-d2H8iYVJQ-LY_3_OsJyBZkA>
Subject: Re: [Int-area] Adam Roach's Discuss on draft-ietf-intarea-provisioning-domains-10: (with DISCUSS and COMMENT)
X-BeenThere: int-area@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: IETF Internet Area Mailing List <int-area.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/int-area>, <mailto:int-area-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/int-area/>
List-Post: <mailto:int-area@ietf.org>
List-Help: <mailto:int-area-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/int-area>, <mailto:int-area-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 23 Jan 2020 00:18:17 -0000


> On Jan 22, 2020, at 4:08 PM, Adam Roach <adam@nostrum.com> wrote:
> 
> Thanks again for the quick turn-around on this.
> 
> Using your proposed 2**(Delay + 10) seems to strike an okay balance, if I'm understanding the situation correctly. Double-check my thinking here: the scope of RA reach from an attacker will be available only on a single local link, which deployments typically limit to on the order of 500 clients or so. If all 500 are triggered at the same time and smooth out their requests over a one-second window, we're looking at a 500 TPS load on a web server. That's about 25% the capacity of a relatively low-end web server (e.g., Apache running on an Atom 1.66), which seems small enough to avoid major issues.

Yes, that sounds right to me. That limits the single burst size in response to a given RA being sent. Each of these hosts wouldn't request again for that PvD ID, even if RAs keep coming telling the host to update, for another ten seconds after that. And, the limit for the number of different hostnames (for a case in which a wildcard host exists, which presumably also has a higher capacity) is 5 different names in the ten second period, so that limits at 2500 TPS across all fetches to any server from a given network.
> 
> So, unless one of my assumptions above is wrong, I think your proposal below is a good solution to the issue. I'll clear my DISCUSS when a new version of the draft comes out (I would propose that you wait for instructions from your AD about when to do so).

Thanks! Yes, I'll wait for the go-ahead from Suresh. I appreciate your helping to work through these important details!

Best,
Tommy
> 
> /a
> 
> On 1/22/20 17:51, Tommy Pauly wrote:
>> Hi Adam,
>> 
>> Thanks for the feedback! The updated paragraph in the retrieval section, to indicate a maximum failure count per attachment, is:
>> 
>> If the request for PvD Additional Information fails due to a TLS error,
>> an HTTP error, or because the retrieved file does not contain valid PvD JSON,
>> hosts MUST close any connection used to fetch the PvD Additional Information,
>> and MUST NOT request the information for that PvD ID again for the duration
>> of the local network attachment. If a host detects 10 or more such failures
>> to fetch PvD Additional Information, the local network is assumed to be
>> misconfigured or under attack, and the host MUST NOT make any further
>> requests for PvD Additional Information, belonging to any PvD ID, for
>> the duration of the local network attachment. For more discussion, see {{security}}.
>> 
>> I've also expanded the security considerations DoS section as follows:
>> 
>> An attacker generating RAs on a local network can use the H-flag and the PvD ID
>> to cause hosts on the network to make requests for PvD Additional Information
>> from servers. This can become a denial-of-service attack, in which an attacker
>> can amplify its attack by triggering TLS connections to arbitrary servers in response
>> to sending UDP packets containing RA messages. To mitigate this attack, hosts
>> MUST:
>> 
>> - limit the rate at which they fetch a particular PvD's Additional Information;
>> - limit the rate at which they fetch any PvD Additional Information on a given local
>> network;
>> - stop making requests for a PvD ID that does not respond with valid JSON;
>> - stop making requests for all PvD IDs once a certain number of failures is reached
>> on a particular network.
>> 
>> Details are provided in {{retr}}. This attack can be targeted at generic web servers,
>> in which case the host behavior of stopping requesting for any server that doesn't
>> behave like a PvD Additional Information server is critical. Limiting requests for
>> a specific PvD ID might not be sufficient if the attacker changes the PvD ID values
>> quickly, so hosts also need to stop requesting if they detect consistent failure when
>> on a network that is under attack. For cases in which an attacker is pointing hosts at
>> a valid PvD Additional Information server (but one that is not actually associated
>> with the local network), the server SHOULD reject any requests that do not originate
>> from the expected IPv6 prefix as described in {{serverop}}.
>> 
>> For the delay calculation, you make a good point that the larger values get pretty unnecessarily large! I'm a bit concerned about making the minimum fetch range be ~4 seconds, as that could end up being user visible for some valid scenarios. How about making the formula "2**(10 + Delay)":
>> 
>> The target time for the delay is calculated
>> as a random time between zero and 2**(10 + Delay) milliseconds,
>> where 'Delay' corresponds to the 4-bit unsigned integer in
>> the last received PvD Option.
>> 
>> This limits it to 1 second as what the RA can request for fastest frequency bound. This isn't incredibly fast, and with the overall limits for how many requests can be made by a client (which provide the larger portion of the DoS prevention, I'd argue), I think this strikes a good balance between usability and precaution. Thoughts?
>> 
>> I've updated the GitHub text for anyone wanting to see the full flow: https://github.com/IPv6-mPvD/mpvd-ietf-drafts/pull/25 <https://github.com/IPv6-mPvD/mpvd-ietf-drafts/pull/25>
>> 
>> Thanks,
>> Tommy
>> 
>>> On Jan 22, 2020, at 2:58 PM, Adam Roach <adam@nostrum.com <mailto:adam@nostrum.com>> wrote:
>>> 
>>> Thanks for the explanation and the further proposed mitigation.
>>> 
>>> Allowing the RA to specify an arbitrarily small "Delay" parameter seems to still allow for a pretty big burst of traffic. If I read the proposed interpretation of the "Delay" bits correctly (2**(Delay * 2)), the current behavior is specified to allow a delay upper bound selected from one of the following (approximate) values:
>>> 
>>> 1 ms
>>> 4 ms
>>> 16 ms
>>> 64 ms
>>> 256 ms
>>> 1 second
>>> 4 seconds
>>> 16 seconds
>>> 1 minute
>>> 4 minutes
>>> 17 minutes
>>> 70 minutes
>>> 4 hours, 40 minutes
>>> 18 hours 38 minutes
>>> 3 days, 3 hours
>>> 1 week, 5 days
>>> 
>>> That's a pretty breathtaking scope, and it's hard to imagine that the first six or so are strictly needed, while all six are in a range that might overload a DDoS target. The final several seem a bit questionable as well, given normal operational timelines for network attachment. If the formula were revised to, e.g., "2**(Delay + 12)" instead of the current formula, you would have an enforced lower bound of roughly four seconds (which should be enough to blunt most DDoS attacks), and an upper bound of roughly 37 hours (which still seems excessive, although not quite as much as the previous upper bound).
>>> 
>>> Assuming the additional mitigation you propose below (10 maximum failures per attachment) as well as some means of achieving a lower-bound for "Delay" on the order of multiple seconds, I think I'm good clearing when a new version comes out.
>>> 
>>> Thanks for your work in thinking through practical solutions to this issue.
>>> 
>>> /a
>>> 
>> 
>