Re: [tsvwg] L4S dual-queue re-ordering and VPNs

[Sebastian Moeller <[e-mail]>]

Hi Bob,

I a still considering the VPN-tunnel from home-network scenario below ([SM]), since I cnsider that to be an important use case where replay-windows seem to be actively used today, by quite a lot of users.

> On May 8, 2021, at 17:05, Bob Briscoe <[e-mail]> wrote:
> 
> Jonathan,
> 
> As just pointed out (my email crossed yours), existing established Diffserv technology is also prone to this problem, and it's lower hanging fruit for an attacker, because the delay-delta is typically larger.


[SM] This is a good point, see https://datatracker.ietf.org/doc/html/rfc4301#page-50:

"If different classes of traffic (distinguished by Differentiated
   Services Code Point (DSCP) bits [NiBlBaBL98], [Gro02]) are sent on
   the same SA, and if the receiver is employing the optional
   anti-replay feature available in both AH and ESP, this could result
   in inappropriate discarding of lower priority packets due to the
   windowing mechanism used by this feature.  Therefore, a sender SHOULD
   put traffic of different classes, but with the same selector values,
   on different SAs to support Quality of Service (QoS) appropriately.
   To permit this, the IPsec implementation MUST permit establishment
   and maintenance of multiple SAs between a given sender and receiver,
   with the same selectors.  Distribution of traffic among these
   parallel SAs to support QoS is locally determined by the sender and
   is not negotiated by IKE.  The receiver MUST process the packets from
   the different SAs without prejudice.  These requirements apply to
   both transport and tunnel mode SAs.  In the case of tunnel mode SAs,
   the DSCP values in question appear in the inner IP header.  In
   transport mode, the DSCP value might change en route, but this should
   not cause problems with respect to IPsec processing since the value
   is not employed for SA selection and MUST NOT be checked as part of
   SA/packet validation.  However, if significant re-ordering of packets
   occurs in an SA, e.g., as a result of changes to DSCP values en
   route, this may trigger packet discarding by a receiver due to
   application of the anti-replay mechanism."

In short, this potential for re-ordering was recognized and the recommendation seems to be to not mix different DSCPs in the same "tunnel/SA"...
Additionally rfc4301 also seems to allow not exposing inner DSCP codepoints on encapsulation at all. 
Also note how the recommendation is NOT just increase the replay-window size until the issue goes away.

So for ipsec the RFCs seem to already tackle the issue of exposed DSCPs "accidentally" resulting in packet reordering through differential prioritization.


Which brings us back to the new re-ordering cause that L4S introduces... 
Extrapolating from rfc4301 there are multiple options:

A) set up two SAs, one for ECT(1) and one for ECT(0) and steer packets into the appropriate one 
	-> this is incomplete since it hits the old CE ambiguity issue that dctcp/L4S introduce. We could fix this by giving L4S a proper 1bit ID nstead of the currently proposed 0.5 bit approach.

B) re-mark the ECN bits to 00 on encapsulation
	-> this foregoes all ECN benefits, but should not cause reordering.

C) heuristically increase the replay-window until the error messages subside...
	-> this is tricky, because the amount of re-ordering an L4S AQM will introduce depends on its local latency targets for the C (and less for the L queue) which can not be unambiguously discovered from endpoints. Currently the recommended value seems 25 ms, but a) that recommendation is not based on numeric analysis, and b) it is one of an L4S AQMs "free" configuration parameters, so it can not be relied to always be 25ms*.
And it invites the question of the cost benefit ratio of large replay windows...



Best Regards
	Sebastian


*) This is essentially the same issue TCP Pragues RTT-debias code has, to undo the DualQ class unfairness, TCP Prague should be configured such that the cwin growing function behaves as if the experienced RTT was true RTT + (C-queue latency target - L-queue latency target), but only the first of these three values can be empirically be deduced... 



> 
> The solution to all these interactions between a poorly configured replay protection window and queues with different delay differences is simply to configure a replay window appropriate to the packet rate of the VPN tunnel, taking account of expected delay differences between queues.
> 
> Whatever, VPNs rarely provide open access for traffic from unauthorized sources to enter the VPN.
> 
> 
> 
> Bob
> 
> On 08/05/2021 15:22, Jonathan Morton wrote:
>>> On 8 May, 2021, at 9:45 am, Pete Heist <[e-mail]> wrote:
>>> 
>>> I noticed that this issue seems to affect tunnels with replay window
>>> sizes of 32 and 64 packets regardless of the bottleneck bandwidth,
>>> likely because the peak C sojourn times can also increase as the
>>> bandwidth decreases. IMO, this seems like a safety concern from the
>>> standpoint that the deployment of DualPI2 can cause harm to
>>> conventional traffic, in IPsec tunnels using common defaults in
>>> particular, beyond that which is caused by DualPI2 itself.
>> It turns out that there's also a Denial of Service attack enabled by this phenomenon - one which does not rely on any legitimate L4S endpoints being deployed or used.  It is a property of the passive deployment of DualQ.
>> 
>> We assume a VPN tunnel upon which Sequence Verification is in use (RFC-4303 §3.4.3), over a path that includes a DualQ-Coupled qdisc (draft-ietf-tsvwg-aqm-dualq-coupled), abbreviated DualQ.  We further assume that an attacker is able to send traffic with chosen ECN codepoints into the input end of this tunnel.  We outline two attack scenarios, the second of which further assumes that the attacker sends traffic directly over the DualQ path, not via the tunnel, which requires that the attacker knows more information than the first attack scenario.  The threat is the ability to deny any useful traffic being conveyed by the VPN tunnel, at a significantly lower cost than a conventional DDoS.
>> 
>> In the first attack scenario, the attacker generates two distinct streams of traffic, named Lambda and Omega, both of which are directly into the input end of the tunnel.  The Lambda stream is bulk data with an ECN codepoint that filters into the C queue - either Not-ECT or ECT(0).  The Omega stream is a low-volume stream of small packets with an ECN codepoint that filters into the L queue - either ECT(1) or CE.  It is not necessary for either stream to be accepted by a host at the receiving end, only by the endpoints of the VPN itself.
>> 
>> The purpose of the Lambda stream is to keep the C queue of the DualQ full, and it is sufficient to send MTU-sized packets in only slightly greater volume than the capacity of the link.  We will assume for analysis that DualQ maintains the C queue at its AQM setpoint, which is 25ms by default.
>> 
>> The purpose of the Omega stream is to bypass the Lambda stream by taking the "fast lane" through the DualQ instance, and thereby control the leading edge of the Sequence Verification window of the VPN receiver.  If the leading edge of the window can be kept so far ahead of the traffic arriving via the C queue that the latter always falls behind the trailing edge of the window, then the amount of useful traffic conveyed by the VPN in the same direction falls to nil - a successful Denial of Service.
>> 
>> The required Sequence Verification window to deny this attack scenario full effectiveness is the VPN packet throughput of the C queue multiplied by the difference in queue sojourn times between the C and L queues, plus the interval between packets delivered in the Omega stream.  Since the default SV window in some popular VPNs is 32 or 64, this attack can be fairly easy to implement.
>> 
>> Another mitigation that the VPN could apply is to limit the throughput of the VPN, such that it cannot by itself be used to saturate the DualQ link and thus keep the C queue full.  We address this with the second attack scenario, which bypasses such a mitigation.
>> 
>> In the second attack scenario, the Lambda stream is sent directly to the DualQ link rather than through the VPN.  The Omega stream is still sent through the VPN.  The properties of the two streams are mostly similar otherwise to those of the first attack scenario, but the volume of the Omega stream must be increased.
>> 
>> The second attack scenario will succeed if the number of Omega packets delivered during any interval of time equal to the difference between the C and L queue sojourn times is greater than the Sequence Verification window.  This is also easy to achieve if the Omega packets are small and the SV window is at the usual defaults of 32 or 64.
>> 
>> The cause of this attack enablement is that DualQ can reorder packets from the same flow under fairly precise control of an attacker.  A straight FIFO, or a flow-aware qdisc that treats a VPN as a single flow (such as RFC-8290 fq_codel or Cake), delivers packets from any given flow in-order, and is thus immune from this attack.  Adding DualQ-like semantics to any of these carries the risk of enabling the attack as described above.
>> 
>>  - Jonathan Morton
>> 
> 
> -- 
> ________________________________________________________________
> Bob Briscoe                               http://bobbriscoe.net/
>