Re: [tsvwg] L4S dual-queue re-ordering and VPNs

[Bob Briscoe <[e-mail]>]

Jonathan,

On 08/05/2021 19:32, Jonathan Morton wrote:
>>>> On 8 May, 2021, at 6:05 pm, Bob Briscoe <[e-mail]> wrote:
>>>>
>>>> As just pointed out (my email crossed yours), existing established Diffserv technology is also prone to this problem, and it's lower hanging fruit for an attacker, because the delay-delta is typically larger.
>>> Perhaps - but usable Diffserv deployments at the last-mile are not very common.

[BB] See email to Sebastian for the global market in business services 
that is built on Diffserv deployments in the last mile. But you are 
right that my point about authorization to use VPNs applies to last-mile 
Diffserv as well - Diffserv's rarely open access. That's why I pointed 
to WMM.

>> [BB] WMM (Wifi MultiMedia).
> That's not the last mile - that's within customer premises.

[BB] Is that relevant? I mean, in the context of this issue (I know it's 
not literally what you said).

[BB] All the reasons you give for WMM not being relevant below seem to 
be clutching at straws.

[BB] I found the same text that Sebastian found in the IPsec 
architecture ([RFC4301] section 4.1) saying that different security 
associations 'SHOULD' be used for each Diffserv codepoint. That would 
help rule out Diffserv's vulnerability, but I'm not sure whether IPsec 
implementations follow that advice - anyone know?

> I would also question how often a DSCP would survive on the outer header of a VPN packet long enough to influence this mechanism.

[BB]
* Upstream: Diffserv (typically an abstraction of Diffserv) is 
invariably the interface that OSs present to applications in order to 
use WMM.
* Downstream: ISPs typically supply an access point, so they will often 
hook into WMM from their QoS regime using Diffserv as the go-between.

In the past I'm sure Greg White has given the results of his informal 
study of just how common different DSCPs are on frames snooped from busy 
WiFi environments (downstream and upstream).

> But I would also note the increase in the use of fq_codel in wifi AP stacks, which would tend to reduce any differential delay to a greater degree than the C queue in DualQ does.  Those, taken together, might be significant reasons why problems have not been noticed up to now.

[BB] One would need to imagine that nearly the whole WMM world has 
nearly always used FQ-CoDel for this to be a reason.

>
> For the particular use case I mentioned, a player connecting over wifi would be ridiculed due to the inconsistent latency that wifi offers, especially in congested radio environments.

[BB] It's the attack traffic that needs to use the lower delay WMM 
access category, not (necessarily) the user.

>
>>> L4S proposes to put a DualQ instance in front of every cable internet subscriber, making it much easier for an attacker to predict how he needs to conduct his attack than the equivalent with Diffserv.
> ...
>
>> My point was that, you seem to want this to be a criticism of L4S, rather than of poorly configured VPNs.
> As often seems to happen, you seem to want to deflect any possible criticism of L4S onto some other piece of technology that it might happen to interact poorly with, so that you don't need to change L4S, only all those other things need to change or just plain go away.

[BB] If L4S is the problem, I'll say. But I'm trying to establish 
whether this problem already exists, and whether solutions to it have 
been proposed.

The approach a researcher takes when they think they have found a new 
attack is to generalize it, and then see if it's already been proposed 
and studied. So far I haven't found any research that uses a 
pre-existing delay-delta on the path like your attack. The following two 
papers are about related attacks exploiting the IPsec replay window, but 
the attacker needs to be on-path:
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.183.2766&rep=rep1&type=pdf
     https://arxiv.org/pdf/0910.3511.pdf

> You don't like FQ, or Codel, or fq_codel, or the continuing deployment of RFC-3168 compliant middleboxes in general, and you *certainly* don't like me personally.  I've got used to that.
>
> The operative phrase here is that you're entitled to your own opinions, but not your own facts.

[BB] A mailing list is for establishing what the facts are.

I should know some facts in this space, because I've been involved in 
the access network QoS business of operators for decades (and I helped 
set up a start-up before operators were doing it). But I also know that 
there's always a lot I don't know. So the normal etiquette is to be 
circumspect about the breadth of your own knowledge, because none of us 
know how much we don't know.

>
>> Surely you are not saying the IETF should halt any advances that reduce delay, like Diffserv and L4S, just because the replay windows of VPNs are poorly configured today?
> Configuring the replay window size is a game of compromises.  If it's too small, then a variety of effects can result in legitimate packets falling outside the window.  If it's too large, then there is presumably a greater risk of the replay attacks that it's designed to protect against being inadvertently permitted.  I don't know enough about the details of that to judge how serious that particular risk is - but the fact that many VPNs default to values near the minimum recommended in RFC-4303 is probably not a gross error of judgement.

[BB] I also didn't know much about the replay window tradeoff until 
yesterday, but I've been reading. It seems to be a lot to do with 
implementation efficiency. The number 64 for the replay window seems to 
come from the variable you would use to hold the bit map of the replay 
window on a 64-bit machine.

 From IPSec [RFC4302] Appx B.1.:

    (No requirement is established for minimum or
    recommended sizes for this window, beyond the 32- and 64-packet
    values already established for 32-bit sequence number windows.
    However, it is suggested that an implementer scale these values
    consistent with the interface speed supported by an implementation
    that makes use of the ESN option.


I found papers proposing techniques to efficiently use larger bit-maps, 
including RFC6479 (the IPR declaration offers non-assertion with 
reciprocity).

The papers I cited earlier attempted to reason about a recommended 
window size. The unstated rule for preventing replay attacks seems to be 
"as small as possible without creating vulnerabilities to other 
attacks". The paper proposes a replay window needs to be "as large as 
the e2e bandwidth-delay product at the rate supported by the VPN over a 
max delay path". But I don't see any justification for that.

>
> I do note that RFC-4303 significantly post-dates most of the Diffserv specifications, so it notionally had the opportunity to consider the implications.

[BB] That assumes the problem was known.
The 1998 version of IPsec  had the same replay prevention window 
approach as the current (2005) version in RFC4301. But until the 2009 
papers linked above (and your proposed attack), IPsec's anti-replay 
protocol seems to have only been considered useful for /preventing/ DoS 
attacks.


> One way that VPNs could sidestep this particular problem in a Diffserv context is to fix a single DSCP on all outer headers, ensuring that they'll go through the same queue - and at least some of them actually do this.

[BB] "You can't do what you want to do" isn't really a solution. When 
the DSCP is actually being used for different levels of QoS, one needs a 
Diffserv tunnel's uniform mode [RFC2983], i.e. propagation and from outer.

>
> But they can't really do the equivalent of this to the ECN field in an L4S context, because that would trigger the CE ambiguity problem without also sending the L4S traffic through the L queue.  Not unless they put Not-ECT on the outer header unconditionally - but that *would* be a step backwards.
>
> I hope you can see the difference here.
>
>>> The Wireguard implementation in Linux has a hard ceiling at 8192 packets - is that enough?
>> [BB] A hard ceiling in /soft/ware?
> It's a value that you can't exceed merely by userspace configuration, only by patching the kernel.  Hence there will be significant inertia in raising it, if it needs to be raised.  Hence my question - *is* it large enough?  And following on from that, what are the downsides of choosing such a large value?

[BB] Yes, those questions arose for me too. See sketchy summary of the 
only paper I can find about this, above.

>
>>> If a player uses a VPN to connect to the game, then the remote end of the VPN is the address given to the other players.  A DoS attack that could be launched into such a VPN endpoint would be highly attractive to a griefer.
>> [BB] So the player has a strong incentive to configure the replay window of her VPN properly.
> For normal operation, the default replay window is big enough.

[BB] Can you justify that statement?

Never mind Diffserv or DualQ. Just the background reordering degree on 
some Internet paths is greater than 64.
     https://www.ietf.org/proceedings/90/slides/slides-90-tcpm-9.pdf
Where this is due to bonded links, even if the reordering degree was 
half a dozen, an attacker could tip it over 64 by sending a stream of 
tiny back-to-back datagrams into the VPN (29B instead of 1500B packets 
could inflate the reordering degree by up to x50). It wouldn't matter if 
the tinygrams were spread across both/all channels; IPsec's replay 
protection would still start deleting genuine packets.

As a feel for the order of magnitude, at only 100Mb/s, 64 injected 
back-to-back min sized datagrams (29B) occupy only 150μs.

> For network paths that deliver packets more-or-less in-order, the default replay window is big enough.  It's only when an attack arises on a vulnerable path - a path *made* vulnerable by the installation of DualQ - that an adjustment becomes necessary.
>
> So where is the cause, exactly?  And where should the cure be applied?

[BB] Still trying to find that answer.

Replay attacks have to be mounted by on-path attackers, and they are 
application-specific but the attacker is blind to what's in the packets. 
Plus e2e security protocols take care about replay attacks themselves 
(salting passwords, etc).
So it seems that it's just useful for a VPN to have /a/ replay window in 
case there might be a replay attack that e2e protocols have not 
protected against.

IPsec replay protection seems like an extra pair of underpants just in case.


Bob



>
>   - Jonathan Morton

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/