Re: [Pals] [Int-area] L2TP sequencing: Commonly disabled for IP data? Or always?

[Giles Heron <[e-mail]>]

Hi Bob,

> On 1 Jul 2021, at 18:16, Bob Briscoe <[e-mail]> wrote:
> 
> Giles,
> 
> On 29/06/2021 17:22, Giles Heron wrote:
>> Re-sending trimmed to 40k
>> 
>>> On 29 Jun 2021, at 17:16, Giles Heron <[e-mail] <mailto:[e-mail]>> wrote:
>>> 
>>> Hi Bob,
>>> 
>>>> On 29 Jun 2021, at 16:55, Bob Briscoe <[e-mail] <mailto:[e-mail]>> wrote:
>>>> 
>>>> Giles,
>>>> 
>>>> On 29/06/2021 12:41, Giles Heron wrote:
>>>>> Hi Bob,
>>>>> 
>>>>>> On 28 Jun 2021, at 00:23, Bob Briscoe <[e-mail] <mailto:[e-mail]>> wrote:
>>>>>> 
>>>>>> Giles, Mark,
>>>>>> 
>>>>>> On 10/06/2021 13:22, Giles Heron wrote:
>>>>>>> So AFAIK SP networks don’t generally reorder packets in the steady state, but of course reordering can occur under rerouting.
>>>>>> 
>>>>>> [BB] The cases I'm concerned about are where the operator
>>>>>> * deliberately reorders packets using a multi-queue scheduler at a node contrived to act as the bottleneck (the BNG)
>>>>>> * AND the node is within an L2TPv2/3 tunnel 
>>>>>> * AND the tunnel needs sequencing at the egress for some other reason. 
>>>>> 
>>>>> If that’s the case then the node almost certainly won’t re-order packets within the tunnel, even if it is deliberately reordering packets between clients.  As far as the node is concerned the L2TPv3 tunnel is one flow within one subscriber’s traffic.   BNG reordering is generally between subscribers (e.g. where subscribers get serviced round-robin to ensure fairness), or potentially within classes of service per subscriber (e.g. strict priority for on-net voice and IPTV over Internet traffic).
>>>> 
>>>> [BB] Not so. There are certainly cases where packets /are/ reordered within a per-customer tunnel. For instance, in the BT wholesale case, the BNG uses the BBF framework to schedule a set of QoS queues for each customer CVLAN. I know this intimately, because it was what I had been asked to simplify when we came up with L4S in the first place. Indeed, I was given permission to include a picture and description of it in this public report we did for the EU-funded project: https://riteproject.files.wordpress.com/2015/12/rite-deliverable-3-3-public1.pdf#12 <https://riteproject.files.wordpress.com/2015/12/rite-deliverable-3-3-public1.pdf#12>
>>> 
>>> Yup - what I said.   BNGs may reorder classes of service per sub.  But one L2TP tunnel where the endpoints are either side of the BNG ought to map to a single CoS, right?  And if it does then it won’t be reordered.
> 
> [BB] The question of whether there's an L2TP tunnel per CoS is part of the question. Wouldn't the only reason {Note 1} for one tunnel per CoS be if IP data was being sequenced. Instead, wouldn't the path of least resistance be to not sequence, and to not have to separate into one tunnel per CoS?

Where the non-broadband end of the tunnel is off-net it’s pretty unlikely that the ISP will honour any DSCP markings.

> Whatever, as originally explained as part of the question, L4S packets are identified by the IP-ECN field, not the DSCP. 

Yup.  But they won’t resequence in the BNG if the BNG only looks at DSCP and if alll packets have the same DSCP.

But yes, reading your original question again, yes - if operators want to do L4S etc. *within* an L2TP tunnel then clearly they need to disable sequencing.  But am not quite sure why they would enable it *within* a tunnel (given that the function of a tunnel is generally to carry the packets inside the tunnel without inspecting them along the tunnel).

> Nonetheless, if there is no IP sequencing, the way that L4S packets are identified is irrelevant to this question. So we ought to stay focused on the question of whether anyone does IP sequencing

Exactly

> 
> {Note 1}: As long as the CoS identifier is either propagated to the outer, or the node acting on the CoS is accessing an inner header. Otherwise, being able to identify the CoS by tunnel ID would be the only other reason for one tunnel per CoS.
> 
>>> 
>>>> Also, there are probably many cases of accidental reordering too, e.g. due to channel bonding or multipath links that are left out of order rather than resequenced, or reroutes, including those at L7, when the traffic source moves to a CDN after a flow starts, etc. etc.
>>> 
>>> Channel bonding and multipath links are generally set up very carefully NOT to reorder packets within one flow.
> 
> [BB] I believe that's a rather nostalgic perspective. For instance, from 2014, see
> https://tools.ietf.org/agenda/91/slides/slides-91-tcpm-5.pdf <https://tools.ietf.org/agenda/91/slides/slides-91-tcpm-5.pdf>

So that’s specific to 3G / 4G.   So not really relevant to BNGs.

And it’d be interesting to see whether that reordering is the result of running IP over a lower-layer network that does error correction in an IP-unaware way, or because of cell-site handoffs.

At any rate I think it's highly unlikely this is due to channel bonding / LACP or to multipath / ECMP, but I do quite like being called “nostalgic” ;)

Take care Bob - and let’s close the debate.

Giles


> FYI, the RACK RFC [RFC8985] was published in Feb'21, which moves stds track TCP to detecting reordering in time (e.g. RTT/8), which scales with flow rate. Previously TCP's reordering threshold was measured as a number of ACKs, e.g. 3 duplicate ACKs, which doesn't scale with rate. Soon after the TCPM WG adopted RACK in 2016, the major OS developers had implemented it. Also, QUIC detects reordering using an equivalent algorithm to RACK. So although there will be plenty of legacy TCP stacks still out there for some time, TCP's 3 Duplicate ACK rule is already starting to become history. 


>>> 
>>> I covered rerouting in my original reply to you.
>>> 
>>> And I think we can safely ignore CDNs in discussion of L2TP ;)
>>> 
>>>> Whatever, the question is the converse: where packets are /not/ deliberately re-ordered by the network operator today, might these operators be sequencing IP data emerging from an L2TP tunnel? If so, they would have to disable sequencing if they wanted to introduce deliberate reordering (to transition to L4S). But (based partly on the insights you've given already) I doubt there is much if any sequencing going on of IP data over tunnels.
> 
> Cheers
> 
> 
> 
> Bob
> 
>>> 
>>> Agreed.
>>> 
>>> Giles
>>> 
>>>>> 
>>>>>> 
>>>>>> In many cases, such a scheduler would be located prior to the tunnel ingress, so not a concern. I believe the DOCSIS rPHY case below falls into that category (both downstream and up).
>>>>> 
>>>>> Agreed
>>>>> 
>>>>>> 
>>>>>> In contrast, where a BNG sits /within/ the span of an L2TP tunnel, I think it will often (or at least sometimes?) have been constructed as the bottleneck. Any operator having designed such a QoS arrangement would not want to support sequencing...
>>>>> 
>>>>> Yes, BNGs are often planned to be the bottleneck.  But as I say that’s about fairness between subscribers and potentially prioritising different traffic for each subscriber.
>>>>> 
>>>>>>> As noted by Derek I’m guessing reordering is an issue for L2TPv2 if stateful PPP compression schemes are in play (which I suspect is unlikely to be the case given we usually have abundant bandwidth in the aggregation network, and given that compression can occur at other layers).  Though given that BNGs inherently keep state per subscriber maybe the NPU scaling issues that Stewart mentioned are less of an issue in that use case than for MPLS PEs doing PWE?
>>>>>> 
>>>>>> My concern was that 'keep it simple' operators that are using L2TPv2/3 and had not previously bothered with the complexities of QoS might want to support L4S, because it has the potential to cut out queue delay for /all/ traffic. Altho' L4S is eventually for all traffic, it still requires two queues at the bottleneck for transition - one for L4S and one for not-yet-L4S ('Classic') traffic flows, and therefore introduces reordering at the aggregate level...
>>>>> 
>>>>> Again I presume any single tunnel being passed *through* a BNG would either be L4S or not-yet-L4S, and hence not subject to reordering?
>>>> 
>>>> [BB] Not so, I'm afraid. L4S is enabled on a per-host basis, and during the transition, some traffic in the IP aggregate to (or from) a customer site will be L4S, and some Classic (not-yet-L4S).
>>>> 
>>>> It's this reordering scenario that begged my original question - whether sequencing might be used anywhere for IP data over a tunnel. 
>>>> 
>>>> Cheers
>>>> 
>>>> 
>>>> 
>>>> Bob
>>>> 
>>>>> 
>>>>> Giles
>>>>> 
>>>>>> 
>>>>>> From the replies so far, even if such 'keep it simple' operators were using compression, I can't see any reason why having to turn off compression and sequencing (in order to support L4S) would be a significant problem nowadays.
>>>>>> 
>>>>>> So, in conclusion, I don't think we even need to raise any concerns about L2TP sequencing in the L4S specs.
>>>>>> 
>>>>>> If anyone here thinks otherwise, pls speak now.
>>>>>> 
>>>>>> Thank you everyone who has contributed to this discussion.
>>>>>> 
>>>>>> Cheers
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Bob
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> 
>>>>>>> From a quick look at the DOCSIS rPHY specs it seems they do require support for L2TPv3 sequence numbers.  Of course in that case the payload is the DOCSIS MAC rather than IP (even though, of course, most DOCSIS frames ultimately carry an IP payload).
>>>>>>> 
>>>>>>> Giles
>>>>>>> 
>>>>>>>> On 10 Jun 2021, at 12:49, Andrew G. Malis <[e-mail] <mailto:[e-mail]>> wrote:
>>>>>>>> 
>>>>>>>> (resending with cc: list trimmed to pass the too many recipients filter)
>>>>>>>> Mark,
>>>>>>>> 
>>>>>>>> The original question was, how many (if any) of these L2TPv2 and v3 use cases use sequencing, especially when carrying IP?
>>>>>>>> 
>>>>>>>> Cheers,
>>>>>>>> Andy
>>>>>>>> 
>>>>>>>> On Wed, Jun 9, 2021 at 6:32 PM Mark Townsley <[e-mail] <mailto:[e-mail]>> wrote:
>>>>>>>> Hi folks,
>>>>>>>> 
>>>>>>>> In addition to the DSL arena, L2TP is still in use with host-based VPN clients as it is embedded in Apple, Android, and Windows based operating systems (new and old). Despite most VPN solutions preferring their own client software that must be installed on hosts to operate, there are still admins that appreciate not having to ask their employees to download an app for the VPN to work - in which case PPTP and L2TP with transport-mode IPsec are your most widely available options. 
>>>>>>>> 
>>>>>>>> Regarding L2TPv3 replacing L2TP: L2TPv2 (RFC 2661) was PPP only. L2TPv3 generalized L2TP to support other L2 (including MPLS, but I don’t want to argue what layer MPLS operates within here). There was never a strong push to replace L2TPv2 used in DSL, Dialup and host VPN software with L2TPv3 (there was one use case for an important L2TP operator that wanted to carry PPPoE over L2TPv3 in DSL, but that was overcome by RFC3817 which achieved the same goal without altering the dataplane). Ironically, I would expect to see very little PPP over L2TPv3 in the wild, though obviously it is possible.
>>>>>>>> 
>>>>>>>> In the cable broadband world, the DOCSIS DEPI “Remote PHY” specification (similar I suppose to the split BNG spec Joel is referring to) standardized on L2TPv3 and is in active use.
>>>>>>>> 
>>>>>>>> I also know of at least one vendor that uses Ethernet over L2TPv3 for some wifi backhaul use cases. 
>>>>>>>> 
>>>>>>>> There could be more, this is just what I am personally aware of off the top of my head. Even I am surprised to see how much L2TP is still out there once you start really looking around ;-)
>>>>>>>> 
>>>>>>>> Best Regards,
>>>>>>>> 
>>>>>>>> - Mark
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>> 
>> 
> 
> -- 
> ________________________________________________________________
> Bob Briscoe                               http://bobbriscoe.net/ <http://bobbriscoe.net/>