Re: [tsvwg] draft-ietf-tsvwg-rfc6040update-shim:SuggestedFragmentation/Reassemblytext

[Markku Kojo <[e-mail]>]

Hi Bob, all,

apologies for the additional delay on this.

Please see inline after Bob's suggestion, my 2 cents on this tricky 
issue tagged [MK].

On Mon, 8 Mar 2021, Bob Briscoe wrote:

> Markku,
> 
> Thx. Unfortunately, this draft is coming up in the mtg this afternoon.
> I take some of the blame -  only re-posting the draft a couple of hours ago, which
> presumably reminded you that you were going to think on this.
> 
> inline tagged [BB]
> 
> On 08/03/2021 13:48, Markku Kojo wrote:
>       Hi Bob,
>
>       this issue and text on it seems acceptable to me.
>
>       However, the other issue with the two contradictory SHOULD's - that I
>       now notice I have never replied to - seems not ok, I think.
> 
> 
> [BB] The question is whether we have to solve this now.
> 
> I have a solution to resolve the contradiction. At the risk of prolonging the
> progress of this draft, I'll say it now. But if we can't resolve this in the next
> couple of days, I think we should go ahead with the two contradictory SHOULDs.
> 
> For the list, here's the two contradictory paras:
>
>    Congestion indications SHOULD be propagated on the basis that an
>    encapsulator or decapsulator SHOULD approximately preserve the
>    proportion of PDUs with congestion indications arriving and leaving.
>
>    The mechanism for propagating congestion indications SHOULD ensure
>    that any incoming congestion indication is propagated immediately,
>    not held awaiting the possibility of further congestion indications
>    to be sufficient to indicate congestion on an outgoing PDU.
> 
> Possible resolution of the contradiction: the "SHOULD approximately preserve the
> proportion" is a rough long term average goal while "SHOULD ensure that incoming
> congestion indication is propagated immediately" is a requirement for after there
> has been some period (TBD) without any marking.
> 
> The big question is where would an implementer set that timescale? It needs to be a
> "typical RTT in the deployment environment" or some such get-out clause. I guess
> this is best left to the implementer.
>

[MK]:

My concerns were and still are mainly for traffic under Standards 
Track congestion control, i.e., majority of the traffic foreseen for a 
long time. So, if not otherwise mentioned, the comments apply for 
handling properly the traffic under Standards Track CC.

The subject line was for the shim draft, but the solution IMO should be 
the same for all cases where "fragments" are decapsulated/reassembled
and my comments therefore mainly address fragmentation & reassembly. i.e., 
when small fragmented packets are under AQM drop and later reassembled.

The suggestion above may work with high fidelity CC traffic like L4S but 
unfortunately not with standard CC traffic. I have doubts about correct 
behaviour with L4S traffic though.

There are two major problems:

1) The suggested approach assumes an AQM that uses propabilistic
    dropping. All AQMs do not employ propabilistic dropping. If
    the decapsulator/node doing reasembly does not know which
    type of AQM marked the PDUs/fragments, it cannot do the right
    decision to not apply the above approach of preserving
    the proportion of PDUs with congestion indications arriving
    and leaving in case of AQM that does not employ propabilistic
    dropping.

2) Majority of the congestion controlled traffic today is
    non-ECT traffic, i.e., traffic under loss-based CC (let's
    but delay-based CC aside now for to keep things simple enough),
    and the above solution does not work for it.

    The guiding principle of the original ECN design was to treat
    ECN traffic eguitably with the loss-based traffic, i.e., not
    to give preference to ECN traffic. If the congestion
    indications marked on small fragments are reduced when
    reassembeled then the ECN traffic is preferred over loss-based
    CC traffic because it is impossible to reduce lost fragments
    but each lost fragment results in loss of the entire packet
    at reassembly and therefore triggers a congestion indication
    without exception.

The suggested solution also would be very hard to configure because RTT 
is not known. The potential RTT range in the Internet is from 
sub-millisecs to over 1 sec. In some specific evironments the typical RTT 
could be known but not for the general case and these docs are targetting 
Standards Track, so the solution should apply for all environments.

In addition, when there are multiple flows sharing a tunnel, the proposal 
would potentially concentrate marks to a smaller number of some flows, 
i.e., ignore/move marks for/from some flows to a smaller number of flows.
This would be extremely undesirable in high load cases, e.g., when several 
flows are in slow start. In such a case it is important to mark packets 
for several flows in order to have fast and strong enough CC reaction.
If the AQM has succesfully spread the marks over a number of flows, but 
this gets supressed to a smaller number of flows the overall congestion 
control reaction at the bottleneck is inappropriately diminished and 
postponed.

After considering this problem, my conclusion is that the only working 
approach I can come up with can be achieved by not applying any 
congestion indication manipulation when reassembling and by taking 
different approach to packet drops with AQMs that employ probabilistic 
dropping. The correct approach for dropping had already been taken by 
the RED design with byte-mode queue measurement and drop (note: with the 
error in calculating the drop propbability corrected).

I'm very well aware of the strong case that RFC 7141 makes for 
packet-mode drop, and it makes this problem even much trickier to solve. 
However, if we concentrate only on the problem of dropping small fragments 
and reassembling them, the byte-mode drop together with reassembly logic 
in RFC 3168 results in the correct outcome.

Why? By giving lower drop probability to small fragments, the byte-mode 
drop ensures that the congestion signal (mark or drop) is given 
(approximately) at the same level of data queued (or: of queuing delay), 
no matter whether the packets are small or large (i.e., on a 
bit-congested bottleneck the operating range of AQM is entered at the 
same time no matter what size the packets in the queue are). Therefore, 
it is ok to use fractional cwnd decrease as Standards Track TCP does.

When the drops/marks at the bottleneck are targetted at small fragments, 
it does not mean that the (TCP) sender operates on small packets, but it 
sends MSS-sized segments and is basically unware of fragmentation. 
Therefore, the sender also increments its cwnd using large MSS-sized 
units and there is no small packet bias in its performance (because 
byte-mode drop at the bottleneck does correct job). Small packets 
(fragments) do not go faster.

The fact that the performance problem with small packets does not 
originate from one reason only but from two main reasons: 1) how drops 
are handled at the bottleneck device and 2) how cwnd is incremented at 
the sender endpoint.

The research that RFC 7141 has used as basis when justifying the choice 
with the drop mode seems not to take this fact properly into account. 
Instead, it tries to solve the small packet propblem entirely at the AQM 
node, which of course is wrong kind of reverse engineering that RFC 7141 
states. However, solving only the problem 1) that an AQM node using 
probabilistic dropping creates by dropping small packets faster at the AQM 
node itself is not reverse engineering. Solving the problem with small 
packet senders, which increment cwnd in smaller units, anywhere except at 
the endpoint is of course not the way to go.

I also do not fully agree with RFC 7141 that giving a lower drop 
probability to smaller packets would notably amplify flooding attacks.
AQMs are not designed to protect against flooding attacks and they 
cannot. There are and need to be other tools for that. Having a higher 
drop probability for smaller packets does not prevent small packet 
attacker from (almost) fully utilizing the bottleneck link capacity (AQMs 
are designed to drop excess packets). Sending unresponsive floods 
will anyway push away almost all competing responsive traffic as the 
responsive traffic reduces its sending rate to less than one packet per 
RTT.

So, unfortunately I'm not able to offer a quick and simple solution to 
this question at hand.

thanks,

/Markku

> Bob
> 
>
>       I'm now occupied for the next few hours, so I'll come back with more
>       detailed reasoning after the tsvwg meeting today.
>
>       Cheers,
>
>       /Markku
>
>       On Mon, 8 Mar 2021, Bob Briscoe wrote:
>
>             Markku, chairs, all,
>
>             Having reached agreement on the text last Dec, I then went
>             and dropped the ball and
>             forgot all about this draft,... and ecn-encap-guidelines.
>
>             == ECN-ENCAP-GUIDELINES ==
>
>             I shall upload a new rev shortly with the following single
>             diff that I noticed
>             during the meeting last Nov, and said I would do at the next
>             rev:
>
>              4.6.  Reframing and Congestion Markings
>
>                 The guidance in this section is worded in terms of
>             framing
>                 boundaries, but it applies equally whether the protocol
>             data units
>             -   are frames, cells, packets or fragments.
>             +   are frames, cells or packets.
>
>             == RFC6040UPDATE-SHIM ==
>
>             I shall upload a new rev shortly, with the following 3 paras
>             at the end of S.5 on
>             ECN fragmentation/reassembly:
>
>             Para 1 is just moved up from the end but otherwise
>             unchanged.
>             Para 2 is unchanged.
>             Para 3 is the text agreed on this list last Dec subject to
>             further checking, with
>             one exception: I removed the citation of RFC3168 after
>             "equivalent".
>                 Reason: The citation of RFC6040 at the end is the
>             relevant one, 'cos it
>             introduced the mechanism for ECT(0) and ECT(1) to be either
>             equivalent or two
>             severity levels. In this respect it updated RFC3168. So it
>             would not be appropriate
>             to cite RFC3168, which only said the two were equivalent. If
>             we cited RFC3168 here,
>             it could be interpreted as if we're saying two RFC give
>             conflicting definitions.
>
>                 Section 5.3 of [RFC3168] defines the process that a
>             tunnel egress
>                 follows to reassemble sets of outer fragments
>                 [I-D.ietf-intarea-tunnels] into packets.
>
>                 During reassembly of outer fragments
>             [I-D.ietf-intarea-tunnels], if
>                 the ECN fields of the outer headers being reassembled
>             into a single
>                 packet consist of a mixture of Not-ECT and other ECN
>             codepoints, the
>                 packet MUST be discarded.
>
>             +   If there is mix of ECT(0) and ECT(1) fragments, then the
>             reassembled
>             +   packet MUST be set to either ECT(0) or ECT(1).  In this
>             case,
>             +   reassembly SHOULD take into account that the RFC series
>             has so far
>             +   ensured that ECT(0) and ECT(1) can either be considered
>             equivalent,
>             +   or they can provide 2 levels of congestion severity,
>             where the
>             +   ranking of severity from highest to lowest is CE,
>             ECT(1), ECT(0)
>             +   [RFC6040].
>
>             If any of this isn't acceptable, I'll have to post another
>             rev, but I think it's
>             what was agreed.
>
>             Cheers
> 
> 
>
>             Bob
> 
>
>             On 14/12/2020 15:44, Markku Kojo wrote:
>                   Hi Bob, all,
>
>                   apologies for the delay, now catching up again.
>
>                   yes, handling mix of ECT(0)/ECT(1) like in the new
>             proposed text below
>                   seems reasonable choice (for now).
>
>                   I'll come back shortly with the issue in the other
>             thread. It seems
>                   less clear, actually seems quite difficult to handle
>             correctly for all
>                   foreseen cases.
>
>                   /Markku
>
>                   On Thu, 3 Dec 2020, Bob Briscoe wrote:
>
>                         Markku, all,
>
>                         I am also only now catching up with the list...
>
>                         On 17/11/2019 08:46, Markku Kojo wrote:
>                               Hi Dave, Joe, All,
>
>                               Catching up ...
>
>                               I agree with the modified new text as well
>             as
>                         treatment of an ECT(0)/ECT(1)
>                               mix as "any".
> 
>
>                         [BB] Thanks. For the list, the current text that
>             Markku is
>                         agreeing with is here:
>                        
>             https://tools.ietf.org/html/draft-ietf-tsvwg-rfc6040update-shim-11#section-5
>
>                         Regarding reassembly of a mix of ECT(0)/ECT(1).
>             I agree
>                         with David that the current text
>                         should handle this case that 3168 doesn't
>             address.
>                         And I agree with Joe that an interim way of
>             handling it is
>                         needed, not just punting until
>                         later.
>
>                         I see that all of Jonathan, David and you Markku
>             are happy
>                         with reassembling a mix of
>                         ECT(0) and ECT(1) to result in either ECT(0) or
>             ECT(1).
>                         (for now). I think we can go one
>                         better than that, still without precluding a
>             more specific
>                         RFC later. Here's proposed
>                         text:
>
>                         After the following para:
>
>                            During reassembly of outer fragments
>                         [I-D.ietf-intarea-tunnels], if
>                            the ECN fields of the outer headers being
>             reassembled
>                         into a single
>                            packet consist of a mixture of Not-ECT and
>             other ECN
>                         codepoints, the
>                            packet MUST be discarded.
>
>                         Add:
>
>                               If there is mix of ECT(0) and ECT(1)
>             fragments, then
>                         the reassembled packet
>                               MUST be set to either ECT(0) or ECT(1). In
>             this case,
>                         reassembly SHOULD take
>                               into account that the RFC series has so
>             far ensured
>                         that ECT(0) and ECT(1)
>                               can either be considered equivalent
>             [RFC3168], or
>                         they can provide 2 levels
>                               of congestion severity, where the ranking
>             of severity
>                         from highest to lowest
>                               is CE, ECT(1), ECT(0) [RFC6040].
> 
>
>                         Rationale: This avoids constraining future RFCs,
>             but at
>                         least lays out all the
>                         interoperabilityrequirements we already have for
>             handling
>                         this mixture. Then if an
>                         implementer wants to just default to choosing
>             one, it hints
>                         that they should choose
>                         ECT(1).
> 
> 
>
>                               I also want to repeat my comment that
>
>                                draft-ietf-tsvwg-ecn-encap-guidelines-13
>
>                               added similar new text that alters RFC
>             3168, and it
>                         should be modified
>                               accordingly.
> 
>
>                         [BB] I'll start another thread for this, rather
>             than make
>                         this thread too unweildy.
> 
> 
>
>                         Bob
> 
> 
>
>                               Thanks,
>
>                               /Markku
>
>                               PS. I missed Bob's response to my comment
>             at the
>                         time, but will reply it
>                               separately at some point.
> 
>
>                               On Wed, 9 Oct 2019, David Black wrote:
>
>                                     At this juncture, for an
>             ECT(0)/ECT(1) mix
>                         across a set of
>                                     fragments being reassembled, I would
>             suggest
>                         using "any" (i.e.,
>                                     either is ok) at this juncture to
>             avoid
>                         constraining what we may
>                                     do in the future; in particular,
>             this allows
>                         use of the value in
>                                     the first or last fragment, both of
>             which are
>                         likely to be
>                                     convenient approaches for some
>             implementations.
>
>                                     Thanks, --David
>
>                                           -----Original Message-----
>                                           From: Joe Touch
>             <[e-mail]>
>                                           Sent: Wednesday, October 9,
>             2019 10:29 AM
>                                           To: Black, David
>                                           Cc: Jonathan Morton;
>             [e-mail]
>                                           Subject: Re: [tsvwg]
>                                          
>             draft-ietf-tsvwg-rfc6040update-shim:
>                         Suggested
>                                           Fragmentation/Reassembly text
> 
>
>                                           [EXTERNAL EMAIL]
>
>                                           Hi, all,
>
>                                           I disagree with the suggestion
>             below.
>
>                                           Pushing this “under the rug”
>             for an
>                         indeterminate
>                                           later date only serves to
>                                           undermine the importance of
>             this issue.
>
>                                           At a MINIMUM, there needs to
>             be direct
>                         guidance in
>                                           place until a “better”
>                                           solution can be developed. For
>             now, that
>                         would mean
>                                           one of the following:
>                                           - use the max of the frag code
>             point
>                         values
>                                           - use the min of the frag code
>             point
>                         values
>                                           - use “any” of the frag code
>             point values
>                                           - pick some other way (first,
>             the one in
>                         the initial
>                                           fragment i.e., offset 0), etc.
>
>                                           One of these needs to be
>             *included at
>                         this time*.
>
>                                           If a clean up doc needs to be
>             issued, it
>                         can override
>                                           individual “scattered”
>                                           recommendations later.
>
>                                           Joe
>
>                                                 On Oct 9, 2019, at 6:33
>             AM, Black,
>                         David
>                                                 <[e-mail]>
>             wrote:
>
>                                                       The one case this
>             doesn't
>                                                       really cover is
>             what happens
>                                                       when a fragment
>
>                                           set
>                                                       has a mixture of
>             ECT(0) and
>                                                       ECT(1)
>             codepoints.  This
>                                                       probably isn't
>             very
>                                                       relevant to
>             current ECN
>                                                       usage, but may
>             become
>                                                       relevant with SCE,
>             in
>
>                                           which
>                                                       middleboxes on the
>             tunnel
>                                                       path may introduce
>             such a
>                                                       mixture to
>             formerly
>                                                       "pure" packets. 
>             From my
>                                                       perspective, a
>             likely
>                                                       RFC-3168 compliant
>                                                       implementation of
>             arbitrarily
>                                                       choosing one
>             fragment's ECN
>                                                       codepoint as
>                                                       authoritative
>             (where it
>                                                       doesn't conflict
>             with other
>                                                       rules) is
>             acceptable, but
>                                                       this doesn't
>             currently seem
>                                                       to be mandatory.
>
>                                                       With the above
>             language, it
>                                                       should be
>             sufficient to
>                                                       update RFC-3168 to
>
>                                           cover
>                                                       this case at an
>             appropriate
>                                                       time, rather than
>             scattering
>                                                       further
>
>                                           requirements
>                                                       in many documents.
> 
>
>                                                 I would concur that
>             using a
>                         separate
>                                                 draft to cover that case
>             at the
>
>                                           appropriate time would be the
>             better
>                         course of
>                                           action.
>
>                                                 Thanks, --David
>
>                                                       -----Original
>             Message-----
>                                                       From: Jonathan
>             Morton
>                                                      
>             <[e-mail]>
>                                                       Sent: Tuesday,
>             October 8,
>                                                       2019 6:55 PM
>                                                       To: Black, David
>                                                       Cc: [e-mail]
>                                                       Subject: Re:
>             [tsvwg]
>                                                      
>                         draft-ietf-tsvwg-rfc6040update-shim:
>                                                       Suggested
>                                                      
>             Fragmentation/Reassembly text
> 
>
>                                                       [EXTERNAL EMAIL]
>
>                                                             On 8 Oct,
>             2019,
>                                                             at 10:51 pm,
>                                                             Black, David
>                                                            
>             <[e-mail]>
>                                                             wrote:
>
>                                                             **NEW**:
>             Beyond
>                                                             those first
>             two
>                                                             paragraphs,
>             I
>                                                             suggest
>             deleting
>                                                             the
>
>                                           rest
>                                                       of Section 5 of
>             the
>                                                       rfc6040update-shim
>             draft and
>                                                       substituting the
>
>                                           following
>                                                       paragraph:
>
>                                                               As a
>             tunnel
>                                                             egress
>                                                             reassembles
>             sets
>                                                             of outer
>                                                             fragments
>                                                              
>                                                            
>                         [I-D.ietf-intarea-tunnels]
>                                                             into
>             packets, it
>                                                             MUST comply
>             with
>                                                               the
>             reassembly
>                                                             requirements
>             in
>                                                             Section 5.3
>             of 
>                                                             RFC 3168 in
>                                                               order to
>             ensure
>                                                             that
>             indications
>                                                             of
>             congestion are
>                                                             not lost.
>
>                                                             It is
>             certainly
>                                                             possible to
>                                                             continue
>             from
>                                                             that text to
>                                                             paraphrase
>             part
>                                                             or all
>
>                                           of
>                                                       Section 5.3 of RFC
>             3168, but
>                                                       I think the above
>             text
>                                                       crisply addresses
>             the
>                                                       problem, and
>             avoids
>                                                       possibilities of
>             subtle
>                                                       divergence.  I do
>             like the
>                                                       “reassembles sets
>             of outer
>                                                       fragments” lead-in
>             text
>                                                       (which I copied
>             from
>
>                                           the
>                                                       current
>             rfc6040shim-update
>                                                       draft) because
>             that text
>                                                       makes it clear
>             that
>                                                       reassembly
>             logically precedes
>                                                       decapsulation at
>             the tunnel
>                                                       egress.
>
>                                                             Comments?
> 
>
>                                                       Looks good to me.
>
>                                                       The one case this
>             doesn't
>                                                       really cover is
>             what happens
>                                                       when a fragment
>
>                                           set
>                                                       has a mixture of
>             ECT(0) and
>                                                       ECT(1)
>             codepoints.  This
>                                                       probably isn't
>             very
>                                                       relevant to
>             current ECN
>                                                       usage, but may
>             become
>                                                       relevant with SCE,
>             in
>
>                                           which
>                                                       middleboxes on the
>             tunnel
>                                                       path may introduce
>             such a
>                                                       mixture to
>             formerly
>                                                       "pure" packets. 
>             From my
>                                                       perspective, a
>             likely
>                                                       RFC-3168 compliant
>                                                       implementation of
>             arbitrarily
>                                                       choosing one
>             fragment's ECN
>                                                       codepoint as
>                                                       authoritative
>             (where it
>                                                       doesn't conflict
>             with other
>                                                       rules) is
>             acceptable, but
>                                                       this doesn't
>             currently seem
>                                                       to be mandatory.
>
>                                                       With the above
>             language, it
>                                                       should be
>             sufficient to
>                                                       update RFC-3168 to
>
>                                           cover
>                                                       this case at an
>             appropriate
>                                                       time, rather than
>             scattering
>                                                       further
>
>                                           requirements
>                                                       in many documents.
>
>                                                       - Jonathan Morton
> 
> 
> 
> 
>
>                         -- 
>                        
>             ________________________________________________________________
>                         Bob Briscoe                              
>                         http://bobbriscoe.net/
>                                        PRIVILEGED AND CONFIDENTIAL
> 
> 
>
>             -- 
>             ________________________________________________________________
>             Bob Briscoe                              
>             http://bobbriscoe.net/
> 
> 
> 
> -- 
> ________________________________________________________________
> Bob Briscoe                               http://bobbriscoe.net/
> 
>