Re: [tsvwg] [Technical Errata Reported] RFC7141 (7237)

[Sebastian Moeller <[e-mail]>]

Hi Gorry,


> On Nov 4, 2022, at 14:51, Gorry Fairhurst <[e-mail]> wrote:
> 
> On 04/11/2022 13:37, Sebastian Moeller wrote:
>> Hi Gorry,
>> 
>> 
>>> On Nov 4, 2022, at 14:03, Gorry Fairhurst <[e-mail]> wrote:
>>> 
>>> On 04/11/2022 12:42, Sebastian Moeller wrote:
>>>> Hi Gorry,
>>>> 
>>>> 
>>>>> On Nov 4, 2022, at 11:56, Gorry Fairhurst <[e-mail]> wrote:
>>>>> 
>>>>> On 04/11/2022 10:43, Sebastian Moeller wrote:
>>>>>> Hi Gorry,
>>>>>> 
>>>>>> See [SM] below.
>>>>>> 
>>>>>> On 4 November 2022 11:20:56 CET, Gorry Fairhurst <[e-mail]> wrote:
>>>>>>> Commenting as an individual on the Errata filing:
>>>>>>> 
>>>>>>> On 04/11/2022 09:40, RFC Errata System wrote:
>>>>>>>> The following errata report has been submitted for RFC7141,
>>>>>>>> "Byte and Packet Congestion Notification".
>>>>>>>> 
>>>>>>>> --------------------------------------
>>>>>>>> You may review the report below and at:
>>>>>>>> https://www.rfc-editor.org/errata/eid7237
>>>>>>>> 
>>>>>>>> --------------------------------------
>>>>>>>> Type: Technical
>>>>>>>> Reported by: Sebastian Moeller <[e-mail]>
>>>>>>>> 
>>>>>>>> Section: 2
>>>>>>>> 
>>>>>>>> Original Text
>>>>>>>> -------------
>>>>>>>> 2.2.  Recommendation on Encoding Congestion Notification
>>>>>>>> 
>>>>>>>>     When encoding congestion notification (e.g., by drop, ECN, or PCN),
>>>>>>>>     the probability that network equipment drops or marks a particular
>>>>>>>>     packet to notify congestion SHOULD NOT depend on the size of the
>>>>>>>>     packet in question.
>>>>>>>> [...]
>>>>>>>> 2.3.  Recommendation on Responding to Congestion
>>>>>>>> 
>>>>>>>>     When a transport detects that a packet has been lost or congestion
>>>>>>>>     marked, it SHOULD consider the strength of the congestion indication
>>>>>>>>     as proportionate to the size in octets (bytes) of the missing or
>>>>>>>>     marked packet.
>>>>>>>> 
>>>>>>>>     In other words, when a packet indicates congestion (by being lost or
>>>>>>>>     marked), it can be considered conceptually as if there is a
>>>>>>>>     congestion indication on every octet of the packet, not just one
>>>>>>>>     indication per packet.
>>>>>>>> 
>>>>>>>>     To be clear, the above recommendation solely describes how a
>>>>>>>>     transport should interpret the meaning of a congestion indication, as
>>>>>>>>     a long term goal.  It makes no recommendation on whether a transport
>>>>>>>>     should act differently based on this interpretation.  It merely aids
>>>>>>>>     interoperability between transports, if they choose to make their
>>>>>>>>     actions depend on the strength of congestion indications.
>>>>>>>> 
>>>>>>>> Corrected Text
>>>>>>>> --------------
>>>>>>>> I am not sure the text is actually salvageable, as it appears ti be a logic disconnect at the core of the recommendations.
>>>>>>>> 
>>>>>>>> Notes
>>>>>>>> -----
>>>>>>>> The recommendations seem not self consistent:
>>>>>>>> A) Section 2.2.  recommends that CE marking should be made independent of packet size, so a CE-mark carries no information about packet size.
>>>>>>> I did not understood that it needed to. This RFC I think was intended to be independent of the transport.  I see the transport sender as responsible for determining the packetisation of the transport segments, and the (S)ACKs can often identify segments, hence the sender can determine the segments that have been acknoweldged or times when ECN marking was seen.
>>>>>> [SM] This assumes that relevant segment size does not change along the path. Which generally is not true. Just think fragmentation, if the sender sends a packet that gets fragmented along the path and only a single fragment gets CE marked the sender will see this as the whole packet being marked. Or from the other side of the issue, if say a Linux router uses GRO/GSO and queues a larger meta packet and CE marks that, receiver and sender at best see a sequence of CE marked packets. So the recommendation would need to be changed to calculate the consecutive sequence of CE marked octets and take these as correlate for congestion strength. So no, the sender really has no reliable knowledge about the size of the data unit the marking node marked.
>>>>>> 
>>>>> I suggest IETF transports treat all IP fragments as one unit of retransmission/congestion at the transport layer.
>>>> 	[SM2] But what if the re-segmentation does not happen at the receiver, but say a fragmenting and CE-marking path tries to act transparently. According to the rules both in RFC3168 and RFC7141 a re-segmented packet containing even a single CE-marked fragment is to be CE-marked (or dropped). So the AQM might have marked a 576 octet segment but all the endpoint sees is a marked ~1460 octet segment.
>>>> 	This also illustrates how section 2.4 of RFC7141 proposes a method that does not achieve its aim, of giving veridical "number of market octets" information. It simply is impossible to do so generally (often it will work, but the endpoints can not even know when it was correct and when not).
>>>> Section 2.4 has more issues BTW, it tries to give recommendation how to deal with splitting and merging but fails to achieve its goals of giving a veridical account of the marked octets:
>>>> 
>>>> Let's see what happens when applying the proposed counter method in regards to number of marked octets under the conditions this section addresses
>>>> Here let's look at a toy problem with 20 byte headers and a total payload of 1200 octets that is split in or merged out of 3 fragments/segments with 400 octets payload each
>>>> 
>>>> Merging multiple segments pre-marking:
>>>> (20+400) + (20+400)-20 + (20+400)-20 -> 1220 total 1200 payload + CE
>>>> -> AQM marks 1220 or 1200 octets
>>>> (12+1200)+CE
>>>> receiver sees 1200 octets with CE and ACKs these with ECE
>>>> sender can assume 1200 octets where marked
>>>> CORRECT
>>>> 
>>>> Merging multiple segments post-marking ():
>>>> -> AQM marks segment 2 of 420 or 400 octets
>>>> (20+400) + (20+400+CE) + (20+400)
>>>> (20+400)+(20+400+CE)-20+(20+400)-20 = 1220 total 1200 payload + CE
>>>> receiver sees 1200 octets with CE and ACKs these with ECE
>>>> sender must assume 1200 octets where marked
>>>> FALSE
>>>> 
>>>> Fragmenting a segment pre-marking
>>>> 1220 -> (20+400) + (20+400) + (20+400)
>>>> -> AQM marks segment 2 of 420 or 400 octets
>>>> (20+400) + (20+400+CE) + (20+400)
>>>> Resegmentation happens before protocol sees marking
>>>> (20+400) + (20+400+CE)-20 + (20+400)-20 -> 1220 total 1200 payload + CE
>>>> receiver sees 1200 octets with CE and ACKs these with ECE
>>>> sender must assume 1200 octets where marked
>>>> FALSE
>>>> 
>>>> Fragmenting a segment post-marking
>>>> (20+1200)
>>>> -> AQM marks 1220 or 1200 octets
>>>> (12+1200)+CE
>>>> fragmentation happens:
>>>> (20+400+CE) + (20+400+CE) + (20+400+CE)
>>>> Resegmentation happens before protocol sees marking
>>>> (20+400=CE) + (20+400+CE)-20 + (20+400+C)-20 -> 1220 total 1200 payload + CE
>>>> receiver sees 1200 octets with CE and ACKs these with ECE
>>>> sender must assume 1200 octets where marked
>>>> CORRECT
>>>> 
>>>> So only in two out of four conditions does the proposed method actually achieves its goal.
>>>> 
>>>> Now add the complication that the RFC fails to mention what it considers marked octets, just the payload or payload+headers.
>>>> This is important as the sum of payload + headers of X fragments is larger than the sum payload + header of the single packet re-constituted out of these fragments. So the de-fragmenting process arguably needs to only look at payload size, but RFC7141 section 2.4 does not make that explicit.
>>>> If an implementation actually uses the full size instead of the payload size now the last condition also gets it wrong:
>>>> 
>>>> Fragmenting a segment post-marking
>>>> (20+1200)
>>>> -> AQM marks 1220 or 1200 octets
>>>> (12+1200)+CE
>>>> fragmentation happens:
>>>> (20+400+CE) + (20+400+CE) + (20+400+CE)
>>>> Resegmentation happens before protocol sees marking
>>>> (20+400+CE) + (20+400+CE) + (20+400+C) -> 1220 total 1200 payload + CE
>>>> but (20+400)*3 = 1260 marked octets
>>>> receiver sees 1200 octets with CE and ACKs these with ECE
>>>> sender must assume 1200 octets where marked
>>>> CORRECT
>>>> But now the left over 40 bytes in the marked-octet budget will result in CE feedback for the next (re-assembled) packet.
>>>> FALSE
>>>> 
>>>> For rfc3168 that will not matter much as ECE is sustained until CWR is received anyway, but L4S style signaling now acquired an erroneous CE mark.
>>> Network fragmentation be it in tunnels, extension headers or IPv4 fragments is indeed thwarted with all manner of issues. Nothing new - the IETF has long recommended the unit of loss/marking to be the same as the end to end PDU. PMTU is tricky, but does have benfits:-)
>> 	[SM3] Not a fan of fragmentation either, but I assume that fragmentation will stay a fact of life over the internet independent of my opinion. My point here is if the IETF proposes a method that aims to correctly account for CE-marked octets, that method should actually deliver on its premise. Failing in 2-3 out of 4 conditions the method is designed to handle is IMHO a sign that the proposed method is/was not in proper shape for becoming an official recommendation.
>> This is fine in an informational RFC as documenting subjective opinion, but problematic in a standards or BCP type document, if like in this case we feel that BCP methods (even incomplete or impossible to achieve ones) are binding precedence that need to be respected in later RFC drafts.
> 
> The BCP you mention was discussed some time ago and written because a decision had to be made about how best to think about this topic.

	[SM4] According to the shepherd (yours) write-up this was intended as informative and only switched to BCP:
This was discussed at IETF-81 and that the status changed from Informational to BCP, because the draft provides guidance to implementors and people configuring routers and hosts.

(Again, I have no issues this being an informative RFC, its BCP status causes problems as it apparently turns an incomplete method into precedence that can not be ignored, resulting in more incomplete and untested methods added to other drafts).

> So, if you now have a different proposal, you can write an ID and see what people think.

	[SM] I do, but I am sure my approach "not think about this in impossible terms" will not go down well, but I guess I should try.

>>>>> GSO/GRO and variants would/could change the fragmentation, that is true and need to be considered.
>>>> 	[SM2] I am confused? How do GRO/GSO affect fragmentation, IMHO these two will cause larger aggregates that exist only locally (Linux will segment meta-packets in the sending process and will not sent out say a large 64K TCP packet in fragments, but will re-segment the meta-packet into a neat sequence of complete self-sustained TCP packets)? IMHO they affect primarily the unit size the AQM might CE-mark on, in a way that is in-transparent to the end points. My point is the unit size an AQM acts on is generally unknowable precisely be the end-points. At which point making the end-points pretend that congestion strength somehow correlates with size of marked packets really stops making sense.
>>>> 
>>> The segment delivered can be a different size to the unit of transmission. This is an implementation optimisation - if this done without regard to the marking, then the results will be different and likely do not deliver what is expected - optimisations need to understand what they optimise.
>> 	[SM3] Yes, we seem to agree that it is impossible for endpoints to veridically measure the amount of octets actually CE-marked for a number of reasons. IMHO from this observation it follows directly that basing end-point decisions on number of marked octets is not going to generally work, as that number is not robustly and reliably available at the end-point. That is end-points do see a number but that number is not guaranteed to actually match what happened at the marking entity, and hence this number can not be a correlate of congestion strength. Interpreting that number never the less as indicator of congestion strength seems hence sub-optimal and not something to unconditionally recommend.
> I don't agree they can't measure. I said optimisations need to understand what they optimise.

	Here are two network paths, please explain how the endpoint can measure the size of the packet the AQM marked CE:


Sender(MSS 1460, 10 full packets) -> nodes(...) -> AQM (with GRO/GSO, using say a 14600 meta-packet and marks CE) -> nodes (...) -> receiver (10 packets with CE marks)

How can the receiver, let alone the sender measure that the AQM marked a single 14600 meta-packet and not 10 1460 sized packets?


Sender(MSS 1460, 1 full packets) -> nodes(smaller MTU 1000 -> fragmentation into 1 packet of size 1000 and one of size 460) -> AQM (CE marks 460 fragment) -> nodes (with re-assembly into 1 size 1460 packet with CE mark) -> receiver (1 packet size 1460 with CE mark)

How can the receiver, let alone the sender measure that the AQM marked a single size 460 packet and not a full 1460 sized packet?

(I only use payload sizes here, let's ignore the overhead). Again rfc3168 type ECN response it whole immune to this, because it actually does not seem to take number of marked octets into account.

So I would agree the end-points can "measure" but they are not guaranteed to get robust and reliable results that actually veridically describe what happened at the marking node that experienced the congestion. Note that any "optimization" here happens outside of the sphere of control of the end-points, so arguing that optimization comes with side-effects, while true does not help the end-points as they can not affect the optimization the rest of the network path might employ.


But I might well be wrong and would be happy to learn how the end points in my two scenarios could actually deduce what really happened at the marking node (preferably within the normal transport protocol functionality not employing out of band measurement traffic).


>> 
>>>>>>>> B) Section 2.3 then recommends to use the size of marked packets as direct indicators of congestion strength.
>>>>>>>> 
>>>>>>>> C) Section 2.3 then later clarifies that transports should interpret the size of CE-marked packets as correlate for congestion strength but are in no way required to take this interpretation into account when acting based on the congestion signal.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> This has several problems:
>>>>>>>> 1) A) and B) are in direct contradiction to each other. If we ask marking nodes to ignore packet size while marking, but end nodes to take it into account we basically create random congestion strength "information" by the pure chance of a specific packet of a specific size "catching" a CE mark. At which point we might as well simply draw a random number at the end-point to interpret congestion strength (except that packet sizes are not distributed randomly).
>>>>>>>> 
>>>>>>>> 2) Asking endpoints to interpret CE_marks in this way but not act on it, is hardly actionable advice for potential implementers. If we can not recommend a specific way, we should refrain from offering recommendations at all to keep things as simple as reasonably possible.
>>>>>>> This doesn't appear to be textual errata, it seems more like the request is for more clarification or motivating an alternative?
>>>>>> [SM] What alternatives to changing incorrect text do exist? I do not think changing the status to historic is a realistic option in spite of the text recommending the impossible.
>>>>>> 
>>>>> Put simply:
>>>>> 
>>>>> An Erratum would normally specify either:
>>>>> 
>>>>>     a direct change of text to fix a mistake in production, but a change of the spec from the original intended method;
>>>>> 
>>>>>     or specify something to inform a future revision.
>>>>> 
>>>>> An update in a new RFC is needed to change the method, or a process request to mark an RFC as historic.
>>>> 	[SM2] Would it also be possible to request to re-classify as informative? This RFC with its impossible recommendations is causing issues with other RFCs and I think it would help if this could be ameliorated by moving away from BCP status.
>>>> 
>>> If there is consensus an RFC shouldn't be associated with a BCP, we can examine what to do. The first thing is to write a (short) ID and see if you can gain sufficient attention from the WG to enable this to be discussed.
>> 	[SM3] Thank you very much. Is there an example for a similar process I could look at and take inspiration from?
> 
> I guess one can make a short ID that describes the problem, and see if other people understand the same problem.
> 
> As a chair, my first step would normally be to see whether there was a good common understanding by the WG of what was proposed and then to look to the WG to tell me if individuals see an issue that they would allocate time to fix/review/etc.

	[SM3] Great, the ball is my court now, and I need to see whether I can come up with a convincing write-up. Or I might just do what apparently every implementer under the sun did and simply ignore RFC7141.


> 
> Best wishes,
> 
> Gorry
> 
> P.S. Dropping the ADs and Errata system, and fixing Bob's email.

	Thanks!


Regards
	Sebastian


>> 
>> Regards
>> 	Sebastian
>> 
>> 
>>> Gorry
>>> 
>>>>> Gorry
>>>>> 
>>>>>>>> Instructions:
>>>>>>>> -------------
>>>>>>>> This erratum is currently posted as "Reported". If necessary, please
>>>>>>>> use "Reply All" to discuss whether it should be verified or
>>>>>>>> rejected. When a decision is reached, the verifying party
>>>>>>>> can log in to change the status and edit the report, if necessary.
>>>>>>>> 
>>>>>>>> --------------------------------------
>>>>>>>> RFC7141 (draft-ietf-tsvwg-byte-pkt-congest-12)
>>>>>>>> --------------------------------------
>>>>>>>> Title               : Byte and Packet Congestion Notification
>>>>>>>> Publication Date    : February 2014
>>>>>>>> Author(s)           : B. Briscoe, J. Manner
>>>>>>>> Category            : BEST CURRENT PRACTICE
>>>>>>>> Source              : Transport Area Working Group
>>>>>>>> Area                : Transport
>>>>>>>> Stream              : IETF
>>>>>>>> Verifying Party     : IESG