Re: [tsvwg] dispute (Re: Results of consensus call on ECT(1) usage), Regarding testing

[Sebastian Moeller <[e-mail]>]

Bob, chairs, list,

Top post, because this is not a on-to-ne response to Bob points, but to the sentiment of his argument about testing, as I understood it; since this was triggered by Bob' response I opt to keep it in thread instead of opening a new thread.


I want to take the opportunity to register my objections to Bob's implicit claim below, that L4S has seen sufficient  and diverse enough testing, and that legacy testing  results with DCTCP fro the L4S's effort early days is simply transferable to TCP Prague. 
	Also I want to note that quantity of testing is a imprecise (bad) proxy for quality of testing. In a Popperian sense, I got the impression, that L4S went out of its way to document where white swans can be found instead of spending time and effort trying to find non-white swans, as a result the hypothesis all swans are white (or precisely the proposed L4S design works robustly and reliably) is still insufficiently tested.

1) Quantity: 
	I agree that the L4S team has presented an impressive array of testing results, so I will concede the sheer number. 

2) Diversity:
	Here the effort is considerably less impressive, it took the SCE's team effort to present data for the natural cases of asymmetric links and bi-directional saturation. For a technique primarily intended to the ISP/end-user edge these two are not extreme corner-cases but rather the bread and butter any AQM/QoS solution needs to deal with acceptably, period.
	I find it immensely troubling that Bob, in one earlier e-mail conceded that L4S would probably not work well with bi-directional saturation, and never ever bothered to make that a testing priority. This is the looking for more white swans approach I allude to above, but seemingly by conscious choice and not by simply overlooking the critical cases.
	I am also puzzled that the WG here seems determined to push L4S/ETC(1) through even without sufficient evidence that all the pie-in-the-sky promises actually are kept. All tire-kicking so far implies that L4S over-promises and under-delivers once one looks beyond the "normal" conditions which admittedly have been tested in considerable quantity, but essentially testing the same condition that L4S deals with well multiple times does not increase the relevant depth of testing, IMHO.

3) Legacy testing:
	The amount of major bugs, that the SCE team found in TCP Prague when doing comparison testing reveals that TCP Prague development does not seem to use regular regression testing. Based on that fact and the fact the TCP Prague by now has diverged from DCTCP in a noticeable fashion. I question whether the old DCTCP testing results have more that historic value by now, as it seems far from clear that TCP Prague will behave similarly, either because of evolutionary divergence or because of newly introduced but yet unfixed bugs. If the L4S team wants to have this testing taken seriously I encourage them to repeat these tests with a current TCP Prague version. Otherwise these tests carry little vsalue for predicting how current L4S implementation will behave.



Best Regards
	Sebastian

P.S.: I realize that engineering is the "art of the possible" and that often perfect is a worse goal than good enough. But my objections against the current L4S implementation & design are that they simply are not :good enough" (yet?), and that I consider it to be an odd situation that the WG seemingly steers down a road of spending the last ECN codepoint on a sheer promise alone. Then again I predicted that outcome immediately after the call for consensus was made to the lists, as the chairs might remember.



> On May 22, 2020, at 22:47, Bob Briscoe <[e-mail]> wrote:
> 
> Paul, Chairs
> 
> On 22/05/2020 04:03, Paul Vixie wrote:
>> On Friday, 22 May 2020 02:32:45 UTC Wesley Eddy wrote:
>>> ...
>>> 
>>> Result of consensus call: TSVWG has interest in working on a low-latency
>>> service using ECT(1) as an input (e.g. L4S) in preference to working on
>>> enabling high-fidelity congestion control using ECT(1) as an network
>>> output signal (e.g. SCE). The chairs see strong support for this
>>> direction, including significant interest from network operators and
>>> vendors.
>> the relevant section of RFC 2026 is as follows:
>> 
>>> 6.5.1 Working Group Disputes
>>> 
>>>    An individual (whether a participant in the relevant Working Group or
>>>    not) may disagree with a Working Group recommendation based on his or
>>>    her belief that either (a) his or her own views have not been
>>>    adequately considered by the Working Group, or (b) the Working Group
>>>    has made an incorrect technical choice which places the quality
>>>    and/or integrity of the Working Group's product(s) in significant
>>>    jeopardy.  The first issue is a difficulty with Working Group
>>>    process;  the latter is an assertion of technical error.  These two
>>>    types of disagreement are quite different, but both are handled by
>>>    the same process of review.
>> on a personal note, my own expressed position was, that if something had to be
>> done it should be an input not an output. so, the chairs' declared result is
>> compatible with what i said. therefore i'm not claiming that my own views were
>> not adequately considered (a, above). rather, i'd concerned about (b) above.
>> 
>> this choice was not made based on technical merit, by the chair's own
>> description (quoted above). both the rough consensus and the running code
>> looked to me more like choice 3 ("more testing"). what the operators and
>> vendors who can afford to attend these meetings are interested in, may differ
>> from the community's long term interests.
>> 
>>>    A person who disagrees with a Working Group recommendation shall
>>>    always first discuss the matter with the Working Group's chair(s),
>>>    who may involve other members of the Working Group (or the Working
>>>    Group as a whole) in the discussion.
>> we are "here." i hope the chairs can explain why choice 3, more testing, was
>> not the rough consensus they think they witnessed,
> 
> [BB] Can I suggest that the chairs ask those asking for more testing to write a review of all the tests that have been done so far (by proponents and opposition), and identify the specific further tests that they believe are necessary. Then the chairs or the WG or whoever is judging this dispute can decide whether the tests they are asking for are critical prerequisites for assignment of an experimental codepoint in the IP header.
> 
> Reasoning: There have been a number of cases where people have complained in general terms about testing, and subsequently it became apparent that they had not read the test reports - some of which had even been verified independently and published by third parties.
> 
>> and also, how the time
>> value of a choice of 1 or 2 "now" is higher than the merit value of getting to
>> an evidence-based consensus "later".
> 
> [BB] I notice that you (Paul) said you'd been following this since the first interim. You may not be aware that this work was brought to the IRTF in March 2015, then brought to the IETF in July 2015 along with thousands of test results. A very large BoF was held in Jul 2016 that led to L4S drafting work being started in tsvwg and tcpm. The debate over the pros and cons of different codepoints was completed around Nov 2017, when ECT(1) was settled on and that debate was written up in the ecn-l4s-id draft. In March 2019, L4S was approaching WGLC, when the SCE proposal for a different codepoint arrangement was first published, and proponents of both approaches have published large volumes of test results in the 14 months since then.
> 
> So those taking part in the recent consensus call had a wealth of test data on which to base their decision, from both proponents and opponents of the outcome that the chairs have just announced.
> 
> 
> In the past, I have been in a similar position, when I have discovered that a major decision was imminent that would close off areas that were important to me. The closest example was when draft-ietf-tsvwg-ecn [RFC3168-to-be] was approaching WGLC. I took it upon myself to thoroughly read the draft and all its references, to catch up on all the mail archives, and to write a draft articulating our concerns (with Jon Crowcroft at that time). As I put our concerns, I could feel the time pressure of everyone involved breathing down my neck. By putting in the effort to put our concerns concisely and rapidly, the WG got them resolved quickly - Jon & I had to concede some aspects in the interest of keeping progress moving. As a result, RFC3168 went through WGLC soon after and was published only 7 months after we first intervened.
> 
> So, yes, there is a huge degree of time pressure now. The closing of this consensus call allows us to get moving in anger. You will have seen the number of companies that plan to deploy. No-one has bottomless project funding. I (and hopefully others) will try to point you at the resources that are relevant for your concerns. But I hope you will understand that, at the closing stages of the debate, we can't be expected to retype all the discussion each time a new person joins.
> 
> Cheers
> 
> 
> 
> Bob
> 
> 
> 
>>>    If the disagreement cannot be resolved in this way, any of the
>>>    parties involved may bring it to the attention of the Area
>>>    Director(s) for the area in which the Working Group is chartered.
>>>    The Area Director(s) shall attempt to resolve the dispute.
>>>        If the disagreement cannot be resolved by the Area Director(s) any of
>>>    the parties involved may then appeal to the IESG as a whole.  The
>>>    IESG shall then review the situation and attempt to resolve it in a
>>>    manner of its own choosing.
>>>        If the disagreement is not resolved to the satisfaction of the
>>>    parties at the IESG level, any of the parties involved may appeal the
>>>    decision to the IAB.  The IAB shall then review the situation and
>>>    attempt to resolve it in a manner of its own choosing.
>> i think a challenge was inevitable no matter what outcome was designated as
>> the consensus, and that this probably is a matter to be settled by IETF as a
>> body rather than by this working group in isolation.
>> 
> 
> -- 
> ________________________________________________________________
> Bob Briscoe                               http://bobbriscoe.net/
>