Re: [tsvwg] plan for L4S issue #29

[Sebastian Moeller <[e-mail]>]

Hi Wes,

thank you, more below in-line.

> On Oct 1, 2020, at 05:03, Wesley Eddy <[e-mail]> wrote:
> 
> Replying to your question to the chairs:
> 
> On 9/30/2020 4:00 AM, Sebastian Moeller wrote:
>> 	[SM] I believe that we are conflating two "experiments" here:
>> 
>> 1) the absolutely required experiments to what degree L4S will realize its claimed charcteristics (or its promises) under a number of conditions that are relevant/prevalent in the real world.
>> 
>> 2) how to deploy L4S in the real world.
>> 
>> If we look carefully at these two experiments it becomes obvious, that an RFC in the experimental track seems necessary for 2), but 1) can and should proceed long before 2) is being addressed. To be blunt, if experiment 1) is not clearly demonstrating improvements over the state of the art, then 2) becomes moot.
>> 
>> @CHAIRS: Could you please let me know, if you agree?
> 
> 
> We've heard very strongly from the working group that enabling low-latency is very important to a wide number of stakeholders right now.  Several teams have been experimenting with this, and it's been reported that the industry is actively working on L4S implementations.

	[SM] I heard the same claims, but I am dubious about how much more than a wish for low latency this WG feeling conveyed. I asked repeatedly on this list for data demonstrating L4S keeping its promise over rather normal internet conditions and did not even get a "we have internal confidential data, that convinced us, but that we are not permitted to share" response, but instead nothing at all. I wonder what value the fact "industry is worlking on it" has, if the same industry developers can not easily produce data answering the "does it work under the to be expected conditions, is it safe, is it robust & reliable" triple-question. 

> 
> We also have clearly heard a number of people that are so far unconvinced of the utility or capabilities in their analysis.

	[SM] This is a somewhat distorted way of phrasing this issue. I have repeatedly asked for data and since none has been forthcoming I am forced to assume, that all the people you categorize as "convinced" in the last sentence can at best BELIEVE in L4S capabilities, and I am certain that in science and engineering a believe can be a decent starting point, but never can substitute real data/evidence. All I am asking for is that the WG makes sure to look at real data first instead of making a hard to revoke decision based on hope/wishful thinking/believe... 


>  I think this shows that the operational considerations work that Greg has started editing is important.  It can help people experimenting with methodology for collecting data and having appropriate risk awareness and management plans.

	[SM] As I said before, none of this is a prerequisite for doing internet-size experiments first (size, not scale, I am not asking team L4S to simulate the whole internet, but rather to convincingly demonstrate that L4S keeps its promises over more than the currently over-tested short RTT links, and also assess the side-effects it causes over such internet-style paths).

> 
> So, from my viewpoint as a chair, I don't think the working group seems to want to block right now on convincing every single person to love L4S,

	[SM] This would be a decent argument, if I would measure L4S against some hypothetical magic solution, but I am constantly trying to compare L4Ss implementation against the claims in the drafts in the implementation so far always falls way short of the promises. Expecting L4S to be loved by everybody seems over the top, agreed. At least making sure its RFCs describe its reality however seems rather essential, unless I misunderstood the goal behind the internet engineering task force completely...

> but rather to make sure that it's setup to be sufficiently safe for the wider experimentation plans that we've heard of in prior meetings.

	[SM] Before this de-rails any further into abstract concepts, could you please answer a few questions:

1) Why you consider it essential for these safety tests that the L4S drafts reach RFC status first? Put differently, what unsurmountable obstacles exist today that make it impossible for the required safety tests to be performed right now?

2) And on the topic of failure, will the WG commit to a set of criteria up-front to assess whether a potential L4S experiment succeeded or failed?

3) What to do in the case of failure? Do you consider it out of scope for the WG to require the L4S drafrs ot include explicit section how to un-roll the experiment in case of failure and the expected longer-term side-effects? If no, pleas elaborate why?

Best Regards
	Sebastian