Re: [tsvwg] L4S drafts: Next Steps

[Jonathan Morton <[e-mail]>]

> On 20 Mar, 2021, at 5:08 pm, Livingood, Jason <[e-mail]> wrote:
> 
>> The lab studies have shown
> 
> IMO the challenge with lab studies is that there are a lot of variables that are artificial and/or the test environment or test traffic does not fully reflect reality (production). 10 or 15 years ago I would have been very focused on extensive QA testing and an elaborate lab test environment. These days its more how to do controlled testing directly in the production network, with a controlled/minimized blast radius in case of problems, easy/quick rollback, rapid code/config iteration.

Ah yes, the fail fast, fail often strategy.  And if your self-driving car happens to run into the side of a couple of 18-wheelers in the process, oh well, we'll do better next year.  Or next decade.  Look behind you, a three-headed monkey!

I know that the lab is an artificial environment in which not all possible conditions can be realistically predicted and simulated.  But if your system can't even cope reliably with conditions that the lab *can* easily simulate, what chance does it have in the much less predictable conditions in the real world?  And how do you debug it when it does fail, without being able to replicate some approximation of those conditions in the lab?

>> far from a promising system that might be ready for a field trial, one that shows worrying failure modes that primarily affect innocent bystanders.
>> What we're asking for is a system that at least behaves reasonably under lab conditions, chosen to represent realistic challenge scenarios likely to occur in the real world.  Only then can we have any confidence that L4S will not cause problems anywhere and everywhere that it is deployed.
> 
> Only one way to find out - test it in a real production experiment with appropriate risk controls. The status quo seems to suggest years more debate without (IMO) sufficient data. An alternative is a controlled production experiment, appropriately risk-managed, between willing participants. I see no downside to allowing that to occur via experimental RFC. We're setting the bar at the proposed standard or standard level, which I think is incorrect. I would say instead enable the experiment to accelerate the learning & improvement & drive data-driven decisions.

I would have somewhat more confidence in this approach if that was what the l4s-ops draft was written to require.  My suggestions to that effect do not appear, at this moment, to have been taken seriously by its author.  To eliminate bystander risk, as you suggest, requires that the experimental traffic be explicitly confined to the participating network(s) at a minimum - but that is not in the draft.

 - Jonathan Morton