[Edm] Protocol testability

[Christopher Wood <[e-mail]>]

Apologies again to everyone for making you sit through my rambling thoughts earlier in the meeting. Here's my attempt to better articulate what I was trying to say at the mic.

The bug that was mentioned in the meeting seems to have manifested in practice because of a lack of "proper" testing, for some definition of proper. This happens all the time, and for good reason: perhaps the easiest way we validate a protocol specification is by deploying it and seeing if it works in practice. A much harder (and, frankly, equally imperfect) way would be to spend all of our energy on improving tests and our test harnesses. 

So the way I see it, there are basically two extremes for how we validate protocol specifications. One extreme is where we could write exactly zero tests and rely entirely on deployment experiments to determine if a protocol is correct. The other extreme is where we could focus entirely on tests and do no deployment to determine correctness. In practice, we wind up somewhere in the middle.

Where different implementations fall on this spectrum varies a lot in practice and is influenced by a number of things:

- How much time pressure there is to ship software.
- How many different implementations exist for the purposes of perfroming interop tests.
- How many resources there are to dedicate on tests and test harnesses, and how feasible it is to implement tests in different environments.
- How complex the protocol is and, consequently, how difficult it is to produce test vectors to exercise different parts of the protocol.

There may certainly be more. I don't think there's a "right" answer as to where anyone should fall on this spectrum of testability, but I wonder if it would be useful for us to retroactively look back at collaborative efforts of different protocols to see if there are any trends worth documenting. QUIC is an obvious candidate here. I mentioned Oblivious HTTP in the meeting. Maybe there are others?

Time provided, I'm willing to help look back at time and collate this information, though I'm curious to hear what folks think about this topic in general.

Best,
Chris