Re: [Edm] Protocol testability

[Lucas Pardue <[e-mail]>]

Hi Chris,

Your thoughts in the room and your email make sense to me.

To speak specifically to the QUIC final size bug that we talked about. It
was mentioned that it was the sort of thing that should have been a unit
test. Indeed, after identifying the root cause and a fix, a new test was
added along with it. The interesting part here was that _in the same
implementation_ the bug was in the sender side, while the receiver side
followed the RFC and effectively had no bug. This is slightly different
from a failure to implement a "feature vertical". Given that this should
have been easy to spot within the same implementation, there was little
barrier to a test catching it.

So to pick up on your 3rd and 4th points, I think another aspect is the
extra dimensionality that arises from packet loss. Potentially every unit
test could be duplicated for a loss condition. This sort of thing adds
cognitive overhead when writing tests. In the meeting we talked about
interop testing and exercising different network conditions. That's doable
and a good thing. What is potentially less common is assistance in writing,
or more ideally just automagically doing, multivariate testing at the level
of unit or integration testing. I'd be interested in hearing other's
experiences here about what might exist to address this part of protocol
development, implementation, and maintenance.

Cheers
Lucas

On Wed, Mar 29, 2023 at 1:27 AM Christopher Wood <[e-mail]>
wrote:

> Apologies again to everyone for making you sit through my rambling
> thoughts earlier in the meeting. Here's my attempt to better articulate
> what I was trying to say at the mic.
>
> The bug that was mentioned in the meeting seems to have manifested in
> practice because of a lack of "proper" testing, for some definition of
> proper. This happens all the time, and for good reason: perhaps the easiest
> way we validate a protocol specification is by deploying it and seeing if
> it works in practice. A much harder (and, frankly, equally imperfect) way
> would be to spend all of our energy on improving tests and our test
> harnesses.
>
> So the way I see it, there are basically two extremes for how we validate
> protocol specifications. One extreme is where we could write exactly zero
> tests and rely entirely on deployment experiments to determine if a
> protocol is correct. The other extreme is where we could focus entirely on
> tests and do no deployment to determine correctness. In practice, we wind
> up somewhere in the middle.
>
> Where different implementations fall on this spectrum varies a lot in
> practice and is influenced by a number of things:
>
> - How much time pressure there is to ship software.
> - How many different implementations exist for the purposes of perfroming
> interop tests.
> - How many resources there are to dedicate on tests and test harnesses,
> and how feasible it is to implement tests in different environments.
> - How complex the protocol is and, consequently, how difficult it is to
> produce test vectors to exercise different parts of the protocol.
>
> There may certainly be more. I don't think there's a "right" answer as to
> where anyone should fall on this spectrum of testability, but I wonder if
> it would be useful for us to retroactively look back at collaborative
> efforts of different protocols to see if there are any trends worth
> documenting. QUIC is an obvious candidate here. I mentioned Oblivious HTTP
> in the meeting. Maybe there are others?
>
> Time provided, I'm willing to help look back at time and collate this
> information, though I'm curious to hear what folks think about this topic
> in general.
>
> Best,
> Chris
> --
> Edm mailing list
> [e-mail]
> https://www.iab.org/mailman/listinfo/edm
>