Re: Add extension work to Interop matrix

[Lucas Pardue <[e-mail]>]

On Tue, Jan 7, 2020 at 3:56 PM Dmitri Tikhonov <[e-mail]>
wrote:

> I did not state that we should exclude any extensions.  I believe
> the list above is exhaustive.  If not, I am all for adding to it.
>

A salient view of QUIC-related I-Ds is at
https://datatracker.ietf.org/wg/quic/documents/ which is 19 documents
strong, albeit some do not relate to specific implementation work e.g. use
cases. No doubt there are others not accounted for on that page.

The recent email threads related to the topic "Re-chartering for extension
work" [1] show there are a lot of people that want to discuss a range of
topics.


> > We're at 19 "codes" right now, and not many implementations satisfy them
> > all. I'd like to find a solution that is more sustainable in the longer
> > term that can support extensions along with QUIC v2 or whatever comes
> next.
>
> The good thing about the current form of the Interop matrix is that
> it is easy to start a new sheet and that older matrices become useless
> rapidly.  The English alphabet has 26 letters, so we are using
> 19 / (26 * 2 + 10) = 31% of all easily readable code available to us.
> Going up to 23 codes (37%) is not a big deal.  I don't expect us to
> run out.  Even if we do, we can always do something new with the
> next version of the matrix.
>
> (I considered proposing E1, E2, or Eq, Eg, but one-letter codes are
> simpler and fit with the current model.)
>
> > I don't know what that looks like but I'd take some inspiration from
> things
> > like the QUIC Network Simulator interop runner and Web Platform Tests (
> > https://wpt.fyi/results/?label=experimental&label=master&aligned).
>
> My proposal is just to capture extension work in a simple, low entry
> cost, manner.  I am not willing to commit to create a framework such
> as to which you refer above.
>

I think the simple approach will scale poorly given the active discussion
about QUIC extensibility. There are three aspects to consider: what to test
and how to test it, where to capture results, and how to represent results.
Today we achieve that with "Implementation drafts" and the interop matrix.
Your proposal addresses the second and third aspects. To address the first
one, we could brake it into a set of questions:

1) Should QUIC adopted extensions be in scope for the next interop?
  * Datagram and Version Negotiation are in calls for adoption ending on
January 15.
2) Should QUIC non-adopted extensions be in scope for the next interop?
  * If so, which ones?
3) Who is willing to help define or review interoperability tests for the
above?
4) Who is interested in testing during the interop event?
5) Is it important to record the results?
  * If so, where?

The answers to these questions then help to decide if testing is
incorporated into an implementation draft and the results included in the
Interop matrix.

Personally, I have interest in interop testing of Datagram. I don't think
it needs to be included in the implementation draft nor the interop matrix
but capturing the results in some other location might be interesting.

Cheers
Lucas