RE: Add extension work to Interop matrix

["Lubashev, Igor" <[e-mail]>]

An interop is not an IETF event, so its “scope and agenda” is whatever the people who show up want to work on.   If more than one implementation is interested in implementing an extension, it seems simple to designate a letter and have them go at it.  We’ll not run out of letters (60+ case-sensitive alphanumerics).


  *   Igor

From: Lucas Pardue <[e-mail]>
Sent: Tuesday, January 7, 2020 12:46 PM

On Tue, Jan 7, 2020 at 3:56 PM Dmitri Tikhonov <[e-mail]<mailto:[e-mail]>> wrote:
I did not state that we should exclude any extensions.  I believe
the list above is exhaustive.  If not, I am all for adding to it.

A salient view of QUIC-related I-Ds is at https://datatracker.ietf.org/wg/quic/documents/<https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_wg_quic_documents_&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=NH5Oty6zwbwiXZ3F7JSL5CuvXbThJpB1Kivfb-biWb8&s=bk8-1eXj1pTq4sN0iOEyGbMwwr4ExjZENuCX2ZBWrQQ&e=> which is 19 documents strong, albeit some do not relate to specific implementation work e.g. use cases. No doubt there are others not accounted for on that page.

The recent email threads related to the topic "Re-chartering for extension work" [1] show there are a lot of people that want to discuss a range of topics.


> We're at 19 "codes" right now, and not many implementations satisfy them
> all. I'd like to find a solution that is more sustainable in the longer
> term that can support extensions along with QUIC v2 or whatever comes next.

The good thing about the current form of the Interop matrix is that
it is easy to start a new sheet and that older matrices become useless
rapidly.  The English alphabet has 26 letters, so we are using
19 / (26 * 2 + 10) = 31% of all easily readable code available to us.
Going up to 23 codes (37%) is not a big deal.  I don't expect us to
run out.  Even if we do, we can always do something new with the
next version of the matrix.

(I considered proposing E1, E2, or Eq, Eg, but one-letter codes are
simpler and fit with the current model.)

> I don't know what that looks like but I'd take some inspiration from things
> like the QUIC Network Simulator interop runner and Web Platform Tests (
> https://wpt.fyi/results/?label=experimental&label=master&aligned<https://urldefense.proofpoint.com/v2/url?u=https-3A__wpt.fyi_results_-3Flabel-3Dexperimental-26label-3Dmaster-26aligned&d=DwMFaQ&c=96ZbZZcaMF4w0F4jpN6LZg&r=Djn3bQ5uNJDPM_2skfL3rW1tzcIxyjUZdn_m55KPmlo&m=NH5Oty6zwbwiXZ3F7JSL5CuvXbThJpB1Kivfb-biWb8&s=kMvFr_9jKMBPTRiarfRxr44oC2kMeGYR3exUj1T8VVA&e=>).

My proposal is just to capture extension work in a simple, low entry
cost, manner.  I am not willing to commit to create a framework such
as to which you refer above.

I think the simple approach will scale poorly given the active discussion about QUIC extensibility. There are three aspects to consider: what to test and how to test it, where to capture results, and how to represent results. Today we achieve that with "Implementation drafts" and the interop matrix. Your proposal addresses the second and third aspects. To address the first one, we could brake it into a set of questions:

1) Should QUIC adopted extensions be in scope for the next interop?
  * Datagram and Version Negotiation are in calls for adoption ending on January 15.
2) Should QUIC non-adopted extensions be in scope for the next interop?
  * If so, which ones?
3) Who is willing to help define or review interoperability tests for the above?
4) Who is interested in testing during the interop event?
5) Is it important to record the results?
  * If so, where?

The answers to these questions then help to decide if testing is incorporated into an implementation draft and the results included in the Interop matrix.

Personally, I have interest in interop testing of Datagram. I don't think it needs to be included in the implementation draft nor the interop matrix but capturing the results in some other location might be interesting.

Cheers
Lucas