[Edm] Experiments for reporting implementation and interop results

[Tommy Pauly <[e-mail]>]

Hi EDM,

As I mentioned in the IABOPEN meeting, a few of us worked on some experiments during last week’s hackathon to report implementation and interop test results.

What we used was:
- A library for presenting a webpage of results that can show different requirements or tests per draft, and show results per implementation. This is based on Mark Nottingham’s HTTP cache tests site.
- Annotation for markdown (kramdown) drafts to add test requirements inline, with a script to generate a file that gets dropped into the webpage source

The goal is to make a generic template that can be added to draft repositories to easily generate and track tests for different versions of drafts, which will inform testing and WG evaluation.

Here’s an example of the library in a GitHub repo for a draft: https://github.com/tfpauly/quic-proxy/tree/main/interop <https://github.com/tfpauly/quic-proxy/tree/main/interop>
The results and tests folders are the parts that will change per document.

The drafts are currently annotated with tags like this:

	{::options req-id=“example-test-id" req-type="must" req=“Example description" /}

…where the types are “must”/“should”/“supported” (for indicating if a feature is supported), and the req-id is a tag for which each implementation reports a true/false result.

What do people think about this kind of approach? What would you like to see? How could such templates and tools help inform our protocol development?

Best,
Tommy