Re: [rfc-i] Table of conformance requirements.

[Christian Huitema <[e-mail]>]

I think Robert's example shows the limit of the exercise. Take this
requirement from the list:

  MUST-3: There are certain operations that an application **MUST** be able to perform when interacting with QUIC streams.

How are you going to test that snippet of text? In the original
document, what follows is the list of these operations that the
application is supposed to perform. But if you just take out of context
the snippet of text surrounding the MUST statement, you get something
meaningless.

-- Christian Huitema

On 6/18/2020 9:31 AM, Robert Sparks wrote:
>
> On 6/18/20 11:29 AM, Larry Masinter wrote:
>> A previous proposal
>> https://tools.ietf.org/html/draft-ietf-newtrk-interop-reports-00
>> and experiment
>> https://tools.ietf.org/html/draft-saintandre-xmpp-interop-report-00
> Yeah, well. Fwiw: https://datatracker.ietf.org/doc/rfc5657/
>>
>>
>> -- 
>> https://LarryMasinter.net https://going-remote.info
>>
>>> -----Original Message-----
>>> From: rfc-interest <[e-mail]> On Behalf Of
>>> Robert Sparks
>>> Sent: Thursday, June 18, 2020 6:38 AM
>>> To: [e-mail]
>>> Subject: Re: [rfc-i] Table of conformance requirements.
>>>
>>> I have a script that builds such lists, and have used it for 20-ish
>>> years at this
>>> point in various contexts.
>>>
>>> Some example output:
>>>
>>> https://www.nostrum.com/~rjsparks/draft-ietf-quick-transport-29-
>>> enumreqs.txt
>>>
>>> https://www.nostrum.com/~rjsparks/rfc3261-enumreqs.txt
>>>
>>> For the most part, its best use has been to help identify when there
>>> are
>>> meaningless, duplicate, or contradictory uses of the BCP14 keywords.
>>>
>>> In interop testing (primarily the SIPit), having these requirements
>>> enumerated this way didn't help drive test behavior or the creation
>>> of test
>>> suites. It turned out that far more context was needed as a
>>> predicate to "I'm
>>> going to check this thing" than the isolated sentence containing the
>>> requirement. And agreeing on that context often turned out to be a
>>> subjective exercise (when that happened, I'd help put pressure on
>>> changing
>>> the spec to reduce the probability of different interpretations).
>>>
>>> So, it's been somewhat helpful in my experience, but not helpful
>>> enough to
>>> try to build extra support for it into, say, the v3 grammar.
>>>
>>> RjS
>>>
>>> On 6/18/20 2:08 AM, Carsten Bormann wrote:
>>>> There is a whole industry around requirements tracking.
>>>> I’m sure the Ribose people can tell us more about that.
>>>>
>>>>> Each MUST in a specification should have at least one
>>>>> corresponding unit
>>> test to check compliance.
>>>> Yeah sure.
>>>>
>>>> RFC 4120:
>>>>      *  Principals MUST keep their secret keys secret.
>>>>
>>>> I’d love to see that unit test :-)
>>>>
>>>> The other problem is that we simply don’t state all requirements in
>>>> BCP14
>>> language.
>>>> Very often, most requirements are stated in describing an
>>>> architecture or a
>>> protocol; they are phrased as statements of fact.  Extracting and
>>> labeling
>>> these requirements for requirements tracking is an art form.
>>>>> And this needs to be expressible somehow in XML.
>>>> I played around some with using kramdown’s auto-indexing feature for
>>> BCP14 keywords.  With today’s xml2rfc, that gives you an index of BCP14
>>> usage.  Not exactly what you want, but just a few lines of kramdown
>>> markdown.
>>>> I never used the results of these experiments in a document, mainly
>>> because the lack of BCP14 language on so many of the actual
>>> requirements
>>> made the result too sketchy.
>>>> Grüße, Carsten
>>>>
>>>> _______________________________________________
>>>> rfc-interest mailing list
>>>> [e-mail]
>>>> https://www.rfc-editor.org/mailman/listinfo/rfc-interest
>>> _______________________________________________
>>> rfc-interest mailing list
>>> [e-mail]
>>> https://www.rfc-editor.org/mailman/listinfo/rfc-interest
> _______________________________________________
> rfc-interest mailing list
> [e-mail]
> https://www.rfc-editor.org/mailman/listinfo/rfc-interest

_______________________________________________
rfc-interest mailing list
[e-mail]
https://www.rfc-editor.org/mailman/listinfo/rfc-interest