Re: [rfc-i] Table of conformance requirements.

Christian Huitema <huitema@huitema.net> Thu, 18 June 2020 16:42 UTC

Return-Path: <rfc-interest-bounces@rfc-editor.org>
X-Original-To: ietfarch-rfc-interest-archive@ietfa.amsl.com
Delivered-To: ietfarch-rfc-interest-archive@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 03C173A0DBE; Thu, 18 Jun 2020 09:42:11 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.65
X-Spam-Level:
X-Spam-Status: No, score=-2.65 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HEADER_FROM_DIFFERENT_DOMAINS=0.249, HTML_MESSAGE=0.001, MAILING_LIST_MULTI=-1, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IDoNwt3ChhdD; Thu, 18 Jun 2020 09:42:08 -0700 (PDT)
Received: from rfc-editor.org (rfc-editor.org [4.31.198.49]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 4386F3A0D43; Thu, 18 Jun 2020 09:42:05 -0700 (PDT)
Received: from rfcpa.amsl.com (localhost [IPv6:::1]) by rfc-editor.org (Postfix) with ESMTP id 47EF9F406D6; Thu, 18 Jun 2020 09:41:59 -0700 (PDT)
X-Original-To: rfc-interest@rfc-editor.org
Delivered-To: rfc-interest@rfc-editor.org
Received: from localhost (localhost [127.0.0.1]) by rfc-editor.org (Postfix) with ESMTP id 20484F406D6 for <rfc-interest@rfc-editor.org>; Thu, 18 Jun 2020 09:41:58 -0700 (PDT)
X-Virus-Scanned: amavisd-new at rfc-editor.org
Received: from rfc-editor.org ([127.0.0.1]) by localhost (rfcpa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id BtEmBe1fDbBN for <rfc-interest@rfc-editor.org>; Thu, 18 Jun 2020 09:41:53 -0700 (PDT)
Received: from mx43-out1.antispamcloud.com (mx43-out1.antispamcloud.com [138.201.61.189]) by rfc-editor.org (Postfix) with ESMTPS id 33D41F406CE for <rfc-interest@rfc-editor.org>; Thu, 18 Jun 2020 09:41:53 -0700 (PDT)
Received: from xse498.mail2web.com ([66.113.197.244] helo=xse.mail2web.com) by mx165.antispamcloud.com with esmtp (Exim 4.92) (envelope-from <huitema@huitema.net>) id 1jlxbo-000Psp-WB for rfc-interest@rfc-editor.org; Thu, 18 Jun 2020 18:41:56 +0200
Received: from xsmtp21.mail2web.com (unknown [10.100.68.60]) by xse.mail2web.com (Postfix) with ESMTPS id 49nnlD4QntzsyK for <rfc-interest@rfc-editor.org>; Thu, 18 Jun 2020 09:41:44 -0700 (PDT)
Received: from [10.5.2.16] (helo=xmail06.myhosting.com) by xsmtp21.mail2web.com with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.92) (envelope-from <huitema@huitema.net>) id 1jlxbk-00088z-Fy for rfc-interest@rfc-editor.org; Thu, 18 Jun 2020 09:41:44 -0700
Received: (qmail 19583 invoked from network); 18 Jun 2020 16:41:44 -0000
Received: from unknown (HELO [192.168.1.107]) (Authenticated-user:_huitema@huitema.net@[172.58.43.64]) (envelope-sender <huitema@huitema.net>) by xmail06.myhosting.com (qmail-ldap-1.03) with ESMTPA for <rfc-interest@rfc-editor.org>; 18 Jun 2020 16:41:44 -0000
To: Robert Sparks <rjsparks@nostrum.com>, Larry Masinter <LMM@acm.org>, rfc-interest@rfc-editor.org
References: <CAMm+LwiMOHMWcxFCYMdW_fsWsPpkC0vTt_0=+MzQfCm4qy=PTw@mail.gmail.com> <D6A8EDCA-D864-48C5-844E-D627F056115C@tzi.org> <ba75c5c6-48f9-c871-ef66-1bf743ddcdf5@nostrum.com> <007a01d6458d$9e469760$dad3c620$@acm.org> <ce30508e-3af3-7486-2bf4-38b8c83981ca@nostrum.com>
From: Christian Huitema <huitema@huitema.net>
Autocrypt: addr=huitema@huitema.net; prefer-encrypt=mutual; keydata= mDMEXtavGxYJKwYBBAHaRw8BAQdA1ou9A5MHTP9N3jfsWzlDZ+jPnQkusmc7sfLmWVz1Rmu0 J0NocmlzdGlhbiBIdWl0ZW1hIDxodWl0ZW1hQGh1aXRlbWEubmV0PoiWBBMWCAA+FiEEw3G4 Nwi4QEpAAXUUELAmqKBYtJQFAl7WrxsCGwMFCQlmAYAFCwkIBwIGFQoJCAsCBBYCAwECHgEC F4AACgkQELAmqKBYtJQbMwD/ebj/qnSbthC/5kD5DxZ/Ip0CGJw5QBz/+fJp3R8iAlsBAMjK r2tmyWyJz0CUkVG24WaR5EAJDvgwDv8h22U6QVkAuDgEXtavGxIKKwYBBAGXVQEFAQEHQJoM 6MUAIqpoqdCIiACiEynZf7nlJg2Eu0pXIhbUGONdAwEIB4h+BBgWCAAmFiEEw3G4Nwi4QEpA AXUUELAmqKBYtJQFAl7WrxsCGwwFCQlmAYAACgkQELAmqKBYtJRm2wD7BzeK5gEXSmBcBf0j BYdSaJcXNzx4yPLbP4GnUMAyl2cBAJzcsR4RkwO4dCRqM9CHpVJCwHtbUDJaa55//E0kp+gH
Message-ID: <075cb077-9104-e3b3-6307-7fe160bd76e2@huitema.net>
Date: Thu, 18 Jun 2020 09:41:43 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.9.0
MIME-Version: 1.0
In-Reply-To: <ce30508e-3af3-7486-2bf4-38b8c83981ca@nostrum.com>
Content-Language: en-US
X-Originating-IP: 66.113.197.244
X-Spampanel-Domain: xsmtpout.mail2web.com
X-Spampanel-Username: 66.113.197.0/24
Authentication-Results: antispamcloud.com; auth=pass smtp.auth=66.113.197.0/24@xsmtpout.mail2web.com
X-Spampanel-Outgoing-Class: unsure
X-Spampanel-Outgoing-Evidence: Combined (0.15)
X-Recommended-Action: accept
X-Filter-ID: Mvzo4OR0dZXEDF/gcnlw0f6LF1GdvkEexklpcFpSF5apSDasLI4SayDByyq9LIhVUZbR67CQ7/vm /hHDJU4RXkTNWdUk1Ol2OGx3IfrIJKywOmJyM1qr8uRnWBrbSAGDatr2qjVEYpI48T/8HRZsoW43 XeAVbFadXxNsR+jIm+4BcORxfCojcTOpEjtvWHp2msgOW4n0ST8vHyg5dOzWzzg9YkzbMy6DOYhG 3MUcvhrxNrJjiSXeBtvVVpEJHS+dtYPQEl46A2i0WhAEzy9RWZnckpWaLvahyBjmQxBKOztp0ugt Zlkw9p/AQdIE0knw7cPlLc4mPp21O6mudLBF4/IRL+spW60mrWiAEnQcn/qslg3CoCHsPz6pr/Ao kdAPhX8u3xIOOtB59sWJ1mr0rcswIlm249d1OOo21hSM9tXB1/2M1shZO++tsIV90EPuboZD5vPF e84pJQGk5dfbFPdfcXgLzsY3kBHC+ZTZl3S5+IwAWiarTQyLhmI9lhnr3l4BTMsdLA+IUUtrsjBa axEfRPm7RhvF5NbroBPxVOCfU642KNtk4n/u8nyV2xsjehIqUczFWeS6sE8e1b5/Uj/i4hYVfUxI FxiN15g3w5yRAuqQYnHwfyNsk2gjIPt8zz/e71VhfE4uVcD0sQ7Dp5OhaW0Ta9nDH26aUmdQxQIH mFDqewO9xyOqCYO8P1aHChSMTZbMtliaYpebNO5BA8gcN4/2gcY8aGcjKnyC/4yau8ag6Ik44xqT UtVxl21yn+b5da+4A6LEnav2a0j9MAM7as6wreE2f8C7bo2P7aXe0gJbYvd2FpmfoWg3GLNyf40f Pb36ViQE1/RgrLBT+TwJWw42swm4bO6gacpMpzIWuf64o43ILkWH8r1/6rOg4UT/PX1KIpuuo3Ji 0a+1F/N4PbnkyTlOmu+sqcsY3XoPGIVnp5TH7r29ZjWOfyD4rnnSv2dzL5wMg5rjmgOQhmUVrBsP GnNiJ83AD/4JscDaZrPqwesCTPjTYTd2k9+TUlP348Oim6i8PNK9YAzt5ulNC4wc2LkM7XQE4YLV klN6giDSqpQHYfnjwHBMA7P/
X-Report-Abuse-To: spam@quarantine11.antispamcloud.com
Subject: Re: [rfc-i] Table of conformance requirements.
X-BeenThere: rfc-interest@rfc-editor.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "A list for discussion of the RFC series and RFC Editor functions." <rfc-interest.rfc-editor.org>
List-Unsubscribe: <https://www.rfc-editor.org/mailman/options/rfc-interest>, <mailto:rfc-interest-request@rfc-editor.org?subject=unsubscribe>
List-Archive: <http://www.rfc-editor.org/pipermail/rfc-interest/>
List-Post: <mailto:rfc-interest@rfc-editor.org>
List-Help: <mailto:rfc-interest-request@rfc-editor.org?subject=help>
List-Subscribe: <https://www.rfc-editor.org/mailman/listinfo/rfc-interest>, <mailto:rfc-interest-request@rfc-editor.org?subject=subscribe>
Content-Type: multipart/mixed; boundary="===============0640348712505413832=="
Errors-To: rfc-interest-bounces@rfc-editor.org
Sender: "rfc-interest" <rfc-interest-bounces@rfc-editor.org>

I think Robert's example shows the limit of the exercise. Take this
requirement from the list:

  MUST-3: There are certain operations that an application **MUST** be able to perform when interacting with QUIC streams.

How are you going to test that snippet of text? In the original
document, what follows is the list of these operations that the
application is supposed to perform. But if you just take out of context
the snippet of text surrounding the MUST statement, you get something
meaningless.

-- Christian Huitema

On 6/18/2020 9:31 AM, Robert Sparks wrote:
>
> On 6/18/20 11:29 AM, Larry Masinter wrote:
>> A previous proposal
>> https://tools.ietf.org/html/draft-ietf-newtrk-interop-reports-00
>> and experiment
>> https://tools.ietf.org/html/draft-saintandre-xmpp-interop-report-00
> Yeah, well. Fwiw: https://datatracker.ietf.org/doc/rfc5657/
>>
>>
>> -- 
>> https://LarryMasinter.net https://going-remote.info
>>
>>> -----Original Message-----
>>> From: rfc-interest <rfc-interest-bounces@rfc-editor.org> On Behalf Of
>>> Robert Sparks
>>> Sent: Thursday, June 18, 2020 6:38 AM
>>> To: rfc-interest@rfc-editor.org
>>> Subject: Re: [rfc-i] Table of conformance requirements.
>>>
>>> I have a script that builds such lists, and have used it for 20-ish
>>> years at this
>>> point in various contexts.
>>>
>>> Some example output:
>>>
>>> https://www.nostrum.com/~rjsparks/draft-ietf-quick-transport-29-
>>> enumreqs.txt
>>>
>>> https://www.nostrum.com/~rjsparks/rfc3261-enumreqs.txt
>>>
>>> For the most part, its best use has been to help identify when there
>>> are
>>> meaningless, duplicate, or contradictory uses of the BCP14 keywords.
>>>
>>> In interop testing (primarily the SIPit), having these requirements
>>> enumerated this way didn't help drive test behavior or the creation
>>> of test
>>> suites. It turned out that far more context was needed as a
>>> predicate to "I'm
>>> going to check this thing" than the isolated sentence containing the
>>> requirement. And agreeing on that context often turned out to be a
>>> subjective exercise (when that happened, I'd help put pressure on
>>> changing
>>> the spec to reduce the probability of different interpretations).
>>>
>>> So, it's been somewhat helpful in my experience, but not helpful
>>> enough to
>>> try to build extra support for it into, say, the v3 grammar.
>>>
>>> RjS
>>>
>>> On 6/18/20 2:08 AM, Carsten Bormann wrote:
>>>> There is a whole industry around requirements tracking.
>>>> I’m sure the Ribose people can tell us more about that.
>>>>
>>>>> Each MUST in a specification should have at least one
>>>>> corresponding unit
>>> test to check compliance.
>>>> Yeah sure.
>>>>
>>>> RFC 4120:
>>>>      *  Principals MUST keep their secret keys secret.
>>>>
>>>> I’d love to see that unit test :-)
>>>>
>>>> The other problem is that we simply don’t state all requirements in
>>>> BCP14
>>> language.
>>>> Very often, most requirements are stated in describing an
>>>> architecture or a
>>> protocol; they are phrased as statements of fact.  Extracting and
>>> labeling
>>> these requirements for requirements tracking is an art form.
>>>>> And this needs to be expressible somehow in XML.
>>>> I played around some with using kramdown’s auto-indexing feature for
>>> BCP14 keywords.  With today’s xml2rfc, that gives you an index of BCP14
>>> usage.  Not exactly what you want, but just a few lines of kramdown
>>> markdown.
>>>> I never used the results of these experiments in a document, mainly
>>> because the lack of BCP14 language on so many of the actual
>>> requirements
>>> made the result too sketchy.
>>>> Grüße, Carsten
>>>>
>>>> _______________________________________________
>>>> rfc-interest mailing list
>>>> rfc-interest@rfc-editor.org
>>>> https://www.rfc-editor.org/mailman/listinfo/rfc-interest
>>> _______________________________________________
>>> rfc-interest mailing list
>>> rfc-interest@rfc-editor.org
>>> https://www.rfc-editor.org/mailman/listinfo/rfc-interest
> _______________________________________________
> rfc-interest mailing list
> rfc-interest@rfc-editor.org
> https://www.rfc-editor.org/mailman/listinfo/rfc-interest
_______________________________________________
rfc-interest mailing list
rfc-interest@rfc-editor.org
https://www.rfc-editor.org/mailman/listinfo/rfc-interest