Re: [rfc-i] Table of conformance requirements.

Robert Sparks <rjsparks@nostrum.com> Thu, 18 June 2020 13:38 UTC

Return-Path: <rfc-interest-bounces@rfc-editor.org>
X-Original-To: ietfarch-rfc-interest-archive@ietfa.amsl.com
Delivered-To: ietfarch-rfc-interest-archive@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id B40DE3A0EEE; Thu, 18 Jun 2020 06:38:31 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.451
X-Spam-Level:
X-Spam-Status: No, score=-2.451 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_INVALID=0.1, DKIM_SIGNED=0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249, MAILING_LIST_MULTI=-1, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=fail (1024-bit key) reason="fail (message has been altered)" header.d=nostrum.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id GpmsKVcTJGU4; Thu, 18 Jun 2020 06:38:30 -0700 (PDT)
Received: from rfc-editor.org (rfc-editor.org [4.31.198.49]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id A63973A0F08; Thu, 18 Jun 2020 06:38:21 -0700 (PDT)
Received: from rfcpa.amsl.com (localhost [IPv6:::1]) by rfc-editor.org (Postfix) with ESMTP id CA93FF406F6; Thu, 18 Jun 2020 06:38:15 -0700 (PDT)
X-Original-To: rfc-interest@rfc-editor.org
Delivered-To: rfc-interest@rfc-editor.org
Received: from localhost (localhost [127.0.0.1]) by rfc-editor.org (Postfix) with ESMTP id 83D40F406F6 for <rfc-interest@rfc-editor.org>; Thu, 18 Jun 2020 06:38:14 -0700 (PDT)
X-Virus-Scanned: amavisd-new at rfc-editor.org
Authentication-Results: rfcpa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=nostrum.com
Received: from rfc-editor.org ([127.0.0.1]) by localhost (rfcpa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1Cp9_MRIndFI for <rfc-interest@rfc-editor.org>; Thu, 18 Jun 2020 06:38:10 -0700 (PDT)
Received: from nostrum.com (raven.nostrum.com [69.55.229.100]) by rfc-editor.org (Postfix) with ESMTPS id 7B260F406F5 for <rfc-interest@rfc-editor.org>; Thu, 18 Jun 2020 06:38:10 -0700 (PDT)
Received: from unescapeable.local ([47.186.30.41]) (authenticated bits=0) by nostrum.com (8.15.2/8.15.2) with ESMTPSA id 05IDbw0n031542 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for <rfc-interest@rfc-editor.org>; Thu, 18 Jun 2020 08:37:59 -0500 (CDT) (envelope-from rjsparks@nostrum.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=nostrum.com; s=default; t=1592487479; bh=ktoYjYB/MnSSYZxKBrNN6r7aK+nNxjyVXH2KHzGl1ew=; h=Subject:To:References:From:Date:In-Reply-To; b=rcqllgoir0t/fDYgzHpHmc9cfUhPEN21yEJ/Xzpbu8Cd2ZP4NP4w3JMcWaIcFI9ee 9iEIn/H3jfUti6P1JGfWsRIFnAkvE9a3c872IaQXt6AW5FyY/1cuA8xiyb6+iheYPk DruUjo3+/jCMHShYbXGdAv6lOen+Vq+3GCxKphgw=
X-Authentication-Warning: raven.nostrum.com: Host [47.186.30.41] claimed to be unescapeable.local
To: rfc-interest@rfc-editor.org
References: <CAMm+LwiMOHMWcxFCYMdW_fsWsPpkC0vTt_0=+MzQfCm4qy=PTw@mail.gmail.com> <D6A8EDCA-D864-48C5-844E-D627F056115C@tzi.org>
From: Robert Sparks <rjsparks@nostrum.com>
Message-ID: <ba75c5c6-48f9-c871-ef66-1bf743ddcdf5@nostrum.com>
Date: Thu, 18 Jun 2020 08:37:58 -0500
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Thunderbird/68.9.0
MIME-Version: 1.0
In-Reply-To: <D6A8EDCA-D864-48C5-844E-D627F056115C@tzi.org>
Content-Language: en-US
Subject: Re: [rfc-i] Table of conformance requirements.
X-BeenThere: rfc-interest@rfc-editor.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "A list for discussion of the RFC series and RFC Editor functions." <rfc-interest.rfc-editor.org>
List-Unsubscribe: <https://www.rfc-editor.org/mailman/options/rfc-interest>, <mailto:rfc-interest-request@rfc-editor.org?subject=unsubscribe>
List-Archive: <http://www.rfc-editor.org/pipermail/rfc-interest/>
List-Post: <mailto:rfc-interest@rfc-editor.org>
List-Help: <mailto:rfc-interest-request@rfc-editor.org?subject=help>
List-Subscribe: <https://www.rfc-editor.org/mailman/listinfo/rfc-interest>, <mailto:rfc-interest-request@rfc-editor.org?subject=subscribe>
Content-Transfer-Encoding: base64
Content-Type: text/plain; charset="utf-8"; Format="flowed"
Errors-To: rfc-interest-bounces@rfc-editor.org
Sender: "rfc-interest" <rfc-interest-bounces@rfc-editor.org>

I have a script that builds such lists, and have used it for 20-ish 
years at this point in various contexts.

Some example output:

https://www.nostrum.com/~rjsparks/draft-ietf-quick-transport-29-enumreqs.txt

https://www.nostrum.com/~rjsparks/rfc3261-enumreqs.txt

For the most part, its best use has been to help identify when there are 
meaningless, duplicate, or contradictory uses of the BCP14 keywords.

In interop testing (primarily the SIPit), having these requirements 
enumerated this way didn't help drive test behavior or the creation of 
test suites. It turned out that far more context was needed as a 
predicate to "I'm going to check this thing" than the isolated sentence 
containing the requirement. And agreeing on that context often turned 
out to be a subjective exercise (when that happened, I'd help put 
pressure on changing the spec to reduce the probability of different 
interpretations).

So, it's been somewhat helpful in my experience, but not helpful enough 
to try to build extra support for it into, say, the v3 grammar.

RjS

On 6/18/20 2:08 AM, Carsten Bormann wrote:
> There is a whole industry around requirements tracking.
> I’m sure the Ribose people can tell us more about that.
>
>> Each MUST in a specification should have at least one corresponding unit test to check compliance.
> Yeah sure.
>
> RFC 4120:
>     *  Principals MUST keep their secret keys secret.
>
> I’d love to see that unit test :-)
>
> The other problem is that we simply don’t state all requirements in BCP14 language.
> Very often, most requirements are stated in describing an architecture or a protocol; they are phrased as statements of fact.  Extracting and labeling these requirements for requirements tracking is an art form.
>
>> And this needs to be expressible somehow in XML.
> I played around some with using kramdown’s auto-indexing feature for BCP14 keywords.  With today’s xml2rfc, that gives you an index of BCP14 usage.  Not exactly what you want, but just a few lines of kramdown markdown.
>
> I never used the results of these experiments in a document, mainly because the lack of BCP14 language on so many of the actual requirements made the result too sketchy.
>
> Grüße, Carsten
>
> _______________________________________________
> rfc-interest mailing list
> rfc-interest@rfc-editor.org
> https://www.rfc-editor.org/mailman/listinfo/rfc-interest
_______________________________________________
rfc-interest mailing list
rfc-interest@rfc-editor.org
https://www.rfc-editor.org/mailman/listinfo/rfc-interest