Re: [rfc-i] Table of conformance requirements.

Robert Sparks <rjsparks@nostrum.com> Thu, 18 June 2020 16:32 UTC

Return-Path: <rfc-interest-bounces@rfc-editor.org>
X-Original-To: ietfarch-rfc-interest-archive@ietfa.amsl.com
Delivered-To: ietfarch-rfc-interest-archive@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 39E5C3A0D16; Thu, 18 Jun 2020 09:32:23 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.451
X-Spam-Level:
X-Spam-Status: No, score=-2.451 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_INVALID=0.1, DKIM_SIGNED=0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249, MAILING_LIST_MULTI=-1, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=fail (1024-bit key) reason="fail (message has been altered)" header.d=nostrum.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JvGkImdVQGVL; Thu, 18 Jun 2020 09:32:21 -0700 (PDT)
Received: from rfc-editor.org (rfc-editor.org [4.31.198.49]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 705143A0CE1; Thu, 18 Jun 2020 09:32:21 -0700 (PDT)
Received: from rfcpa.amsl.com (localhost [IPv6:::1]) by rfc-editor.org (Postfix) with ESMTP id 7F1C6F406D6; Thu, 18 Jun 2020 09:32:15 -0700 (PDT)
X-Original-To: rfc-interest@rfc-editor.org
Delivered-To: rfc-interest@rfc-editor.org
Received: from localhost (localhost [127.0.0.1]) by rfc-editor.org (Postfix) with ESMTP id 514E7F406D6 for <rfc-interest@rfc-editor.org>; Thu, 18 Jun 2020 09:32:14 -0700 (PDT)
X-Virus-Scanned: amavisd-new at rfc-editor.org
Authentication-Results: rfcpa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=nostrum.com
Received: from rfc-editor.org ([127.0.0.1]) by localhost (rfcpa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PZ0dAUKm33RU for <rfc-interest@rfc-editor.org>; Thu, 18 Jun 2020 09:32:09 -0700 (PDT)
Received: from nostrum.com (raven.nostrum.com [69.55.229.100]) by rfc-editor.org (Postfix) with ESMTPS id C230BF406D3 for <rfc-interest@rfc-editor.org>; Thu, 18 Jun 2020 09:32:09 -0700 (PDT)
Received: from unescapeable.local ([47.186.30.41]) (authenticated bits=0) by nostrum.com (8.15.2/8.15.2) with ESMTPSA id 05IGVlBG065421 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 18 Jun 2020 11:31:47 -0500 (CDT) (envelope-from rjsparks@nostrum.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=nostrum.com; s=default; t=1592497908; bh=yG++btYYWghaRAUoEzf56W47Yl/ZZo3kk38Ri8NjEv4=; h=Subject:To:References:From:Date:In-Reply-To; b=JMqZYCgzi9mVer1WnFGpbOsGMLpCVajK8EMOaxik3pSozHYzzO1pG7DrVoMxws3lj sE5jZ2/CQj/sJ5LeGyPk4I9aOyASAt/+vSGzuSpHypHonjX0TuOpYCIpHBNwCiKQD4 iPHMtdRfxIF2CvNAAf72WLX7wX5pMawFi3b9Nz5s=
X-Authentication-Warning: raven.nostrum.com: Host [47.186.30.41] claimed to be unescapeable.local
To: Larry Masinter <LMM@acm.org>, rfc-interest@rfc-editor.org
References: <CAMm+LwiMOHMWcxFCYMdW_fsWsPpkC0vTt_0=+MzQfCm4qy=PTw@mail.gmail.com> <D6A8EDCA-D864-48C5-844E-D627F056115C@tzi.org> <ba75c5c6-48f9-c871-ef66-1bf743ddcdf5@nostrum.com> <007a01d6458d$9e469760$dad3c620$@acm.org>
From: Robert Sparks <rjsparks@nostrum.com>
Message-ID: <ce30508e-3af3-7486-2bf4-38b8c83981ca@nostrum.com>
Date: Thu, 18 Jun 2020 11:31:46 -0500
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Thunderbird/68.9.0
MIME-Version: 1.0
In-Reply-To: <007a01d6458d$9e469760$dad3c620$@acm.org>
Content-Language: en-US
Subject: Re: [rfc-i] Table of conformance requirements.
X-BeenThere: rfc-interest@rfc-editor.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "A list for discussion of the RFC series and RFC Editor functions." <rfc-interest.rfc-editor.org>
List-Unsubscribe: <https://www.rfc-editor.org/mailman/options/rfc-interest>, <mailto:rfc-interest-request@rfc-editor.org?subject=unsubscribe>
List-Archive: <http://www.rfc-editor.org/pipermail/rfc-interest/>
List-Post: <mailto:rfc-interest@rfc-editor.org>
List-Help: <mailto:rfc-interest-request@rfc-editor.org?subject=help>
List-Subscribe: <https://www.rfc-editor.org/mailman/listinfo/rfc-interest>, <mailto:rfc-interest-request@rfc-editor.org?subject=subscribe>
Content-Transfer-Encoding: base64
Content-Type: text/plain; charset="utf-8"; Format="flowed"
Errors-To: rfc-interest-bounces@rfc-editor.org
Sender: "rfc-interest" <rfc-interest-bounces@rfc-editor.org>

On 6/18/20 11:29 AM, Larry Masinter wrote:
> A previous proposal
> https://tools.ietf.org/html/draft-ietf-newtrk-interop-reports-00
> and experiment
> https://tools.ietf.org/html/draft-saintandre-xmpp-interop-report-00
Yeah, well. Fwiw: https://datatracker.ietf.org/doc/rfc5657/
>
>
> --
> https://LarryMasinter.net https://going-remote.info
>
>> -----Original Message-----
>> From: rfc-interest <rfc-interest-bounces@rfc-editor.org> On Behalf Of
>> Robert Sparks
>> Sent: Thursday, June 18, 2020 6:38 AM
>> To: rfc-interest@rfc-editor.org
>> Subject: Re: [rfc-i] Table of conformance requirements.
>>
>> I have a script that builds such lists, and have used it for 20-ish years at this
>> point in various contexts.
>>
>> Some example output:
>>
>> https://www.nostrum.com/~rjsparks/draft-ietf-quick-transport-29-
>> enumreqs.txt
>>
>> https://www.nostrum.com/~rjsparks/rfc3261-enumreqs.txt
>>
>> For the most part, its best use has been to help identify when there are
>> meaningless, duplicate, or contradictory uses of the BCP14 keywords.
>>
>> In interop testing (primarily the SIPit), having these requirements
>> enumerated this way didn't help drive test behavior or the creation of test
>> suites. It turned out that far more context was needed as a predicate to "I'm
>> going to check this thing" than the isolated sentence containing the
>> requirement. And agreeing on that context often turned out to be a
>> subjective exercise (when that happened, I'd help put pressure on changing
>> the spec to reduce the probability of different interpretations).
>>
>> So, it's been somewhat helpful in my experience, but not helpful enough to
>> try to build extra support for it into, say, the v3 grammar.
>>
>> RjS
>>
>> On 6/18/20 2:08 AM, Carsten Bormann wrote:
>>> There is a whole industry around requirements tracking.
>>> I’m sure the Ribose people can tell us more about that.
>>>
>>>> Each MUST in a specification should have at least one corresponding unit
>> test to check compliance.
>>> Yeah sure.
>>>
>>> RFC 4120:
>>>      *  Principals MUST keep their secret keys secret.
>>>
>>> I’d love to see that unit test :-)
>>>
>>> The other problem is that we simply don’t state all requirements in BCP14
>> language.
>>> Very often, most requirements are stated in describing an architecture or a
>> protocol; they are phrased as statements of fact.  Extracting and labeling
>> these requirements for requirements tracking is an art form.
>>>> And this needs to be expressible somehow in XML.
>>> I played around some with using kramdown’s auto-indexing feature for
>> BCP14 keywords.  With today’s xml2rfc, that gives you an index of BCP14
>> usage.  Not exactly what you want, but just a few lines of kramdown
>> markdown.
>>> I never used the results of these experiments in a document, mainly
>> because the lack of BCP14 language on so many of the actual requirements
>> made the result too sketchy.
>>> Grüße, Carsten
>>>
>>> _______________________________________________
>>> rfc-interest mailing list
>>> rfc-interest@rfc-editor.org
>>> https://www.rfc-editor.org/mailman/listinfo/rfc-interest
>> _______________________________________________
>> rfc-interest mailing list
>> rfc-interest@rfc-editor.org
>> https://www.rfc-editor.org/mailman/listinfo/rfc-interest
_______________________________________________
rfc-interest mailing list
rfc-interest@rfc-editor.org
https://www.rfc-editor.org/mailman/listinfo/rfc-interest