Re: [rfc-i] Table of conformance requirements.

Larry Masinter <LMM@acm.org> Thu, 18 June 2020 17:17 UTC

Return-Path: <rfc-interest-bounces@rfc-editor.org>
X-Original-To: ietfarch-rfc-interest-archive@ietfa.amsl.com
Delivered-To: ietfarch-rfc-interest-archive@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id D406F3A0A6C; Thu, 18 Jun 2020 10:17:37 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.45
X-Spam-Level:
X-Spam-Status: No, score=-2.45 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_INVALID=0.1, DKIM_SIGNED=0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249, HTML_MESSAGE=0.001, MAILING_LIST_MULTI=-1, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=fail (2048-bit key) reason="fail (message has been altered)" header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id dwzOiysT9H3C; Thu, 18 Jun 2020 10:17:35 -0700 (PDT)
Received: from rfc-editor.org (rfc-editor.org [4.31.198.49]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id D75BF3A09F1; Thu, 18 Jun 2020 10:17:35 -0700 (PDT)
Received: from rfcpa.amsl.com (localhost [IPv6:::1]) by rfc-editor.org (Postfix) with ESMTP id 1B9A4F40709; Thu, 18 Jun 2020 10:17:30 -0700 (PDT)
X-Original-To: rfc-interest@rfc-editor.org
Delivered-To: rfc-interest@rfc-editor.org
Received: from localhost (localhost [127.0.0.1]) by rfc-editor.org (Postfix) with ESMTP id 3B0A4F406F6 for <rfc-interest@rfc-editor.org>; Thu, 18 Jun 2020 10:17:29 -0700 (PDT)
X-Virus-Scanned: amavisd-new at rfc-editor.org
Authentication-Results: rfcpa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from rfc-editor.org ([127.0.0.1]) by localhost (rfcpa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id E2KIjxq2i3Ot for <rfc-interest@rfc-editor.org>; Thu, 18 Jun 2020 10:17:24 -0700 (PDT)
Received: from mail-pf1-f196.google.com (mail-pf1-f196.google.com [209.85.210.196]) by rfc-editor.org (Postfix) with ESMTPS id A511EF40718 for <rfc-interest@rfc-editor.org>; Thu, 18 Jun 2020 10:17:24 -0700 (PDT)
Received: by mail-pf1-f196.google.com with SMTP id j12so848053pfn.10 for <rfc-interest@rfc-editor.org>; Thu, 18 Jun 2020 10:17:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:references:in-reply-to:subject:date:message-id :mime-version:content-language:thread-index; bh=vZ3pxZgCPn+0wERKGiaNSUqk59MAnwDxwVT1W6jaxgc=; b=CyeX008oB6fJPnFLhOp1BVlnYgHzEe+HY9EB4x3wwXknqzQ/XtGsSBwqKn0p1TAwdV z835/CfCxTD/nXC0iY1qxVbtQY2EjV4Jo2jeAdAseNaoZbWbilBvvsNEiZ9z2doXn0GB gXAJNN0B2ttyb8B/57ITazczUnK5tl13SDXN2X7fqkUYBZL0rLdZeNy7jzWJvxFiZemV nGJQp9GNnG6K/HQ5Tm6KP5Ls7g7spRuDGVl3yUk31r/yZZmQ+N87eZr1h5k0/1WE2kXV au+HIZ117Z82bDZubCsFmRObo8ZXneVfQDLzgIUcYdFlmj8vWRufnGN3hLkrHYME2xnK RouQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:references:in-reply-to:subject :date:message-id:mime-version:content-language:thread-index; bh=vZ3pxZgCPn+0wERKGiaNSUqk59MAnwDxwVT1W6jaxgc=; b=duK06VFHmkx6YBpkiihxkFvWazEnwlvjff01wDWN53EKqsr931RTMjhng0aRAgoHkL KgkCMe7kBBVe18SDyllaQwHoawQ3nRo38dt5MOWEw7Gxehk3j9AE5EM4nEh9bRbQRuhy vZpJ/wdhrkUnUZys06rcz8sdEcWIGAaX17+THlsgJaVOdVqZ2ln/gz72bivPbcj8Ao8r ku9oxJUt/hMHjf30/rajGGM8vEme35bkwWLiJZCAs3iJs+xKmgr9gc2Jmj20elDfyt6P 02C7ZW9b/KKb7o8P6i66cT4FVFDfKUxsQcgr6R+6vYkQ2jSBCOcP8xPeRn6ZvqYfXprJ k/NQ==
X-Gm-Message-State: AOAM531ZjM2HzOLu9NVQsme/VP5zDMuB+asWj9/d0HBDD1eF3kK7mvPc QVlG4iK3PQrPTD1lKX5Bw2E=
X-Google-Smtp-Source: ABdhPJw3cU5NX2NBrIC0AiDsns17R5kW6WctEU1pv6kI2A9w2oWXoFl4YrBDCl8cmxUY656vCFz4sw==
X-Received: by 2002:a62:9246:: with SMTP id o67mr4333815pfd.109.1592500589528; Thu, 18 Jun 2020 10:16:29 -0700 (PDT)
Received: from TVPC (c-67-169-101-78.hsd1.ca.comcast.net. [67.169.101.78]) by smtp.gmail.com with ESMTPSA id z24sm3497833pfk.29.2020.06.18.10.16.28 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 18 Jun 2020 10:16:28 -0700 (PDT)
From: Larry Masinter <LMM@acm.org>
X-Google-Original-From: "Larry Masinter" <lmm@acm.org>
To: "'Christian Huitema'" <huitema@huitema.net>, "'Robert Sparks'" <rjsparks@nostrum.com>, <rfc-interest@rfc-editor.org>
References: <CAMm+LwiMOHMWcxFCYMdW_fsWsPpkC0vTt_0=+MzQfCm4qy=PTw@mail.gmail.com> <D6A8EDCA-D864-48C5-844E-D627F056115C@tzi.org> <ba75c5c6-48f9-c871-ef66-1bf743ddcdf5@nostrum.com> <007a01d6458d$9e469760$dad3c620$@acm.org> <ce30508e-3af3-7486-2bf4-38b8c83981ca@nostrum.com> <075cb077-9104-e3b3-6307-7fe160bd76e2@huitema.net>
In-Reply-To: <075cb077-9104-e3b3-6307-7fe160bd76e2@huitema.net>
Date: Thu, 18 Jun 2020 10:16:27 -0700
Message-ID: <00d901d64594$34439910$9ccacb30$@acm.org>
MIME-Version: 1.0
X-Mailer: Microsoft Outlook 16.0
Content-Language: en-us
Thread-Index: AQKsY7PWm7dz3N0N9w3eZk85D/xO4ALKabyUApBqIBMCAkm44AHHDXEkAYJ3Ll2m3STG4A==
Subject: Re: [rfc-i] Table of conformance requirements.
X-BeenThere: rfc-interest@rfc-editor.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "A list for discussion of the RFC series and RFC Editor functions." <rfc-interest.rfc-editor.org>
List-Unsubscribe: <https://www.rfc-editor.org/mailman/options/rfc-interest>, <mailto:rfc-interest-request@rfc-editor.org?subject=unsubscribe>
List-Archive: <http://www.rfc-editor.org/pipermail/rfc-interest/>
List-Post: <mailto:rfc-interest@rfc-editor.org>
List-Help: <mailto:rfc-interest-request@rfc-editor.org?subject=help>
List-Subscribe: <https://www.rfc-editor.org/mailman/listinfo/rfc-interest>, <mailto:rfc-interest-request@rfc-editor.org?subject=subscribe>
Content-Type: multipart/mixed; boundary="===============9084978308202992850=="
Errors-To: rfc-interest-bounces@rfc-editor.org
Sender: "rfc-interest" <rfc-interest-bounces@rfc-editor.org>

At the time, and also now, being clear enough about conformance requirements would be a requirement for the STD series, not the RFC series.

 

Given that you can’t just concatenate XML of separate RFCs to make a STD which has more than one component, there is a need for some other kind of way of specifying Internet Standards.

I imagine  Internet Standards which are a document series which might start out as pointers to the relevant RFCs (and errata) but which could be updated as the standard evolved.

 

 

 

--

 <https://larrymasinter.net/> https://LarryMasinter.net  <https://going-remote.info/> https://going-remote.info

 

From: rfc-interest <rfc-interest-bounces@rfc-editor.org> On Behalf Of Christian Huitema
Sent: Thursday, June 18, 2020 9:42 AM
To: Robert Sparks <rjsparks@nostrum.com>om>; Larry Masinter <LMM@acm.org>rg>; rfc-interest@rfc-editor.org
Subject: Re: [rfc-i] Table of conformance requirements.

 

I think Robert's example shows the limit of the exercise. Take this requirement from the list:

  MUST-3: There are certain operations that an application **MUST** be able to perform when interacting with QUIC streams.

How are you going to test that snippet of text? In the original document, what follows is the list of these operations that the application is supposed to perform. But if you just take out of context the snippet of text surrounding the MUST statement, you get something meaningless.

 

-- Christian Huitema

 

On 6/18/2020 9:31 AM, Robert Sparks wrote:


On 6/18/20 11:29 AM, Larry Masinter wrote: 



A previous proposal 
https://tools.ietf.org/html/draft-ietf-newtrk-interop-reports-00 
and experiment 
https://tools.ietf.org/html/draft-saintandre-xmpp-interop-report-00 

Yeah, well. Fwiw: https://datatracker.ietf.org/doc/rfc5657/ 





-- 
https://LarryMasinter.net https://going-remote.info 




-----Original Message----- 
From: rfc-interest  <mailto:rfc-interest-bounces@rfc-editor.org> <rfc-interest-bounces@rfc-editor.org> On Behalf Of 
Robert Sparks 
Sent: Thursday, June 18, 2020 6:38 AM 
To: rfc-interest@rfc-editor.org <mailto:rfc-interest@rfc-editor.org>  
Subject: Re: [rfc-i] Table of conformance requirements. 

I have a script that builds such lists, and have used it for 20-ish years at this 
point in various contexts. 

Some example output: 

https://www.nostrum.com/~rjsparks/draft-ietf-quick-transport-29- 
enumreqs.txt 

https://www.nostrum.com/~rjsparks/rfc3261-enumreqs.txt 

For the most part, its best use has been to help identify when there are 
meaningless, duplicate, or contradictory uses of the BCP14 keywords. 

In interop testing (primarily the SIPit), having these requirements 
enumerated this way didn't help drive test behavior or the creation of test 
suites. It turned out that far more context was needed as a predicate to "I'm 
going to check this thing" than the isolated sentence containing the 
requirement. And agreeing on that context often turned out to be a 
subjective exercise (when that happened, I'd help put pressure on changing 
the spec to reduce the probability of different interpretations). 

So, it's been somewhat helpful in my experience, but not helpful enough to 
try to build extra support for it into, say, the v3 grammar. 

RjS 

On 6/18/20 2:08 AM, Carsten Bormann wrote: 



There is a whole industry around requirements tracking. 
I’m sure the Ribose people can tell us more about that. 




Each MUST in a specification should have at least one corresponding unit 

test to check compliance. 



Yeah sure. 

RFC 4120: 
     *  Principals MUST keep their secret keys secret. 

I’d love to see that unit test :-) 

The other problem is that we simply don’t state all requirements in BCP14 

language. 



Very often, most requirements are stated in describing an architecture or a 

protocol; they are phrased as statements of fact.  Extracting and labeling 
these requirements for requirements tracking is an art form. 



And this needs to be expressible somehow in XML. 

I played around some with using kramdown’s auto-indexing feature for 

BCP14 keywords.  With today’s xml2rfc, that gives you an index of BCP14 
usage.  Not exactly what you want, but just a few lines of kramdown 
markdown. 



I never used the results of these experiments in a document, mainly 

because the lack of BCP14 language on so many of the actual requirements 
made the result too sketchy. 



Grüße, Carsten 

_______________________________________________ 
rfc-interest mailing list 
rfc-interest@rfc-editor.org <mailto:rfc-interest@rfc-editor.org>  
https://www.rfc-editor.org/mailman/listinfo/rfc-interest 

_______________________________________________ 
rfc-interest mailing list 
rfc-interest@rfc-editor.org <mailto:rfc-interest@rfc-editor.org>  
https://www.rfc-editor.org/mailman/listinfo/rfc-interest 

_______________________________________________ 
rfc-interest mailing list 
rfc-interest@rfc-editor.org <mailto:rfc-interest@rfc-editor.org>  
https://www.rfc-editor.org/mailman/listinfo/rfc-interest 

_______________________________________________
rfc-interest mailing list
rfc-interest@rfc-editor.org
https://www.rfc-editor.org/mailman/listinfo/rfc-interest