Re: [rfc-i] Table of conformance requirements.

Larry Masinter <LMM@acm.org> Thu, 18 June 2020 16:59 UTC

Return-Path: <rfc-interest-bounces@rfc-editor.org>
X-Original-To: ietfarch-rfc-interest-archive@ietfa.amsl.com
Delivered-To: ietfarch-rfc-interest-archive@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id E0FD03A0A78; Thu, 18 Jun 2020 09:59:41 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.451
X-Spam-Level:
X-Spam-Status: No, score=-2.451 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_INVALID=0.1, DKIM_SIGNED=0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.249, MAILING_LIST_MULTI=-1, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=fail (2048-bit key) reason="fail (message has been altered)" header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kddjzpk0bC5U; Thu, 18 Jun 2020 09:59:39 -0700 (PDT)
Received: from rfc-editor.org (rfc-editor.org [4.31.198.49]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id D43903A0A4D; Thu, 18 Jun 2020 09:59:39 -0700 (PDT)
Received: from rfcpa.amsl.com (localhost [IPv6:::1]) by rfc-editor.org (Postfix) with ESMTP id B846FF406D6; Thu, 18 Jun 2020 09:59:33 -0700 (PDT)
X-Original-To: rfc-interest@rfc-editor.org
Delivered-To: rfc-interest@rfc-editor.org
Received: from localhost (localhost [127.0.0.1]) by rfc-editor.org (Postfix) with ESMTP id 8AEE2F406D6 for <rfc-interest@rfc-editor.org>; Thu, 18 Jun 2020 09:30:21 -0700 (PDT)
X-Virus-Scanned: amavisd-new at rfc-editor.org
Authentication-Results: rfcpa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from rfc-editor.org ([127.0.0.1]) by localhost (rfcpa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ADhePir-s1n8 for <rfc-interest@rfc-editor.org>; Thu, 18 Jun 2020 09:30:17 -0700 (PDT)
Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) by rfc-editor.org (Postfix) with ESMTPS id 11C23F406D3 for <rfc-interest@rfc-editor.org>; Thu, 18 Jun 2020 09:30:17 -0700 (PDT)
Received: by mail-pj1-f54.google.com with SMTP id ne5so2755593pjb.5 for <rfc-interest@rfc-editor.org>; Thu, 18 Jun 2020 09:30:22 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:references:in-reply-to:subject:date:message-id :mime-version:content-transfer-encoding:content-language :thread-index; bh=HRZjDca7KMwwccTlzotLctdFfjGtDdMn+2ixYMpb2rQ=; b=EGp+Whql2HCLCaVBHdHl27EBIzCW8t58K1x9FdIiEhwS2PrOOIXTGiq09hOpQg7j4L HUuXJA0s8cKlgCiK2099MyOmSxWHDQGTtB0DLQFMfLYAKVq5mI0fobJ27++HBW8IDfnj cejq3DGrzX6VkdwawEMMMM7VTtGIeLiLNx3U4qyWaMHH3i2fS3oAH8rKSaYCLdyU+6pi X/RsMsRs52P3axMs0RbLauvhSTCJdPMfYxniGPooL1rf3dOlDA+lBRTNlV5fcmrkyfTs gSiGc+ncyyYmaQkkNDNvOXIAMUUzemRZ0ss4hlyOFNVXMtqgCYkht8QRj6f8+hX68Ndm sbaQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:references:in-reply-to:subject :date:message-id:mime-version:content-transfer-encoding :content-language:thread-index; bh=HRZjDca7KMwwccTlzotLctdFfjGtDdMn+2ixYMpb2rQ=; b=SUN4z0dqCqyeJJz6WN5ojv2P143Jh2qwUKPVfDgwBr/gXO4A8iLLkdjYtdelh92A1n YGkTx3UAN6Td9XtbyD5a8oaSNjm2MIlc5FRs3GWTqHHt6jswHbKdTIFl3SWKibdMH2t/ UbrFteQKvw805vmJepGezPWDWVaaZUrP5CHEXYGabgkSEQYF+S3jpk4meyenRlxditE+ 17RPoH15nOgiSEC1TdWZVETR9uuSDRXS3HR3v13uR4/odxXVdPH58Fl+h54RbF17RGJL tpMQ5lVBDRrgIygWnfjGrvZo7WdQcTAT4MCP4Z/Q/9OB7tx9LCLXYRvw6XgI8SJXxWXo +m7A==
X-Gm-Message-State: AOAM5338U5m3cbSs+6cv3rPvluR0Jqjb6AqYbiaSDLOmFJJyqgLP5suM Pow0wrd8wdtajb360W0yD6ghNLbE
X-Google-Smtp-Source: ABdhPJziIp2x8M5L24p3E0dqjxTedAAU5VkWGMm+kd+2nU+fmLUfPgZy9oUyk9GfhYRqwufFqS2qTw==
X-Received: by 2002:a17:902:710b:: with SMTP id a11mr4400517pll.156.1592497761173; Thu, 18 Jun 2020 09:29:21 -0700 (PDT)
Received: from TVPC (c-67-169-101-78.hsd1.ca.comcast.net. [67.169.101.78]) by smtp.gmail.com with ESMTPSA id o16sm2946059pgg.57.2020.06.18.09.29.19 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 18 Jun 2020 09:29:20 -0700 (PDT)
From: Larry Masinter <LMM@acm.org>
X-Google-Original-From: "Larry Masinter" <lmm@acm.org>
To: "'Robert Sparks'" <rjsparks@nostrum.com>, <rfc-interest@rfc-editor.org>
References: <CAMm+LwiMOHMWcxFCYMdW_fsWsPpkC0vTt_0=+MzQfCm4qy=PTw@mail.gmail.com> <D6A8EDCA-D864-48C5-844E-D627F056115C@tzi.org> <ba75c5c6-48f9-c871-ef66-1bf743ddcdf5@nostrum.com>
In-Reply-To: <ba75c5c6-48f9-c871-ef66-1bf743ddcdf5@nostrum.com>
Date: Thu, 18 Jun 2020 09:29:19 -0700
Message-ID: <007a01d6458d$9e469760$dad3c620$@acm.org>
MIME-Version: 1.0
X-Mailer: Microsoft Outlook 16.0
Content-Language: en-us
Thread-Index: AQKsY7PWm7dz3N0N9w3eZk85D/xO4ALKabyUApBqIBOnB3hToA==
X-Mailman-Approved-At: Thu, 18 Jun 2020 09:59:32 -0700
Subject: Re: [rfc-i] Table of conformance requirements.
X-BeenThere: rfc-interest@rfc-editor.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "A list for discussion of the RFC series and RFC Editor functions." <rfc-interest.rfc-editor.org>
List-Unsubscribe: <https://www.rfc-editor.org/mailman/options/rfc-interest>, <mailto:rfc-interest-request@rfc-editor.org?subject=unsubscribe>
List-Archive: <http://www.rfc-editor.org/pipermail/rfc-interest/>
List-Post: <mailto:rfc-interest@rfc-editor.org>
List-Help: <mailto:rfc-interest-request@rfc-editor.org?subject=help>
List-Subscribe: <https://www.rfc-editor.org/mailman/listinfo/rfc-interest>, <mailto:rfc-interest-request@rfc-editor.org?subject=subscribe>
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
Errors-To: rfc-interest-bounces@rfc-editor.org
Sender: "rfc-interest" <rfc-interest-bounces@rfc-editor.org>

A previous proposal
https://tools.ietf.org/html/draft-ietf-newtrk-interop-reports-00 
and experiment
https://tools.ietf.org/html/draft-saintandre-xmpp-interop-report-00 


--
https://LarryMasinter.net https://going-remote.info

> -----Original Message-----
> From: rfc-interest <rfc-interest-bounces@rfc-editor.org> On Behalf Of
> Robert Sparks
> Sent: Thursday, June 18, 2020 6:38 AM
> To: rfc-interest@rfc-editor.org
> Subject: Re: [rfc-i] Table of conformance requirements.
> 
> I have a script that builds such lists, and have used it for 20-ish years at this
> point in various contexts.
> 
> Some example output:
> 
> https://www.nostrum.com/~rjsparks/draft-ietf-quick-transport-29-
> enumreqs.txt
> 
> https://www.nostrum.com/~rjsparks/rfc3261-enumreqs.txt
> 
> For the most part, its best use has been to help identify when there are
> meaningless, duplicate, or contradictory uses of the BCP14 keywords.
> 
> In interop testing (primarily the SIPit), having these requirements
> enumerated this way didn't help drive test behavior or the creation of test
> suites. It turned out that far more context was needed as a predicate to "I'm
> going to check this thing" than the isolated sentence containing the
> requirement. And agreeing on that context often turned out to be a
> subjective exercise (when that happened, I'd help put pressure on changing
> the spec to reduce the probability of different interpretations).
> 
> So, it's been somewhat helpful in my experience, but not helpful enough to
> try to build extra support for it into, say, the v3 grammar.
> 
> RjS
> 
> On 6/18/20 2:08 AM, Carsten Bormann wrote:
> > There is a whole industry around requirements tracking.
> > I’m sure the Ribose people can tell us more about that.
> >
> >> Each MUST in a specification should have at least one corresponding unit
> test to check compliance.
> > Yeah sure.
> >
> > RFC 4120:
> >     *  Principals MUST keep their secret keys secret.
> >
> > I’d love to see that unit test :-)
> >
> > The other problem is that we simply don’t state all requirements in BCP14
> language.
> > Very often, most requirements are stated in describing an architecture or a
> protocol; they are phrased as statements of fact.  Extracting and labeling
> these requirements for requirements tracking is an art form.
> >
> >> And this needs to be expressible somehow in XML.
> > I played around some with using kramdown’s auto-indexing feature for
> BCP14 keywords.  With today’s xml2rfc, that gives you an index of BCP14
> usage.  Not exactly what you want, but just a few lines of kramdown
> markdown.
> >
> > I never used the results of these experiments in a document, mainly
> because the lack of BCP14 language on so many of the actual requirements
> made the result too sketchy.
> >
> > Grüße, Carsten
> >
> > _______________________________________________
> > rfc-interest mailing list
> > rfc-interest@rfc-editor.org
> > https://www.rfc-editor.org/mailman/listinfo/rfc-interest
> _______________________________________________
> rfc-interest mailing list
> rfc-interest@rfc-editor.org
> https://www.rfc-editor.org/mailman/listinfo/rfc-interest

_______________________________________________
rfc-interest mailing list
rfc-interest@rfc-editor.org
https://www.rfc-editor.org/mailman/listinfo/rfc-interest