Re: [yang-doctors] automating yang doctor reviews

Kent Watsen <kwatsen@juniper.net> Wed, 25 April 2018 00:56 UTC

Return-Path: <kwatsen@juniper.net>
X-Original-To: yang-doctors@ietfa.amsl.com
Delivered-To: yang-doctors@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 56FD412D9FF for <yang-doctors@ietfa.amsl.com>; Tue, 24 Apr 2018 17:56:20 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.7
X-Spam-Level:
X-Spam-Status: No, score=-2.7 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=juniper.net
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kZNDNu6KxQpm for <yang-doctors@ietfa.amsl.com>; Tue, 24 Apr 2018 17:56:18 -0700 (PDT)
Received: from mx0b-00273201.pphosted.com (mx0b-00273201.pphosted.com [67.231.152.164]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 61F0D126D85 for <yang-doctors@ietf.org>; Tue, 24 Apr 2018 17:56:18 -0700 (PDT)
Received: from pps.filterd (m0108162.ppops.net [127.0.0.1]) by mx0b-00273201.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w3P0ru8a022035; Tue, 24 Apr 2018 17:56:17 -0700
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=juniper.net; h=from : to : cc : subject : date : message-id : references : in-reply-to : content-type : content-id : content-transfer-encoding : mime-version; s=PPS1017; bh=UJeLUyKYUSF2AKKaIIHd4YEunqco8B/KBhzuY8L6i9U=; b=fXxxq8WtEOpzQpMSHXTAf6V/3adQj727n7RGBzlB7iF9uoncUQIQS/4N+LVYq8/4NvDz vUfI2KXJ4uElhtLAR4FJ/DT47L/6/u4Xzk0NOxq8BAROsxfh+tdwMzJQctVDTSU4kHp8 UFdbUVYYmV0sE0nbSpeyHKlY+y/jpJPWcv3hzIQRnnqgEuRCZhI7m+bDAkqf98Qn8fQw m+vxwDyCCWQY7cGvU6IE6vdIDyMweNvHDgZMtcYGv6Zgb9+i9/9SSGOZspt26NNc222H loDb9To6iJwyuoBiDjWXs3bS3tUNJfQMP/P50gu8iSa88lZ4n84iSP6buZ/B41FutHKj 2w==
Received: from nam02-bl2-obe.outbound.protection.outlook.com (mail-bl2nam02lp0082.outbound.protection.outlook.com [207.46.163.82]) by mx0b-00273201.pphosted.com with ESMTP id 2hjdu4g5nn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Tue, 24 Apr 2018 17:56:16 -0700
Received: from DM5PR05MB3484.namprd05.prod.outlook.com (10.174.240.147) by DM5PR05MB3435.namprd05.prod.outlook.com (10.174.240.26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.715.7; Wed, 25 Apr 2018 00:56:13 +0000
Received: from DM5PR05MB3484.namprd05.prod.outlook.com ([fe80::173:36cf:42b7:5965]) by DM5PR05MB3484.namprd05.prod.outlook.com ([fe80::173:36cf:42b7:5965%6]) with mapi id 15.20.0715.015; Wed, 25 Apr 2018 00:56:13 +0000
From: Kent Watsen <kwatsen@juniper.net>
To: Martin Bjorklund <mbj@tail-f.com>
CC: "yang-doctors@ietf.org" <yang-doctors@ietf.org>
Thread-Topic: [yang-doctors] automating yang doctor reviews
Thread-Index: AQHT11P33vrEHgU4zkSHD96AJV2846QQbu+A////xIA=
Date: Wed, 25 Apr 2018 00:56:13 +0000
Message-ID: <B4E19045-7239-408E-87B5-11747BFD25B7@juniper.net>
References: <FF0CC106-9C22-4E09-8940-E759753D0AC0@juniper.net> <20180424.225703.1856993049395921058.mbj@tail-f.com>
In-Reply-To: <20180424.225703.1856993049395921058.mbj@tail-f.com>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
user-agent: Microsoft-MacOutlook/f.20.0.170309
x-originating-ip: [66.129.241.14]
x-ms-publictraffictype: Email
x-microsoft-exchange-diagnostics: 1; DM5PR05MB3435; 7:tTP0yD+gsz2+p7iN7ZjfNgyppfBF4ATAKJHBNhLj/Z6M8tpXDqGkGgdA8/h9s96Tvt/M8TOW0NEa0RkCzjBFpOf9GuxzQo+2+BJ2FISTWud3mhJW47aD420gn8fgSZ2MwTLOPIS6nTAaYjTFttBmUlDGisIINgz+X2c6NYklHOlUmeOymlM2tQ/05KyyqSrxIGkkYFGYtVqM2lA77x/wDKELW2q1+PawSoJ1wJRr1znJSl8F0D8uXsj2x3Ce6HyB
x-ms-exchange-antispam-srfa-diagnostics: SOS;
x-ms-office365-filtering-ht: Tenant
x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(48565401081)(5600026)(4534165)(4627221)(201703031133081)(201702281549075)(2017052603328)(7153060)(7193020); SRVR:DM5PR05MB3435;
x-ms-traffictypediagnostic: DM5PR05MB3435:
x-microsoft-antispam-prvs: <DM5PR05MB343523718D589A7A0EDF3255A58F0@DM5PR05MB3435.namprd05.prod.outlook.com>
x-exchange-antispam-report-test: UriScan:(138986009662008)(788757137089);
x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(5005006)(8121501046)(93006095)(93001095)(3002001)(10201501046)(3231232)(944501410)(52105095)(6055026)(6041310)(20161123558120)(20161123564045)(20161123560045)(20161123562045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(6072148)(201708071742011); SRVR:DM5PR05MB3435; BCL:0; PCL:0; RULEID:; SRVR:DM5PR05MB3435;
x-forefront-prvs: 06530126A4
x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(396003)(346002)(39380400002)(39860400002)(366004)(376002)(199004)(189003)(66066001)(102836004)(186003)(68736007)(486006)(476003)(33656002)(561944003)(2906002)(14454004)(11346002)(446003)(2616005)(106356001)(2900100001)(105586002)(6116002)(97736004)(3660700001)(478600001)(59450400001)(6512007)(82746002)(3280700002)(3846002)(25786009)(6916009)(7736002)(8676002)(8936002)(81166006)(81156014)(6486002)(6436002)(305945005)(5250100002)(6246003)(4326008)(5660300001)(76176011)(316002)(229853002)(83716003)(26005)(6506007)(53936002)(58126008)(86362001)(99286004)(36756003)(502464002); DIR:OUT; SFP:1102; SCL:1; SRVR:DM5PR05MB3435; H:DM5PR05MB3484.namprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1;
received-spf: None (protection.outlook.com: juniper.net does not designate permitted sender hosts)
x-microsoft-antispam-message-info: qrVSs1G4lMGMBa59B6hnkNh2ilP+ugZkrQHVoOvtNNICfYVPCmv0cWUXUXvBxofj/7k3WghWKRTeY6WNpZ/OAHn8Hb2V8RNhwQb04rLlFY75qYMdXpajU6nha4CnQjypMlX8/N7uNYUh5vbfQMne+1IlZarV+DfEIfBxtV6hjyNzsl2qMpQyK2iDSt7+jMEe
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: text/plain; charset="utf-8"
Content-ID: <253D47D3CCE98D43812D687877471964@namprd05.prod.outlook.com>
Content-Transfer-Encoding: base64
MIME-Version: 1.0
X-MS-Office365-Filtering-Correlation-Id: 03d6d4cf-83ff-4f04-aed4-08d5aa4757ee
X-OriginatorOrg: juniper.net
X-MS-Exchange-CrossTenant-Network-Message-Id: 03d6d4cf-83ff-4f04-aed4-08d5aa4757ee
X-MS-Exchange-CrossTenant-originalarrivaltime: 25 Apr 2018 00:56:13.7153 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: bea78b3c-4cdb-4130-854a-1d193232e5f4
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR05MB3435
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-04-24_07:, , signatures=0
X-Proofpoint-Spam-Details: rule=outbound_spam_notspam policy=outbound_spam score=0 priorityscore=1501 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1711220000 definitions=main-1804250006
Archived-At: <https://mailarchive.ietf.org/arch/msg/yang-doctors/H_oVwH1cxSCVTyYqQIjSwjOd6js>
Subject: Re: [yang-doctors] automating yang doctor reviews
X-BeenThere: yang-doctors@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Email list of the yang-doctors directorate <yang-doctors.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/yang-doctors>, <mailto:yang-doctors-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/yang-doctors/>
List-Post: <mailto:yang-doctors@ietf.org>
List-Help: <mailto:yang-doctors-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/yang-doctors>, <mailto:yang-doctors-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 25 Apr 2018 00:56:20 -0000

Martin,

Regarding the problem trying to be solved, the end of the Section 1 reads:

   While the solution presented in this document facilitates "doctor"
   reviews (MIB, YANG, etc.), experience shows that doctor reviews are
   often out of synch with the document submitted for publication, some
   times by several draft revisions.  Thusly, it is currently common
   practice for the draft's shepherd to make some attempt at verifying
   the correctness of artwork containing structured code.  And, of
   course, as the document progresses in the publication process, it 
   may be subsequently updated by IESG and/or RFC Editor reviews.  By
   enabling the verification [and tree-diagram generation] to be 
   automated, correctness can be more assured throughout the 
   publication process.

That said, I'll grant you that there could be a bug in the author's
specification of the command line (or script) to validate the 
<sourcecode> element (e.g., <sourcecode validate="true"> to hack
it), but perhaps the YANG Doctor could at least verify that much
in their review, with the reasonable assumption that it will not
be removed (or become out-of-date) later. 

Kent

===== original message =====

Hi Kent,

While I appreciate the effort, I'm not sure this will actually make
the YANG doctor's job any easier.  For example, re including the
command to generate the tree diagram - if people knew how to do this
properly, they could as easily write a Makefile (or similar) that
always generate the diagram so that the tree diagram is always up to
date.  But we know that this is not the case.  So why do we think that
the addition of the 'gen' attribute will make people produce good
drafts?

Also, I'm not sure it helps to include specific commands like this --
what it tells me as a YD is that the probability that the examples
validate is somewhat higher than w/o this attribute.  It does not tell
me that the examples are valid.

And sometimes examples and tree diagrams are manually edited for
formatting reason (e.g., replace chunks of noise with "...").   This
is difficult to capture formally, and people will still have to
validate examples.

As have been pointed out, there is often dependencies to other
modules; capturing all this seems a bit tricky.  And the question is
what problem is actually solves.

Another approach might be to encourage people to keep their draft work
on github, have separate files for examples, use 'make' to generate
tree diagrams and to validate examples.  This is less formal, but
achieves the same goal.


/martin



Kent Watsen <kwatsen@juniper.net>; wrote:
> 
> Doctors,
> 
> Here's a stab at how we might automate the basic parts of a YANG
> Doctor review, something I've mentioned wanting at the YD-lunch
> meeting at the last two IETF meetings.
> 
> I'll be the first to say that this proposal has issues, but hopefully
> it's in the ballpark, and we can finish it off together, assuming
> there is interest in bringing it forward at all...
> 
> Note, I assume that this document will be AD-sponsored, just like RFC
> 7991 was, hence why the draft name is what it is.
> 
> PS: I'm falling into a black hole, and may not reply to any responses
> until early next week.
> 
> Kent
>  
>