Re: [yang-doctors] automating yang doctor reviews

Mahesh Jethanandani <mjethanandani@gmail.com> Thu, 26 April 2018 19:51 UTC

Return-Path: <mjethanandani@gmail.com>
X-Original-To: yang-doctors@ietfa.amsl.com
Delivered-To: yang-doctors@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id DB8E4128D2E for <yang-doctors@ietfa.amsl.com>; Thu, 26 Apr 2018 12:51:57 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.999
X-Spam-Level:
X-Spam-Status: No, score=-1.999 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RprPCuzdl0tT for <yang-doctors@ietfa.amsl.com>; Thu, 26 Apr 2018 12:51:55 -0700 (PDT)
Received: from mail-pg0-x22d.google.com (mail-pg0-x22d.google.com [IPv6:2607:f8b0:400e:c05::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 85F33126C26 for <yang-doctors@ietf.org>; Thu, 26 Apr 2018 12:51:55 -0700 (PDT)
Received: by mail-pg0-x22d.google.com with SMTP id e12so16708857pgn.9 for <yang-doctors@ietf.org>; Thu, 26 Apr 2018 12:51:55 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=2LVmcwpNB/BMCSPxPWXZjGdDkuOGOXwo1kiRQH9EoZI=; b=Ltt5eUUtEV4oIHHobOkJuS38+WGaotYBm4ufeFsJJv9Tno+vUk8IKHrNSn5jN8SiKX by/WUH8iOWQ3sjg6A6tqIauOwGKpromEi6oPPVRztpVzDwJORkWIXfPr0u6ScPf+/Kch fWOUkXeCgHYVmNIzxJyOgIsjiRcqQZY521aYSRnyjFN2q8NqZMoPgJIIHyW725xnG23h cwCWyzTaYq+MN5s5dqvsssrfhhLR19SaQomaCSiP7pyJbKKfuQfOla/jLwlDUqaWWrc0 KQRRIErCuC3kJcAAIQY4NaKBbh40AMrufkHUZ7wrMhhHqXgk00RxArDmVEghJgoS0u8H TeWg==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=2LVmcwpNB/BMCSPxPWXZjGdDkuOGOXwo1kiRQH9EoZI=; b=p+3Cy+lUdrQwYC4dh5cTOatujxeXZ7iQngkAIUYsQ2GMclx+PdeeUKIV+8qOlcUTgT TbQibhxwJtMSatE75M3boumTldurziBRO4OdJOZqFOR0nw2xgJ6EdNJTqNv9OHZ1KiVs Kbycsdus+HRvK0E+8bRZXdSHyhaWCvCHBbzXQp/aywRLvi+CgulMV+c2gGz+YvsrIXGA YuTXY5AJjmLls/P6/RbBJlR+8+6ibFI4dYzXP+5cAPhPkNSFyd4KDuhA2ltOoQL0Lpap 7+yYy35WWYxNUlBvrBtyT87KSLz+wlDRfVCae816FYUqOYAVWJcliAEy1AFO7GOBoyvi /isA==
X-Gm-Message-State: ALQs6tCwe5AN2C8eGlyZt/Pau8XMBbro6hNcsb55FWrYLL1YHv68g+N9 Tu5kqA1z0NO8uWjdRx7Dp1g=
X-Google-Smtp-Source: AIpwx48QJnaOISa2ke4mR6sEu7YUqzFd7OQ1cJ4O0ZPPwZNOrZkIXoQL0B+R7eoGLZgDkk5ASoaahg==
X-Received: by 2002:a17:902:125:: with SMTP id 34-v6mr35877507plb.42.1524772315099; Thu, 26 Apr 2018 12:51:55 -0700 (PDT)
Received: from ?IPv6:2601:647:4700:1280:1472:5de2:3242:8a08? ([2601:647:4700:1280:1472:5de2:3242:8a08]) by smtp.gmail.com with ESMTPSA id n10sm34599438pgc.4.2018.04.26.12.51.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 26 Apr 2018 12:51:54 -0700 (PDT)
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 11.3 \(3445.6.18\))
From: Mahesh Jethanandani <mjethanandani@gmail.com>
In-Reply-To: <86441E66-EA63-4896-9BE5-DBC16F5A9BAD@cisco.com>
Date: Thu, 26 Apr 2018 12:51:53 -0700
Cc: Kent Watsen <kwatsen@juniper.net>, Martin Bjorklund <mbj@tail-f.com>, "yang-doctors@ietf.org" <yang-doctors@ietf.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <9AA65A45-32DA-44DA-9B8F-88BA492DF9C0@gmail.com>
References: <FF0CC106-9C22-4E09-8940-E759753D0AC0@juniper.net> <20180424.225703.1856993049395921058.mbj@tail-f.com> <B4E19045-7239-408E-87B5-11747BFD25B7@juniper.net> <86441E66-EA63-4896-9BE5-DBC16F5A9BAD@cisco.com>
To: Reshad Rehman <rrahman@cisco.com>
X-Mailer: Apple Mail (2.3445.6.18)
Archived-At: <https://mailarchive.ietf.org/arch/msg/yang-doctors/zGGKbvO1j0bNPn2ekHR4xMnh4bs>
Subject: Re: [yang-doctors] automating yang doctor reviews
X-BeenThere: yang-doctors@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: Email list of the yang-doctors directorate <yang-doctors.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/yang-doctors>, <mailto:yang-doctors-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/yang-doctors/>
List-Post: <mailto:yang-doctors@ietf.org>
List-Help: <mailto:yang-doctors-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/yang-doctors>, <mailto:yang-doctors-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 26 Apr 2018 19:51:58 -0000

I have borrowed heavily scripts and Makefiles that Kent and others have developed, to validate YANG modules, generate the tree diagram, and verify the examples against the YANG module BEFORE assembling them for producing the draft. This is simple to do when YANG module, examples, tree diagrams, and the draft text exist as separate files. I would recommend that we publish such a template in GitHub and have YANG module developers follow the template, perhaps with some guidance in 6087bis or in this new draft. Such steps will reduce the time a YANG doctor will need to spend reviewing the model.

This may not be possible for existing drafts, which is where I guess Kent is suggesting taking existing drafts and disassembling it so the individual files that can be validated/verified/generated.

If I had to choose, I would focus on the tools and steps for new drafts and leave existing drafts for manual review.

Cheers.

> On Apr 26, 2018, at 8:07 AM, Reshad Rahman (rrahman) <rrahman@cisco.com>; wrote:
> 
> Hi,
> 
> I'm all for "encouraging" Makefiles which generate trees, validate examples which are in separate etc. On 2 YANG models I work on, Mahesh pushed for this and it made a big difference.
> 
> For things such as whether import statements contain a reference and whether the reference is correct, it'd be good if we could have a tool to do the verification. It is very tedious work and error-prone.
> 
> Regards,
> Reshad.
> 
> On 2018-04-24, 8:56 PM, "yang-doctors on behalf of Kent Watsen" <yang-doctors-bounces@ietf.org on behalf of kwatsen@juniper.net>; wrote:
> 
>    Martin,
> 
>    Regarding the problem trying to be solved, the end of the Section 1 reads:
> 
>       While the solution presented in this document facilitates "doctor"
>       reviews (MIB, YANG, etc.), experience shows that doctor reviews are
>       often out of synch with the document submitted for publication, some
>       times by several draft revisions.  Thusly, it is currently common
>       practice for the draft's shepherd to make some attempt at verifying
>       the correctness of artwork containing structured code.  And, of
>       course, as the document progresses in the publication process, it 
>       may be subsequently updated by IESG and/or RFC Editor reviews.  By
>       enabling the verification [and tree-diagram generation] to be 
>       automated, correctness can be more assured throughout the 
>       publication process.
> 
>    That said, I'll grant you that there could be a bug in the author's
>    specification of the command line (or script) to validate the 
>    <sourcecode> element (e.g., <sourcecode validate="true"> to hack
>    it), but perhaps the YANG Doctor could at least verify that much
>    in their review, with the reasonable assumption that it will not
>    be removed (or become out-of-date) later. 
> 
>    Kent
> 
>    ===== original message =====
> 
>    Hi Kent,
> 
>    While I appreciate the effort, I'm not sure this will actually make
>    the YANG doctor's job any easier.  For example, re including the
>    command to generate the tree diagram - if people knew how to do this
>    properly, they could as easily write a Makefile (or similar) that
>    always generate the diagram so that the tree diagram is always up to
>    date.  But we know that this is not the case.  So why do we think that
>    the addition of the 'gen' attribute will make people produce good
>    drafts?
> 
>    Also, I'm not sure it helps to include specific commands like this --
>    what it tells me as a YD is that the probability that the examples
>    validate is somewhat higher than w/o this attribute.  It does not tell
>    me that the examples are valid.
> 
>    And sometimes examples and tree diagrams are manually edited for
>    formatting reason (e.g., replace chunks of noise with "...").   This
>    is difficult to capture formally, and people will still have to
>    validate examples.
> 
>    As have been pointed out, there is often dependencies to other
>    modules; capturing all this seems a bit tricky.  And the question is
>    what problem is actually solves.
> 
>    Another approach might be to encourage people to keep their draft work
>    on github, have separate files for examples, use 'make' to generate
>    tree diagrams and to validate examples.  This is less formal, but
>    achieves the same goal.
> 
> 
>    /martin
> 
> 
> 
>    Kent Watsen <kwatsen@juniper.net>; wrote:
>> 
>> Doctors,
>> 
>> Here's a stab at how we might automate the basic parts of a YANG
>> Doctor review, something I've mentioned wanting at the YD-lunch
>> meeting at the last two IETF meetings.
>> 
>> I'll be the first to say that this proposal has issues, but hopefully
>> it's in the ballpark, and we can finish it off together, assuming
>> there is interest in bringing it forward at all...
>> 
>> Note, I assume that this document will be AD-sponsored, just like RFC
>> 7991 was, hence why the draft name is what it is.
>> 
>> PS: I'm falling into a black hole, and may not reply to any responses
>> until early next week.
>> 
>> Kent
>> 
>> 
> 
> 
>    _______________________________________________
>    yang-doctors mailing list
>    yang-doctors@ietf.org
>    https://www.ietf.org/mailman/listinfo/yang-doctors
> 
> 
> _______________________________________________
> yang-doctors mailing list
> yang-doctors@ietf.org
> https://www.ietf.org/mailman/listinfo/yang-doctors

Mahesh Jethanandani
mjethanandani@gmail.com