Re: [ippm] Genart last call review of draft-ietf-ippm-pam-06

Greg Mirsky <gregimirsky@gmail.com> Wed, 18 October 2023 16:53 UTC

Return-Path: <gregimirsky@gmail.com>
X-Original-To: ippm@ietfa.amsl.com
Delivered-To: ippm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 6AC3DC15152F; Wed, 18 Oct 2023 09:53:34 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -7.093
X-Spam-Level:
X-Spam-Status: No, score=-7.093 tagged_above=-999 required=5 tests=[AC_DIV_BONANZA=0.001, BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-5, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_HTML_ATTACH=0.01, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_BLOCKED=0.001, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id M-oj8jHiSRTz; Wed, 18 Oct 2023 09:53:30 -0700 (PDT)
Received: from mail-yb1-xb2b.google.com (mail-yb1-xb2b.google.com [IPv6:2607:f8b0:4864:20::b2b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 3D7ECC15152E; Wed, 18 Oct 2023 09:53:30 -0700 (PDT)
Received: by mail-yb1-xb2b.google.com with SMTP id 3f1490d57ef6-d9ca471cf3aso260958276.2; Wed, 18 Oct 2023 09:53:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1697648009; x=1698252809; darn=ietf.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=ZoP0s709BErKx8PzSNiRXLJIYwnA4uFPyQQWgurKxoM=; b=d1YNAcgxm2clcOYBi5lmWo5z5MLj3NiG2XSfH8djAOhHtX64PlEHIKojhbeExtet6B nhAPeqAc4D0NRIRBUC38fTKc3IJ84hgUkF+Kfa280QwQIOB+iId7y1sftSF74/6++lOO uMki1zQJCWSrZQfY4cKND3dJTsgt2pg7TS3vPhw47Jrpqh1gMDC3MObY4qtpf96J+Qj5 kUCrI2tZAw4/0yirIaeiJsP84RRW53SrrqvYwPVJoLQnuMsgOYIJeMEpvKu51e9KK8dR LaX82zJrUpwe4eT8R6StVwySPDsD4fEoKW1BDOY4uAPO3ggXxE3NYJSvI+qewYZ9vj+G KdcA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697648009; x=1698252809; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ZoP0s709BErKx8PzSNiRXLJIYwnA4uFPyQQWgurKxoM=; b=SK9Ww2OfEw0rT5Vs0ZCZruyDxWJfYJpK43KpJ2r4LWw7UEOPgRRpIubFBxyT5VLFW7 r2+Y4EyrskkeeNI8DXa/x1g9cSIpkJZubPzcWci04QJHffI1YR0y3qdW+quzZRZ0Z8F1 7JEv+zbgmMpC22AVDgNipjLsd1HX/E/lh6yAsELmanhyJad4l3cZUKCRaokyKOIFoCNs nBkQIMkwH8txoobyEFFZH4Ig50ngnFTyqRrlfZTzoA8PyygomSOp/28X0cniZwK42wut VEiyfg+qtjqLQvkk4Xnp+trguGI541cjOAnzuK42SgZzJDA61wKm4vLXn9V15mRvktu2 dJQw==
X-Gm-Message-State: AOJu0Yz3A9Sfj51H5yJylsMQaZ92u2A1I0OQVrIPM2eYPNdSzdBM9odZ mAt6ZBIvzGp+Vmux4prNlOlsL8wMMBzGSf25QcR9kGf1
X-Google-Smtp-Source: AGHT+IFLtq+/9fzC1KnEg4O8k1eiHWMkVvXUaNIOEfAC+g+FwGNdWgW+q/mGzNRQSueOjrL4f9SKvQLA6HNZNC+gCyw=
X-Received: by 2002:a25:84c8:0:b0:d43:a84f:a6aa with SMTP id x8-20020a2584c8000000b00d43a84fa6aamr5326256ybm.39.1697648008965; Wed, 18 Oct 2023 09:53:28 -0700 (PDT)
MIME-Version: 1.0
References: <169691296935.62200.10650014107445592447@ietfa.amsl.com> <CA+RyBmXSBON4j3K0EXV6w1PpzP2-zZWVm3k6ZUn6wbcCFH5R9A@mail.gmail.com> <007801d9fdf0$4388afb0$ca9a0f10$@akayla.com>
In-Reply-To: <007801d9fdf0$4388afb0$ca9a0f10$@akayla.com>
From: Greg Mirsky <gregimirsky@gmail.com>
Date: Wed, 18 Oct 2023 09:53:17 -0700
Message-ID: <CA+RyBmVBNOwzxv_+aDSh6L=8AdL0GdE7-6e+joPgTcrP5UmiAg@mail.gmail.com>
To: peter@akayla.com
Cc: gen-art <gen-art@ietf.org>, draft-ietf-ippm-pam.all@ietf.org, IETF IPPM WG <ippm@ietf.org>, Last Call <last-call@ietf.org>
Content-Type: multipart/mixed; boundary="000000000000874e400608007af8"
Archived-At: <https://mailarchive.ietf.org/arch/msg/ippm/5wP4yGhprzKhGt8O3o2gCJaGaLs>
Subject: Re: [ippm] Genart last call review of draft-ietf-ippm-pam-06
X-BeenThere: ippm@ietf.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: IETF IP Performance Metrics Working Group <ippm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/ippm>, <mailto:ippm-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ippm/>
List-Post: <mailto:ippm@ietf.org>
List-Help: <mailto:ippm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ippm>, <mailto:ippm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 18 Oct 2023 16:53:34 -0000

Hi Peter,
thank you for your kind consideration of my notes. I have added a few new
notes below under the GIM2>> tag. Attached, is the diff highlighting the
updates resulting from our discussion.

Regards,
Greg

On Fri, Oct 13, 2023 at 9:13 AM <peter@akayla.com> wrote:

> Greg,
>
>
>
> Thanks for considering my review comments. My responses to specific points
> are found below.
>
>
>
>                                 Kind regards,
>
>                                 -Peter
>
>
>
> *From:* Greg Mirsky <gregimirsky@gmail.com>
> *Sent:* Wednesday, October 11, 2023 1:54 AM
> *To:* Peter Yee <peter@akayla.com>
> *Cc:* gen-art <gen-art@ietf.org>; draft-ietf-ippm-pam.all@ietf.org; IETF
> IPPM WG <ippm@ietf.org>; Last Call <last-call@ietf.org>
> *Subject:* Re: Genart last call review of draft-ietf-ippm-pam-06
>
>
>
> Hi Peter,
>
> thank you for your thorough review and detailed comments. Please find my
> notes below under the GIM>> tag. I've attached the diff highlighting all
> the updates.
>
>
>
> Regards,
>
> Greg
>
>
>
> On Tue, Oct 10, 2023 at 5:42 AM Peter Yee via Datatracker <
> noreply@ietf.org> wrote:
>
> Reviewer: Peter Yee
> Review result: Ready with Issues
>
> I am the assigned Gen-ART reviewer for this draft. The General Area
> Review Team (Gen-ART) reviews all IETF documents being processed
> by the IESG for the IETF Chair.  Please treat these comments just
> like any other last call comments.
>
> For more information, please see the FAQ at
>
> <https://wiki.ietf.org/en/group/gen/GenArtFAQ>.
>
> Document: draft-ietf-ippm-pam-06
> Reviewer: Peter Yee
> Review Date: 2023-10-09
> IETF LC End Date: 2023-10-20
> IESG Telechat date: Not scheduled for a telechat
>
> Summary: This specification offers a scheme for metrics that are used in
> determining if what you’re getting on a network is what you contracted
> for. I
> have some issues with the draft and the usual nits. [Ready with issues]
>
> Major issues: None
>
> Minor issues:
>
> General: For something that's Standards Track, I find the heavy use of
> "can",
> "could", and “might” leaving me wondering if this specification should be
> Informational instead. It all seems a bit loose. You could do this or
> that, if
> you wanted… I get that the specific metrics are defined here, but even so,
> there's not a lot of meat on the bone.
>
> GIM>> As the result of the discussion with our AD Martin Duke, the -07
> version of the document was set onto the Informational track
> <https://mailarchive.ietf.org/arch/msg/ippm/E5FEuv5gQ3_rwotJCXpwascEsuQ/>.
> In your opinion, the wording you've pointed out could be appropriate for an
> informational document?
>
>
>
> PEY> The wording used and the lack of specificity made me think the
> document was useful conceptually, giving guidance to the general idea of
> PAMs but not something from which I would directly drive an implementation.
>
>
>
>
> Page 4, 1st full paragraph, 2nd sentence: I've not followed this work, so
> pardon me if I'm repeating a previous argument. However, I would find terms
> like "compliance", "compliant", and "non-compliant" (maybe "incompliant")
> preferable over "availability", "available", and "unavailable". Sure, you
> get
> PCM for an initialism instead of a nice acronym like PAM, but "available"
> also
> has a common meaning that makes it less desirable, in my opinion.
> Compliance is
> already used in the previous paragraph and seems like it could be
> substituted
> without a lot of drama. Of course, this will also affect section 3.3,
> should
> you choose the change the terms.
>
> GIM>> We, the authors, over the course of developing the document,
> discussed the terminology among ourselves,e.g., the use of "conformance"
> and "compliance", and with the WG. Current terminology, as I understand it,
> is consistent and reflects the consensus of the WG.
>
>
>
> PEY> I’m completely fine with that and realize that I lack the knowledge
> of he discussions that went on before.
>
>
>
>
> Page 8, 2nd paragraph, 3rd sentence: I don’t find availability based on
> successive intervals all that good a way of making that determination. A
> service with an unavailability threshold set would be deemed "available"
> if it
> had a pattern of "unavailability threshold" - 1 interval violations
> followed by
> one VFI, rinse and repeat. Might you not want something like a percentage
> of
> VIs over a larger window of time instead? I realize this a couched in
> permissive language, but then the following text runs with the idea that
> successive intervals is the way to go. This also goes back to my general
> issue
> above about the looseness of this specification.
>
> GIM>> Thank you for the question. Violated Interval Ratio is one of the
> PAM metrics that could be defined as noted in the last paragraph of Section
> 3.3. That and other metrics can be used to improve the understanding of the
> state of a service regarding the defined SLO thresholds.
>
>
>
> PEY> Thanks for that answer. I think the language in that paragraph also
> reinforces my feeling that this draft is better as an Informational
> specification.
>
>
>
>
> Nits/editorial comments:
>
> Page 2, center header: I'm not sure how “PAM for Multi-SLO” applies as a
> compression of the title. “Multi-SLO” never appears in any other context
> than
> the header of the document.
>
> GIM>> The intention of the shortened title was to highlight that PAM can
> be particularly useful for services governed by more than one SLO, multiple
> SLOs, as in the full title of the document - "Precision Availability
> Metrics for Services Governed by Service Level Objectives (SLOs)". If you
> find the short title not reflecting the intent of the document, could you
> kindly suggest an alternative?
>
>
>
> PEY> Touché. I’m not entirely sure what I would use. It seemed to me that
> the document wasn’t really emphasizing the multiplicity of SLOs, although
> it certainly does use the plural for several terms such as parameter or
> SLOs. On the other hand, it also works were there a single SLO being
> measured. When reading the document, I wasn’t struck by a need for there to
> be multiple SLOs or that there was any difference in whether one or
> multiple SLOs were being measured. The SLOs seem to be independent
> variables (at least at the high level of this document), so I was surprised
> at the emphasis on the multiple part in the short title. Perhaps,
> “Guidelines for PAMs”?
>
GIM2>> Thank you for the suggestion. It helped me a lot. What if the short
name states "PAM Framework"?

>
>
> Page 4, 1st full paragraph, 3rd sentence: delete “, and which” along with
> “therefore” and if that parses better. Otherwise, the sentence feels like a
> phrase.
>
> GIM>> That text has been edited in the current version -07:
>
> OLD TEXT in -06:
>
>     "Precision" refers to the fact that services whose end-to-end service
>
>    levels are governed by SLOs, and which must therefore be precisely
>
>    delivered according to the associated quality and performance
>
>    requirements.
>
> NEW TEXT in -07:
>
>    "Precision" refers to services whose end-to-end service levels are
>
>    governed by SLOs and must be delivered precisely according to the
>
>    associated quality and performance requirements.
>
> Would you find the current version clearer and acceptable?
>
>
>
> PEY> Yes, that helps.
>
>
>
> Page 4, 2nd full paragraph, 1st sentence: change “is” to “are”.
>
> GIM>> Agreed. Also s/a separate topic/separate topics/ Right?
>
>
>
> PEY> Indeed!
>
>
>
> Page 5, section 3.1, 1st paragraph, 3rd sentence: “second” is given of an
> example of a different granularity, but that’s the granularity given in the
> second sentence, so it doesn’t make a good example of a different
> granularity.
> Decamillisecond is fine.
>
> GIM>> Removed "second or". Is that acceptable?
>
>
>
> PEY> Yes.
>
>
>
> Page 6, definitions for “VPC” and “SVPC”: How is a severely VPC different
> from
> a VPC?
>
> GIM>> Although, the applicability of these two metrics is the same, they
> reflect aspects of different severity.
>
>
>
> PEY> Okay.
>
>
>
> It’s also not clear to me from the text that the performance parameters
> for VI/SVI apply to individual packets, so how is this count generated?
> Only
> from parameters that can be applied on a per-packet basis (so not things
> like
> jitter)?
>
> GIM>> Indeed, some listed metrics in this list provided as examples of
> what could be used. Our intention is to define new metrics in the future,
> in a separate standard document. Do you see that as a reasonable approach
> for the informational document or would suggest another path forward?
>
>
>
> PEY> That approach is fine to me.
>
>
>
> Page 7, 6th and 7th bullet items: I’d change “measurement interval” to
> “measurement session” (session was used previously) or “period”.
> Otherwise, the
> term “interval” becomes unnecessarily overloaded.
>
> GIM>> I think that using "interval" in these sentences is intentional to
> reference VI, SVI, and VFI. Could you kindly suggest how we can make it
> clearer?
>
>
>
> PEY> My argument would be for “measurement session”, as used in the first
> paragraph of 3.1. Interval is used here as the period for a VI, SVI, or
> VFI. I see the session as the period covering all of the intervals.
>
GIM2>> Thank you for the clarification. I agree. Updated text as you've
suggested.

>
>
>  Id-nits says:
>
>   == There is 1 instance of lines with non-ascii characters in the
> document.
>
>
> I’m not sure where that character is and I admit to not searching for it.
>
> GIM>> It appears that the warning is triggered by 'ø' in the following in
> Acknowledgment:
>
> The authors greatly appreciate review and comments by Bjørn Ivar
>
> I hope that this is tolerable.
>
>
>
> PEY> Completely. I just couldn’t easily find (on the particular OS I was
> using) what was causing idnits to complain.
>