Re: [ippm] Genart last call review of draft-ietf-ippm-pam-06

[Greg Mirsky <[e-mail]>]

Hi Peter,
thank you for your kind consideration of my notes. I have added a few new
notes below under the GIM2>> tag. Attached, is the diff highlighting the
updates resulting from our discussion.

Regards,
Greg

On Fri, Oct 13, 2023 at 9:13 AM <[e-mail]> wrote:

> Greg,
>
>
>
> Thanks for considering my review comments. My responses to specific points
> are found below.
>
>
>
>                                 Kind regards,
>
>                                 -Peter
>
>
>
> *From:* Greg Mirsky <[e-mail]>
> *Sent:* Wednesday, October 11, 2023 1:54 AM
> *To:* Peter Yee <[e-mail]>
> *Cc:* gen-art <[e-mail]>; [e-mail]; IETF
> IPPM WG <[e-mail]>; Last Call <[e-mail]>
> *Subject:* Re: Genart last call review of draft-ietf-ippm-pam-06
>
>
>
> Hi Peter,
>
> thank you for your thorough review and detailed comments. Please find my
> notes below under the GIM>> tag. I've attached the diff highlighting all
> the updates.
>
>
>
> Regards,
>
> Greg
>
>
>
> On Tue, Oct 10, 2023 at 5:42 AM Peter Yee via Datatracker <
> [e-mail]> wrote:
>
> Reviewer: Peter Yee
> Review result: Ready with Issues
>
> I am the assigned Gen-ART reviewer for this draft. The General Area
> Review Team (Gen-ART) reviews all IETF documents being processed
> by the IESG for the IETF Chair.  Please treat these comments just
> like any other last call comments.
>
> For more information, please see the FAQ at
>
> <https://wiki.ietf.org/en/group/gen/GenArtFAQ>.
>
> Document: draft-ietf-ippm-pam-06
> Reviewer: Peter Yee
> Review Date: 2023-10-09
> IETF LC End Date: 2023-10-20
> IESG Telechat date: Not scheduled for a telechat
>
> Summary: This specification offers a scheme for metrics that are used in
> determining if what you’re getting on a network is what you contracted
> for. I
> have some issues with the draft and the usual nits. [Ready with issues]
>
> Major issues: None
>
> Minor issues:
>
> General: For something that's Standards Track, I find the heavy use of
> "can",
> "could", and “might” leaving me wondering if this specification should be
> Informational instead. It all seems a bit loose. You could do this or
> that, if
> you wanted… I get that the specific metrics are defined here, but even so,
> there's not a lot of meat on the bone.
>
> GIM>> As the result of the discussion with our AD Martin Duke, the -07
> version of the document was set onto the Informational track
> <https://mailarchive.ietf.org/arch/msg/ippm/E5FEuv5gQ3_rwotJCXpwascEsuQ/>.
> In your opinion, the wording you've pointed out could be appropriate for an
> informational document?
>
>
>
> PEY> The wording used and the lack of specificity made me think the
> document was useful conceptually, giving guidance to the general idea of
> PAMs but not something from which I would directly drive an implementation.
>
>
>
>
> Page 4, 1st full paragraph, 2nd sentence: I've not followed this work, so
> pardon me if I'm repeating a previous argument. However, I would find terms
> like "compliance", "compliant", and "non-compliant" (maybe "incompliant")
> preferable over "availability", "available", and "unavailable". Sure, you
> get
> PCM for an initialism instead of a nice acronym like PAM, but "available"
> also
> has a common meaning that makes it less desirable, in my opinion.
> Compliance is
> already used in the previous paragraph and seems like it could be
> substituted
> without a lot of drama. Of course, this will also affect section 3.3,
> should
> you choose the change the terms.
>
> GIM>> We, the authors, over the course of developing the document,
> discussed the terminology among ourselves,e.g., the use of "conformance"
> and "compliance", and with the WG. Current terminology, as I understand it,
> is consistent and reflects the consensus of the WG.
>
>
>
> PEY> I’m completely fine with that and realize that I lack the knowledge
> of he discussions that went on before.
>
>
>
>
> Page 8, 2nd paragraph, 3rd sentence: I don’t find availability based on
> successive intervals all that good a way of making that determination. A
> service with an unavailability threshold set would be deemed "available"
> if it
> had a pattern of "unavailability threshold" - 1 interval violations
> followed by
> one VFI, rinse and repeat. Might you not want something like a percentage
> of
> VIs over a larger window of time instead? I realize this a couched in
> permissive language, but then the following text runs with the idea that
> successive intervals is the way to go. This also goes back to my general
> issue
> above about the looseness of this specification.
>
> GIM>> Thank you for the question. Violated Interval Ratio is one of the
> PAM metrics that could be defined as noted in the last paragraph of Section
> 3.3. That and other metrics can be used to improve the understanding of the
> state of a service regarding the defined SLO thresholds.
>
>
>
> PEY> Thanks for that answer. I think the language in that paragraph also
> reinforces my feeling that this draft is better as an Informational
> specification.
>
>
>
>
> Nits/editorial comments:
>
> Page 2, center header: I'm not sure how “PAM for Multi-SLO” applies as a
> compression of the title. “Multi-SLO” never appears in any other context
> than
> the header of the document.
>
> GIM>> The intention of the shortened title was to highlight that PAM can
> be particularly useful for services governed by more than one SLO, multiple
> SLOs, as in the full title of the document - "Precision Availability
> Metrics for Services Governed by Service Level Objectives (SLOs)". If you
> find the short title not reflecting the intent of the document, could you
> kindly suggest an alternative?
>
>
>
> PEY> Touché. I’m not entirely sure what I would use. It seemed to me that
> the document wasn’t really emphasizing the multiplicity of SLOs, although
> it certainly does use the plural for several terms such as parameter or
> SLOs. On the other hand, it also works were there a single SLO being
> measured. When reading the document, I wasn’t struck by a need for there to
> be multiple SLOs or that there was any difference in whether one or
> multiple SLOs were being measured. The SLOs seem to be independent
> variables (at least at the high level of this document), so I was surprised
> at the emphasis on the multiple part in the short title. Perhaps,
> “Guidelines for PAMs”?
>
GIM2>> Thank you for the suggestion. It helped me a lot. What if the short
name states "PAM Framework"?

>
>
> Page 4, 1st full paragraph, 3rd sentence: delete “, and which” along with
> “therefore” and if that parses better. Otherwise, the sentence feels like a
> phrase.
>
> GIM>> That text has been edited in the current version -07:
>
> OLD TEXT in -06:
>
>     "Precision" refers to the fact that services whose end-to-end service
>
>    levels are governed by SLOs, and which must therefore be precisely
>
>    delivered according to the associated quality and performance
>
>    requirements.
>
> NEW TEXT in -07:
>
>    "Precision" refers to services whose end-to-end service levels are
>
>    governed by SLOs and must be delivered precisely according to the
>
>    associated quality and performance requirements.
>
> Would you find the current version clearer and acceptable?
>
>
>
> PEY> Yes, that helps.
>
>
>
> Page 4, 2nd full paragraph, 1st sentence: change “is” to “are”.
>
> GIM>> Agreed. Also s/a separate topic/separate topics/ Right?
>
>
>
> PEY> Indeed!
>
>
>
> Page 5, section 3.1, 1st paragraph, 3rd sentence: “second” is given of an
> example of a different granularity, but that’s the granularity given in the
> second sentence, so it doesn’t make a good example of a different
> granularity.
> Decamillisecond is fine.
>
> GIM>> Removed "second or". Is that acceptable?
>
>
>
> PEY> Yes.
>
>
>
> Page 6, definitions for “VPC” and “SVPC”: How is a severely VPC different
> from
> a VPC?
>
> GIM>> Although, the applicability of these two metrics is the same, they
> reflect aspects of different severity.
>
>
>
> PEY> Okay.
>
>
>
> It’s also not clear to me from the text that the performance parameters
> for VI/SVI apply to individual packets, so how is this count generated?
> Only
> from parameters that can be applied on a per-packet basis (so not things
> like
> jitter)?
>
> GIM>> Indeed, some listed metrics in this list provided as examples of
> what could be used. Our intention is to define new metrics in the future,
> in a separate standard document. Do you see that as a reasonable approach
> for the informational document or would suggest another path forward?
>
>
>
> PEY> That approach is fine to me.
>
>
>
> Page 7, 6th and 7th bullet items: I’d change “measurement interval” to
> “measurement session” (session was used previously) or “period”.
> Otherwise, the
> term “interval” becomes unnecessarily overloaded.
>
> GIM>> I think that using "interval" in these sentences is intentional to
> reference VI, SVI, and VFI. Could you kindly suggest how we can make it
> clearer?
>
>
>
> PEY> My argument would be for “measurement session”, as used in the first
> paragraph of 3.1. Interval is used here as the period for a VI, SVI, or
> VFI. I see the session as the period covering all of the intervals.
>
GIM2>> Thank you for the clarification. I agree. Updated text as you've
suggested.

>
>
>  Id-nits says:
>
>   == There is 1 instance of lines with non-ascii characters in the
> document.
>
>
> I’m not sure where that character is and I admit to not searching for it.
>
> GIM>> It appears that the warning is triggered by 'ø' in the following in
> Acknowledgment:
>
> The authors greatly appreciate review and comments by Bjørn Ivar
>
> I hope that this is tolerable.
>
>
>
> PEY> Completely. I just couldn’t easily find (on the particular OS I was
> using) what was causing idnits to complain.
>