Re: [ippm] Call for Adoption of draft-mhmcsfh-ippm-pam

[Greg Mirsky <[e-mail]>]

Dear Tommy and Marcus,
I don't see Benoit responding to Alex and my notes. That does not appear as
a constructive position. I would greatly appreciate it if you considered
the situation and Benoit's unwillingness to work with the authors of the
draft and suggested the next steps to progress this work.

Regards,
Greg

On Wed, Nov 16, 2022 at 4:32 PM Alexander L Clemm <[e-mail]> wrote:

> Hi all,
>
> I wanted to respond to this thread to address and hopefully get closure on
> Benoit's comments, inline, delimited by <ALEX>.  My apologies for the
> belated response.  In summary, I do think the scope and the problem are
> clear enough for this work to be adopted, recognizing that there is clearly
> work that remains to be done.  One area certainly for discussion concerns
> the actual set of "second order" metrics to include, but again I am
> wondering whether this determination needs to be made now versus once
> adopted.
>
> --- Alex
> On 9/14/2022 1:50 AM, Greg Mirsky wrote:
>
> Hi Benoit,
> thank you for your comments and questions. Please find my notes in-lined
> below under the GIM>> tag. I am looking forward to continuing our
> discussion.
>
> Regards,
> Greg
>
> On Mon, Sep 5, 2022 at 9:21 AM Benoit Claise <benoit.claise=
> [e-mail]> wrote:
>
>> Dear all,
>>
>> I don't dispute the importance of this work. However, the scope of this
>> work is not clear yet IMO.
>>
>>  - SLO, sure,  but what's not clear to me is: SLO per customer, per
>> service, per class of service, per flow, per application
>> I found "Precision Availability Metrics (PAM), aimed at capturing
>> end-to-end service levels for a flow, specifically the degree to which
>> flows comply with the SLOs that are in effect".
>> So OK, we speak about flow. So what is your flow definition?
>>
> GIM>> The scope of monitoring is the same as the scope of SLA that is
> composed of the set of SLOs.
>
> <ALEX> To Benoit: The SLO will apply at the level of a service instance,
> generally be at the level of the flow.  Performing metering at that level
> is straightforward; at the end of the day metrics such a Violated Intervals
> are yet another metric that could be maintained as part of flow statistics.
> That said, clearly you can construct more complex service models and the
> availability metrics will apply equally regardless of the particular scope
> with which you define the SLO.  Whatever the scope, the metrics answer the
> question whether the service that was being delivered was in fact available
> at all times, i.e. compliant with the terms of the SLO.
>
> We can refine the text, but at the end of the day I think the intent is
> pretty clear and this is one of the items that can be refined as the work
> is adopted.
>
> </ALEX>
>
> - Btw, based on the previous quoted sentence, I don't understand this PAM
>> name. No mention of SLA, no mention of flow, no notion of service.
>>  Basically, you report a service level indicator (SLI). You confused me
>> with PAM
>>
> GIM>> The intention is to report not raw SLI, i.e., measurable metric, but
> rather how the SLI is conforming to its SLO.
>
> <ALEX> PAM stands for Precision Availability Metric.  I do not think there
> is a need to get into SLAs (which contain a lot more stuff than is relevant
> here); let's focus just on the compliance of the service being delivered
> with its stated service level objective(s).  The reason we chose the term
> "availability" is because of its analogy to system availability.  We
> consider the service as "available" when it is delivered per the
> agreed-upon quality (i.e. SLO).
>
> I am not sure what you refer to with "service level indicator".  If you
> consider "system availability" a service level indicator, then sure, you
> can consider it an SLI.  To me, however, SLIs would be better reserved to
> refer to things like latency or less.  What makes the Precision
> Availability Metrics "special" is the fact that they are not "absolute"
> (i.e. RTT=276msec or such), but relative to an SLO, capturing violations.
> Regular SLIs don't do that - they simply say what was delivered, without
> indicating whether or not that was in compliance or not, which would
> require additional postprocessing.
>
> </ALEX>
>
>
>
>> - How are you going to report this flow definition, along with the SLI?
>> IPFIX key fields? With a YANG model?
>> This section 6 content is key to understand how to use those SLIs in an
>> operational environment
>>
>>    The following is a list of items for which further discussion is
>>    needed as to whether they should be included in the scope of this
>>    specification:
>>
>>    *  A YANG data model.
>>
>>    *  A set of IPFIX Information Elements.
>>
>>    *  Statistical metrics: e.g., histograms/buckets.
>>
>> GIM>> We welcome collaboration on all or any of these problems.
>
>
>> - I am not a big fan to specify some level of thresholding in
>> specifications.
>>
>>    *  VI is a time interval during which at least one of the performance
>>       parameters degraded below its pre-defined optimal level threshold.
>>
>>    *  SVI is a time interval during which at least one the performance
>>       parameters degraded below its pre-defined critical threshold.
>>
>>
>> Based on my experience, most of the time, we don't get the threshold
>> values/names right, and we don't get the number of them right.
>>     ex: violated, severely violated ... why not extremely violated,
>> catastrophically violated?
>>
> <ALEX> I do agree that we need to keep thresholding and such (which I
> consider secondary metrics) separate from the primary metrics.  The primary
> metrics are what is IMHO the most important here; secondary metrics are
> nice to have (but an add-on which offers its own complexity, may depend on
> individual operational policies, etc).  We can certainly discuss which of
> the metrics to include, but again this is a determination that IMHO should
> not make or break adoption. Re: tresholding, this is in no way central in
> this draft; we are aware of the associated issues and this may be one of
> the items that could be taken out or isolated from the other aspects.
>
> </ALEX>
>
> GIM>> Agree that it might take several iterations to set thresholds right.
> Would note that draft-ietf-teas-ietf-network-slices
> <https://datatracker.ietf.org/doc/draft-ietf-teas-ietf-network-slices/> gives
> and example of SLO in Section 4.1 using target/bound values, i.e.,
> thresholds, as following:
>    *  A Service Level Objective (SLO) is a target value or range for the
>       measurements returned by observation of an SLI.  For example, an
>       SLO may be expressed as "SLI <= target", or "lower bound <= SLI <=
>       upper bound".  A customer can determine whether the provider is
>       meeting the SLOs by performing measurements on the traffic.
>
>
>> Trying to express, from the measurement aspects, whether the observations
>> are SEVERELY impacting (that's the way I read SVI) is not the right
>> approach IMO.
>> This is maybe you open issues in section 6
>>     * Policies regarding the definition of "violated" and "severely
>> violated" time interval.
>>
> GIM>> Yes, that is our intention to further work on improving these
> definitions.
>
>>
>> Bottom line:
>> Granted, IPPM is about performance metrics but specifying metrics without
>> specifying how they will be used in an operational environment is not the
>> right way IMO.
>> I believe the scope of this document is NOT clear enough to be adopted.
>> In other words, I don't know what I'm signing for...
>>
>> <ALEX> I hope this does provide clarification.  I do think that what the
> document is trying to accomplish is sufficiently clear, the fact that some
> additional work remains to be done after adoption notwithstanding.  But we
> are not asking for Last Call, just for adoption at this point.
>
> </ALEX>
>
>
> Regards, Benoit
>>
>> On 9/1/2022 7:25 PM, Tommy Pauly wrote:
>>
>> Hello IPPM,
>>
>> As discussed at IETF 114, we’re starting an adoption call for Precision
>> Availability Metrics for SLO-Governed End-to-End Services,
>> draft-mhmcsfh-ippm-pam.
>>
>> https://datatracker.ietf.org/doc/draft-mhmcsfh-ippm-pam/
>>
>> The current version is here:
>>
>> https://www.ietf.org/archive/id/draft-mhmcsfh-ippm-pam-02.html
>>
>> Please reply to this email by *Thursday, September 15*, to indicate
>> whether you support adoption of this draft.
>>
>> Best,
>> Tommy & Marcus
>>
>>
>> _______________________________________________
>> ippm mailing [e-mail]://www.ietf.org/mailman/listinfo/ippm
>>
>>
>> _______________________________________________
>> ippm mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/ippm
>>
>
> _______________________________________________
> ippm mailing [e-mail]://www.ietf.org/mailman/listinfo/ippm
>
>