Re: [ippm] Call for Adoption of draft-mhmcsfh-ippm-pam

[Alexander L Clemm <[e-mail]>]

Hi all,

I wanted to respond to this thread to address and hopefully get closure 
on Benoit's comments, inline, delimited by <ALEX>. My apologies for the 
belated response.  In summary, I do think the scope and the problem are 
clear enough for this work to be adopted, recognizing that there is 
clearly work that remains to be done.  One area certainly for discussion 
concerns the actual set of "second order" metrics to include, but again 
I am wondering whether this determination needs to be made now versus 
once adopted.

--- Alex

On 9/14/2022 1:50 AM, Greg Mirsky wrote:
> Hi Benoit,
> thank you for your comments and questions. Please find my notes 
> in-lined below under the GIM>> tag. I am looking forward to continuing 
> our discussion.
>
> Regards,
> Greg
>
> On Mon, Sep 5, 2022 at 9:21 AM Benoit Claise 
> <benoit.claise=[e-mail]> wrote:
>
>     Dear all,
>
>     I don't dispute the importance of this work. However, the scope of
>     this work is not clear yet IMO.
>
>      - SLO, sure,  but what's not clear to me is: SLO per customer,
>     per service, per class of service, per flow, per application
>     I found "Precision Availability Metrics (PAM), aimed at capturing
>     end-to-end service levels for a flow, specifically the degree to
>     which flows comply with the SLOs that are in effect".
>     So OK, we speak about flow. So what is your flow definition?
>
> GIM>> The scope of monitoring is the same as the scope of SLA that is 
> composed of the set of SLOs.

<ALEX> To Benoit: The SLO will apply at the level of a service instance, 
generally be at the level of the flow. Performing metering at that level 
is straightforward; at the end of the day metrics such a Violated 
Intervals are yet another metric that could be maintained as part of 
flow statistics. That said, clearly you can construct more complex 
service models and the availability metrics will apply equally 
regardless of the particular scope with which you define the SLO.  
Whatever the scope, the metrics answer the question whether the service 
that was being delivered was in fact available at all times, i.e. 
compliant with the terms of the SLO.

We can refine the text, but at the end of the day I think the intent is 
pretty clear and this is one of the items that can be refined as the 
work is adopted.

</ALEX>

>     - Btw, based on the previous quoted sentence, I don't understand
>     this PAM name. No mention of SLA, no mention of flow, no notion of
>     service.
>      Basically, you report a service level indicator (SLI). You
>     confused me with PAM
>
> GIM>> The intention is to report not raw SLI, i.e., measurable metric, 
> but rather how the SLI is conforming to its SLO.

<ALEX> PAM stands for Precision Availability Metric.  I do not think 
there is a need to get into SLAs (which contain a lot more stuff than is 
relevant here); let's focus just on the compliance of the service being 
delivered with its stated service level objective(s).  The reason we 
chose the term "availability" is because of its analogy to system 
availability.  We consider the service as "available" when it is 
delivered per the agreed-upon quality (i.e. SLO).

I am not sure what you refer to with "service level indicator". If you 
consider "system availability" a service level indicator, then sure, you 
can consider it an SLI.  To me, however, SLIs would be better reserved 
to refer to things like latency or less.  What makes the Precision 
Availability Metrics "special" is the fact that they are not "absolute" 
(i.e. RTT=276msec or such), but relative to an SLO, capturing 
violations.  Regular SLIs don't do that - they simply say what was 
delivered, without indicating whether or not that was in compliance or 
not, which would require additional postprocessing.

</ALEX>

>     - How are you going to report this flow definition, along with the
>     SLI? IPFIX key fields? With a YANG model?
>     This section 6 content is key to understand how to use those SLIs
>     in an operational environment
>
>             The following is a list of items for which further discussion is
>             needed as to whether they should be included in the scope of this
>             specification:
>
>             *  A YANG data model.
>
>             *  A set of IPFIX Information Elements.
>
>             *  Statistical metrics: e.g., histograms/buckets.
>
> GIM>> We welcome collaboration on all or any of these problems.
>
>     - I am not a big fan to specify some level of thresholding in
>     specifications.
>
>         *  VI is a time interval during which at least one of the performance
>            parameters degraded below its pre-defined optimal level threshold.
>
>         *  SVI is a time interval during which at least one the performance
>            parameters degraded below its pre-defined critical threshold.
>
>     Based on my experience, most of the time, we don't get the
>     threshold values/names right, and we don't get the number of them
>     right.
>         ex: violated, severely violated ... why not extremely
>     violated, catastrophically violated?
>
<ALEX> I do agree that we need to keep thresholding and such (which I 
consider secondary metrics) separate from the primary metrics.  The 
primary metrics are what is IMHO the most important here; secondary 
metrics are nice to have (but an add-on which offers its own complexity, 
may depend on individual operational policies, etc).  We can certainly 
discuss which of the metrics to include, but again this is a 
determination that IMHO should not make or break adoption. Re: 
tresholding, this is in no way central in this draft; we are aware of 
the associated issues and this may be one of the items that could be 
taken out or isolated from the other aspects.

</ALEX>

> GIM>> Agree that it might take several iterations to set thresholds 
> right. Would note that draft-ietf-teas-ietf-network-slices 
> <https://datatracker.ietf.org/doc/draft-ietf-teas-ietf-network-slices/> gives 
> and example of SLO in Section 4.1 using target/bound values, i.e., 
> thresholds, as following:
>    *  A Service Level Objective (SLO) is a target value or range for the
>       measurements returned by observation of an SLI.  For example, an
>       SLO may be expressed as "SLI <= target", or "lower bound <= SLI <=
>       upper bound".  A customer can determine whether the provider is
>       meeting the SLOs by performing measurements on the traffic.
>
>     Trying to express, from the measurement aspects, whether the
>     observations are SEVERELY impacting (that's the way I read SVI) is
>     not the right approach IMO.
>     This is maybe you open issues in section 6
>         * Policies regarding the definition of "violated" and
>     "severely violated" time interval.
>
> GIM>> Yes, that is our intention to further work on improving these 
> definitions.
>
>
>     Bottom line:
>     Granted, IPPM is about performance metrics but specifying metrics
>     without specifying how they will be used in an operational
>     environment is not the right way IMO.
>     I believe the scope of this document is NOT clear enough to be
>     adopted. In other words, I don't know what I'm signing for...
>
<ALEX> I hope this does provide clarification.  I do think that what the 
document is trying to accomplish is sufficiently clear, the fact that 
some additional work remains to be done after adoption notwithstanding.  
But we are not asking for Last Call, just for adoption at this point.

</ALEX>


>     Regards, Benoit
>
>     On 9/1/2022 7:25 PM, Tommy Pauly wrote:
>>     Hello IPPM,
>>
>>     As discussed at IETF 114, we’re starting an adoption call for
>>     Precision Availability Metrics for SLO-Governed End-to-End
>>     Services, draft-mhmcsfh-ippm-pam.
>>
>>     https://datatracker.ietf.org/doc/draft-mhmcsfh-ippm-pam/
>>
>>     The current version is here:
>>
>>     https://www.ietf.org/archive/id/draft-mhmcsfh-ippm-pam-02.html
>>
>>     Please reply to this email by *Thursday, September 15*, to
>>     indicate whether you support adoption of this draft.
>>
>>     Best,
>>     Tommy & Marcus
>>
>>
>>     _______________________________________________
>>     ippm mailing list
>>     [e-mail]
>>     https://www.ietf.org/mailman/listinfo/ippm
>
>     _______________________________________________
>     ippm mailing list
>     [e-mail]
>     https://www.ietf.org/mailman/listinfo/ippm
>
>
> _______________________________________________
> ippm mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/ippm