Re: [lmap] FW: AD review of draft-ietf-lmap-use-cases-03

[Benoit Claise <[e-mail]>]

Hi Phil,

I'm happy with the way you addressed my feedback.
Just one point needs correction: the last one.

    Pervasive Monitoring, RFC 7528. This RFC should at least be
    referenced, for example around this sentence:

        The other approach is passive measurements

        on the customer's ordinary traffic; the advantage is that it measures

        what the customer actually does, but it creates extra variability

        (different traffic mixes give different results) and especially it

        raises privacy concerns.

It's actually my fault. Wrong reference! it should be RFC 7258

Regards, Benoit
> cfi - in case there needs to be any discussion during Monday's 
> interim, here are the proposed resolutions to Benoit's comments.
> best wishes,
> phil
> ------------------------------------------------------------------------
> *From:* [e-mail] [[e-mail]]
> *Sent:* 03 September 2014 14:20
> *To:* [e-mail]; [e-mail]
> *Subject:* RE: [lmap] AD review of draft-ietf-lmap-use-cases-03
>
> Benoit,
>
> Thank-you very much for your review. Sorry for the slow response - 
> crossed wires amongst the authors
>
> I've proposed resolutions to all your comments below -- except for one 
> that I don't understand
>
> <<- Section 3.1
>
>     Operators want to understand the quality of experience (QoE) of their
>     broadband customers.
>
> There is a QoE reference in RFC 5481
>
> >>
>
> [phil] This was clarified:-
>
> The QoE reference is 
> actuallyhttps://tools.ietf.org/html/rfc6390#section-2.4
>
> [to try and close the issues more rapidly, am emailing the use cases 
> authors rather than lmap list.]
>
> Thanks!
>
> phil
>
> *From:*lmap [mailto:[e-mail]] *On Behalf Of *Benoit Claise
> *Sent:* 24 July 2014 15:30
> *To:* [e-mail]
> *Subject:* [lmap] AD review of draft-ietf-lmap-use-cases-03
>
> Dear all,
>
> Here is my AD review
>
> - OLD
>
> Abstract
>   
>     Measuring broadband performance on a large scale is important for
>     network diagnostics by providers and users, as well as for public
>     policy.  Understanding the various scenarios and users of measuring
>     broadband performance is essential to development of the framework,
>     information model and protocol. This document details two use cases
>     that can assist to developing that framework.  The details of the
>     measurement metrics themselves are beyond the scope of this document.
>
> NEW:
>
> Abstract
>   
>     Measuring broadband performance on a large scale is important for
>     network diagnostics by providers and users, as well as for public
>     policy.  Understanding the various scenarios and users of measuring
>     broadband performance is essential to development of the_Large-scale Measurement_
> _    of Broadband Performance (LMAP)_  framework,
>     information model and protocol. This document details two use cases
>     that can assist to developing that framework.  The details of the
>     measurement metrics themselves are beyond the scope of this document.
>
> Yes -- agree.
>
> -  Section 2.1
>
>     An ISP, or indeed another network operator, needs to understand the
>     performance of their networks, the performance of the suppliers
>     (downstream and upstream networks), the performance of services, and
>     the impact that such performance has on the experience of their
>     customers.
>
> "indeed _another _network operator"?
>
> NEW:
>
>     A network operator needsto understand the
>     performance of their networks, the performance of the suppliers
>     (downstream and upstream networks), the performance of services, and
>     the impact that such performance has on the experience of their
>     customers.
>
>
> - Section 2.1
>
>        Identifying, isolating and fixing problems in the network,
>        services or with CPE and end user equipment.
>   
> I can't parse the sentence
>   
> NEW
>        Identifying, isolating and fixingproblems, which may be in the network,
>        with the service provider, or in theend user equipment.
>   
>
>
> -  Section 2.1
>
> This sentence speaks about "services"
>
>        o Identifying, isolating and fixing problems in the network,
>        services or with CPE and end user equipment.
>   
> Section 3.2 speaks about new services.
> So, this paragraph also includes "services":
>   
>        o Understanding the impact and operation of new devices and
>        technology. As a new product is deployed, or a new technology
>        introduced into the network, it is essential that its operation
>        and impact on other services is measured. This also helps to
>        quantify the advantage that the new technology is bringing and
>        support the business case for larger roll-out.
>   
>
> NEW:
>
>        o Understanding the impact and operation of new devices,
>        technology,_or services._  As a new product is deployed, or a new technology
>        introduced into the network, it is essential that its operation
>        and impact on other services is measured. This also helps to
>        quantify the advantage that the new technology is bringing and
>        support the business case for larger roll-out.
>
> I think 'services' here & in S3.2 is mainly being used in the sense of 
> 'network services', which is confusing.
>
> NEW
>
>        o Understanding the impact and operation of new devices and
>        technology. As a new product is deployed, or a new technology
>        introduced into the network, it is essential that its operation
>        andits impact ismeasured. This also helps to
>        quantify the advantage that the new technology is bringing and
>        support the business case for larger roll-out.
>
> and in S3.2, first sentence
>
> OLD
>
> Another type of measurement is to test new capabilities and services
>
>    before they are rolled out.
>
> NEW
>
> Another type of measurement is to test new capabilities
>
>    before they are rolled out.
>
>
> - I'm confused by:
>
>   2  <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#section-2>   Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . .3  <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#page-3>
>       2.1  <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#section-2.1>  Internet Service Provider (ISP) Use Case . . . . . . . . . .3  <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#page-3>
>       2.2  <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#section-2.2>  Regulators . . . . . . . . . . . . . . . . . . . . . . . . .4  <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#page-4>
>       2.3  <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#section-2.3>  Implementation options . . . . . . . . . . . . . . . . . . .5  <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#page-5>
>
> What is the link between the "Implementation options" subsection and 
> use cases?
> Should "implementation options" have its own section?
>
> I agree. Think it fits best as a section near the end.
>
> NEW
>
> 5 <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#section-5> 
> Implementation options . . . . . . . . . . . . . . . . . . . 12 
> <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#page-12>
>
>    6  Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . 
> 12 <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#page-12>
>
>
>
> -
>
>     Another approach involves implementing the measurement capability as
>     a webpage or an "app" that end users are encouraged to download onto
>     their mobile phone or computing device. Measurements are triggered by
>     the end user, for example the user interface may have a button to
>     "test my broadband now". Compared with the previous approach, the
>     system is much more loosely controlled, as the panel of end users and
>     the schedule of tests are determined by the end users themselves
>     rather than the measurement system. It would be easier to get large-
>     scale, however it is harder to get comparable benchmarks as the
>     measurements are affected by the home network and also the population
>     is self-selecting and so potentially biased towards those who think
>     they have a problem. This could be alleviated by stimulating
>     widespread downloading of the app and careful post-processing of the
>     results to reduce biases.
>
> This end-user triggered measurement provides a huge advantage: it 
> reflects the end-user QoE, as opposed to the broadband device QoE 
> (*).  In combination with LMAP measurements in the broadband device, 
> the end-user triggered measurement might prove that the home network 
> is the source of performance degradations.
> You might want to add those points either in section "Implementation 
> options" or around the section 3.1 3rd paragraph.
> This point is partly covered in last paragraph of section 3.5
>
> (*) except if the end-user triggered measurement is executed directly 
> on the broadband device ... For example, downloading a page, or 
> initiating a test from the web server on the broadband device, as you 
> mentioned:
>
>     There are several other possibilities. For example, as a variant on
>     the first approach, the measurement capability could be implemented
>     as software embedded in the home gateway, which would make it more
>     viable to have the capability on every user line. As a variant on the
>     second approach, the end user could initiate measurements in response
>     to a request from the measurement system.
>
> NEW
>
>     Another approach involves implementing the measurement capability as
>     a webpage or an "app" that end users are encouraged to download onto
>     their mobile phone or computing device. Measurements are triggered by
>     the end user, for example the user interface may have a button to
>     "test my broadband now".One advantage of this approach is that the performance is measured to the end user, rather than to the home gateway, and so includes the home network. Another difference is thatthe
>     system is much more loosely controlled, as the panel of end users and
>     the schedule of tests are determined by the end users themselves
>     rather than the measurement system. It would be easier to get large-
>     scale, however it is harder to get comparable benchmarks as the
>     measurements are affected by the home network and also the population
>     is self-selecting and so potentially biased towards those who think
>     they have a problem. This could be alleviated by stimulating
>     widespread downloading of the app and careful post-processing of the
>     results to reduce biases.
>
>
>
> - In connection with the previous point,  I was confused while reading 
> the document.
> I don't see"end-user triggered measurement" in section 2.X next to the 
> ISP and operators use case. fine, but it's still discussed throughout 
> the draft. For example
>
>     Measurements are triggered by
>     the end user, for example the user interface may have a button to
>     "test my broadband now".
>   
>     OR
>     As a variant on the
>     second approach, the end user could initiate measurements in response
>     to a request from the measurement system.
>   
>     OR
>   
>     The operator really wants to understand the end-to-end service
>     experience. However, the home network (Ethernet, WiFi, powerline) is
>     highly variable and outside its control. To date, operators (and
>     regulators) have instead measured performance from the home gateway.
>     However, mobile operators clearly must include the wireless link in
>     the measurement.
>   
>     OR
>   
>     The operator would like the
>     end-to-end view of the service, rather than (say) just the access
>     portion
>   
>
> So, I've been wondering whether this end-user triggered measurement 
> was in scope or not.
> I only found my answer in the conclusion section
>
>     There are
>     other use cases that are not the focus of the initial LMAP charter
>     (although it is expected that the mechanisms developed would be
>     readily applied), for example end users would like to use
>     measurements to help identify problems in their home network and to
>     monitor the performance of their broadband provider.
>
> You should make it very clear, upfront, that the end-user triggered 
> measurement is not in scope.
> Something like section 5.6.1 End-user-controlled measurement system in 
> the framework document
>
> Suggest changing S1 Introduction
>
> OLD
>
>    This document describes two use cases for the Large-scale Measurement
>
>    of Broadband Performance (LMAP), in particular use cases for ISPs and
>
>    regulators.  Although there are many other use cases for large-scale
>
>    measurements systems, the two described here are the consensus
>
>    starting point for defining the system.
>
> NEW
>
>    This document describes two use cases for the Large-scale Measurement
>
>     of Broadband Performance (LMAP). Firstly, to enable network operators to understand the performance of the network and the quality experienced by customers. Secondly, to enable regulators to provide information on the performance of the ISPs in their jurisdiction. There are other use cases that are not the focus of the initial LMAP work, for example end users would like to use measurements to help identify problems in their home network and to monitor the performance of their broadband provider; it is expected that the same mechanisms are applicable.
>
>
> - Section 3.1
>
>     Operators want to understand the quality of experience (QoE) of their
>     broadband customers.
>
> There is a QoE reference in RFC 5481
>
> I don't understand this comment, sorry. I couldn't find "QoE" in RFC5481
>
>
>
> - Section 3.5
>
>     One example was described in [IETF85-Plenary]. The operator was
>     running a measurement panel for reasons discussed in sub use case #1.
>
> Should we all review [IETF85-Plenary] to understand what sub use case 
> # 1 is? ;-)
>
> This meant sub use case #1 in draft-lmap-use-cases. However, I think 
> the current text can be improved.
>
> OLD
>
>    One example was described in [IETF85-Plenary 
> <http://tools.ietf.org/html/draft-ietf-lmap-use-cases-03#ref-IETF85-Plenary>]. 
> The operator was
>
>    running a measurement panel for reasons discussed in sub use case #1.
>
>    It was noticed that the performance of some lines had unexpectedly
>
>    degraded. This led to a detailed (off-line) investigation which
>
>    discovered that a particular home gateway upgrade had caused a
>
>    (mistaken!) drop in line rate.
>
>    Another example is that occasionally some internal network management
>
>    event (like re-routing) can be customer-affecting (of course this is
>
>    unusual). This affects a whole group of customers, for instance those
>
>    on the same DSLAM. Understanding this will help an operator fix the
>
>    fault more rapidly and/or allow the affected customers to be informed
>
>    what's happening and/or request them to re-set their home hub
>
>    (required to cure some conditions). More accurate information enables
>
>    the operator to reassure customers and take more rapid and effective
>
>    action to cure the problem.
>
>    There may also be problems unique to a single user line (e.g. copper
>
>    access) that need to be identified.
>
> NEW
>
> An operator can obtain useful information without measuring the 
> performance on every broadband line. By measuring a subset, the 
> operator identify problems that affect a group of customers. For 
> example, the issue could be at a shared point in the network topology 
> (such as an exchange) or common to a vendor or equipment type 
> ([IETF85-Plenary] describes a case where a particular home gateway 
> upgrade had caused a (mistaken!) drop in line rate). A more extensive 
> deployment of the measurement capability to every broadband line would 
> enable an operator to identify issues unique to a single customer. 
> Overall, large-scale measurements can help an operator help an 
> operator fix the fault more rapidly and/or allow the affected 
> customers to be informed what's happening. More accurate information 
> enables the operator to reassure customers and take more rapid and 
> effective action to cure the problem.
>
>
> - Section 4.1
>
>     The published information needs to be:
>   
>        o  Accurate - the measurement results must be correct and not
>        influenced by errors or side effects. The results should be
>        reproducible and consistent over time.
>   
>        o  Comparable - common metrics should be used across different
>        ISPs and service offerings so that measurement results can be
>        compared.
>   
>        o  Meaningful - the metrics used for measurements need to reflect
>        what end users value about their broadband Internet access service
>   
>        o  Reliable - the number and distribution of measurement agents,
>        and the statistical processing of the raw measurement raw data,
>        needs to be appropriate
>
>
> Not sure if this was discussed, but one goal behind the regulator use 
> case is to strongly suggest the ISPs to use the same metrics in their 
> SLC (Service Level Contracts).
> If you ever tried to compare SLC between ISPs, well, go luck.
> This would be a nice additional 4.x section
>
> Rather than a new section, I suggest an additional sentence in the 
> 2^nd para of S4.1:
>
> NEW
>
>    End users need effective transparency to be able to make informed
>
>    choices throughout the different stages of their relationship with
>
>    ISPs, when selecting Internet access service offers, and when
>
>    considering switching service offer within an ISP or to an
>
>    alternative ISP. Quality information about service offers could
>
>    include speed, delay, and jitter. Regulators can publish such
>
>    information to facilitate end users' choice of service provider and
>
>    offer. It may also encourage ISPs to use the same metrics in their 
> service level contracts, which would further help end users to choose 
> an ISP. Finally, transparency may help content, application, service 
> and device
>
>    providers develop their Internet offerings.
>
>
>
> - Pervasive Monitoring, RFC 7528. This RFC should at least be 
> referenced, for example around this sentence:
>
>     The other approach is passive measurements
>     on the customer's ordinary traffic; the advantage is that it measures
>     what the customer actually does, but it creates extra variability
>     (different traffic mixes give different results) and especially it
>     raises privacy concerns.
>
>
>   Do you mean RFC6973? (Privacy Considerations for Internet Protocols)
>
> NEW
>
>     The other approach is passive measurements
>     on the customer's ordinary traffic; the advantage is that it measures
>     what the customer actually does, but it creates extra variability
>     (different traffic mixes give different results) and especially it
>     raises privacy concerns.[RFC6973] discusses privacy considerations for Internet protocols in general, whilst [framework] discusses them specifically for large-scale measurement systems.
>
> And add to references:
>
>    [RFC6973]  Cooper, A., Tschofenig, H., Aboba, B., Peterson, J.,
>
>               Morris, J., Hansen, M., and R. Smith, "Privacy
>
>               Considerations for Internet Protocols", RFC 6973 
> <http://tools.ietf.org/html/rfc6973>, July
>
>               2013.
>
> Regards, Benoit
>
> --
>
> Nits spotted:
>
> S2.2
>
> Programs /programmes (twice)
>
> S2.2, end of penultimate paragraph is missing a full stop.
>
> S3.5 Large scale measurements can help provide a more nuance view that
>
> èLarge-scale measurements can help provide a more nuanced view that
>
> S4.3 take away for LMAP purposes are that policy-makers are looking for
>
> ètake-away for LMAP purposes is that policy-makers are looking for
>
>
>
> _______________________________________________
> lmap mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/lmap