Re: [lmap] Proposed LMAP Charter

[Brian Trammell <[e-mail]>]

Greetings, all,

I'm behind the curve a bit on this thread, so I thought I'd sum up from my point of view... Please note I'm not wearing my IPPM co-chair hat here; comments on what belongs where below are personal opinion only. 

Executive summary: I broadly agree with Phil and Trevor's proposal.

There seem to be a few areas of work to build something like the union of the visions of LMAP I've seen on this list; in determining how to charter a WG, the question for each of these is does it already have a home in another WG (here I'm thinking IPPM), does it fit into a charter simple enough to make LMAP likely to complete its initial milestones in a timely fashion, or should it be declared out of scope? 

(1) A common definition of singleton metrics to be produced by a single measurement agents. In my opinion, this is key to ensuring interoperability, and is pretty clearly in the scope of IPPM; new development required by LMAP should probably be done by IPPM. As you pointed out, there are two variants of a new registry of these under consideration in IPPM, as well: draft-bagnulo-ippm-new-registry and draft-bagnulo-ippm-new-registry-independent. I repeat your call for those interested in LMAP to have a look at these and provide comments to the IPPM list.

(2) A common definition of sampling/statistics over singleton metrics to be produced by a single measurement agent. This is also pretty clearly in the scope of IPPM.

(3) A common definition of statistics over singleton metrics to be produced by multiple measurement agents. Some specific cases, especially as regards the derivation of whole-path metrics from subpath metrics or whole-interval metrics from sub-interval metrics, have already been covered by IPPM (RFC 5835 defines a Framework for Metric Composition, and RFC 6049, Spatial Composition of Metrics). As for other statistics, IPPM could provide a home for this work, depending on how close to the assumptions of RFC 5835 those statistics hold.

(4) A set of reference sample plans referencing these metrics and statistics in (1)-(3); this provides the basis for comparison among multiple tests run by different test providers. 

(5) A definition of a control protocol for setting up measurements at measurement agents, with standard representations for the sampling plans in (4), based on metrics and statistics in (1)-(3). Key for interoperability in a multi-vendor environment (as you're likely to have with measurement agents on user end systems or CPE), and more so in a multi-measurement-operator environment (even if this group decides it's out of scope). Such work also implies the definition of a reference architecture so everyone can be sure what the terminology referenced by the protocol means. This seems to be the focus of much of the energy in LMAP to date, and probably makes sense to put this central in the charter of a new working group should one be formed. This is broadly in line with the charter proposal under discussion, so, so far, so good.

(6) A definition (or selection) of a data export protocol for moving measurements from agents to collectors. If the data export protocol is part of the measurement specification the controller gives to the agent, this is perhaps less crucial (though then you do need to make sure your agent and collector protocols match at deployment time). Otherwise, this must be defined along with (5), or you have no common representation for interoperability between collectors and agents.

(7) Cross-domain reporting of results in (6) above. This adds anonymization/pseudonymization for privacy preservation and authorization to receive reports to the issues that must be addressed. 

(8) Cross-domain control of measurement agents to support (7) above. As above, plus you have a whole host of new AAA problems to solve, too. 

(9) Best practices in the management of relationships among the entities managing each piece of an LMAP infrastructure. This touches on the legal conversation on this subject, which is somewhat outside of my area of expertise, but an applicability statement is probably a good starting point here.

(10) Definition of best practices for data processing and archiving (e.g., outliner removal from statistics, specification of metadata, etc). There is some work in this area under consideration for IPPM (draft-mathis-ippm-data-curation), but no plan as yet to address this point comprehensively. 

I'm not sure where the line between IPPM and not-IPPM should be for composition of metrics across multiple measurement agents as in (3). I'm inclined to say that operation on IPPM metrics that doesn't fit into anything that we need that's not covered in existing IPPM frameworks should probably be done as new work in IPPM.  Similarly, (10) gets into the management of the data at the semantic level; as such, it's kind of tied to the definitions of the metrics (1) - (3). I'm inclined to say this is IPPM work, but maybe that's only because there's already a draft on the subject up for consideration in IPPM.

Recommendations for sample plans (4) should follow from the LMAP use cases, and as such are the first thing "up the stack" that, in my mind, is not clearly IPPM work. Definitions are only necessary to provided a basis for comparison of tests from multiple providers, but would be useful as recommendations in any case.

I think focusing on (4), then, plus the control protocol (5), and determining to what extent we can reuse/extend existing protocols for data export (6) is probably a good starting point for an LMAP charter. Cross-domain issues (7) and (8) are of interest to some potential users of the control protocol (myself included), but are probably best considered after the core is done. Work on (9) might be necessary in any case other than single-owner, single-operator infrastructures (i.e., even if you don't need interdomain control you may need to think about who's responsible for what in order to ensure the protocol supports the ownership structures you require), so that should probably appear in an applicability statement, as well. 

Best regards,

Brian

On 24 Jan 2013, at 13:53 , MORTON JR., ALFRED (AL) wrote:

> Hi Trevor,
> 
> You are right that post-singleton measurement processing is part of
> IPPM's work (singletons, samples, statistics). But LMAP could easily 
> use what has been specified in IPPM RFCs to specify some form of
> results format. So I tend to agree with Arturo.
> 
> For me, the simplest LMAP framework is the one which ensures
> the same measurements are performed according to the same sampling plan
> (schedule) and reported the same way by each organization in their network.
> Thus, there is freedom to choose the best implementation within a
> network.
> 
> regards,
> Al
> 
> 
>> -----Original Message-----
>> From: [e-mail] [mailto:[e-mail]] On Behalf Of
>> [e-mail]
>> Sent: Thursday, January 24, 2013 6:46 AM
>> To: [e-mail]; [e-mail]
>> Cc: [e-mail]; [e-mail]; [e-mail];
>> [e-mail]
>> Subject: Re: [lmap] Proposed LMAP Charter
>> 
>> 
>> I would argue that the scope of LMAP is to define the overall measurement
>> framework. If this framework is used to run the same (IPPM defined) tests
>> then it will yield comparable results.
>> 
>> How to process these results into equivalent statistics is post-
>> measurement and beyond the scope of LMAP (and would probably fall in the
>> remit of IPPM in any case). Similar argument for data sharing - data can
>> be anonymised/aggregated (if necessary) and shared after collection.
>> 
>> Trevor.
>> 
>>> -----Original Message-----
>>> From: Arturo Servin [mailto:[e-mail]]
>>> Sent: 24 January 2013 10:51
>>> To: [e-mail]
>>> Cc: Matthew Ford; Eardley,PL,Philip,TUB8 R; Burbridge,T,Trevor,TUB8 R;
>>> [e-mail]; Brian Trammell
>>> Subject: Re: [lmap] Proposed LMAP Charter
>>> 
>>> Hi al,
>>> 
>>> 	I was going to ask something similar.
>>> 
>>> 	For example, we have a project where we have done some
>>> measurements but those are confined to a specific region of the world. It
>>> would be interesting to share that data among other organizations doing
>> the
>>> same but today we can't because there is not a standard (that I asume we
>>> are intent to produce) and because privacy implications (that I am not
>> sure
>>> that we are addressing in the chapter).
>>> 
>>> 	So, is a WG goal to address the problem of sharing and anomyzing
>>> data (or address any privacy concern)?
>>> 
>>> Regards,
>>> as
>>> 
>>> On 23 Jan 2013, at 10:15, Brian Trammell wrote:
>>> 
>>>> hi Mat, Phil, all,
>>>> 
>>>> I agree as well, though other comments on finer points may follow. One
>>> inline below.
>>>> 
>>>> On 23 Jan 2013, at 13:07 , Matthew Ford wrote:
>>>> 
>>>>> Phil,
>>>>> 
>>>>> Nice post, which I think I completely agree with.
>>>>> 
>>>>> On 23 Jan 2013, at 09:54, [e-mail] wrote:
>>>>> 
>>>>>> *         We assume that Measurement Agent, tests Controller and
>> results
>>> Collector are all under control of same organisation (eg Samknows, FCC,
>> BT).
>>> Reason: Inter-organisation interactions (for example to share results) is
>>> better handled by business-level negotiation than a control protocol.
>>>>> 
>>>>> I think we also need an assumption that nothing will preclude having
>>> collectors, for example, be under control of third-parties. I think it is
>> worth
>>> explicitly stating that, unless there is disagreement on this point?
>>>> 
>>>> +1 on this point. I'd suggest that we leave the component
>>> ownership/control issues completely aside; i.e., we don't necessarily
>> want to
>>> preclude organization(s) from agreeing to use a measurement
>> infrastructure
>>> in place (which uses this control protocol) for sharing of intermediate
>> or final
>>> measurement results (in support of an existing business-level agreement).
>>>> 
>>>> What's _really_ important about this assumption is that it makes the
>>> problems of inter-domain measurement control and exchange explicitly out
>>> of scope for the charter, and simplifies issues of trust among the
>>> components. I think it should probably be explicitly phrased as an issue
>> of
>>> scope rather than an assumption about deployment.
>>>> 
>>>> Cheers,
>>>> 
>>>> Brian
>>>> 
>>>> 
>>>> _______________________________________________
>>>> lmap mailing list
>>>> [e-mail]
>>>> https://www.ietf.org/mailman/listinfo/lmap
>> 
>> _______________________________________________
>> lmap mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/lmap
> _______________________________________________
> lmap mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/lmap