[ippm] Re: Minutes from the Chicago meeting

IPPM group,

> Below you'll find a first version of the minutes of the Chicago
> meeting.  Thanks to Al Morton for taking most of the notes.  Comments
> to the chairs and/or the list before August 24 please.

There have been no comments on the minutes, so this version will be
submitted to the secretariat for the proceedings.

Henk

> 
> Matt & Henk
> 
> -=-=-=-
> IP Performance Metrics WG (ippm)
> Tuesday, July 24, 09:00--11:30 CDT
> 
> This session was chaired by Henk Uijterwaal and Matt Zekauskas. Al 
> Morton acted as
> scribe, and his notes (along with notes taken by Matt) were edited into 
> these
> minutes by the Chairs.
> 
> AGENDA:
> 
> 1. Administrivia
> 2. TWAMP updates, and next steps
> 3. Duplication Draft Update
> 4. Composition Framework and Spatial Composition Additions
> 5. Multimetrics Draft Developments
> 6. Delay Variation Draft Developments
> 7. Any Other Business
> 
> 1. Administrivia
>    Agenda bashing, Scribe, Minutes, Blue Sheets
>    Status of drafts and milestones
> 
> Henk opened the meeting.  There were no changes to the agenda.
> 
> During the discussion of drafts and milestones, Henk menioned that we 
> are still
> sitting on the implementation draft awaiting guidance on progressing 
> metrics along
> the standards track. While initially the thought was we should just 
> publish the
> docoument as informational, Lars skimmed 
> draft-bradner-metricstest-02.txt, and it
> said metrics would progress when test equipment from different vendors 
> had results
> in controlled experiment that were verifyably equivalent. However, the 
> definition
> of "verifyably equivalent" is still open, and the results of that 
> definition might
> imply that changes need to be made to the draft.  Therefore we will let 
> the draft
> sit, and encourage this group (and bmwg) to contribute to the definition of
> "verifyably equivalent".
> 
> Henk asked if Al had any idea of realistic dates for his composition 
> drafts, since
> the milestones for those drafts had passed.  Al thought that the 
> framework was in
> good shape, but didn't want to progress it before one of the composition 
> drafts
> was ready.  The big barrier is that it needs more review by working 
> group members.
> We need some rigorous reviews. December 2007 is probably too soon, 
> mid-2008 is a
> more realistic target. Folks in the group are requested to read and 
> commend on the
> mailing list.
> 
> Emile said that the multimetrics draft is more-or-less complete and has 
> been
> stable.  It too needs review, and he thought a WGLC would be the means 
> to coerce
> folks to review.  However, he has been speaking with Jurgen Quittek, and 
> Jurgen
> has a series of comments he is going to send in.  Therefore, he requests 
> that the
> working group review the document after the next revision. Al offered to 
> help with
> readability going in to the next version as well.  The big issue with 
> the draft
> are the passive aspects, which were discussed during the draft 
> discussion below.
> 
> 2. TWAMP updates, and next steps
>    --Kaynam Hedayat
> 
>    http://tools.ietf.org/id/draft-ietf-ippm-twamp-04.txt
>    http://tools.ietf.org/wg/ippm/draft-ietf-ippm-twamp/
> 
> Kaynam presented a single slide on TWAMP status.  At the last meeting 
> they agreed
> to add sneder TTL into test packets, as well as clarifying security text 
> with
> respect to TWAMP-Light, and both have been done.  In addition, text has 
> been
> clarified for sender port and receiver port.
> 
> Someone brought up setting DSCP on the ML.  What does the group feel we 
> should do?
> Matt asked if Kaynam had seen a use for it, and he said he did, but more 
> for
> one-way than two-way tests.  It seems like a fine idea, but should we 
> make such a
> change now?  No one felt strongly that we should do so.  Therefore, no 
> further
> changes for DSCP are proposed.  This question was also asked on the list 
> before
> the meeting, with no response.
> 
> There were additional clarifications that were suggested on the ML and a 
> new
> revision is in the works to address those.
> 
> Matt mentioned that he was supposed to help the authors come up with 
> some IANA
> text for OWAMP-Control commands, since this document adds a new value.  
> He said he
> would do that by the end of the week.
> 
> 3. Duplication Draft Update
>    --Henk Uijterwaal
> 
>    http://tools.ietf.org/id/draft-ietf-ippm-duplicate-01.txt
>    http://tools.ietf.org/wg/ippm/draft-ietf-ippm-duplicate/
> 
> Henk next presented updates on the duplication draft.  Based on previous 
> comments,
> a new version was created in April.  There are two statistics, and two 
> ways to
> express duplication.  This draft spun off from the original reporting 
> draft, which
> wanted to reference duplication but there was no definition.
> 
> There is a placeholder Y.1540 section.  However, references have been 
> made along
> the way, so Henk does not see the need to keep the section.  Al agreed, 
> although
> he also stated that it might not be a bad idea to see where we could 
> harmonize the
> definitions, and make the differnces go away (Y.1540 is currently under 
> revision,
> see the final topic.)
> 
> Al had reviewed the draft and made a few comments. He thought the names 
> for some
> metrics might be change to be more descriptive. For example,
> "type-P-one-way-packet-duplication-average" was new in this version and 
> didn't
> seem to capture the meaning of the metric.
> 
> Al also noted that the real singleton here is not "duplication", but an 
> arrival
> counter.  It starts at 0, and goes to 1, 2, 3, etc, for each packet.  
> Including
> "duplication" in the name seems like forming a conclusion that we 
> haven't reached
> yet.  WE should evaluate the arrival counter and see if a packet only 
> once or more
> than once. Only those that arrive more than once should be called 
> duplicates. Al
> offered to send his comments to the list.  They may be found here:
> http://www1.ietf.org/mail-archive/ippm/current/msg01174.html
> 
> 4. Composition Framework and Spatial Composition Additions
>    -- Al Morton
> 
>    http://tools.ietf.org/id/draft-ietf-ippm-framework-compagg-04.txt
>    http://tools.ietf.org/wg/ippm/draft-ietf-ippm-framework-compagg/
>    http://tools.ietf.org/id/draft-ietf-ippm-spatial-composition-04.txt
>    http://tools.ietf.org/wg/ippm/draft-ietf-ippm-spatial-composition/
> 
> Al started by reviewing the composition framework.  Overall, it's been 
> fairly
> stable; the definitions have been refined, and a maintenance update has 
> been
> performed.  However, there has not been much feedback recently.  Al said 
> he is
> declaring this stable, pending feedback or one of the composition drafts 
> to be
> complete.
> 
> Al noted the Metro Ethernet Forum is specifying some end-to-end performance
> parameters, and they are going to need a solution to this problem as well.
> 
> Jurgen Quittek and Emile Stephan offered to read the draft.
> 
> Al then did an overview of the spatial composition draft, and
> the metrics that are defined there.  The metric names have been
> simplified, allowing for easier exposition.
> 
> Scott Bradner and Loki Jorgenson offered to read this draft.
> 
> Matt admitted he had not read the draft recently, but the description 
> triggered a
> thought -- are there auxilliary metrics or parameters to specifcy when a
> composition is valid?  For example, min composition works as long as the 
> path is
> relatively stable and uncongested. If the path is congested, the min 
> will jump
> around.  Al thought that it might just mean the "error bars" are larger, 
> but you
> can still get an estimate of the minimum.
> 
> Craig White remarked: Given a large enough sample size, the minimum will 
> tend to
> be the one that occurs the most frequently, as long as the path is 
> relatively
> stable.  Even if the path is not stable, you do get some bimodal 
> distributions
> (for example if a SONET ring goes to its protect side, and then back.
> 
> Al said that bimodal distributions are dealt with in the document. You 
> don't want
> to mix them together because the minimum for one is different than the 
> other.  The
> intent is to measure for a long time, if you see some path changes, 
> segregate the
> measurements into different intervals.
> 
> Emile said that in his opinion you can't consider making an aggregation 
> that
> consists of measurements where the path changes -- the result would be 
> unusable.
> Perhaps we need to say that in the framework or in the specific metrics.
> 
> Al said that path stability needs to be mentioned when we talk about 
> measuremetn
> validity; he thought it went to an overall section since it applies to all
> parameters.
> 
> 5. Multimetrics Draft Developments
>    -- Emile Stephan
> 
>    http://tools.ietf.org/id/draft-ietf-ippm-multimetrics-04.txt
>    http://tools.ietf.org/wg/ippm/draft-ietf-ippm-multimetrics/
> 
> Emile started with a quick review of the multimetrics draft, starting 
> with the
> spatial metrics, and then multicast metrics.
> 
> For the spatial metrics, there has been some harmonization and 
> discussion of
> scalability aspects.  These metrics are used for either path composition 
> or by the
> end user.  Emile felt that these descriptions are now stable, and he 
> didn't plan
> to make more changes except to clean up wording.
> 
> The segment metrics have been renamed so that it is clear that they look at
> performance between any two hops on the network. They are purely passive.
> 
> One participant had an issue with purely passive measurements, 
> especially those
> based only on user packets.  Did we want to discuss this now or later?
> 
> Jurgen said that he had read the document, and think we run into 
> problems if we
> focus on passive measurements.  A lot of the terminology has been 
> inherited from
> other drafts, and doesn't apply to passive measurements.  If there are 
> active
> probes, there is one sender and one receiver.  If you passively examine 
> user
> traffic, there are many senders and receivers that pass by.  Emile 
> thought that
> each flow had one sender and one receiver.  There is some time when the 
> packet was
> sent, even if it is not measurable.  You want to know the performance 
> between two
> points, you don't need to know the original sending time, you just need 
> the two
> points. The terminology could be clarified.
> 
> This would also provide psamp with some purely passive metrics; the 
> section could
> be removed if it doesn't make sense.
> 
> Emile thought that 80% of the wording is OK, and we need strong input, 
> such as
> that Jurgen is giving.  He would like a WGLC to force people to read the 
> document.
> 
> Al said that he wanted to make a pass over the document first, to 
> improve grammar.
> He thought the document could benefit form an editorial pass to make it 
> more
> readable, and as a co-author felt that he was responsible for that.  
> After that,
> he agreed a WGLC would be a good idea.
> 
> Al also wanted to know the groups point of view with respect to purely 
> passive
> metrics.  There was some discussion among the authors.  The IPPM charter 
> says that
> the "metrics will not characterize traffic", and thus would not be used to
> characterize VOIP traffic, for example.  He reads this as passive is out 
> of bounds
> for the working group.  However, other people are entitled to give 
> interpretations
> of that.
> 
> Emile siad that it is not easy to mix passive and active metrics, but 
> that we
> needed some ground truth.  If we want to compose metrics, we will need 
> to make
> approximations, and we need to include passive metrics in the 
> algorithms.  The
> spatial definition relies on passive technieuqs in nodes the packet cros.
> 
> If there is an active source that provides packets that can be observed 
> along the
> path, that was still an active measurement. The discussion is about 
> qualifying the
> stream you are using for the basis of measurement.
> 
> One opinion was that we haven't done this yet, and that there is enough 
> material
> to create a separate draft on that topic.
> 
> Lars (our area director) agreed that the charter text has an underlying 
> notion
> that the metrics are based on active sampling.  The working group could 
> of course
> decide to take on passive measurements. The previous presentation goes 
> in that
> direction.  But he thought it was out of scope in the current charter, 
> and he
> would like to see us finish the metrics and related documents on active 
> techniques
> first. He thought passive measuremetns were a pretty large piece of new 
> work -- in
> short, he agreed with Al.
> 
> Emile reiterated his opinion that ignoring passive metrics is a mistake, 
> and that
> the best way to have some "ground-truth" measurements is to have somee 
> passive
> measurements to compare to.
> 
> Alan Clark said he agreed with both positions, in that if you look at 
> composition
> and define it using active measurements, the basic metrics are equally 
> applicable
> to passive techniques.  Work in composing delay, delay variation, and 
> others, can
> be picked up and used with passive measurements.  He felt that once the 
> basic
> ideas have been developed, that the group should look at passive 
> applications.
> However, it is difficult to make passive measurements of an IP stream, 
> because you
> have to decode protocols, and that runs the risk of overlapping with 
> other groups.
> 
> Al Morton thought that was an interesting point; that the working group has
> focused on IP, dabbled a bit in TCP, but has focussed on active live 
> measurements
> of the IP layer of the stack, and that's our core charter.  If you work 
> on passive
> metrics, and start to look at things like sequence numbers in higher 
> layers, that
> definitely goes beyond our charter.
> 
> Al also wanted to note that there are other gaps, places where people 
> want to grow
> the group beyond the charter, both here and in BMWG. Interested folks 
> should
> follow the "application performance metrics" BOF developments.
> 
> Jurgen said that he agreed with Emile, that it would be good to have a 
> methodology
> to apply.  However, if the working group wanted to pursue this, it would 
> take more
> than just a paragraph; there are many things to consier.  There should be a
> framework for how we do passive measurements in general, before we apply 
> them to
> any specific document.
> 
> Emile feld we had to fix this before going further in composition, and that
> something was needed now to promote comparable metrics in this space.  
> He also
> agreed that squeezing it in as a subsection of a draft isn't the right 
> place for
> this to be.
> 
> There was a comment that at least with the active measurements, we would be
> combining metrics where we understand the characteristics; if we combine 
> passive
> measurements, there may be less we can rely on to combine them 
> meaningfully.
> 
> Emile noted that companies and people will do this combination anyway, 
> so we
> should be working to defining standard metrics.
> 
> There was a bit of further discussion on how large this effort might be 
> and where
> the right place to do it would be; there was no definitive conclusion.
> 
> Next, Karsten Fleischhauer presented a multimetrics use case provided by 
> Ruediger
> Geib, who could not attend this meeting.  The goal in this case it to 
> improve
> multicast monitoring within a provider's network.  It would give an 
> operator
> increased understanding of what was happening with multicast, and an 
> easy view of
> where failures occur.
> 
> Several simple multicast trees, and failures were presented, along with 
> showing
> how the metrics, especially the segment metrics, would improve 
> understanding.
> (See slides for details.)
> 
> Emile pointed out that the current draft does not mix segment metrics and
> multicast metrics, which these examples seem to do.  Karsten said that his
> understanding is that there are no requests to change the draft, the 
> example is
> just showing how the metrics can be used together.
> 
> 
> 6. Delay Variation Draft Developments
>    -- Al Morton
> 
>    http://tools.ietf.org/wg/ippm/draft-morton-ippm-delay-var-as-03.txt
> 
> Next Al Morton presented the delay variation applicability statement 
> draft, which
> is not yet a working group item, but has had extensive review in the 
> group.  He
> spent some time presenting the problem space and showing how the 
> different metric
> formulations are useful for different problems.  See the slides for 
> details.
> 
> There was a recent section that made suggestions on what could be done 
> to reduce
> variation.  This provoked a discussion of whether the section was 
> appropriate for
> the draft.  Scott Bradner mentioned that there was a different set of 
> expertise
> involved in fixing a problem than how to describe and measure the 
> problem clearly.
> We would need to reach out to operators in different environments including
> enterprise and ISP for validation.  He thought this section was better in a
> different document, or even a Wiki to keep advice up to date.
> 
> Al noted that Roman Krzanowski, who first asked for this work, worked 
> for an
> operator, and that Al himself was "tier-5 support" at his company.
> 
> Wenji Wu asked about applicability of this work, combining it with loss to
> indicate congestion.  Al said that the draft was not trying to combine 
> delay
> variation with packet loss in this analysis; loss affects the delay 
> variation
> metric, however.  In the extreme case of losing every other packet you 
> cannot
> produce a inter-packet delay variation result.
> 
> Wenji said that if you have the minimum delay, and an indication of the 
> maximum
> delay, then you have an indication of the minimum latency from source to
> destination, and an indication of queueing. Have there been thoughts to
> standardize this application? Al said that the draft includes an 
> estimate of queue
> occupation, in orther words, this is a task that is already identified.  
> Al asked
> that Wenji read the draft and comment.
> 
> Lars made the comment that there are transport protocols that are trying 
> to infer
> congestion by looking at both delay and loss changes. They are 
> experimental.
> There are other causes for loss than congestion, and other causes for 
> delay than
> congestion, for example a wireless link changing it's rate.  He said 
> that there is
> ongoing research here but there is nothing baked enough to become a 
> metric. is
> research on this,but not ready.
> 
> Al noted that one thing the draft could talk about with respect to 
> minimum delay
> is that it is an indication of path changes and could be used to separate a
> bimodal distribution.  If we can detect that a path has changed 
> positivly by TTL
> changes, and correlate that with bursts of loss and delay change (and 
> possibly by
> reordering caused by "microloops" during a change) you may be able to 
> reset the
> minimum delay at that point.  If path changes are frequent, delay 
> variation is
> probably not your biggest problem.
> 
> Resetting the minimum at appropriate points would be good for long-term
> measurements.
> 
> Alan said that at the last meeting he raised a possible case where 
> inter-packet
> delay variation might be useful, and that is the packet video buffer 
> smoothing
> problem.  Video packets have timestamps in them, but the rate is highly 
> variable
> due to big complete frames being mixed with packets indicating changes 
> since the
> last complete frame.  There is a smoothing buffer to try and even things 
> out. You
> need to take fragmentation and MTU size into account in this case.  He's 
> thought
> about it a bit, but has no conclusion yet.
> 
> Alan also noted another interesting problem, not specific to video, is 
> where the
> bandwith of test signal, whatever it may be, is sufficient to cause some 
> delay
> variation itself.  Can you separate this test signal induction (due to 
> bandwidth
> limits in the network) from those from other sources in the network?  It 
> doesn't
> matter what the absolute value of variation is in this case.  He has 
> been doing
> some simulation that might apply, with scene characteristics of video 
> streams.
> 
> In summary, Al said that he'll go through the comparisions again, and 
> felt he was
> still trying to acheive consensus, but that he has not seen much 
> resistance to the
> basic results.
> 
> Speaking to slide 9, Alan Clarke made the observation that rather than 
> calling
> these guidelines in the document, we should call them measurement 
> consderations;
> to him guidelines are directing test vendors to do something rather than 
> giving
> them advice about things to watch out for.  Packet Delay Variation is 
> one of the
> most difficult things to measure and understand properly, so he thinks a 
> section
> like this is extremely useful.  Scott Bradner noted that his earlier 
> reaction to
> the guidance in the appendix did not apply here, measurement 
> considerations are
> good to state.
> 
> Al concluded noting that there had been significant support in the room, 
> comments
> from group members both here and on the mailing list and that this draft 
> had
> already been cited to satisfy an item the charter, so it's time to make 
> this
> document a working group item.
> 
> Henk took a quick poll of the room; about 1/2 indicated support for the 
> document.
> When asked who thought it was a bad idea to continue with this document, 
> there was
> no opposition.  Therefore Henk said the Chairs would verify the sense of 
> the room
> on the mailing list, and assuming no major change would work with the area
> director to make this a working group draft.
> 
> An group member thought the document should be broadend from just 
> focusing on
> delay variation.  Al felt that it was better to stay focused in order to 
> finish
> the document and complete the milestone.  [Later note by MZ: we have had 
> a couple
> of other starts at applicability statements that did not have enough 
> support to
> follow through to completion; others are welcome to make suggestions, 
> but we
> should just focus on finishing the document in an area where there is a 
> need and
> support from the working group.]
> 
> 7. Any Other Business
> 
> Henk mentioned a passive measurement draft that had been sent to the list:
> draft-kikuchi-passive-measure-00.txt. No group members had read the 
> draft; given
> the earlier discussion about passive measurements and the upcoming 
> "Application
> Performance Metrics BOF", it seems that the draft is currently out of 
> scope for
> this working group but potentially in scope for the results of the BOF, 
> Henk will
> send out a note to that effect.
> 
> Al Morton made a comment about his ITU-T liaison work.  Years ago, Al 
> was put in
> charge of the IP Performance question in SG 12. We have been working to 
> reduce the
> differences between the work done there and here, harmonizing the 
> results as much
> as possible.  The scope of SG 12 is different and wider -- it produces 
> numerical
> objectives based on performance perameters (from metrics); this group is 
> focussed
> on the metrics.
> 
> One way to reduce the differences between the groups is to share 
> information so
> that each group can comment on the other groups doucments. Following the 
> model of
> several other ITU-T study groups, SG 12 has created an easily accessable 
> FTP site.
> See: www1.ietf.org/mail-archive/web/ippm/current/msg1163.html and
> www.itu.int/ITU-T/studygroups/com12/packet/index.html.
> 
> Scott Bradner noted that working documents are what are covered by the 
> FTP site;
> right now finished documents are available for free.  Brian Carpenter 
> noted that
> the plan to distribute finished documents for free was approved last 
> January for
> one year, so it is not yet clear what will happen in 2008.
> 
> SG 12 would specifically like feedback on two documents that are now in the
> repository.  The first is a revised version of Y.1540, the IP 
> performance metrics
> document. This revision has more metrics, with reordering pulled into the
> normative text, and new text on duplication and replication.  We should 
> try to
> harmonize on terminology if we can.  At the moment the group is 
> grappling with the
> problem of how to include a markov model description of burst loss.  
> There are two
> competing proposals of how to define states (like burst or gap, 
> alternatively
> stable or un-stable).
> 
> The current burst/gap proposal needs to evaluate packets in sending 
> order and
> requires an evaluation interval whose length has a dependece on the 
> application of
> interests.  An alternative proposal might be to use arrival order 
> instad, so it
> would include the effects of reordering.
> 
> The other document is a multicast metrics draft.  Within the ITU the 
> scope is
> expanded so that it covers the IGMP aspects of multicast, not just 
> packet transfer
> performance.  This is a complement to Y.1540. It goes "above and below" 
> to look at
> session establishment and teardown of IGMP.  It would be useful to get 
> feedback
> from IETF folks.
> 
> If one wanted to post a comment document, they could send it to Al Morton,
> acmorton at att.com.  The IPPM chairs and also the Transport AD also 
> have the
> ability to post to the site. If there is a document that is relvant to 
> the whole
> community we will post it to the web site.
> 
> Someone asked about the intellectual property rules of documents posted 
> to that
> space.  Al said he did not know, but would look into it.  Alan noted 
> that one of
> the differences between the ITU and IETF; here people are encouraged to 
> disclose
> IPR as early as possible.  Within the ITU, the onus on members is to 
> formally
> offer to license IPR, if any exists, prior to any recommendation being 
> approved.
> There is less pressure to disclose early.
> 
> Scott Bradner replied that this is accurate, except... IETF has no 
> requirment to
> say what licensing is but ITU does, everything else is exactly right
> 
> APM
> application performance metrics.  tomorrow afternoon 1510
> 
> 
> 
> 
> 

-- 
------------------------------------------------------------------------------
Henk Uijterwaal                           Email: henk.uijterwaal(at)ripe.net
RIPE Network Coordination Centre          http://www.amsterdamned.org/~henk
P.O.Box 10096          Singel 258         Phone: +31.20.5354414
1001 EB Amsterdam      1016 AB Amsterdam  Fax: +31.20.5354445
The Netherlands        The Netherlands    Mobile: +31.6.55861746
------------------------------------------------------------------------------

# Lawyer: "Now sir, I'm sure you are an intelligent and honest man--"
# Witness: "Thank you. If I weren't under oath, I'd return the compliment."

_______________________________________________
ippm mailing list
ippm@ietf.org 
https://www1.ietf.org/mailman/listinfo/ippm