[ippm] Minutes from the Chicago meeting

IPPM Group,

Below you'll find a first version of the minutes of the Chicago
meeting.  Thanks to Al Morton for taking most of the notes.  Comments
to the chairs and/or the list before August 24 please.

Matt & Henk

-=-=-=-
IP Performance Metrics WG (ippm)
Tuesday, July 24, 09:00--11:30 CDT

This session was chaired by Henk Uijterwaal and Matt Zekauskas. Al Morton acted as
scribe, and his notes (along with notes taken by Matt) were edited into these
minutes by the Chairs.

AGENDA:

1. Administrivia
2. TWAMP updates, and next steps
3. Duplication Draft Update
4. Composition Framework and Spatial Composition Additions
5. Multimetrics Draft Developments
6. Delay Variation Draft Developments
7. Any Other Business

1. Administrivia
    Agenda bashing, Scribe, Minutes, Blue Sheets
    Status of drafts and milestones

Henk opened the meeting.  There were no changes to the agenda.

During the discussion of drafts and milestones, Henk menioned that we are still
sitting on the implementation draft awaiting guidance on progressing metrics along
the standards track. While initially the thought was we should just publish the
docoument as informational, Lars skimmed draft-bradner-metricstest-02.txt, and it
said metrics would progress when test equipment from different vendors had results
in controlled experiment that were verifyably equivalent. However, the definition
of "verifyably equivalent" is still open, and the results of that definition might
imply that changes need to be made to the draft.  Therefore we will let the draft
sit, and encourage this group (and bmwg) to contribute to the definition of
"verifyably equivalent".

Henk asked if Al had any idea of realistic dates for his composition drafts, since
the milestones for those drafts had passed.  Al thought that the framework was in
good shape, but didn't want to progress it before one of the composition drafts
was ready.  The big barrier is that it needs more review by working group members.
We need some rigorous reviews. December 2007 is probably too soon, mid-2008 is a
more realistic target. Folks in the group are requested to read and commend on the
mailing list.

Emile said that the multimetrics draft is more-or-less complete and has been
stable.  It too needs review, and he thought a WGLC would be the means to coerce
folks to review.  However, he has been speaking with Jurgen Quittek, and Jurgen
has a series of comments he is going to send in.  Therefore, he requests that the
working group review the document after the next revision. Al offered to help with
readability going in to the next version as well.  The big issue with the draft
are the passive aspects, which were discussed during the draft discussion below.

2. TWAMP updates, and next steps
    --Kaynam Hedayat

    http://tools.ietf.org/id/draft-ietf-ippm-twamp-04.txt
    http://tools.ietf.org/wg/ippm/draft-ietf-ippm-twamp/

Kaynam presented a single slide on TWAMP status.  At the last meeting they agreed
to add sneder TTL into test packets, as well as clarifying security text with
respect to TWAMP-Light, and both have been done.  In addition, text has been
clarified for sender port and receiver port.

Someone brought up setting DSCP on the ML.  What does the group feel we should do?
Matt asked if Kaynam had seen a use for it, and he said he did, but more for
one-way than two-way tests.  It seems like a fine idea, but should we make such a
change now?  No one felt strongly that we should do so.  Therefore, no further
changes for DSCP are proposed.  This question was also asked on the list before
the meeting, with no response.

There were additional clarifications that were suggested on the ML and a new
revision is in the works to address those.

Matt mentioned that he was supposed to help the authors come up with some IANA
text for OWAMP-Control commands, since this document adds a new value.  He said he
would do that by the end of the week.

3. Duplication Draft Update
    --Henk Uijterwaal

    http://tools.ietf.org/id/draft-ietf-ippm-duplicate-01.txt
    http://tools.ietf.org/wg/ippm/draft-ietf-ippm-duplicate/

Henk next presented updates on the duplication draft.  Based on previous comments,
a new version was created in April.  There are two statistics, and two ways to
express duplication.  This draft spun off from the original reporting draft, which
wanted to reference duplication but there was no definition.

There is a placeholder Y.1540 section.  However, references have been made along
the way, so Henk does not see the need to keep the section.  Al agreed, although
he also stated that it might not be a bad idea to see where we could harmonize the
definitions, and make the differnces go away (Y.1540 is currently under revision,
see the final topic.)

Al had reviewed the draft and made a few comments. He thought the names for some
metrics might be change to be more descriptive. For example,
"type-P-one-way-packet-duplication-average" was new in this version and didn't
seem to capture the meaning of the metric.

Al also noted that the real singleton here is not "duplication", but an arrival
counter.  It starts at 0, and goes to 1, 2, 3, etc, for each packet.  Including
"duplication" in the name seems like forming a conclusion that we haven't reached
yet.  WE should evaluate the arrival counter and see if a packet only once or more
than once. Only those that arrive more than once should be called duplicates. Al
offered to send his comments to the list.  They may be found here:
http://www1.ietf.org/mail-archive/ippm/current/msg01174.html

4. Composition Framework and Spatial Composition Additions
    -- Al Morton

    http://tools.ietf.org/id/draft-ietf-ippm-framework-compagg-04.txt
    http://tools.ietf.org/wg/ippm/draft-ietf-ippm-framework-compagg/
    http://tools.ietf.org/id/draft-ietf-ippm-spatial-composition-04.txt
    http://tools.ietf.org/wg/ippm/draft-ietf-ippm-spatial-composition/

Al started by reviewing the composition framework.  Overall, it's been fairly
stable; the definitions have been refined, and a maintenance update has been
performed.  However, there has not been much feedback recently.  Al said he is
declaring this stable, pending feedback or one of the composition drafts to be
complete.

Al noted the Metro Ethernet Forum is specifying some end-to-end performance
parameters, and they are going to need a solution to this problem as well.

Jurgen Quittek and Emile Stephan offered to read the draft.

Al then did an overview of the spatial composition draft, and
the metrics that are defined there.  The metric names have been
simplified, allowing for easier exposition.

Scott Bradner and Loki Jorgenson offered to read this draft.

Matt admitted he had not read the draft recently, but the description triggered a
thought -- are there auxilliary metrics or parameters to specifcy when a
composition is valid?  For example, min composition works as long as the path is
relatively stable and uncongested. If the path is congested, the min will jump
around.  Al thought that it might just mean the "error bars" are larger, but you
can still get an estimate of the minimum.

Craig White remarked: Given a large enough sample size, the minimum will tend to
be the one that occurs the most frequently, as long as the path is relatively
stable.  Even if the path is not stable, you do get some bimodal distributions
(for example if a SONET ring goes to its protect side, and then back.

Al said that bimodal distributions are dealt with in the document. You don't want
to mix them together because the minimum for one is different than the other.  The
intent is to measure for a long time, if you see some path changes, segregate the
measurements into different intervals.

Emile said that in his opinion you can't consider making an aggregation that
consists of measurements where the path changes -- the result would be unusable.
Perhaps we need to say that in the framework or in the specific metrics.

Al said that path stability needs to be mentioned when we talk about measuremetn
validity; he thought it went to an overall section since it applies to all
parameters.

5. Multimetrics Draft Developments
    -- Emile Stephan

    http://tools.ietf.org/id/draft-ietf-ippm-multimetrics-04.txt
    http://tools.ietf.org/wg/ippm/draft-ietf-ippm-multimetrics/

Emile started with a quick review of the multimetrics draft, starting with the
spatial metrics, and then multicast metrics.

For the spatial metrics, there has been some harmonization and discussion of
scalability aspects.  These metrics are used for either path composition or by the
end user.  Emile felt that these descriptions are now stable, and he didn't plan
to make more changes except to clean up wording.

The segment metrics have been renamed so that it is clear that they look at
performance between any two hops on the network. They are purely passive.

One participant had an issue with purely passive measurements, especially those
based only on user packets.  Did we want to discuss this now or later?

Jurgen said that he had read the document, and think we run into problems if we
focus on passive measurements.  A lot of the terminology has been inherited from
other drafts, and doesn't apply to passive measurements.  If there are active
probes, there is one sender and one receiver.  If you passively examine user
traffic, there are many senders and receivers that pass by.  Emile thought that
each flow had one sender and one receiver.  There is some time when the packet was
sent, even if it is not measurable.  You want to know the performance between two
points, you don't need to know the original sending time, you just need the two
points. The terminology could be clarified.

This would also provide psamp with some purely passive metrics; the section could
be removed if it doesn't make sense.

Emile thought that 80% of the wording is OK, and we need strong input, such as
that Jurgen is giving.  He would like a WGLC to force people to read the document.

Al said that he wanted to make a pass over the document first, to improve grammar.
He thought the document could benefit form an editorial pass to make it more
readable, and as a co-author felt that he was responsible for that.  After that,
he agreed a WGLC would be a good idea.

Al also wanted to know the groups point of view with respect to purely passive
metrics.  There was some discussion among the authors.  The IPPM charter says that
the "metrics will not characterize traffic", and thus would not be used to
characterize VOIP traffic, for example.  He reads this as passive is out of bounds
for the working group.  However, other people are entitled to give interpretations
of that.

Emile siad that it is not easy to mix passive and active metrics, but that we
needed some ground truth.  If we want to compose metrics, we will need to make
approximations, and we need to include passive metrics in the algorithms.  The
spatial definition relies on passive technieuqs in nodes the packet cros.

If there is an active source that provides packets that can be observed along the
path, that was still an active measurement. The discussion is about qualifying the
stream you are using for the basis of measurement.

One opinion was that we haven't done this yet, and that there is enough material
to create a separate draft on that topic.

Lars (our area director) agreed that the charter text has an underlying notion
that the metrics are based on active sampling.  The working group could of course
decide to take on passive measurements. The previous presentation goes in that
direction.  But he thought it was out of scope in the current charter, and he
would like to see us finish the metrics and related documents on active techniques
first. He thought passive measuremetns were a pretty large piece of new work -- in
short, he agreed with Al.

Emile reiterated his opinion that ignoring passive metrics is a mistake, and that
the best way to have some "ground-truth" measurements is to have somee passive
measurements to compare to.

Alan Clark said he agreed with both positions, in that if you look at composition
and define it using active measurements, the basic metrics are equally applicable
to passive techniques.  Work in composing delay, delay variation, and others, can
be picked up and used with passive measurements.  He felt that once the basic
ideas have been developed, that the group should look at passive applications.
However, it is difficult to make passive measurements of an IP stream, because you
have to decode protocols, and that runs the risk of overlapping with other groups.

Al Morton thought that was an interesting point; that the working group has
focused on IP, dabbled a bit in TCP, but has focussed on active live measurements
of the IP layer of the stack, and that's our core charter.  If you work on passive
metrics, and start to look at things like sequence numbers in higher layers, that
definitely goes beyond our charter.

Al also wanted to note that there are other gaps, places where people want to grow
the group beyond the charter, both here and in BMWG. Interested folks should
follow the "application performance metrics" BOF developments.

Jurgen said that he agreed with Emile, that it would be good to have a methodology
to apply.  However, if the working group wanted to pursue this, it would take more
than just a paragraph; there are many things to consier.  There should be a
framework for how we do passive measurements in general, before we apply them to
any specific document.

Emile feld we had to fix this before going further in composition, and that
something was needed now to promote comparable metrics in this space.  He also
agreed that squeezing it in as a subsection of a draft isn't the right place for
this to be.

There was a comment that at least with the active measurements, we would be
combining metrics where we understand the characteristics; if we combine passive
measurements, there may be less we can rely on to combine them meaningfully.

Emile noted that companies and people will do this combination anyway, so we
should be working to defining standard metrics.

There was a bit of further discussion on how large this effort might be and where
the right place to do it would be; there was no definitive conclusion.

Next, Karsten Fleischhauer presented a multimetrics use case provided by Ruediger
Geib, who could not attend this meeting.  The goal in this case it to improve
multicast monitoring within a provider's network.  It would give an operator
increased understanding of what was happening with multicast, and an easy view of
where failures occur.

Several simple multicast trees, and failures were presented, along with showing
how the metrics, especially the segment metrics, would improve understanding.
(See slides for details.)

Emile pointed out that the current draft does not mix segment metrics and
multicast metrics, which these examples seem to do.  Karsten said that his
understanding is that there are no requests to change the draft, the example is
just showing how the metrics can be used together.

6. Delay Variation Draft Developments
    -- Al Morton

    http://tools.ietf.org/wg/ippm/draft-morton-ippm-delay-var-as-03.txt

Next Al Morton presented the delay variation applicability statement draft, which
is not yet a working group item, but has had extensive review in the group.  He
spent some time presenting the problem space and showing how the different metric
formulations are useful for different problems.  See the slides for details.

There was a recent section that made suggestions on what could be done to reduce
variation.  This provoked a discussion of whether the section was appropriate for
the draft.  Scott Bradner mentioned that there was a different set of expertise
involved in fixing a problem than how to describe and measure the problem clearly.
We would need to reach out to operators in different environments including
enterprise and ISP for validation.  He thought this section was better in a
different document, or even a Wiki to keep advice up to date.

Al noted that Roman Krzanowski, who first asked for this work, worked for an
operator, and that Al himself was "tier-5 support" at his company.

Wenji Wu asked about applicability of this work, combining it with loss to
indicate congestion.  Al said that the draft was not trying to combine delay
variation with packet loss in this analysis; loss affects the delay variation
metric, however.  In the extreme case of losing every other packet you cannot
produce a inter-packet delay variation result.

Wenji said that if you have the minimum delay, and an indication of the maximum
delay, then you have an indication of the minimum latency from source to
destination, and an indication of queueing. Have there been thoughts to
standardize this application? Al said that the draft includes an estimate of queue
occupation, in orther words, this is a task that is already identified.  Al asked
that Wenji read the draft and comment.

Lars made the comment that there are transport protocols that are trying to infer
congestion by looking at both delay and loss changes. They are experimental.
There are other causes for loss than congestion, and other causes for delay than
congestion, for example a wireless link changing it's rate.  He said that there is
ongoing research here but there is nothing baked enough to become a metric. is
research on this,but not ready.

Al noted that one thing the draft could talk about with respect to minimum delay
is that it is an indication of path changes and could be used to separate a
bimodal distribution.  If we can detect that a path has changed positivly by TTL
changes, and correlate that with bursts of loss and delay change (and possibly by
reordering caused by "microloops" during a change) you may be able to reset the
minimum delay at that point.  If path changes are frequent, delay variation is
probably not your biggest problem.

Resetting the minimum at appropriate points would be good for long-term
measurements.

Alan said that at the last meeting he raised a possible case where inter-packet
delay variation might be useful, and that is the packet video buffer smoothing
problem.  Video packets have timestamps in them, but the rate is highly variable
due to big complete frames being mixed with packets indicating changes since the
last complete frame.  There is a smoothing buffer to try and even things out. You
need to take fragmentation and MTU size into account in this case.  He's thought
about it a bit, but has no conclusion yet.

Alan also noted another interesting problem, not specific to video, is where the
bandwith of test signal, whatever it may be, is sufficient to cause some delay
variation itself.  Can you separate this test signal induction (due to bandwidth
limits in the network) from those from other sources in the network?  It doesn't
matter what the absolute value of variation is in this case.  He has been doing
some simulation that might apply, with scene characteristics of video streams.

In summary, Al said that he'll go through the comparisions again, and felt he was
still trying to acheive consensus, but that he has not seen much resistance to the
basic results.

Speaking to slide 9, Alan Clarke made the observation that rather than calling
these guidelines in the document, we should call them measurement consderations;
to him guidelines are directing test vendors to do something rather than giving
them advice about things to watch out for.  Packet Delay Variation is one of the
most difficult things to measure and understand properly, so he thinks a section
like this is extremely useful.  Scott Bradner noted that his earlier reaction to
the guidance in the appendix did not apply here, measurement considerations are
good to state.

Al concluded noting that there had been significant support in the room, comments
from group members both here and on the mailing list and that this draft had
already been cited to satisfy an item the charter, so it's time to make this
document a working group item.

Henk took a quick poll of the room; about 1/2 indicated support for the document.
When asked who thought it was a bad idea to continue with this document, there was
no opposition.  Therefore Henk said the Chairs would verify the sense of the room
on the mailing list, and assuming no major change would work with the area
director to make this a working group draft.

An group member thought the document should be broadend from just focusing on
delay variation.  Al felt that it was better to stay focused in order to finish
the document and complete the milestone.  [Later note by MZ: we have had a couple
of other starts at applicability statements that did not have enough support to
follow through to completion; others are welcome to make suggestions, but we
should just focus on finishing the document in an area where there is a need and
support from the working group.]

7. Any Other Business

Henk mentioned a passive measurement draft that had been sent to the list:
draft-kikuchi-passive-measure-00.txt. No group members had read the draft; given
the earlier discussion about passive measurements and the upcoming "Application
Performance Metrics BOF", it seems that the draft is currently out of scope for
this working group but potentially in scope for the results of the BOF, Henk will
send out a note to that effect.

Al Morton made a comment about his ITU-T liaison work.  Years ago, Al was put in
charge of the IP Performance question in SG 12. We have been working to reduce the
differences between the work done there and here, harmonizing the results as much
as possible.  The scope of SG 12 is different and wider -- it produces numerical
objectives based on performance perameters (from metrics); this group is focussed
on the metrics.

One way to reduce the differences between the groups is to share information so
that each group can comment on the other groups doucments. Following the model of
several other ITU-T study groups, SG 12 has created an easily accessable FTP site.
See: www1.ietf.org/mail-archive/web/ippm/current/msg1163.html and
www.itu.int/ITU-T/studygroups/com12/packet/index.html.

Scott Bradner noted that working documents are what are covered by the FTP site;
right now finished documents are available for free.  Brian Carpenter noted that
the plan to distribute finished documents for free was approved last January for
one year, so it is not yet clear what will happen in 2008.

SG 12 would specifically like feedback on two documents that are now in the
repository.  The first is a revised version of Y.1540, the IP performance metrics
document. This revision has more metrics, with reordering pulled into the
normative text, and new text on duplication and replication.  We should try to
harmonize on terminology if we can.  At the moment the group is grappling with the
problem of how to include a markov model description of burst loss.  There are two
competing proposals of how to define states (like burst or gap, alternatively
stable or un-stable).

The current burst/gap proposal needs to evaluate packets in sending order and
requires an evaluation interval whose length has a dependece on the application of
interests.  An alternative proposal might be to use arrival order instad, so it
would include the effects of reordering.

The other document is a multicast metrics draft.  Within the ITU the scope is
expanded so that it covers the IGMP aspects of multicast, not just packet transfer
performance.  This is a complement to Y.1540. It goes "above and below" to look at
session establishment and teardown of IGMP.  It would be useful to get feedback
from IETF folks.

If one wanted to post a comment document, they could send it to Al Morton,
acmorton at att.com.  The IPPM chairs and also the Transport AD also have the
ability to post to the site. If there is a document that is relvant to the whole
community we will post it to the web site.

Someone asked about the intellectual property rules of documents posted to that
space.  Al said he did not know, but would look into it.  Alan noted that one of
the differences between the ITU and IETF; here people are encouraged to disclose
IPR as early as possible.  Within the ITU, the onus on members is to formally
offer to license IPR, if any exists, prior to any recommendation being approved.
There is less pressure to disclose early.

Scott Bradner replied that this is accurate, except... IETF has no requirment to
say what licensing is but ITU does, everything else is exactly right

APM
application performance metrics.  tomorrow afternoon 1510

-- 
------------------------------------------------------------------------------
Henk Uijterwaal                           Email: henk.uijterwaal(at)ripe.net
RIPE Network Coordination Centre          http://www.amsterdamned.org/~henk
P.O.Box 10096          Singel 258         Phone: +31.20.5354414
1001 EB Amsterdam      1016 AB Amsterdam  Fax: +31.20.5354445
The Netherlands        The Netherlands    Mobile: +31.6.55861746
------------------------------------------------------------------------------

# Lawyer: "Now sir, I'm sure you are an intelligent and honest man--"
# Witness: "Thank you. If I weren't under oath, I'd return the compliment."

_______________________________________________
ippm mailing list
ippm@ietf.org 
https://www1.ietf.org/mailman/listinfo/ippm