[ippm] Responsiveness under working conditions

[Chris Box <[e-mail]>]

Hi everyone

I watched Stuart's presentation on Monday and I'm personally convinced that
getting this responsiveness test right is one of the most important tasks
of internet engineering right now. Responsiveness is the next frontier of
quality improvement, and the current experience of most is plentiful
bandwidth but frequently poor responsiveness. We can do better, and this
test is the key enabler for that.

My personal aims sound very similar to Stuart's:

Primary goals (must have): Relevance and Repeatability
Secondary goal (desirable): Convenience


When I heard Jonathan describe the once-per-minute wifi freeze, I concluded
this is a case where we ought to sacrifice convenience in favour of
relevance and repeatability. If end users are being impacted by that 500ms
gap, then the test ought to measure that. It's much less helpful if it
ignores that degradation. Of course periodic (or even sporadic) disruptions
can occur on longer timescales and we have to draw a line somewhere. But
perhaps we can consider a "full test" and a "quick test" mode.

The other major open question is how to convert from a large set of samples
to a single RPM value. I agree with Abhishek that the message from Bjorn's
QoO distribution is that the outliers have the most significant effect on
user experience. Those little black circles are the ones we notice the
most. So I suggest we do not discard any values. They all count. My view is
that the packet with the maximum delay should form a significant component
of the RPM. It's not everything of course, and repeatability probably
implies we need some way to compute a "worst delay" figure from multiple
packets, e.g. the worst 10 or 20.

ITU Recommendation Y.1514 defines network performance requirements. It says
for class 0 audio over a measurement period of one minute, the mean IP
Packet Transfer Delay should not exceed 100ms, and IP Packet Delay
Variation (RFC3393) should not exceed 50ms. To have good enough
responsiveness for a video conference, this means mean delay in each
direction <100ms, and no audio-bearing packets should arrive later than
150ms. With responsiveness we're measuring the sum of both directions, so
it will depend on the assymetry of delay, for example in a mobile network
uplink packets need to wait until granted permission. What we can say is
that for all networks, http_l of <100 mean and <150 max is sufficient to
meet the class 0 requirement. For a symmetrical network, http_l of <200
mean and <300 max is sufficient.

Also ITU G.114 discusses delay due to the use of an IP delay variation
buffer. It says that for planning purposes, it is recommended to assume
that a de-jitter buffer adds one half of its peak delay to the mean network
delay.  A jitter buffer designed to compensate for 50 ms packet delay
variation range will introduce 25 ms additional delay, on average.

So what should we compute the RPM from? I suggest it needs to be some
combination of the worst delay and the typical delay, that in testing
proves itself to be both Relevant and Repeatable.

Chris