[ippm] Re: Responsiveness under working conditions

[[e-mail]]

Hi Chris,

capturing QoE in a simple and comprehensive way which is linked to a what a person perceives just experiencing performance issues isn’t simple, I think.

Sure, offering two or three dimensions minimum delay, buffer caused latency and packet loss helps. Looking at the work invested by ITU-T in MOS modeling, I’d by surprised if any simple metric can be built. Voice suffers more from serious jitter than from low percentage drop. Other apps may suffer more from packet drop than they do from jitter. During the IETF meeting, I noted that slides and talk for remote presentations are more important, than slides and presenter picture. I think, models for interactive remote control start to be built and there picture quality degradation may be more acceptable, than freeze.

I appreciate efforts to improve situation, but as I mentioned, there may be no simple one-approach-fits-all solution.

The ITU-T recommendation mentioned by you is likely Y.1541, https://www.itu.int/itu-t/recommendations/rec.aspx?rec=11462. I’d guess later ITU-T work on streaming and voice QoE is more relevant. Examples are
P.863: Perceptual objective listening quality prediction, I think, and
P.1200-P.1299: Models and tools for quality assessment of streamed media.

Work related to VR and modeling of interactive services is ongoing in ITU-T.

Regards,

Rüdiger

Von: Chris Box <[e-mail]>
Gesendet: Samstag, 9. November 2024 10:25
An: [e-mail]
Betreff: [ippm] Responsiveness under working conditions

Hi everyone

I watched Stuart's presentation on Monday and I'm personally convinced that getting this responsiveness test right is one of the most important tasks of internet engineering right now. Responsiveness is the next frontier of quality improvement, and the current experience of most is plentiful bandwidth but frequently poor responsiveness. We can do better, and this test is the key enabler for that.

My personal aims sound very similar to Stuart's:
Primary goals (must have): Relevance and Repeatability
Secondary goal (desirable): Convenience

When I heard Jonathan describe the once-per-minute wifi freeze, I concluded this is a case where we ought to sacrifice convenience in favour of relevance and repeatability. If end users are being impacted by that 500ms gap, then the test ought to measure that. It's much less helpful if it ignores that degradation. Of course periodic (or even sporadic) disruptions can occur on longer timescales and we have to draw a line somewhere. But perhaps we can consider a "full test" and a "quick test" mode.

The other major open question is how to convert from a large set of samples to a single RPM value. I agree with Abhishek that the message from Bjorn's QoO distribution is that the outliers have the most significant effect on user experience. Those little black circles are the ones we notice the most. So I suggest we do not discard any values. They all count. My view is that the packet with the maximum delay should form a significant component of the RPM. It's not everything of course, and repeatability probably implies we need some way to compute a "worst delay" figure from multiple packets, e.g. the worst 10 or 20.

ITU Recommendation Y.1514 defines network performance requirements. It says for class 0 audio over a measurement period of one minute, the mean IP Packet Transfer Delay should not exceed 100ms, and IP Packet Delay Variation (RFC3393) should not exceed 50ms. To have good enough responsiveness for a video conference, this means mean delay in each direction <100ms, and no audio-bearing packets should arrive later than 150ms. With responsiveness we're measuring the sum of both directions, so it will depend on the assymetry of delay, for example in a mobile network uplink packets need to wait until granted permission. What we can say is that for all networks, http_l of <100 mean and <150 max is sufficient to meet the class 0 requirement. For a symmetrical network, http_l of <200 mean and <300 max is sufficient.

Also ITU G.114 discusses delay due to the use of an IP delay variation buffer. It says that for planning purposes, it is recommended to assume that a de-jitter buffer adds one half of its peak delay to the mean network delay.  A jitter buffer designed to compensate for 50 ms packet delay variation range will introduce 25 ms additional delay, on average.

So what should we compute the RPM from? I suggest it needs to be some combination of the worst delay and the typical delay, that in testing proves itself to be both Relevant and Repeatable.

Chris