[ippm] Re: Responsiveness under working conditions

["Henrik Nydell (hnydell)" <[e-mail]>]

When looking at metrics where there is a jitter buffer involved it usually helps to treat packets that violate the jitter buffer characteristic as being lost.
As an example, if the jitter buffer for an application is 50ms, and you have p95 delay reporting at 50ms then 5% of the packets are too late for the buffer, so these should be considered as 5% packet loss.
Using something like the e-model for various VoIP codecs you can then add the 5% loss due to jitter buffer overflow to get a more representative view of the voice degradation.

For other applications than common voice codecs, a more detailed investigation of the loss sensitivity due to jitter buffer overflows need to be done since this loss sensitivity varies a lot depending on the application.

From: Magnus Olden <[e-mail]>
Date: Monday, 11 November 2024 at 14:37
To: Chris Box <[e-mail]>, Bjørn Ivar Teigen <[e-mail]>
Cc: [e-mail] <[e-mail]>
Subject: [ippm] Re: Responsiveness under working conditions
Hi Chris,

Reading your email, it seems you are running into a problem we ran into when designing QoO <https://datatracker.ietf.org/wg/ippm/about/> , which later became an important pillar of the design.

When mapping network metrics to app performance, there will be multiple relevant requirements. As you mention, the mean/median and max latency will play a role. Going down the rabbit hole, you see that some apps can drop some packets without meaningful app degradation, hence the 99th percentile latency may be more important than the max, but not always.

Some apps have jitter buffers that are tuned by the 5% worst latencies, which means the 95th percentile may also be important, other apps implement their jitter buffers in different ways (e.g., like you mention with a fixed jitter buffer). It seems to us that you are bound to end up with a set of network requirements rather than one, for example:

Network Requirement 1: Median latency < 100 ms
Network Requirement 2:  95th percentile latency  < 150 ms
...
Network Requirement N-1: 99th percentile latency < 250ms
Network Requirement N: Max (100th) percentile latency < 300 ms

Where the app performance is bound by the worst network requirement.

Not sure how this is directly relatable to RPM calculations, but I figured it was useful input anway. Both  @Bjørn Ivar Teigen<mailto:[e-mail]> and I would be happy to walk you through our approach, and the underlying maths<https://wiki.broadband-forum.org/display/RESOURCES/Broadband+Forum+Published+Resources#tf-filters=%7B%22selectfilters%22%3A%5B%5D%2C%22userfilters%22%3A%5B%22Number%22%5D%2C%22numberfilters%22%3A%5B%5D%2C%22datefilters%22%3A%5B%5D%2C%22globalfilter%22%3Atrue%2C%22columnhider%22%3Afalse%2C%22iconfilters%22%3A%5B%5D%2C%22defaults%22%3A%5B%22TR-452.2%22%2C%22%22%5D%2C%22width%22%3A%5B%22150%22%2C%22150%22%5D%2C%22inverse%22%3A%5Bfalse%2Cfalse%5D%2C%22order%22%3A%5B0%2C1%5D%2C%22ddSeparator%22%3A%5B%5D%2C%22ddOperator%22%3A%5B%5D%2C%22sorts%22%3A%5B%22Date%20%E2%87%A9%22%5D%7D> and justifications.


Best Regards,
Magnus Olden | CTO | Domos | Bridging the gap between applications and networks

The content of this email is intended for the person or entity to which it is addressed only. This email may contain confidential information. If you are not the person to whom this message is addressed, be aware that any use, reproduction, or distribution of this message is strictly prohibited. If you received this in error, please contact the sender and immediately delete this email and any attachments.


On Sat, 9 Nov 2024 at 10:25, Chris Box <[e-mail]<mailto:[e-mail]>> wrote:
Hi everyone

I watched Stuart's presentation on Monday and I'm personally convinced that getting this responsiveness test right is one of the most important tasks of internet engineering right now. Responsiveness is the next frontier of quality improvement, and the current experience of most is plentiful bandwidth but frequently poor responsiveness. We can do better, and this test is the key enabler for that.

My personal aims sound very similar to Stuart's:
Primary goals (must have): Relevance and Repeatability
Secondary goal (desirable): Convenience

When I heard Jonathan describe the once-per-minute wifi freeze, I concluded this is a case where we ought to sacrifice convenience in favour of relevance and repeatability. If end users are being impacted by that 500ms gap, then the test ought to measure that. It's much less helpful if it ignores that degradation. Of course periodic (or even sporadic) disruptions can occur on longer timescales and we have to draw a line somewhere. But perhaps we can consider a "full test" and a "quick test" mode.

The other major open question is how to convert from a large set of samples to a single RPM value. I agree with Abhishek that the message from Bjorn's QoO distribution is that the outliers have the most significant effect on user experience. Those little black circles are the ones we notice the most. So I suggest we do not discard any values. They all count. My view is that the packet with the maximum delay should form a significant component of the RPM. It's not everything of course, and repeatability probably implies we need some way to compute a "worst delay" figure from multiple packets, e.g. the worst 10 or 20.

ITU Recommendation Y.1514 defines network performance requirements. It says for class 0 audio over a measurement period of one minute, the mean IP Packet Transfer Delay should not exceed 100ms, and IP Packet Delay Variation (RFC3393) should not exceed 50ms. To have good enough responsiveness for a video conference, this means mean delay in each direction <100ms, and no audio-bearing packets should arrive later than 150ms. With responsiveness we're measuring the sum of both directions, so it will depend on the assymetry of delay, for example in a mobile network uplink packets need to wait until granted permission. What we can say is that for all networks, http_l of <100 mean and <150 max is sufficient to meet the class 0 requirement. For a symmetrical network, http_l of <200 mean and <300 max is sufficient.

Also ITU G.114 discusses delay due to the use of an IP delay variation buffer. It says that for planning purposes, it is recommended to assume that a de-jitter buffer adds one half of its peak delay to the mean network delay.  A jitter buffer designed to compensate for 50 ms packet delay variation range will introduce 25 ms additional delay, on average.

So what should we compute the RPM from? I suggest it needs to be some combination of the worst delay and the typical delay, that in testing proves itself to be both Relevant and Repeatable.

Chris


_______________________________________________
ippm mailing list -- [e-mail]<mailto:[e-mail]>
To unsubscribe send an email to [e-mail]<mailto:[e-mail]>