[ippm] Re: Responsiveness under working conditions

[Magnus Olden <[e-mail]>]

Hi Chris,

Reading your email, it seems you are running into a problem we ran into
when designing QoO <https://datatracker.ietf.org/wg/ippm/about/>, which
later became an important pillar of the design.

When mapping network metrics to app performance, there will be multiple
relevant requirements. As you mention, the mean/median and max latency will
play a role. Going down the rabbit hole, you see that some apps can drop
some packets without meaningful app degradation, hence the 99th percentile
latency may be more important than the max, but not always.

Some apps have jitter buffers that are tuned by the 5% worst latencies,
which means the 95th percentile may also be important, other apps implement
their jitter buffers in different ways (e.g., like you mention with a fixed
jitter buffer). It seems to us that you are bound to end up with a set of
network requirements rather than one, for example:

Network Requirement 1: Median latency < 100 ms
Network Requirement 2:  95th percentile latency  < 150 ms
...
Network Requirement N-1: 99th percentile latency < 250ms
Network Requirement N: Max (100th) percentile latency < 300 ms

Where the app performance is bound by the worst network requirement.

Not sure how this is directly relatable to RPM calculations, but I figured
it was useful input anway. Both  @Bjørn Ivar Teigen <[e-mail]> and
I would be happy to walk you through our approach, and the underlying maths
<https://wiki.broadband-forum.org/display/RESOURCES/Broadband+Forum+Published+Resources#tf-filters=%7B%22selectfilters%22%3A%5B%5D%2C%22userfilters%22%3A%5B%22Number%22%5D%2C%22numberfilters%22%3A%5B%5D%2C%22datefilters%22%3A%5B%5D%2C%22globalfilter%22%3Atrue%2C%22columnhider%22%3Afalse%2C%22iconfilters%22%3A%5B%5D%2C%22defaults%22%3A%5B%22TR-452.2%22%2C%22%22%5D%2C%22width%22%3A%5B%22150%22%2C%22150%22%5D%2C%22inverse%22%3A%5Bfalse%2Cfalse%5D%2C%22order%22%3A%5B0%2C1%5D%2C%22ddSeparator%22%3A%5B%5D%2C%22ddOperator%22%3A%5B%5D%2C%22sorts%22%3A%5B%22Date%20%E2%87%A9%22%5D%7D>
and
justifications.


Best Regards,
Magnus Olden | CTO | Domos | Bridging the gap between applications and
networks

The content of this email is intended for the person or entity to which it
is addressed only. This email may contain confidential information. If you
are not the person to whom this message is addressed, be aware that any
use, reproduction, or distribution of this message is strictly prohibited.
If you received this in error, please contact the sender and immediately
delete this email and any attachments.


On Sat, 9 Nov 2024 at 10:25, Chris Box <[e-mail]> wrote:

> Hi everyone
>
> I watched Stuart's presentation on Monday and I'm personally convinced
> that getting this responsiveness test right is one of the most important
> tasks of internet engineering right now. Responsiveness is the next
> frontier of quality improvement, and the current experience of most is
> plentiful bandwidth but frequently poor responsiveness. We can do better,
> and this test is the key enabler for that.
>
> My personal aims sound very similar to Stuart's:
>
> Primary goals (must have): Relevance and Repeatability
> Secondary goal (desirable): Convenience
>
>
> When I heard Jonathan describe the once-per-minute wifi freeze, I
> concluded this is a case where we ought to sacrifice convenience in favour
> of relevance and repeatability. If end users are being impacted by that
> 500ms gap, then the test ought to measure that. It's much less helpful if
> it ignores that degradation. Of course periodic (or even sporadic)
> disruptions can occur on longer timescales and we have to draw a line
> somewhere. But perhaps we can consider a "full test" and a "quick test"
> mode.
>
> The other major open question is how to convert from a large set of
> samples to a single RPM value. I agree with Abhishek that the message from
> Bjorn's QoO distribution is that the outliers have the most significant
> effect on user experience. Those little black circles are the ones we
> notice the most. So I suggest we do not discard any values. They all count.
> My view is that the packet with the maximum delay should form a significant
> component of the RPM. It's not everything of course, and repeatability
> probably implies we need some way to compute a "worst delay" figure from
> multiple packets, e.g. the worst 10 or 20.
>
> ITU Recommendation Y.1514 defines network performance requirements. It
> says for class 0 audio over a measurement period of one minute, the mean IP
> Packet Transfer Delay should not exceed 100ms, and IP Packet Delay
> Variation (RFC3393) should not exceed 50ms. To have good enough
> responsiveness for a video conference, this means mean delay in each
> direction <100ms, and no audio-bearing packets should arrive later than
> 150ms. With responsiveness we're measuring the sum of both directions, so
> it will depend on the assymetry of delay, for example in a mobile network
> uplink packets need to wait until granted permission. What we can say is
> that for all networks, http_l of <100 mean and <150 max is sufficient to
> meet the class 0 requirement. For a symmetrical network, http_l of <200
> mean and <300 max is sufficient.
>
> Also ITU G.114 discusses delay due to the use of an IP delay variation
> buffer. It says that for planning purposes, it is recommended to assume
> that a de-jitter buffer adds one half of its peak delay to the mean network
> delay.  A jitter buffer designed to compensate for 50 ms packet delay
> variation range will introduce 25 ms additional delay, on average.
>
> So what should we compute the RPM from? I suggest it needs to be some
> combination of the worst delay and the typical delay, that in testing
> proves itself to be both Relevant and Repeatable.
>
> Chris
>
>
> _______________________________________________
> ippm mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]
>