[ippm] Re: Responsiveness under working conditions

[Joachim Fabini <[e-mail]>]

Hi Chris, Hi Rüdiger

I'd add some (technical and partly philosophical) aspects to the 
observations of Rüdiger. Some topics worth considering complicate 
matters even more:

1. Traffic & loss patterns: Metrics like loss period and loss distance 
(RFC3357) have a huge impact on the perceived QoS. One can easily design 
streams having identical loss percentage but fundamentally distinct QoE. 
Isolating QoE observations from application-layer information (like the 
codec - how good it is in concealing the losses) is nearly impossible. 
Not even mentioning the difference in terms of QoE when losing i-frames 
vs. losing p-frames within a video stream. So let's stick to 
network-layer measurements.

2. On-demand capacity allocation is key to the scalability of today's 
mobile cellular access networks. Additional uncertainty factors like 
users-in-a-cell, capacity-shared-within-a-cell, etc. contribute to the 
network state and render measurements in a mobile network a one-time 
experience. We can attempt to restrict uncertainty factors - but this 
limits the representativity of measurements to the specific scenario.

3. More then ten years ago, Al Morton and I have summarized some of 
these observed (and confirmed) peculiarities in RFC 7312 - in particular 
considerations on *repeatability*, *continuity*, and *conservative*. 
These three are closely related to the terms that you used, 
repeatability and relevance. You can find more details by searching for 
the contributions to IRTF RAIM 2015 (slides including measurement 
results and references to publications at various conferences and journals).

Key to the RFC2330-defined term repeatability is the concept of 
"identical conditions". Today's real networks are stateful and much too 
complex (i.e. involve too many uncertainty factors that 
users/measurements can not control). Main metric for vendors and 
operators is overall network capacity optimization (=revenue) - at the 
cost of high non-determinism from a single user's perspective. I'm 
convinced that now and in the future we can and will *never* safeguard 
identical conditions even for network-layer measurements (except for 
fully isolated measurements, or unless the principles of network design 
change fundamentally). With application-layer semantics assigned to 
specific packets (some are more "valuable" than others) this becomes 
even more complex.

So I'd like to stress that repeatability is a purely theoretical concept 
that measurements should aim at. I do not say that we should refrain 
from designing better measurement methods, on the contrary. But my 
impression is that we're attempting/expecting the impossible. The 
design, implementation and configuration of today's network technologies 
is the reason why neither users nor measurements can anticipate or 
extrapolate network behavior. What measurements can do is to assess how 
the network behavior was for one specific stream or traffic under 
consideration that was monitored. Any inference beyond this is 
fortunetelling (imho slightly beyond the scope of the IPPM charter): 
this includes in particular inference on the behavior/results of the 
same network at a distinct point in time, on the experience of other 
users, or on the network behavior of a distinct stream of the same user.

One valid conclusion is, however, to request network vendors and 
operators to improve the determinism of their networks. Ultimately this 
shortcoming is the main challenge, both to users and measurements.

kind regards,
Joachim



On 11.11.24 17:10, [e-mail] wrote:
> Hi Chris,
> 
> capturing QoE in a simple and comprehensive way which is linked to a 
> what a person perceives just experiencing performance issues isn’t 
> simple, I think.
> 
> Sure, offering two or three dimensions minimum delay, buffer caused 
> latency and packet loss helps. Looking at the work invested by ITU-T in 
> MOS modeling, I’d by surprised if any simple metric can be built. Voice 
> suffers more from serious jitter than from low percentage drop. Other 
> apps may suffer more from packet drop than they do from jitter. During 
> the IETF meeting, I noted that slides and talk for remote presentations 
> are more important, than slides and presenter picture. I think, models 
> for interactive remote control start to be built and there picture 
> quality degradation may be more acceptable, than freeze.
> 
> I appreciate efforts to improve situation, but as I mentioned, there may 
> be no simple one-approach-fits-all solution.
> 
> The ITU-T recommendation mentioned by you is likely Y.1541, 
> https://www.itu.int/itu-t/recommendations/rec.aspx?rec=11462 
> <https://www.itu.int/itu-t/recommendations/rec.aspx?rec=11462>. I’d 
> guess later ITU-T work on streaming and voice QoE is more relevant. 
> Examples are
> 
> P.863: Perceptual objective listening quality prediction, I think, and
> 
> P.1200-P.1299: Models and tools for quality assessment of streamed media.
> 
> Work related to VR and modeling of interactive services is ongoing in ITU-T.
> 
> Regards,
> 
> Rüdiger
> 
> *Von:* Chris Box <[e-mail]>
> *Gesendet:* Samstag, 9. November 2024 10:25
> *An:* [e-mail]
> *Betreff:* [ippm] Responsiveness under working conditions
> 
> Hi everyone
> 
> I watched Stuart's presentation on Monday and I'm personally convinced 
> that getting this responsiveness test right is one of the most important 
> tasks of internet engineering right now. Responsiveness is the next 
> frontier of quality improvement, and the current experience of most is 
> plentiful bandwidth but frequently poor responsiveness. We can do 
> better, and this test is the key enabler for that.
> 
> My personal aims sound very similar to Stuart's:
> 
>     Primary goals (must have): Relevance and Repeatability
> 
>     Secondary goal (desirable): Convenience
> 
> When I heard Jonathan describe the once-per-minute wifi freeze, I 
> concluded this is a case where we ought to sacrifice convenience in 
> favour of relevance and repeatability. If end users are being impacted 
> by that 500ms gap, then the test ought to measure that. It's much less 
> helpful if it ignores that degradation. Of course periodic (or even 
> sporadic) disruptions can occur on longer timescales and we have to draw 
> a line somewhere. But perhaps we can consider a "full test" and a "quick 
> test" mode.
> 
> The other major open question is how to convert from a large set of 
> samples to a single RPM value. I agree with Abhishek that the message 
> from Bjorn's QoO distribution is that the outliers have the most 
> significant effect on user experience. Those little black circles are 
> the ones we notice the most. So I suggest we do not discard any values. 
> They all count. My view is that the packet with the maximum delay should 
> form a significant component of the RPM. It's not everything of course, 
> and repeatability probably implies we need some way to compute a "worst 
> delay" figure from multiple packets, e.g. the worst 10 or 20.
> 
> ITU Recommendation Y.1514 defines network performance requirements. It 
> says for class 0 audio over a measurement period of one minute, the mean 
> IP Packet Transfer Delay should not exceed 100ms, and IP Packet Delay 
> Variation (RFC3393) should not exceed 50ms. To have good enough 
> responsiveness for a video conference, this means mean delay in each 
> direction <100ms, and no audio-bearing packets should arrive later than 
> 150ms. With responsiveness we're measuring the sum of both directions, 
> so it will depend on the assymetry of delay, for example in a mobile 
> network uplink packets need to wait until granted permission. What we 
> can say is that for all networks, http_l of <100 mean and <150 max is 
> sufficient to meet the class 0 requirement. For a symmetrical network, 
> http_l of <200 mean and <300 max is sufficient.
> 
> Also ITU G.114 discusses delay due to the use of an IP delay variation 
> buffer. It says that for planning purposes, it is recommended to assume 
> that a de-jitter buffer adds one half of its peak delay to the mean 
> network delay.  A jitter buffer designed to compensate for 50 ms packet 
> delay variation range will introduce 25 ms additional delay, on average.
> 
> So what should we compute the RPM from? I suggest it needs to be some 
> combination of the worst delay and the typical delay, that in testing 
> proves itself to be both Relevant and Repeatable.
> 
> Chris
> 
> 
> _______________________________________________
> ippm mailing list -- [e-mail]
> To unsubscribe send an email to [e-mail]