Re: [pm-dir] request for an early PMDIR review

[Varun Singh <[e-mail]>]

Hi Al,

Thanks for the review. Comments inline.

Cheers,
Varun

On Aug 26, 2013, at 5:12 PM, MORTON JR., ALFRED C (AL) wrote:

> Hi Dan,
> 
> As promised, here's an early pm-dir review of the draft:
> https://datatracker.ietf.org/doc/draft-huang-xrblock-rtcweb-rtcp-xr-metrics/
> 
> Since the purpose of this draft is to make recommendations 
> for measurements to satisfy the WebRTC requirement cited in the draft:
> 
>   "F38    The browser MUST be able to collect statistics, related to
>   the transport of audio and video between peers, needed to estimate
>   quality of service."
> 
> "Statistics" seemed a bit vague to me, especially in a requirement.
> One supporting reference is:
> 
>   [RTCWEB-STAT] Alvestrand, H., "A Registry for WebRTC statistics
>                 identifiers", September 24, 2012.
> 
> I took a look at Harald's draft. "RTCWeb Media Statistics" are indeed the 
> result of some performance measurement, and the registry he proposes 
> appears to assume that all needed metric definition details can be
> found in a specification or supplied by the expert who prepares the
> registry entry. There would likely be overlap between the
> "RTCWeb Media Statistics" registry and other registries containing
> performance metrics.
> 

Our intention with the upcoming versions of the draft was to add to 
the registry proposed by Harald.

> There's a *missing* aspect of the WebRTC requirement F38:
> Specification of the *minimum* set of metrics that MUST be provided
> to accomplish the stated goal: estimate quality of service
> (and not quality of experience!).
> 
> It seems to me that draft-huang-xrblock-rtcweb-rtcp-xr-metrics
> *could* be revised to fulfill that role. Otherwise, the intent and
> goal of F38 will never be realized because implementers can make
> different choices of statistics, and possibly include a set which
> is insufficient. In addition, the draft currently takes the position 
> that some metrics are needed for network performance diagnosis.
> This is a valuable set of metrics too, but they need to be distinguished
> from the minimum set of metrics needed to estimate QoS and 
> justified through explanation of their diagnostic power 
> (as has been done in some cases already in the draft).
> Again, emphasis should be on a *minimum* set of diagnostic metrics,
> and which applications and common circumstances.
> 

I agree we should define a minimum set of metrics for the next version, 
the current version of the draft was mainly to organize all the available 
metrics/measures and to facilitate discussion. Would your suggestion 
be to: just define the minimum set or keep the current discussion and then 
define the minimum set?

> Some more specific comments follow:
> 
> 3.2.1 Retransmitted Packet Count Metric
> I'm surprised to see this suggested, especially as the first in the memo.
> In true Real-time conversational applications, there's little opportunity
> for retransmission to be effective. The draft says:
>   In low delay networks with low
>   loss rates, retransmissions have great value without incurring
>   additional complexity.
> If the loss ratio is low, it seems retransmission is not very valuable,
> and low delay networks are a very restricted set. I don't see what 
> additional diagnostic value this metric has, beyond the other loss metrics.
> How widely is the NACK capability deployed and used?
> Note: there could be link-layer retransmissions below RTP-layer,
> but these are not visible above the link layer, except as 
> reordered packets.
> 

Short answer, Chrome uses NACK or PLI (packet loss indication) extensively. 
My measurements show that there are cases in low delay where the actual packet loss 
is 10-20% and by using retx they are able to bring it down to <1%, these measurements 
are basically within northern europe (end-to-end delay up to 50ms, AFAIK, the chrome
implementation tries to meet the 150ms one-way delay guarantees).

They also use FEC, which can sometimes be 10-20% of the overall bit rate. We try 
to capture this in the post-repair metric in 3.2.5. Perhaps, we may have to define
a new Post-repair metric which is not RLE, but a measure of number of packets 
recovered by using FEC.


> 
> 3.2.3 Loss, Discard and Duplicate Packet Count Metric
> I agree that Loss and Discard are valuable for QoS and diagnostic
> purposes, but in my experience Duplicate packets are rare and have
> little effect on QoS (they are usually discarded silently, but they
> should not appear in the discard counts).
> 
Ok, remove duplicate and keep the discard and loss metric?


> 3.2.5 Run Length Encoded Metrics for Loss, Discard and Post-repair
> 
>   Run-length encoding uses a bit vector to encode information about the
>   packet. Each bit in the vector represents a packet and depending on
>   the signaled metric it defines if the packet was lost, duplicated,
>   discarded, or repaired. An endpoint typically uses the run length
>   encoding to accurately communicate the status of each packet in the
>   interval to the other endpoint.
> This seems like an enormous amount of data to convey between
> the receiver and sender, a fairly complete record of transmission.
> It might be a diagnostic capability turned-on in some specific 
> troubleshooting cases, but I don't see RLE as part of the 
> QoS estimation metrics set.
> 

I agree, RLE may be an overkill for the general monitoring.

> It's fine to include Jitter and De-jitter buffer metrics in the QoS 
> estimation set. Others could be discussed, but it's probably good to
> stop here for now.
> 

Thanks again for the comments.

> hope this helps,
> Al
> PS I'm switching to other topics, and will probably delay any responses
> on this thread for a couple of weeks.
> 
> 
>> -----Original Message-----
>> From: Romascanu, Dan (Dan) [mailto:[e-mail]]
>> Sent: Thursday, August 01, 2013 3:01 PM
>> To: MORTON JR., ALFRED C (AL)
>> Cc: Huangyihong (Rachel); Varun Singh; Roni Even; Shida Schubert
>> Subject: request for an early PMDIR review
>> 
>> Hi Al,
>> 
>> Following our discussion earlier today, I would kindly ask for an early
>> PM-DIR review of https://datatracker.ietf.org/doc/draft-huang-xrblock-
>> rtcweb-rtcp-xr-metrics/.
>> 
>> Thanks and Regards,
>> 
>> Dan
>> 
>> 
>>