Re: [pm-dir] request for an early PMDIR review

["Huangyihong (Rachel)" <[e-mail]>]

Hi Al,

Thank you for these valuable comments. Please see some additional replies inline.

Best regards,
Rachel

-----Original Message-----
From: MORTON JR., ALFRED C (AL) [mailto:[e-mail]] 
Sent: Monday, August 26, 2013 10:13 PM
To: Romascanu, Dan (Dan); [e-mail]
Cc: Huangyihong (Rachel); Varun Singh; Roni Even; Shida Schubert
Subject: RE: request for an early PMDIR review

Hi Dan,

As promised, here's an early pm-dir review of the draft:
https://datatracker.ietf.org/doc/draft-huang-xrblock-rtcweb-rtcp-xr-metrics/

Since the purpose of this draft is to make recommendations 
for measurements to satisfy the WebRTC requirement cited in the draft:

   "F38    The browser MUST be able to collect statistics, related to
   the transport of audio and video between peers, needed to estimate
   quality of service."

"Statistics" seemed a bit vague to me, especially in a requirement.
One supporting reference is:

   [RTCWEB-STAT] Alvestrand, H., "A Registry for WebRTC statistics
                 identifiers", September 24, 2012.

I took a look at Harald's draft. "RTCWeb Media Statistics" are indeed the 
result of some performance measurement, and the registry he proposes 
appears to assume that all needed metric definition details can be
found in a specification or supplied by the expert who prepares the
registry entry. There would likely be overlap between the
"RTCWeb Media Statistics" registry and other registries containing
performance metrics.

There's a *missing* aspect of the WebRTC requirement F38:
Specification of the *minimum* set of metrics that MUST be provided
to accomplish the stated goal: estimate quality of service
(and not quality of experience!).

It seems to me that draft-huang-xrblock-rtcweb-rtcp-xr-metrics
*could* be revised to fulfill that role. Otherwise, the intent and
goal of F38 will never be realized because implementers can make
different choices of statistics, and possibly include a set which
is insufficient. In addition, the draft currently takes the position 
that some metrics are needed for network performance diagnosis.
This is a valuable set of metrics too, but they need to be distinguished
from the minimum set of metrics needed to estimate QoS and 
justified through explanation of their diagnostic power 
(as has been done in some cases already in the draft).
Again, emphasis should be on a *minimum* set of diagnostic metrics,
and which applications and common circumstances.

[Rachel]: Are you suggestion that we should focus the metrics evaluating QoS while remove the QoE related ones? Actually, the reason why we consider QoE metrics (such as frame impairment metrics) is because they are useful to diagnostic why media quality degrades. We think it's important because people care about the quality. But I agree with you that they're not common enough to all applications. It's fine to take them out.

Some more specific comments follow:

3.2.1 Retransmitted Packet Count Metric
I'm surprised to see this suggested, especially as the first in the memo.
In true Real-time conversational applications, there's little opportunity
for retransmission to be effective. The draft says:
   In low delay networks with low
   loss rates, retransmissions have great value without incurring
   additional complexity.
If the loss ratio is low, it seems retransmission is not very valuable,
and low delay networks are a very restricted set. I don't see what 
additional diagnostic value this metric has, beyond the other loss metrics.
How widely is the NACK capability deployed and used?
Note: there could be link-layer retransmissions below RTP-layer,
but these are not visible above the link layer, except as 
reordered packets.

[Rachel]: The metrics are not ordered by their importance, instead, they are listed from transport level to application level. I'm sorry it confused you. We can adjust the order in the next version. I have no additional comments other than Varun. 


3.2.3 Loss, Discard and Duplicate Packet Count Metric
I agree that Loss and Discard are valuable for QoS and diagnostic
purposes, but in my experience Duplicate packets are rare and have
little effect on QoS (they are usually discarded silently, but they
should not appear in the discard counts).

[Rachel]: Okay. Agree.


3.2.5 Run Length Encoded Metrics for Loss, Discard and Post-repair

   Run-length encoding uses a bit vector to encode information about the
   packet. Each bit in the vector represents a packet and depending on
   the signaled metric it defines if the packet was lost, duplicated,
   discarded, or repaired. An endpoint typically uses the run length
   encoding to accurately communicate the status of each packet in the
   interval to the other endpoint.
This seems like an enormous amount of data to convey between
the receiver and sender, a fairly complete record of transmission.
It might be a diagnostic capability turned-on in some specific 
troubleshooting cases, but I don't see RLE as part of the 
QoS estimation metrics set.

[Rachel]: Okay.

It's fine to include Jitter and De-jitter buffer metrics in the QoS 
estimation set. Others could be discussed, but it's probably good to
stop here for now.


hope this helps,
Al
PS I'm switching to other topics, and will probably delay any responses
on this thread for a couple of weeks.


> -----Original Message-----
> From: Romascanu, Dan (Dan) [mailto:[e-mail]]
> Sent: Thursday, August 01, 2013 3:01 PM
> To: MORTON JR., ALFRED C (AL)
> Cc: Huangyihong (Rachel); Varun Singh; Roni Even; Shida Schubert
> Subject: request for an early PMDIR review
> 
> Hi Al,
> 
> Following our discussion earlier today, I would kindly ask for an early
> PM-DIR review of https://datatracker.ietf.org/doc/draft-huang-xrblock-
> rtcweb-rtcp-xr-metrics/.
> 
> Thanks and Regards,
> 
> Dan
> 
> 
>