Re: [bmwg] FW: IETF BMWG Work Item Proposal for SIP PerformanceBenchmarking

[Magnus Westerlund <[e-mail]>]

Hi,

As AVT WG chair I do have some opinions on this topic.

Vijay K. Gurbani wrote:

>> I am not saying that RTP media plane benchmarking is not useful, but I
>> believe that it should be within the scope of a different document. 
> 
> I am curious: is there such an existing document?  Has there been
> any work in avt or mmusic for RTP benchmarking?  Would the bench-
> mark be specific to a codec?

There are at least two performance monitoring frameworks for RTP. The 
first one is RTCP with additional metrics, see RFC 3550 and RFC 3611. 
There is also the RMONMIB work, which provides another framework for 
real-time monitoring of performance.
http://www.ietf.org/internet-drafts/draft-ietf-rmonmib-raqmon-framework-16.txt

There is a very difficult problem when trying to benchmark end to end 
performance of media. The reason is that the performance depends on the 
users subjective quality impression. Errors affects the subjective 
quality different, and are also dependent on the consuming individual. 
Thus the connection between objective measurements and subjective 
quality is very difficult. A common method to get somewhere is to 
calibrate a objective measurement method towards subjective assessments. 
That way one can get decent quality of the objective measurement. 
However the important factor here is that theses measurements are only 
valid within the confines of the method used in the test. Thus certain 
system changes or simply values outside of the correlated values lack 
meaning.

The problem with the media plane is that beyond the network 
characteristics there is so much implementation dependent behavior that 
affect quality. It is also very difficult to setup probes in meaningful 
ways within these implementations that are comparable between 
implementations.

> 
> It would be nice to gauge some consensus in the WG around whether
> the "Media Plane" is inline with the notion of SIP benchmarking
> or not.
> 

In my knowledge there there is clearly separate parts which are linked 
in that the full performance of a VoIP "system" clearly is the sum of 
the signalling and the media plane behavior. The debate about how to 
correlate the signalling with the media plane measurements that we have 
had regarding the RFC 3611 update in AVT WG clearly shows there is 
important factors to consider here. The bottom line is that it is 
important to be able to correlate a signalling measurement with the 
media plane measurements to determine the system performance.

Cheers

Magnus Westerlund

Multimedia Technologies, Ericsson Research EAB/TVA/A
----------------------------------------------------------------------
Ericsson AB                | Phone [phone]
Torshamsgatan 23           | Fax   [phone]
S-164 80 Stockholm, Sweden | mailto: [e-mail]

_______________________________________________
bmwg mailing list
[e-mail]
https://www1.ietf.org/mailman/listinfo/bmwg