Re: [AVTCORE] Subject: AD review of draft-ietf-avtcore-monarch-13

[Albrecht Schwarz <[e-mail]>]

I would support that direction and structuring proposal.

Side comment to the notion of "end system metric":
We should be carefully because it could be interpretated in two directions:
a) metric related to RTP topology "RTP end system"
b) metric related to user equipment

There's a subtle difference between both, the "RTP end system" FUNCTION could be located in multiple different physical entities like user equipment, but also RTP-to-X gateway equipment. The later ones are not located at the end of the overall communication path.
Conclusion for this example: e.g. QoE metrics (as application level metrics) might be monitored in an "RTP end system" when located in user equipment, but not if "RTP end system" would be located in network level equipment. Consequently, the association of QoE metrics as "end system metrics" would be ambiguous.

My feeling:
we don't need necessarily the concept of an "end system metric", but we need a location concept for RTP monitors with respect to the topology of the end-to-end communication path ("which again might be completely e2e RTP based or some segments might be non-RTP ...").

-Albrecht





>________________________________
> Von: Colin Perkins <[e-mail]>
>An: Qin Wu <[e-mail]> 
>CC: Albrecht Schwarz <[e-mail]>; [e-mail]; [e-mail] 
>Gesendet: 23:21 Mittwoch, 23.Mai 2012
>Betreff: Re: [AVTCORE] Subject: AD review of draft-ietf-avtcore-monarch-13
> 
>Qin,
>
>The text you propose is better, but I'm still not sure it's clear. I wonder if it may be worth splitting that section into several paragraphs, and giving a little more description. One paragraph could discuss the possible locations for monitors: in end-systems, in middleboxes that are an active part of an RTP session (e.g., RTP mixers or translators), or in third-party devices that passively monitor an RTP session. For each of these locations, you could then include a paragraph that briefly explains the uses for monitors in that location, and the types of metrics they're suited to monitoring, perhaps with examples. 
>
>Colin
>
>
>
>
>On 23 May 2012, at 04:22, Qin Wu wrote:
>> Hi,
>> Take Robert,Colin and Albrechet's comments into account, I propose to rewrite section 3.3 with the following changes:
>> OLD TEXT:
>> "
>> 3.3.  RTP Sender/Receiver entities located in network nodes
>> 
>>   The location of the RTP Sender/Receiver entities may impact a set of
>>   meaningful metrics.  For instance, application level metrics for QoE
>>   related performance parameters are under most conditions measured at
>>   the user device that receives RTP data packets.  However in some
>>   cases, given the factors ( "measurement point location", "measurement
>>   model location", "awareness of content information", etc [P.NAMS])
>>   taken into account, such metrics may be measured in a network node
>>   instead of a user device.
>> 
>> "
>> NEW TEXT:
>> "
>> 3.3. Measurement point of the RTP monitor within the network.
>> 
>> The RTP monitors are spread across a single domain(e.g., network  domain) or 
>> multiple domains (e.g., between network domain and service domain)in the network.
>> The performance Metrics reported from the RTP monitor can be divided into end system 
>> metrics, application level metrics, transport level metrics. Some of these metrics 
>> may be specific to the measurement point of the RTP monitor or depend on where 
>> the RTP monitors are located in the network. For instance, end system metrics are only
>> available or measured at the RTP end system or terminal device. Application level
>> metrics that reflect end to end user experience at the application-level are measured 
>> more precise or less complicated at the RTP end system than measured from intermediary 
>> system (e.g., the RTP media translator) in the nework. For transport level metrics, takes
>> one way delay as an example, in the asymmetric network, the one way delay measured 
>> at the RTP sender entity is different from one way delay measured at the RTP 
>> receiver entity. 
>> "
>> 
>> Regards!
>> -Qin
>> ----- Original Message ----- 
>> From: "Colin Perkins" <[e-mail]>
>> To: "Qin Wu" <[e-mail]>
>> Cc: "Albrecht Schwarz" <[e-mail]>; <[e-mail]>
>> Sent: Tuesday, May 22, 2012 7:04 PM
>> Subject: Re: [AVTCORE] Subject: AD review of draft-ietf-avtcore-monarch-13
>> 
>> 
>> I also don't agree with deleting Section 3.3 from the monarch draft. 
>> 
>> The architecture needs to be clear that there are valid scenarios for RTP monitoring within a network, either in some middlebox that comprises part of an RTP session, or in a third-party monitoring device used for network management purposes. Section 3.3 is a reasonable place to discuss these, and to explain why the metrics that can be reported from middleboxes are different to those that can be captured at end systems.
>> 
>> Colin
>> 
>> 
>> 
>> On 22 May 2012, at 07:53, Qin Wu wrote:
>>> Hi,:
>>> ----- Original Message ----- 
>>> From: "Albrecht Schwarz" <[e-mail]>
>>> To: <[e-mail]>
>>> Cc: <[e-mail]>; <[e-mail]>
>>> Sent: Tuesday, May 22, 2012 11:18 AM
>>> Subject: Re: [AVTCORE] Subject: AD review of draft-ietf-avtcore-monarch-13
>>> 
>>> 
>>> Dear All,
>>> 
>>> concerning the resolution proposal for clause 3.3:
>>> I don't agree in just deleting again that section!
>>> There was quite a long debate for incorporating such a clause.
>>> The original text proposal was much more detailed. Perhaps the shortening to just a single paragraph is the main cause for unclarity.
>>> Thus, instead of removing previous agreed text, we just rather extend it again.
>>> 
>>> 
>>> [Qin]: 
>>> The reason I propose to remove the section 3.3 is 
>>> a. I think Robert is correct. The set of meaningful metrics isn't changed at all  wherever you collect them. It seems little sense 
>>>   to say "the location of the RTP Sender/Receiver entities may impact a set of meaningful metrics. " 
>>> b. Where to gather or measure metrics can be indicated by the definition of Transport metrics, End system metrics and application level metrics.
>>> c. Section 3.3 seems to discuss where to gather what metrics. Recommendation ITU-T G.1081 have already defined five monitoring points 
>>>   where the performance measurements can be made.
>>> 
>>>> I think the definition of application level metrics has already clarified that  application level metrics is normally gathered from user endpoint. 
>>> 
>>> No. We did discuss this already extensively.
>>> An RTP media translator could provide application level metrics because e.g. terminating the application protocol layer.
>>> Such an RTP topology is NOT located in an user endpoint!
>>> 
>>> [Qin]: Sorry, I should correct me to say QoE metric is ususally measured at the user endpoint.
>>> 
>>> Regards,
>>> Albrecht
>>> ________________________________
>>> [Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] 
>>> Re: [AVTCORE] Subject: AD review of draft-ietf-avtcore-monarch-13
>>> ________________________________
>>> 
>>> * To: Robert Sparks <rjsparks at nostrum.com>, IETF AVTCore WG <avt at ietf.org>, <draft-ietf-avtcore-monarch at tools.ietf.org>
>>> * Subject: Re: [AVTCORE] Subject: AD review of draft-ietf-avtcore-monarch-13
>>> * From: Qin Wu <bill.wu at huawei.com>
>>> * Date: Fri, 18 May 2012 18:37:56 +0800
>>> * Delivered-to: avt at ietfa.amsl.com
>>> * List-archive: <http://www.ietf.org/mail-archive/web/avt>
>>> * List-help: <mailto:[e-mail]?subject=help>
>>> * List-id: Audio/Video Transport Core Maintenance <avt.ietf.org>
>>> * List-post: <mailto:[e-mail]>
>>> * List-subscribe: <https://www.ietf.org/mailman/listinfo/avt>, <mailto:[e-mail]?subject=subscribe>
>>> * List-unsubscribe: <https://www.ietf.org/mailman/options/avt>, <mailto:[e-mail]?subject=unsubscribe>
>>> * References: <4FB2E323.3090406 at nostrum.com>
>>> ________________________________
>>> 
>>> Hi,Robert:
>>> Thank for your valuable review. please see my replies inline belows. Regards!
>>> -Qin
>>> ----- Original Message ----- 
>>> From: "Robert Sparks" <rjsparks at nostrum.com>
>>> To: "IETF AVTCore WG" <avt at ietf.org>; <draft-ietf-avtcore-monarch at tools.ietf.org>
>>> Sent: Wednesday, May 16, 2012 7:13 AM
>>> Subject: [AVTCORE] Subject: AD review of draft-ietf-avtcore-monarch-13 > Summary: There are issues to resolve with a revised ID before 
>>>> progressing to IETF LC.
>>>> 
>>>> It's a struggle to see the _architecture_ this document is supposed to 
>>>> describe. I suspect most of the working group energy went into Section 5 
>>>> (the guidelines for writing XR block definitions) - the rest of the 
>>>> document needs a similar level of attention to detail.
>>>> 
>>>> There are two major issues that I've tagged with a (*). I followed a 
>>>> mostly document-order flow instead of grouping the major issues at the top.
>>>> 
>>>> (*) Given the document's title, it should be very easy to answer "What 
>>>> is the monitoring architecture?" for people who are reading this draft 
>>>> for the first time.
>>>> The key message is that the monitoring architecture consists of Monitors 
>>>> that exchange information using RTCP Metric Blocks. [Qin]: Agree. >The text should say 
>>>> that. [Qin]: Okay. > There is a lot of prose in section 3 and visual detail in the 
>>>> diagram in Figure 1 that makes that extremely difficult to see. 
>>>> Simplifying the figure by removing the details about applications and 
>>>> transports, or at least moving them to a second figure, making the 
>>>> Monitor boxes stand out, and making the types of information flows (the 
>>>> arrows) visually distinctive would help. Simplify the prose (much of it 
>>>> could be removed) to focus on that key message. [Qin]: Agree, I propose to remove the figure 1 and squeeze the section 3.1 as follows: " 3.1. Overview The RTP monitoring architecture comprises the following two key functional components shown below: o RTP Monitor o RTP Metric Block Structure RTP Monitor is the functional component defined in the Real-time Transport Protocol [RFC3550]. It acts as a source of information gathered for monitoring purposes and exchanges information with the other RTP monitors using RTCP Metric Blocks. According to the definition of monitor in the RTP Protocol [RFC3550], the end system that runs an application program that sends or receives RTP data packets, an intermediate- system that forwards RTP packets to End-devices or a third party that observes the RTP and RTCP traffic but does not make itself visible to the RTP Session participants (i.e., the third party monitor depicted in figure 1) can act as the
>>> RTP monitor. The third party monitor should be placed on the RTP/RTCP paths between the sender, intermediate and the receiver. The RTP monitor also exposes real time Application QoS/QoE metric information in the appropriate report block format (e.g.,RTCP XR or othe non-RTP means) to the management system. Such information can be formulated as different metric types, e.g.,application level metric, transport level metric, end system metric, interval metric, cumulative metric, sampled metric, direct metric, composed metric,etc. Both the RTCP or RTCP XR can be extended to convey these metrics. The details on transport protocols for these metric are described in Section 3.2. RTP may be used to multicast groups, both Any Source Multicast (ASM) and Source Specific Multicast (SSM). These groups can be monitored using RTCP. In the ASM case, the monitor is a member of the multicast group and listens to RTCP XR reports from all members of the ASM
>>> group. In the SSM case, there is a unicast feedback target that receives RTCP feedback from receivers and distributes it to other members of the SSM group (see figure 1 of RFC5760). The monitor will need to be co-located with the feedback target to receive all feedback from the receivers (this may also be an intermediate system). In both ASM and SSM scenarios, receivers can send RTCP XR reports to enhance the reception quality reporting. " > In the definition of Composed metrics, the sentence "An example is a 
>>>> metric calculated based on derived metrics that have been measured." 
>>>> makes little sense. [Qin] The derived metrics in the sentence should been corrected as "direct metric". >Did you mean something like "An example is a metric 
>>>> derived by algorithmically combining measured metrics."? [Qin]:I agree with your suggested text and like to adopt your suggestion. > The definitions of Interval, Cumulative, and Sampled metrics are not clear. [Qin]: I agree the definition for these three metrics need to be more cleaer, I proposes to do the following changes: NEW Text: " Interval metrics Metrics of which the reported values are measured in the most recent measurement interval duration between successive metrics reports. Such measurement interval is equal to one RTCP report interval (RFC3550,Section 6.2). Cumulative metrics Metrics of which the reported values apply to the accumulation period characteristic of cumulative measurements. The reported values are measured during the accumulation period. The accumulation period spans over more than one RTCP report intervals. Sampled metrics Metrics of which the reported values only apply to the value of a continuously measured or
>>> calculated metric that has been sampled at any given instance of the RTCP report interval. One example of such metric is the inter-arrival jitter described in the section 6.4.1 of RFC3550. " > If the discussion at the end of section 3.1 is kept, the reference to 
>>>> RFC5760 should be a real reference, and RFC5760 should appear in the 
>>>> reference section. [Qin]: Okay. > The point of section 3.2 is not obvious. How does it contribute to the 
>>>> description of the architecture. It mixes talking about what report 
>>>> blocks are with occasionally needing to send them more frequently. Those 
>>>> points could be made separately and more simply. [Qin]: Okay. I like to incorporate the key points in the section 3.2 into section 3.1 > Section 3.3 is not clear - it starts off saying that the location of the 
>>>> sender and receiver may change what metrics are meaningful. It's not 
>>>> clear what you mean by location. You certainly don't mean geospatial 
>>>> coordinates. Do you mean anything other than whether the element is 
>>>> trusted or untrusted by the operations network? If so, say that. The 
>>>> rest of the paragraph only speaks to what element collects the metrics. [Qin] Section 3.3 is only used to describe where or from which RTP entity to gather the metrics (i.e., measurement point) in the physical network and is not focus of this document. Secondly, I think the definition of application level metrics has already clarified that application level metrics is normally gathered from user endpoint. > It's not clear that the set of meaningful metrics is changed at all - 
>>>> just where you collect them. If the set of meaningful metrics does in 
>>>> fact change, there needs to be additional discussion here. [Qin]: The metrics actually doesn't change.Since the definition of application level metrics Have already conveyed the meaning of what section 3.3 wants to describe, I propose to remove this section. > Section 4's first paragraph: Is "probably" the best the working group 
>>>> can say? [Qin]:No, will remove this word. > The last two sentences of the second bullet in section 4 do not parse. 
>>>> Please simplify them. [Qin]: Okay, how about change them as follows: NEW Text: " With these minimal number of RTCP statistics parameters mapped to non-RTCP measurements, it is hard to provide accurate measures of real time application quality,conduct detailed data analysis and creates alerts timly to the users. Therefore correlation between RTCP XR and non-RTP data should be provided. " > The first full sentence of the third bullet in section 4 does not make 
>>>> sense. The whole bullet could be clarified. [Qin]: Agree and will remove the first sentence. > The fourth bullet in section 4 and section 5.5 should call out that 
>>>> RFC3611 already set Block Type 255 aside for future extensions, 
>>>> anticipating the potential need to extend the block type space. [Qin]: Okay, How about revising the section 4,bullet 4 as follows: " Consumption of XR block code points. The RTCP XR block namespace is limited by the 8-bit block type field in the RTCP XR header. Space exhaustion may be a concern in the future. Anticipating the potential need to extend the block type space, it is noted in [RFC3611] that Block Type 255 is reserved for future extensions, " > In section 5, consider naming the subsections with the actual guideline 
>>>> instead of the problem the guideline is trying to address.
>>>> 5.1. Metric blocks should contain a single metric [Qin]: I suggest to change it as "Contain the single metrics in the metric block" > 5.2. Include the payload type and format parameters in the metric block [Qin]: Okay. > 5.3. Use RTCP SDES to correlate XR reports with non-RTP data [Qin]: Okay. > 5.4. Don't repeat measurement information across metric blocks
>>>> 
>>> [Qin]: It is not possible completely to avoid measurement information repeatition. e.g., for packet loss robustness if the report blocks for the same interval span over more than one RTCP packet then each must have the same measurement identity information So I suggest to change it as: "reduce measurement information across metric blocks" > This points out a structural problem with the section. Why is 5.5 here? 
>>>> It's not a guideline. [Qin]: Agree and also as discussed,Space exhaustion is not big issue. How about remove this section? > (*) And it raises a question about what the document is trying to 
>>>> achieve with section 5.4. That section claims the document is proposing 
>>>> a new XR block type to help avoid duplication, but it doesn't actually 
>>>> define the new block type. Is the guideline saying simply don't repeat 
>>>> measurements or its the guideline saying "use this new XR block type we 
>>>> haven't defined yet"? [Qin]: Just to clarify, we have already defined one new XR Block type to help avoid or reduce duplication. It is measurement information block. The relevant draft is available at: http://tools.ietf.org/html/draft-ietf-xrblock-rtcp-xr-meas-identity-06 >The section needs more clarity. If it really is 
>>>> trying to define a new type, there's more work to be done. [Qin]: In order to avoid such confusion, I proposed to do the following change to the section 5.4: OLD Text: " This memo proposes to define a new XR Block that will be used to convey the common time period and the number of packets sent during this period. " New text: " This memo proposes to define a new XR Block (i.e., the Measurement information block[I.D-ietf-xrblock-rtcp-xr-meas-identity]) that will be used to convey the common time period and the number of packets sent during this period. " > The Security Considerations section, particularly the third paragraph, 
>>>> needs to be clearer. What exactly do you mean by "monitoring should be 
>>>> prohibited", and how is the fact that third party monitors don't send 
>>>> RTCP relevant? Did you intend those to be separate points? [Qin]: Yes, they are on the different port. Such monitor may receive RTP packet but does not send RTCP packet. > Editorial Nits and Suggestions:
>>>> 
>>>> The first paragraph of the introduction is overly complex, and won't age 
>>>> well. I suggest replacing it with: "Multimedia services using the 
>>>> Real-Time Protocol (RTP) are seeing increased use. Standard methods for 
>>>> gathering RTP performance metrics from these applications are needed to 
>>>> manage uncertainties in the behavior and availability of their 
>>>> services." Then replace "These rapidly emerging standards," with 
>>>> "Standards". [Qin]: Okay. > Introduction paragraph 3 sentence 2 should start "The RTCP Guidelines 
>>>> [RFC5968]". It would be even better to use the full title from 5968: 
>>>> 'The "Guidelines for Extending the RTP Control Protocol (RTCP)" [RFC5968]' [Qin]: Okay. > This may become moot as section 3 is modified to make its main point 
>>>> more obvious, but RFC3550 does not characterize Monitors as "a source of 
>>>> information". That could be misunderstood. Please choose another word. 
>>>> Perhaps "repository"? [Qin]: Okay, repository looks good to me. > In bullet lists, starting an entry with "or" or "and" is unnecessary. It 
>>>> should be clear from the introduction to the list whether the individual 
>>>> entries should be combined or selected from. [Qin]: Okay. > In section 3.1 where you say "RTP may be used to multicast groups" did 
>>>> you mean "RTP may be sent to multicast groups" or "RTP may be used with 
>>>> multicast groups"? [Qin]: I think it should be the latter case,i.e.," RTP may be used with 
>>> multicast groups". > 
>>>> In section 3.2 "The following are four use cases but are not limited 
>>>> to:" does not parse. The bullets aren't really use cases as much as they 
>>>> are mechanisms. Would it make more sense to say "Here are four 
>>>> mechanisms that have been created to address the need for more frequent 
>>>> reporting. This list is not intended to be exhaustive." The third bullet 
>>>> in that list should say "RTCP and RTCP XR are" instead of "RTCP or RTCP 
>>>> XR is". [Qin]: Okay, but I have incorporated the keypoints of section 3.2 into section 3.1. > The bullets in section 4 would be better as subsections with titles 
>>>> (taken from the first phrase of the bullet). [Qin]: Okay. > I suggest this alternative for the first full sentence in the first 
>>>> bullet of section 4 : "A compound metrics block is designed to contain a 
>>>> large number of parameters from different classes for a specific 
>>>> application in a single block." [Qin]: Okay. > Section 5's title should be "Guidelines for reporting" instead of 
>>>> "Guideline for reporting" [Qin]: Okay. > In section 5.1 I suggest "justify the overhead," instead of "justify the 
>>>> overheads," [Qin]: Okay. > Section 7 - the last sentence of the first paragraph does not parse. The 
>>>> second paragraph (which is a single sentence) should be clarified - an 
>>>> MCU is not a topology. [Qin]: Okay. How about revising them as follows: OLD Text: " The topologies in this category are Topo-Video-Switch-MCU and Topo-RTCP-terminating-MCU. Such topologies based on systems do not behave according to the RTP protocol [RFC3550]. Considering the MCU and translator are two typical topologies in the two categories mentioned above, this document will take them as two typical examples to explain how RTCP XR report works in different RFC5117 topologies. " NEW Text: " The topologies in this category are Topo-Video-Switch-MCU and Topo-RTCP-terminating-MCU. Such topologies based on systems (e.g.,MCU) do not behave according to the RTP protocol [RFC3550]. Considering the MCU and translator are two typical intermediary systems in the two categories mentioned above, this document will take them as two typical examples to explain how RTCP XR report works in different RFC5117 topologies. " > The last two sentences of the first
>>> paragraph of the security 
>>>> considerations section add nothing to the document. I suggest deleting them. [Qin]: Okay. > _______________________________________________
>>>> Audio/Video Transport Core Maintenance
>>>> avt at ietf.org
>>>> https://www.ietf.org/mailman/listinfo/avt >
>>> ________________________________
>>> 
>>> * References: 
>>> * [AVTCORE] Subject: AD review of draft-ietf-avtcore-monarch-13 
>>> * From: Robert Sparks
>>> * Prev by Date: [AVTCORE] Protocol Action: 'Explicit Congestion Notification (ECN) for RTP over 
>>> UDP' to Proposed Standard (draft-ietf-avtcore-ecn-for-rtp-08.txt) 
>>> * Previous by thread: [AVTCORE] Subject: AD review of draft-ietf-avtcore-monarch-13 
>>> * Next by thread: [AVTCORE] Protocol Action: 'Explicit Congestion Notification (ECN) for RTP over 
>>> UDP' to Proposed Standard (draft-ietf-avtcore-ecn-for-rtp-08.txt) 
>>> * Index(es): 
>>> * Date
>>> * Thread
>>> _______________________________________________
>>> Audio/Video Transport Core Maintenance
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/avt
>> 
>> 
>> 
>> -- 
>> Colin Perkins
>> http://csperkins.org/
>> 
>> 
>
>
>
>-- 
>Colin Perkins
>http://csperkins.org/
>
>
>
>
>
>