Re: [MMUSIC] [avtext] framemarking: add frame size info

Miguel París Díaz <> Thu, 10 November 2016 14:18 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 088B812950C; Thu, 10 Nov 2016 06:18:58 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -2.699
X-Spam-Status: No, score=-2.699 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id Fth8xCk4_jbo; Thu, 10 Nov 2016 06:18:55 -0800 (PST)
Received: from ( [IPv6:2a00:1450:400c:c09::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 62E121296AC; Thu, 10 Nov 2016 06:18:52 -0800 (PST)
Received: by with SMTP id c184so52992021wmd.0; Thu, 10 Nov 2016 06:18:52 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=5rWsDzhyfhf2rwSxRC4O3TYyC0whgcNH5xQ3fsfxeCs=; b=jWoO0P+U0OcwDpARENCjSBLAl7khRzdt2WLur15f+pUp8T5VDjxKY59tw6lEUUQZDm 7x/r6lJnWSdizUoWcC4PtD7PLbnp0hgqYn7YeF/4QgMM9762Nmb86tZafZT9M7vIV0zK YnBGZT365wW2LTOw4KtaF6QU03LtfC4mhwgt1DK6L58NUyoXP481QGVw/Uhm98vudKzI S4Cs+6OPJqJUqjmsyMdn+W+gvvFED3Z6wwT/tVTJy8fJK52E9RywAW22VLlH8ZytfJpS 4NO/vlk7fCcsJUDJCQR2vGHPY2sWPGlK5XxlN28okU/idNEYxM24Ywl7XjH1h6LgZNqz I2xw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=5rWsDzhyfhf2rwSxRC4O3TYyC0whgcNH5xQ3fsfxeCs=; b=FZOA8gZKsHQgg08GgcX8EE2D/RXBle9WrfOL84Xj3qWbDkn+edWC6wTxbd7Aljie4d d4os32tQolWU2RC5h/9nlbkS1uZBbE+fY8AodOi+9TLBBqgsFxw1SaNsbviOZI6c7UzU +0aBIaO/JQTHKD5Jj7bA9Lho0mz0Hd6byDVkBDBatBM0A6ROeyEBia0J0vyB6FXTIiDE FsHOaRmxMIIHF7jrfWqvz+2AGmT50Rcuju0HkWPPf8rQeTgs1vaDuzF/BpZ2G0kxt2SG 8wG7C55SSunzEn5DO8Qp27fwIiTqrLTsnPbBytE7hl+AcvX1fsJPjOCGb+wCQUynSdM4 wZ3g==
X-Gm-Message-State: ABUngvcBB5mF+xeZm8rsAGOmu2rsYqsgsEymN+CtwVC15/L6fA/OMwYpFBWcfnNfyyPNr6aXiELUoQmiACREvQ==
X-Received: by with SMTP id i71mr25468496wmd.100.1478787530850; Thu, 10 Nov 2016 06:18:50 -0800 (PST)
MIME-Version: 1.0
Received: by with HTTP; Thu, 10 Nov 2016 06:18:50 -0800 (PST)
In-Reply-To: <>
References: <> <em8de2860d-9b70-44ce-87e5-3c6ecb1fb1ee@sydney> <> <>
From: =?UTF-8?Q?Miguel_Par=C3=ADs_D=C3=ADaz?= <>
Date: Thu, 10 Nov 2016 15:18:50 +0100
Message-ID: <>
To: Jonathan Lennox <>
Content-Type: multipart/alternative; boundary=001a11442c02798e3b0540f30dcd
Archived-At: <>
Cc: "" <>, mmusic <>
Subject: Re: [MMUSIC] [avtext] framemarking: add frame size info
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Multiparty Multimedia Session Control Working Group <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Thu, 10 Nov 2016 14:18:58 -0000

in the new draft of sdp-simulcast an "RTP Aspect" section [1] has been
added, which explains how the media is handled on RTP level.

Specifically, In the Media-Switching Mixer section [2] the same thoughts I
exposed are said:

   This section discusses the behavior in cases where the RTP middlebox
   behaves like the Media-Switching Mixer (Section 3.6.2
in RTP
   Topologies [RFC7667 <>].  The
fundamental aspect here is that the media
   sources delivered from the middlebox will be the mixer's conceptual
   or functional ones.  For example, one media source may be the main
   speaker in high resolution video, while a number of other media
   sources are thumbnails of each participant.

   The above results in that the RTP stream produced by the mixer is one
   that switches between a number of received incoming RTP streams for
   different media sources and in different simulcast versions.  The
   mixer selects the media source to be sent as one of the RTP streams,
   and then selects among the available simulcast streams for the most
   appropriate one.  The selection criteria include available bandwidth
   on the mixer to receiver path and restrictions based on the
   functional usage of the RTP stream delivered to the receiver.  An
   example of the latter, is that it is unnecessary to forward a full HD
   video to a receiver if the display area is just a thumbnail.  Thus,
   restrictions may exist to not allow some simulcast streams to be
   forwarded for some of the mixer's media sources.

In our case to provide this feature, currently we have to depay the RTP
packets, and apply different types of parses (depending on the codec) to
read the frame size, which reduces the scalability of the system and hinder
the implementation a lot.
Because of that, I think that having frame size (width and height) info in
the Frame Marking RTP header extension is quite interesting to implement
this kind of use cases in a easy and efficient way (the same that an
audio-level extension header is provided to avoid analysing it in the
middlebox side).

I am adding MMUSIC group in the thread, because I think that this also
should be discussed in the context of the simulcast case.



2016-08-30 12:37 GMT+02:00 Miguel París Díaz <>om>:

> I assume that the media distributor has the information from the SDP (it
> performs the SDP negotiation which each "client"), but the point is that
> encoders may change the video size depending on the available bandwidth,
> the complexivity of the video source, etc., unless the sender forces the
> encoders' configuration with a fix frame size...
> 2016-08-26 19:07 GMT+02:00 Jonathan Lennox <>om>:
>> (As an individual.)
>> In the latest version of simulcast the media distributor would need the
>> RID values, not the PT values, but the idea is the same — it needs the SDP.
>> Note that if the media distributor doesn’t have information from the SDP
>> it can’t reliably identify the frame marking header extension at all, since
>> header extension IDs are negotiated. So I’m not sure how much benefit there
>> is to putting the size in the header extension.
>> That said, if we envision a scenario where encoders might be frequently
>> changing their video size (in response to available network bandwidth, or
>> the like), it might be useful for encoders to be able to indicate the
>> current size they’re encoding without needing to send updated SDP all the
>> time.
>> On Aug 26, 2016, at 12:52 PM, Paul E. Jones <>
>> wrote:
>> Miguel,
>> You make the assumption that the media distributor will not see the SDP,
>> I suppose.  While certainly a valid model, I'll admit that I had personally
>> assumed any media forwarding function would see the SDP (or at least be
>> told the PT values and any relevant flow information similar to what RFC
>> 6236 provides) and would thus know which PT values correspond to what video
>> resolutions if simulcast is employed.
>> Paul
>> ------ Original Message ------
>> From: "Miguel París Díaz" <>
>> To:
>> Sent: 8/25/2016 10:12:48 AM
>> Subject: [avtext] framemarking: add frame size info
>> Hello,
>> it would be great having frame size (width and height) info in the Frame
>> Marking RTP header extension [1].
>> Why?
>> For example, in the case of using simulcast in an SFU, selecting the
>> stream by the size would ease the application development and improve the
>> experience of the users.
>> Application developers don't usually have deep knowledge about media like
>> bitrate, etc., but they know which video size has to be rendered in the
>> GUI, which may depend on the client where the app is running: a mobile, a
>> PC with a 13"· screen, a PC with 27" screen, etc.
>> In this way and taking a videoconference app as example, if a participant
>> select another participant to be rendered as main video, the app could ask
>> the SFU to select the video quality that better matches to 800x600 size.
>> What do you think about this idea?
>> Thanks and best regards!!
>> Refs
>> [1]
>> --
>> Miguel París Díaz
>> ------------------------------------------------------------------------
>> Computer/Software engineer.
>> Researcher and architect in
>> ------------------------------------------------------------------------
>> _______________________________________________
>> avtext mailing list
> --
> Miguel París Díaz
> ------------------------------------------------------------------------
> Computer/Software engineer.
> Researcher and architect in
> ------------------------------------------------------------------------

Miguel París Díaz
Computer/Software engineer.
Researcher and architect in