Re: [rtcweb] [MMUSIC] RTP demux in JSEP and BUNDLE

Bernard Aboba <> Sat, 07 January 2017 00:48 UTC

Return-Path: <>
Received: from localhost (localhost []) by (Postfix) with ESMTP id 9CDCE1296D6; Fri, 6 Jan 2017 16:48:27 -0800 (PST)
X-Virus-Scanned: amavisd-new at
X-Spam-Flag: NO
X-Spam-Score: -1.999
X-Spam-Status: No, score=-1.999 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=ham autolearn_force=no
Authentication-Results: (amavisd-new); dkim=pass (2048-bit key)
Received: from ([]) by localhost ( []) (amavisd-new, port 10024) with ESMTP id wrLO2EuHVU6T; Fri, 6 Jan 2017 16:48:24 -0800 (PST)
Received: from ( [IPv6:2607:f8b0:400c:c08::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by (Postfix) with ESMTPS id 1309B12966F; Fri, 6 Jan 2017 16:48:24 -0800 (PST)
Received: by with SMTP id i68so316174363uad.0; Fri, 06 Jan 2017 16:48:24 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=edd4bW3v15+fZqChu6iZBDAfDphbZKnXuv4wIj0fSuo=; b=T0RKUeB3CY8T3HO+36OHy+vpwwkFqJ0CG860PoTlSMOFZt0XiKQ52GVU//NkblU8E4 96cCXlGWjF8rQ5rbi1RDnm7uxS2kt2l6RbQDvJxpBvvhdq+q3Dq9aLCxfjQaBFSTNv2q NvSsyzk0b0sfJbpmBqem2WdQ0FjLi1KTR9ULWo0qUeClmDZ9c6H7Z0tWsn455+IaHfLI PLBOGM31pPzNIcy7ysji7zHsIL6HSTjMc7lAe2YDi08IjheuyL5hXvviDfz3OJWo939t GwvH+t8QDuOUCMv3bRgQuy8tn/R+qV+ZRhKMUKFxWQH9pGyVfNKMNfliVa6lbOL42VwL pSiQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=edd4bW3v15+fZqChu6iZBDAfDphbZKnXuv4wIj0fSuo=; b=WfZaIm5uRJBzttriWRmnk7KcY4SiJ65ozuOCcqDnvzOkjn2eXvzzDZA1zBB/C+xcHv x1GuYxSLY5OU9xIn0YROax1eE8UpBOXPOTQCAkENmj6YCmqPzvY+x91CT6JMZPZTEZmN zqdQ9Q59D5REl27edxUkkUxVSwG7Q6RwolvMqJqxjPLLZ9vB4BLWfrfhn6PVlQGDIStE rQgrCY9xuA3UTg0DlqlvjZcK7VNVsTI2AxtP0ZObp+w4l+z3Z0DE0RmcbZPDPv1SR1Va GBIXkSewoFqjqc5NAs20nTPLjv6/kiPLQUtvxU3RX3aQxHhNlFJcrn9Zw5Lyf2G0A+Xp p/sA==
X-Gm-Message-State: AIkVDXIv7ORLdYVylX2Lil0o6CMOgP1XRc0xOZ5dX1zDMWsH++DeekS0R1PmonrVacAbmb8Pcr3O9mZnzotjjw==
X-Received: by with SMTP id l10mr49935579uab.166.1483750102763; Fri, 06 Jan 2017 16:48:22 -0800 (PST)
MIME-Version: 1.0
Received: by with HTTP; Fri, 6 Jan 2017 16:48:02 -0800 (PST)
In-Reply-To: <>
References: <> <> <>
From: Bernard Aboba <>
Date: Fri, 06 Jan 2017 16:48:02 -0800
Message-ID: <>
To: Peter Thatcher <>
Content-Type: multipart/alternative; boundary="f403045ddf7acf8ed60545767d23"
Archived-At: <>
Cc: "mmusic (E-mail)" <>,, "" <>, Colin Perkins <>,
Subject: Re: [rtcweb] [MMUSIC] RTP demux in JSEP and BUNDLE
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Real-Time Communication in WEB-browsers working group list <>
List-Unsubscribe: <>, <>
List-Archive: <>
List-Post: <>
List-Help: <>
List-Subscribe: <>, <>
X-List-Received-Date: Sat, 07 Jan 2017 00:48:27 -0000

Peter said:

In practice, I think this is mostly word smithing and doesn't require a
virtual interim.  The big questions, which perhaps to require a virtual
interim, are:

1.  Do we want to include PT demux in the algorithm (when neither SSRC nor
MID match)?
2.  Do we want to include signaled SSRCs in the demux algorithm (the SSRC
table is loaded with signaled SSRCs)?

[BA]  I believe that we need to cover both #1 and #2.

1. There have been bugs filed against implementations that relate to PT
demux (e.g. SSRC change scenario).  So IMHO covering PT demux is
necessary.  JSEP-17 was pretty close, but tere are a few issues that still
need to be discussed:
     a.  SSRC latching on a PT match.  If the PT can change without SSRC
changing (e.g. in a video scenario where all codecs have the same
clockrate), it seems that latching could result in mis-routing.
     b. Whether the PT_table is filled if SSRCs are signaled in codecs
and/or RTX/FEC.  Implementations are doing different things here, and that
is going to cause problems.

2. Including signaled SSRCs in the demux algorithm is a practical
requirement because SSRC signaling is widely implemented and MID isn't
yet.  Also, when discussing RTCP segment routing, doesn't the SSRC_table
come into play (e.g. in deciding which receiver to route a Sender Report

On Fri, Jan 6, 2017 at 2:49 PM, Peter Thatcher <> wrote:

> I made a PR for JSEP that incorporates your text *and* adds a more
> specific algorithm which only uses MID and SSRC latching (my interpretation
> of your text, which I'm still not sure is correct):
> In practice, I think this is mostly word smithing and doesn't require a
> virtual interim.  The big questions, which perhaps to require a virtual
> interim, are:
> 1.  Do we want to include PT demux in the algorithm (when neither SSRC nor
> MID match)?
> 2.  Do we want to include signaled SSRCs in the demux algorithm (the SSRC
> table is loaded with signaled SSRCs)?
> The question to both of those is "yes" in PR 411 and "no" in PR 489.
> On Fri, Jan 6, 2017 at 11:23 AM Peter Thatcher <>
> wrote:
>> When I read your text, I get the impression that you want to simplify the
>> algorithm to only supporting MIDs for demux with SSRC latching.  No PT
>> demux.   No SSRC demux (except for ones latched by MID).   In other words,
>> if you want to use BUNDLE, you must use MID and only MID.
>> But it's not clear if that's the case.  Perhaps I missed something.  I
>> agree with others that a more explicit algorithm would make your intention
>> clear.
>> To help, here is my interpretation of your text as an algorithm.  Please
>> let me know if it's a correct interpretation:
>>     <section title="Appendix B" anchor="sec.appendix-b">
>>       <t>To prepare for demultiplexing RTP packets to the correct "m="
>>       line, the following steps MUST be followed for each BUNDLE
>>       group.</t>
>>       <t>
>>         <list>
>>           <t>Construct a table mapping MID to "m=" line for each "m="
>>           line in this BUNDLE group.</t>
>>           <t>Construct an empty table mapping incoming SSRC to "m=" line.
>> </t>
>>         </list>
>>       </t>
>>       <t>As "m=" lines are added or removed from the BUNDLE groups, or
>>       their configurations are changed, the tables above must also be
>>       updated.</t>
>>       <t>For each RTP packet received, the following steps MUST be
>>       followed to route the packet to the correct "m=" section within
>>       a BUNDLE group:</t>
>>       <t>
>>         <list>
>>           <t>If the packet has a MID and that MID is not in the table
>>           mapping MID to "m=" line, drop the packet and stop.</t>
>>           <t>If the packet has a MID and that MID is in the table
>>           mapping MID to "m=" line, update the incoming SSRC mapping
>>           table to include an entry that maps the packet's SSRC to the
>>           "m=" line for that MID.</t>
>>           <t>If the packet's SSRC is in the incoming SSRC mapping
>>           table, route the packet to the associated "m=" line and
>>           stop.</t>
>>           <t>Otherwise, drop the packet.</t>
>>         </list>
>>       </t>
>> ... similar for RTCP ...
>>     </section>
>> On Thu, Dec 8, 2016, 2:55 AM Colin Perkins <> wrote:
>> Hi,
>> [cc’ing RTCWEB and MMUSIC, since this relates to both JSEP and BUNDLE
>> drafts]
>> There’s been a lot of discussion on the lists, and at the meeting in
>> Seoul, around how RTP streams are mapped onto higher-level, application
>> meaningful, semantic roles. In particular, around how RTP streams map onto
>> JSEP objects for WebRTC. Magnus Westerlund and I would like to propose the
>> following updates to JSEP and BUNDLE to try to clarify the behaviour.
>> Comments and feedback very welcome.
>> Cheers,
>> Colin
>> # Replacement for Section 6 of draft-ietf-rtcweb-jsep-17
>>  <section title="Processing RTP/RTCP" anchor="sec.rtp.demux">
>>     <t>As described in <xref target="RFC3550"/>, RTP packets are
>>     associated with RTP streams <xref target="RFC7656"/>. Metadata
>>     about those streams, including source description information
>>     and reception quality feedback is conveyed in RTCP packets.
>>     Each RTP stream is identified by an SSRC, and each RTP packet
>>     carries an SSRC value that is used to associate the packet with
>>     the correct RTP stream. RTCP packets also use SSRCs to identify
>>     the RTP streams that the reports and metadata relate to.  RTCP
>>     packets generally carry multiple SSRC values and report on, or
>>     deliver source description information relating to, several RTP
>>     streams.</t>
>>     <t>Each incoming RTP stream, identified by its SSRC, is mapped to
>>     an m= section in the SDP. The SDP m= sections then correspond to
>>     RtpReceiver objects. This allows each RTP stream to be associated
>>     with an RtpTransceiver. Further processing of the RTP stream can
>>     then be done at the RtpTransceiver level.  This includes using
>>     RID <xref target="I-D.ietf-mmusic-rid"/> to distinguish between
>>     multiple Encoded Streams, as well as determine which Source RTP
>>     stream should be repaired by a given Redundancy RTP stream.</t>
>>     <t>The process of mapping RTP streams onto m= sections depends on
>>     whether streams are bundled or not. If the SDP BUNDLE extension
>>     is in use, then RTP streams are mapped onto m= sections based on
>>     the MID values as described in
>>     <xref target="I-D.ietf-mmusic-sdp-bundle-negotiation"/>.  If the
>>     SDP BUNDLE extension is not in use, each m= section corresponds
>>     to a transport layer connection and the RTP streams received on
>>     that connection correspond to the m= section.</t>
>>     <t>Incoming RTCP packets contain metadata including reception
>>     quality feedback, source description information, and other
>>     signalling relating to RTP streams. The RTCP packets are parsed,
>>     the associated RTP streams are identified based on the included
>>     SSRC values, and the metadata relating to those RTP streams is
>>     updated (this might include updating the MID information, used
>>     to associate RTP streams with m= sections, if the SDP BUNDLE
>>     extension is in use). This updated metadata is available to the
>>     RtpTransceiver objects associated with those RTP streams.
>>     </t>
>>  </section>
>> # Replacement text for section 10.2 of draft-ietf-mmusic-sdp-bundle-
>> negotiation-36
>>      <section anchor="sec-rtp-pt"
>>               title="Associating RTP Streams With Correct SDP Media
>> Description"
>>               toc="default">
>>        <t>As described in <xref format="default" pageno="false"
>>        target="RFC3550"/>, RTP packets are associated with RTP streams
>> <xref
>>        format="default" pageno="false" target="RFC7656"/>. Each RTP
>> stream is
>>        identified by an SSRC value, and each RTP packet carries an SSRC
>> value
>>        that is used to associate the packet with the correct RTP stream.
>>        packets also uses SSRCs to identify on which RTP streams any
>> report or
>>        feedback relate to. Thus, an RTCP packet will commonly carry
>> multiple
>>        SSRC values, and might therefore be providing feedback or report on
>>        multiple RTP streams. </t>
>>        <t>In order to be able to process received RTP packets correctly it
>>        must be possible to associate an RTP stream with the correct "m="
>>        line, as the "m=" line and SDP attributes associated with the "m="
>>        line contain information needed to process the packets.</t>
>>        <t>As all RTP streams associated with a BUNDLE group are part of
>> the
>>        same RTP session and using the same address:port combination for
>>        sending and receiving RTP/RTCP packets, the local address:port
>>        combination cannot be used to associate an RTP stream with the
>> correct
>>        "m=" line. In addition, multiple RTP streams might be associated
>> with
>>        the same "m=" line.</t>
>>        <t>Also, as described in <xref format="default" pageno="false"
>>        target="sec-rtp-sessions-pt"/>, the same payload type value might
>> be
>>        used by multiple RTP streams, in which case the payload type value
>>        cannot be used to associate an RTP stream with the correct "m="
>> line.
>>        However, there are cases where each "m=" line has unique payload
>> type
>>        values, and then the payload type could serve as hint to the
>> relevant
>>                     "m=" line the RTP stream is associated with.</t>
>>        <t>An offerer and answerer can inform each other which SSRC values
>>        they will use for an RTP stream by using the SDP 'ssrc' attribute
>>        <xref format="default" pageno="false" target="RFC5576"/>. However,
>> an
>>        offerer will not know which SSRC values the answerer will use until
>>        the offerer has received the answer providing that information.
>> Due to
>>        this, before the offerer has received the answer, the offerer will
>> not
>>        be able to associate an RTP stream with the correct "m=" line using
>>        the SSRC value associated with the RTP stream. In addition, the
>>        offerer and answerer may start using new SSRC values mid-session,
>>        without informing each other using the SDP 'ssrc' attribute.</t>
>>        <t>In order for an offerer and answerer to always be able to
>> associate
>>        an RTP stream with the correct "m=" line, the offerer and answerer
>>        using the BUNDLE extension MUST support the mechanism defined in
>> <xref
>>        format="default" pageno="false" target="sec-receiver-id"/>, where
>> the
>>        offerer and answerer includes the identification-tag (provided by
>> the
>>        remote peer) associated with an "m=" line in the RTP Streams and in
>>        RTCP SDES packets part of a BUNDLE group.</t>
>>        <t>The mapping from an SSRC to an identification-tag is carried in
>>        RTCP SDES packets or in RTP header extensions (<xref
>> format="default"
>>        pageno="false" target="sec-receiver-id"/>). Since a compound RTCP
>>        packet can contain multiple RTCP SDES packets, and each RTCP SDES
>>        packet can contain multiple chunks, an RTCP packet can contain
>> several
>>        SSRC to identification-tag mappings. The offerer and answerer
>> maintain
>>        tables mapping RTP streams identified by SSRC to "m=" lines
>> identified
>>        by the identification-tag.
>>        When receiving an RTP packet carrying a MID header extension
>>        with the identification-tag, or an RTCP packet carrying one or
>>        more SDES MID items, the offerer or answerer creates a mapping
>>        table entry between the SSRC value and the identification-tag,
>>        in order to associate the RTP stream associated with that SSRC
>>        value with the "m=" line corresponding to the
>> identification-tag.</t>
>>        <t>The mapping between the SSRC an identification-tag might change
>>        mid-session if, for a given SSRC value, a different
>> identification-tag
>>        is provided in an RTP or RTCP packet. In that case these tables are
>>        updated each time an RTP/RTCP packet containing a new mappings from
>>        SSRC to identification-tag is received. Some considerations for
>>        avoiding update flaps are provided in Section 4.2.6 of <xref
>>        target="RFC7941"/> which should be followed. </t>
>>        <t>If an offerer and answerer is not able to associate an RTP
>> stream
>>        with an "m=" line (using the mechanisms described in this section,
>> or
>>        using other appropriate mechanism, e.g., based on the payload type
>>        value if it is unique to a single "m=" line), it MUST either drop
>> the
>>        RTP packets associated with the RTP stream, or process them in an
>>        application specific manner, once non-stream specific processing
>>        (e.g., related to congestion control) of the RTP packets have
>>        occurred.</t>
>>        <t>When compound RTCP packets are received, they are split
>>        into their component RTCP packets and those component RTCP
>>        packets are processed based on their RTCP packet type, in
>>        the order in which they were placed into the compound RTCP
>>        packet. Non-compound RTCP packets are processed based on
>>        their RTCP packet type, in the order they are received. Information
>>        in each RTCP packet can relate to one or more RTP streams.
>>        For example, RTCP Sender Report (SR) and Receiver Report (RR)
>>        packets include an SSRC of sender field that indicates the
>>        identity of the participant that sent the RTCP packet, along
>>        with a list of Report Blocks. Each report contains data on the
>>        reception quality of a single RTP stream, identified by SSRC,
>>        as received by the SSRC that sent the RTCP packet. Other RTCP
>>        packet types similarly contain references to the SSRC of the
>>        sender of the RTCP packet, and the RTP streams to which it
>>        refers.</t>
>>        <t>It should always be possible to process RTCP packets, and
>>        store the received information in a data structure associated
>>        with an RTP stream, identified by SSRC, for later access and
>>        use. It is possible that RTCP packets relating to an SSRC can
>>        be received before RTP packets relating to that SSRC, so the
>>        data structures relating to an SSRC might need to be created
>>        before the corresponding RTP stream is received.</t>
>>        <t>Similarly, information relating to an RTP stream might be
>>        received before the data needed to map it onto an m= line is
>>        received. Information carried in RTCP packets relating to such
>>        an RTP stream that is application and/or "m=" line dependent
>>        MAY be dropped until the SSRCs is associated with a particular
>>        "m=" line. However, information to generate RTCP report blocks
>>        and other basic transport level feedback or reporting needs to
>>        be retained, so RTCP reports relating to the stream can be
>>        generated.</t>
>>      </section>
>> --
>> Colin Perkins
>> _______________________________________________
>> mmusic mailing list
> _______________________________________________
> mmusic mailing list