Re: [MMUSIC] Fwd: New Version Notification for draft-abhishek-mmusic-overlay-grouping-00.txt

[Paul Kyzivat <[e-mail]>]

Stephan,

On 11/10/20 11:50 AM, Stephan Wenger wrote:
> Hi Paul,
> 
> I'm not Rohit, but we work closely together.  Please see inline in blue.
> 
> Thanks,
> 
> Stephan
> 
> On 11/3/20, 07:32, "mmusic on behalf of Paul Kyzivat" 
> <[e-mail] on behalf of [e-mail]> wrote:
> 
>      Rohit,
> 
>      IMO a lot more content is needed to clarify what you are proposing.
> 
> Agreed.
> 
>      For one thing, I gather this draft only covers signaling for the 
> overlay
> 
>      streams, not for the immersive video stream.
> 
> The draft covers the relationship between a “base” or “background” 
> stream (which could be immersive, see below), and one or more overlays.
> 
> I find almost no
> 
>      information on how the immersive video is signaled. (I see the 
> reference
> 
>      to [ISO23090], but apparently I have to pay to get a copy of that.)
> 
> Forget that reference for now.  The background of this draft relates to 
> work in 3GPP SA4, specifically ITT4RT (*Support of Immersive 
> Teleconferencing and Telepresence for Remote Terminals 
> <https://portal.3gpp.org/desktopmodules/WorkItem/WorkItemDetails.aspx?workitemId=820003>*).  
> They talk a lot about the background being immersive, projected video, 
> but for our purposes here that background could also be a simple planar 
> video stream.
> 
> ISTM
> 
>      that it is pretty important to associate the overlays with the 
> immersive
> 
>      stream in the signaling.
> 
> Beyond the information that an overlay is to be displayed “on top” of 
> whatever the background may be?  I’m not sure that is necessarily 
> correct. Making an oversimplifying analogy, when you draw something in 
> your favorite drawing program, you can send objects “to the front” or  
> “to the back”.  That’s all the functionality we are currently after 
> (though we are considering adding, to this draft, information that would 
> establish a hierarchy of overlays beyond their relationship with the 
> background, and also some geometric information.

My concern is that by not explicitly associating the overlay with the 
background you are assuming that there *is* a single well defined 
background that both ends agree on. What if there are multiple video 
streams in the session other than the overlay?

>      You apparently are aware of CLUE. It didn't contemplate exactly the 
> case
> 
>      you are interested in, but it is clearly very closely related, in that
> 
>      it has mechanisms to associate multiple streams together as parts of a
> 
>      telepresence session and to describe their relationship to one another.
> 
>      I think it would be straightforward to extend CLUE to cover your
> 
>      immersive telepresence. (It might take some work to incorporate the
> 
>      geometry / coordinate system of the immersive stream into the CLUE
> 
>      framework.)
> 
>      Of course CLUE hasn't been an overwhelming success. :-(
> 
>      But if there is a new interest in *immersive* telepresence then 
> perhaps
> 
>      there is some motivation to reopen the work.
> 
> Yes, we looked at CLUE.  CLUE could probably be extended to support such 
> use cases, but it would be a pretty heavy operation on a protocol suite 
> that struggles to find acceptance outside the narrowly defined, high-end 
> environment it was designed for.  Specifically, the issue at hand is 
> that CLUE describes the geometric relationship of media sources (camera 
> positions, for example), which can be expressed as a point of origin and 
> some angles.  Those areas of capture can, and typically do, overlap at 
> least to some extent.  No foreground/background relationship is defined 
> in CLUE between these sources, nor is transparency (which may be 
> something to look into for our draft). 

Yes. Clearly it would require enhancements to the clue model.

> Our equivalent of CLUE sources 
> are video streams—planes of reconstructed samples.  Their relationship 
> with the background is undefined.  Where those planes are rendered is a 
> user interface decision in the receiver.  We are not trying to re-invent 
> scene descriptions here.
> 
> https://www.3gpp.org/ftp/tsg_sa/WG4_CODEC/TSGS4_110-e/Docs/S4-201194.zip 
> <https://www.3gpp.org/ftp/tsg_sa/WG4_CODEC/TSGS4_110-e/Docs/S4-201194.zip> 
> covers the current thinking in 3GPP (phase 1—requirements phase), and 
> the figs in section 3.8 may be helpful.
> 
>      If you intend this as an *alternative* to CLUE for telepresence, then
> 
>      you need a lot more info in this document, or an associated framework
> 
>      document, to lay out the complete environment in which this overlay
> 
>      piece resides. Perhaps that is all defined in ISO documents. If so 
> then
> 
>      it will be difficult to do this work in IETF without access to the
> 
>      relevant ISO documents.
> 
> Agreed, but I hope I managed to describe above why an extension to the 
> CLUE protocol suite would be a lot heavier work than something as simple 
> as described in the draft at hand—using a reference to a document that’s 
> available to everyone.

Without getting into the weeds I can accept your conclusion that CLUE 
isn't appropriate here. (It might be good to put a few words saying why 
CLUE isn't a suitable base for this.)

I only ask that you resolve ambiguities in the document.

	Thanks,
	Paul

>                  Thanks,
> 
>                  Paul
> 
>      On 11/2/20 11:18 AM, Rohit Abhishek wrote:
> 
>      > Hi Christer,
> 
>      >
> 
>      > Thanks for reading the draft and your reviews. I really 
> appreciate your
> 
>      > feedback. The questions here suggest more explanatory text about
> 
>      > motivation is needed. I will address these in the next version.
> 
>      >
> 
>      > *From: *Christer Holmberg <[e-mail]>
> 
>      > *Date: *Thursday, October 29, 2020 at 7:46 AM
> 
>      > *To: *Rohit Abhishek <[e-mail]>, "[e-mail]"
> 
>      > <[e-mail]>
> 
>      > *Subject: *RE: [MMUSIC] Fwd: New Version Notification for
> 
>      > draft-abhishek-mmusic-overlay-grouping-00.txt
> 
>      >
> 
>      > Hi Rohit,
> 
>      >
> 
>      > A few initial questions for clarifications:
> 
>      >
> 
>      > *RA: The assumption here is for an immersive telepresence 
> session, one
> 
>      > immersive video will be transmitted, whereas other streams will be
> 
>      > transmitted as overlays. For ex, consider 10 people in a conference
> 
>      > call; a user may stream an immersive video from one user, whereas 2D
> 
>      > videos as overlays from other 8 participants (plus additional 
> streams if
> 
>      > any participant shares any additional media) . The total overlay 
> streams
> 
>      > may vary, for example, a session with a large number of audiences.*
> 
>      >
> 
>      > Q1:                      The draft references the telepresence 
> use-case
> 
>      > spec (RFC7205), but there is no text on why the actual CLUE 
> mechanism
> 
>      > isn’t feasible for what you want to do.
> 
>      >
> 
>      > *RA: I had referred to RFC7205 for definition purposes. CLUE enables
> 
>      > interoperability between telepresence systems by exchanging 
> information
> 
>      > about the systems’ characteristics. Here, the cameras and screens 
> may be
> 
>      > arranged to provide a panoramic view of the remote room.  However,
> 
>      > overlays were not considered in the CLUE mechanism.*
> 
>      >
> 
>      > Q2:                      It is unclear why you need to group the 
> overlay
> 
>      > streams together. If you only want to indicate that a stream is for
> 
>      > overlay you can e.g., use an SDP attribute for that. You don’t need
> 
>      > grouping.
> 
>      >
> 
>      > *RA: Of course, when a single or relatively fewer overlays are
> 
>      > transmitted, grouping won’t be needed. However, when relatively 
> larger
> 
>      > numbers of users are in a session, it may be more favorable to use a
> 
>      > grouping framework than using SDP attribute for each stream.*
> 
>      >
> 
>      > Q3:                      The draft talks about the overlay 
> streams being
> 
>      > associated with “immersive video”. But, there is no text about 
> how this
> 
>      > immersive video stream is represented, and how the overlay 
> streams are
> 
>      > associated with that immersive video stream.
> 
>      >
> 
>      > *RA: Thanks for the suggestion. I will update this in the next 
> version*
> 
>      >
> 
>      > Q4:                      I assume the overlay streams can be 
> multiplexed
> 
>      > using the BUNDLE mechanism?
> 
>      >
> 
>      > *RA: There maybe multiple RTP sessions whose streams may have to be
> 
>      > grouped. Using BUNDLE mechanism will only allow to group media 
> streams
> 
>      > within a single RTP session.  However, multiple media streams from
> 
>      > various endpoints may be multiplexed into a single RTP session 
> but would
> 
>      > require RTP-level middle boxes. These middle boxes may add delay 
> if a
> 
>      > large number of streams are to be multiplexed, which even if very 
> less,
> 
>      > may not be favorable for telepresence sessions.*
> 
>      >
> 
>      > *Best Regards,*
> 
>      >
> 
>      > *Rohit*
> 
>      >
> 
>      > Regards,
> 
>      >
> 
>      > Christer
> 
>      >
> 
>      > *From:* mmusic <[e-mail]> *On Behalf Of *Rohit 
> Abhishek
> 
>      > *Sent:* keskiviikko 28. lokakuuta 2020 0.23
> 
>      > *To:* [e-mail]
> 
>      > *Subject:* [MMUSIC] Fwd: New Version Notification for
> 
>      > draft-abhishek-mmusic-overlay-grouping-00.txt
> 
>      >
> 
>      > Dear All,
> 
>      >
> 
>      > I am currently working on an SDP extension facilitating overlays for
> 
>      > immersive telepresence.
> 
>      >
> 
>      > An individual draft has been posted at
> 
>      >
> 
>      > 
> https://www.ietf.org/archive/id/draft-abhishek-mmusic-overlay-grouping-00.txt 
> 
> 
>      > 
> <https://www.ietf.org/archive/id/draft-abhishek-mmusic-overlay-grouping-00.txt>
> 
>      >
> 
>      > Comments are welcome
> 
>      >
> 
>      > Best Regards,
> 
>      >
> 
>      > Rohit
> 
>      >
> 
>      > ---------- Forwarded message ---------
> 
>      > From: <[e-mail] <mailto:[e-mail]>>
> 
>      > Date: Tue, Oct 27, 2020 at 2:56 PM
> 
>      > Subject: New Version Notification for
> 
>      > draft-abhishek-mmusic-overlay-grouping-00.txt
> 
>      > To: Rohit Abhishek <[e-mail]
> 
>      > <mailto:[e-mail]>>
> 
>      >
> 
>      >
> 
>      >
> 
>      >
> 
>      > A new version of I-D, draft-abhishek-mmusic-overlay-grouping-00.txt
> 
>      > has been successfully submitted by Rohit Abhishek and posted to the
> 
>      > IETF repository.
> 
>      >
> 
>      > Name:           draft-abhishek-mmusic-overlay-grouping
> 
>      > Revision:       00
> 
>      > Title:          SDP Overlay Grouping framework for immersive
> 
>      > telepresence media streams
> 
>      > Document date:  2020-10-27
> 
>      > Group:          Individual Submission
> 
>      > Pages:          8
> 
>      > URL:
> 
>      > 
> https://www.ietf.org/archive/id/draft-abhishek-mmusic-overlay-grouping-00.txt 
> 
> 
>      > 
> <https://www.ietf.org/archive/id/draft-abhishek-mmusic-overlay-grouping-00.txt>
> 
>      > Status:
> 
>      > 
> https://datatracker.ietf.org/doc/draft-abhishek-mmusic-overlay-grouping/
> 
>      > 
> <https://datatracker.ietf.org/doc/draft-abhishek-mmusic-overlay-grouping/>
> 
>      > Htmlized:
> 
>      > 
> https://datatracker.ietf.org/doc/html/draft-abhishek-mmusic-overlay-grouping 
> 
> 
>      > 
> <https://datatracker.ietf.org/doc/html/draft-abhishek-mmusic-overlay-grouping>
> 
>      > Htmlized:
> 
>      > 
> https://tools.ietf.org/html/draft-abhishek-mmusic-overlay-grouping-00
> 
>      > 
> <https://tools.ietf.org/html/draft-abhishek-mmusic-overlay-grouping-00>
> 
>      >
> 
>      >
> 
>      > Abstract:
> 
>      >     This document defines semantics that allow for signalling a 
> new SDP
> 
>      >     group "OL" for overlays in an immersive telepresence 
> session.  The
> 
>      >     "OL" attribute can be used by the application to relate all the
> 
>      >     overlay media streams enabling them to be added as overlay on 
> top of
> 
>      >     the immersive video.  The overlay grouping semantics is 
> required, if
> 
>      >     the media data is seperate and transported via different 
> protocols.
> 
>      >
> 
>      >
> 
>      >
> 
>      >
> 
>      > Please note that it may take a couple of minutes from the time of 
> submission
> 
>      > until the htmlized version and diff are available at tools.ietf.org
> 
>      > <http://tools.ietf.org>.
> 
>      >
> 
>      > The IETF Secretariat
> 
>      >
> 
>      >
> 
>      > _______________________________________________
> 
>      > mmusic mailing list
> 
>      > [e-mail]
> 
>      > https://www.ietf.org/mailman/listinfo/mmusic
> 
>      >
> 
>      _______________________________________________
> 
>      mmusic mailing list
> 
>      [e-mail]
> 
>      https://www.ietf.org/mailman/listinfo/mmusic
>