Re: [MMUSIC] Fwd: New Version Notification for draft-abhishek-mmusic-overlay-grouping-00.txt

[Christer Holmberg <[e-mail]>]

Hi,

Based on my understanding, and what I have heard and read, I would also agree that CLUE is not needed. However, as Paul also suggested, it would be good to add some text about that.

I also agree that it must explicitly be possible to associate the overlay streams with a given "main stream". We shall not assume that there will only be one main stream, eventhough that may be the case 3GPP is currently looking into.

If we want to use the grouping mechanism, one way would be to place the main stream and the overlay stream into the same group, and then e.g., use an attribute to indicate which is main and which is overlay. But again, let's get the requirements clear before we start designing the protocol mechanism :)

Regards,

Christer


-----Original Message-----
From: mmusic <[e-mail]> On Behalf Of Paul Kyzivat
Sent: tiistai 10. marraskuuta 2020 19.40
To: Stephan Wenger <[e-mail]>; [e-mail]
Subject: Re: [MMUSIC] Fwd: New Version Notification for draft-abhishek-mmusic-overlay-grouping-00.txt

Stephan,

On 11/10/20 11:50 AM, Stephan Wenger wrote:
> Hi Paul,
> 
> I'm not Rohit, but we work closely together.  Please see inline in blue.
> 
> Thanks,
> 
> Stephan
> 
> On 11/3/20, 07:32, "mmusic on behalf of Paul Kyzivat" 
> <[e-mail] on behalf of [e-mail]> wrote:
> 
>      Rohit,
> 
>      IMO a lot more content is needed to clarify what you are proposing.
> 
> Agreed.
> 
>      For one thing, I gather this draft only covers signaling for the 
> overlay
> 
>      streams, not for the immersive video stream.
> 
> The draft covers the relationship between a “base” or “background” 
> stream (which could be immersive, see below), and one or more overlays.
> 
> I find almost no
> 
>      information on how the immersive video is signaled. (I see the 
> reference
> 
>      to [ISO23090], but apparently I have to pay to get a copy of 
> that.)
> 
> Forget that reference for now.  The background of this draft relates 
> to work in 3GPP SA4, specifically ITT4RT (*Support of Immersive 
> Teleconferencing and Telepresence for Remote Terminals 
> <https://protect2.fireeye.com/v1/url?k=11f71834-4e6c2105-11f758af-86e2237f51fb-2bcb4dbf2fce02d0&q=1&e=230e16d8-da72-481d-8298-2d6153cfc91e&u=https%3A%2F%2Fportal.3gpp.org%2Fdesktopmodules%2FWorkItem%2FWorkItemDetails.aspx%3FworkitemId%3D820003>*).
> They talk a lot about the background being immersive, projected video, 
> but for our purposes here that background could also be a simple 
> planar video stream.
> 
> ISTM
> 
>      that it is pretty important to associate the overlays with the 
> immersive
> 
>      stream in the signaling.
> 
> Beyond the information that an overlay is to be displayed “on top” of 
> whatever the background may be?  I’m not sure that is necessarily 
> correct. Making an oversimplifying analogy, when you draw something in 
> your favorite drawing program, you can send objects “to the front” or 
> “to the back”.  That’s all the functionality we are currently after 
> (though we are considering adding, to this draft, information that 
> would establish a hierarchy of overlays beyond their relationship with 
> the background, and also some geometric information.

My concern is that by not explicitly associating the overlay with the background you are assuming that there *is* a single well defined background that both ends agree on. What if there are multiple video streams in the session other than the overlay?

>      You apparently are aware of CLUE. It didn't contemplate exactly 
> the case
> 
>      you are interested in, but it is clearly very closely related, in 
> that
> 
>      it has mechanisms to associate multiple streams together as parts 
> of a
> 
>      telepresence session and to describe their relationship to one another.
> 
>      I think it would be straightforward to extend CLUE to cover your
> 
>      immersive telepresence. (It might take some work to incorporate 
> the
> 
>      geometry / coordinate system of the immersive stream into the 
> CLUE
> 
>      framework.)
> 
>      Of course CLUE hasn't been an overwhelming success. :-(
> 
>      But if there is a new interest in *immersive* telepresence then 
> perhaps
> 
>      there is some motivation to reopen the work.
> 
> Yes, we looked at CLUE.  CLUE could probably be extended to support 
> such use cases, but it would be a pretty heavy operation on a protocol 
> suite that struggles to find acceptance outside the narrowly defined, 
> high-end environment it was designed for.  Specifically, the issue at 
> hand is that CLUE describes the geometric relationship of media 
> sources (camera positions, for example), which can be expressed as a 
> point of origin and some angles.  Those areas of capture can, and 
> typically do, overlap at least to some extent.  No 
> foreground/background relationship is defined in CLUE between these 
> sources, nor is transparency (which may be something to look into for our draft).

Yes. Clearly it would require enhancements to the clue model.

> Our equivalent of CLUE sources
> are video streams—planes of reconstructed samples.  Their relationship 
> with the background is undefined.  Where those planes are rendered is 
> a user interface decision in the receiver.  We are not trying to 
> re-invent scene descriptions here.
> 
> https://protect2.fireeye.com/v1/url?k=62e03972-3d7b0043-62e079e9-86e22
> 37f51fb-28c26b67238f4c8f&q=1&e=230e16d8-da72-481d-8298-2d6153cfc91e&u=
> https%3A%2F%2Fwww.3gpp.org%2Fftp%2Ftsg_sa%2FWG4_CODEC%2FTSGS4_110-e%2F
> Docs%2FS4-201194.zip 
> <https://protect2.fireeye.com/v1/url?k=cbce09a5-94553094-cbce493e-86e2
> 237f51fb-b2974a8cc1dbc52f&q=1&e=230e16d8-da72-481d-8298-2d6153cfc91e&u
> =https%3A%2F%2Fwww.3gpp.org%2Fftp%2Ftsg_sa%2FWG4_CODEC%2FTSGS4_110-e%2
> FDocs%2FS4-201194.zip> covers the current thinking in 3GPP (phase 
> 1—requirements phase), and the figs in section 3.8 may be helpful.
> 
>      If you intend this as an *alternative* to CLUE for telepresence, 
> then
> 
>      you need a lot more info in this document, or an associated 
> framework
> 
>      document, to lay out the complete environment in which this 
> overlay
> 
>      piece resides. Perhaps that is all defined in ISO documents. If 
> so then
> 
>      it will be difficult to do this work in IETF without access to 
> the
> 
>      relevant ISO documents.
> 
> Agreed, but I hope I managed to describe above why an extension to the 
> CLUE protocol suite would be a lot heavier work than something as 
> simple as described in the draft at hand—using a reference to a 
> document that’s available to everyone.

Without getting into the weeds I can accept your conclusion that CLUE isn't appropriate here. (It might be good to put a few words saying why CLUE isn't a suitable base for this.)

I only ask that you resolve ambiguities in the document.

	Thanks,
	Paul

>                  Thanks,
> 
>                  Paul
> 
>      On 11/2/20 11:18 AM, Rohit Abhishek wrote:
> 
>      > Hi Christer,
> 
>      >
> 
>      > Thanks for reading the draft and your reviews. I really 
> appreciate your
> 
>      > feedback. The questions here suggest more explanatory text 
> about
> 
>      > motivation is needed. I will address these in the next version.
> 
>      >
> 
>      > *From: *Christer Holmberg <[e-mail]>
> 
>      > *Date: *Thursday, October 29, 2020 at 7:46 AM
> 
>      > *To: *Rohit Abhishek <[e-mail]>, "[e-mail]"
> 
>      > <[e-mail]>
> 
>      > *Subject: *RE: [MMUSIC] Fwd: New Version Notification for
> 
>      > draft-abhishek-mmusic-overlay-grouping-00.txt
> 
>      >
> 
>      > Hi Rohit,
> 
>      >
> 
>      > A few initial questions for clarifications:
> 
>      >
> 
>      > *RA: The assumption here is for an immersive telepresence 
> session, one
> 
>      > immersive video will be transmitted, whereas other streams will 
> be
> 
>      > transmitted as overlays. For ex, consider 10 people in a 
> conference
> 
>      > call; a user may stream an immersive video from one user, 
> whereas 2D
> 
>      > videos as overlays from other 8 participants (plus additional 
> streams if
> 
>      > any participant shares any additional media) . The total 
> overlay streams
> 
>      > may vary, for example, a session with a large number of 
> audiences.*
> 
>      >
> 
>      > Q1:                      The draft references the telepresence 
> use-case
> 
>      > spec (RFC7205), but there is no text on why the actual CLUE 
> mechanism
> 
>      > isn’t feasible for what you want to do.
> 
>      >
> 
>      > *RA: I had referred to RFC7205 for definition purposes. CLUE 
> enables
> 
>      > interoperability between telepresence systems by exchanging 
> information
> 
>      > about the systems’ characteristics. Here, the cameras and 
> screens may be
> 
>      > arranged to provide a panoramic view of the remote room.  
> However,
> 
>      > overlays were not considered in the CLUE mechanism.*
> 
>      >
> 
>      > Q2:                      It is unclear why you need to group 
> the overlay
> 
>      > streams together. If you only want to indicate that a stream is 
> for
> 
>      > overlay you can e.g., use an SDP attribute for that. You don’t 
> need
> 
>      > grouping.
> 
>      >
> 
>      > *RA: Of course, when a single or relatively fewer overlays are
> 
>      > transmitted, grouping won’t be needed. However, when relatively 
> larger
> 
>      > numbers of users are in a session, it may be more favorable to 
> use a
> 
>      > grouping framework than using SDP attribute for each stream.*
> 
>      >
> 
>      > Q3:                      The draft talks about the overlay 
> streams being
> 
>      > associated with “immersive video”. But, there is no text about 
> how this
> 
>      > immersive video stream is represented, and how the overlay 
> streams are
> 
>      > associated with that immersive video stream.
> 
>      >
> 
>      > *RA: Thanks for the suggestion. I will update this in the next
> version*
> 
>      >
> 
>      > Q4:                      I assume the overlay streams can be 
> multiplexed
> 
>      > using the BUNDLE mechanism?
> 
>      >
> 
>      > *RA: There maybe multiple RTP sessions whose streams may have 
> to be
> 
>      > grouped. Using BUNDLE mechanism will only allow to group media 
> streams
> 
>      > within a single RTP session.  However, multiple media streams 
> from
> 
>      > various endpoints may be multiplexed into a single RTP session 
> but would
> 
>      > require RTP-level middle boxes. These middle boxes may add 
> delay if a
> 
>      > large number of streams are to be multiplexed, which even if 
> very less,
> 
>      > may not be favorable for telepresence sessions.*
> 
>      >
> 
>      > *Best Regards,*
> 
>      >
> 
>      > *Rohit*
> 
>      >
> 
>      > Regards,
> 
>      >
> 
>      > Christer
> 
>      >
> 
>      > *From:* mmusic <[e-mail]> *On Behalf Of *Rohit 
> Abhishek
> 
>      > *Sent:* keskiviikko 28. lokakuuta 2020 0.23
> 
>      > *To:* [e-mail]
> 
>      > *Subject:* [MMUSIC] Fwd: New Version Notification for
> 
>      > draft-abhishek-mmusic-overlay-grouping-00.txt
> 
>      >
> 
>      > Dear All,
> 
>      >
> 
>      > I am currently working on an SDP extension facilitating 
> overlays for
> 
>      > immersive telepresence.
> 
>      >
> 
>      > An individual draft has been posted at
> 
>      >
> 
>      >
> https://www.ietf.org/archive/id/draft-abhishek-mmusic-overlay-grouping
> -00.txt
> 
> 
>      >
> <https://www.ietf.org/archive/id/draft-abhishek-mmusic-overlay-groupin
> g-00.txt>
> 
>      >
> 
>      > Comments are welcome
> 
>      >
> 
>      > Best Regards,
> 
>      >
> 
>      > Rohit
> 
>      >
> 
>      > ---------- Forwarded message ---------
> 
>      > From: <[e-mail] 
> <mailto:[e-mail]>>
> 
>      > Date: Tue, Oct 27, 2020 at 2:56 PM
> 
>      > Subject: New Version Notification for
> 
>      > draft-abhishek-mmusic-overlay-grouping-00.txt
> 
>      > To: Rohit Abhishek <[e-mail]
> 
>      > <mailto:[e-mail]>>
> 
>      >
> 
>      >
> 
>      >
> 
>      >
> 
>      > A new version of I-D, 
> draft-abhishek-mmusic-overlay-grouping-00.txt
> 
>      > has been successfully submitted by Rohit Abhishek and posted to 
> the
> 
>      > IETF repository.
> 
>      >
> 
>      > Name:           draft-abhishek-mmusic-overlay-grouping
> 
>      > Revision:       00
> 
>      > Title:          SDP Overlay Grouping framework for immersive
> 
>      > telepresence media streams
> 
>      > Document date:  2020-10-27
> 
>      > Group:          Individual Submission
> 
>      > Pages:          8
> 
>      > URL:
> 
>      >
> https://www.ietf.org/archive/id/draft-abhishek-mmusic-overlay-grouping
> -00.txt
> 
> 
>      >
> <https://www.ietf.org/archive/id/draft-abhishek-mmusic-overlay-groupin
> g-00.txt>
> 
>      > Status:
> 
>      >
> https://datatracker.ietf.org/doc/draft-abhishek-mmusic-overlay-groupin
> g/
> 
>      >
> <https://datatracker.ietf.org/doc/draft-abhishek-mmusic-overlay-groupi
> ng/>
> 
>      > Htmlized:
> 
>      >
> https://datatracker.ietf.org/doc/html/draft-abhishek-mmusic-overlay-gr
> ouping
> 
> 
>      >
> <https://datatracker.ietf.org/doc/html/draft-abhishek-mmusic-overlay-g
> rouping>
> 
>      > Htmlized:
> 
>      >
> https://tools.ietf.org/html/draft-abhishek-mmusic-overlay-grouping-00
> 
>      >
> <https://tools.ietf.org/html/draft-abhishek-mmusic-overlay-grouping-00
> >
> 
>      >
> 
>      >
> 
>      > Abstract:
> 
>      >     This document defines semantics that allow for signalling a 
> new SDP
> 
>      >     group "OL" for overlays in an immersive telepresence 
> session.  The
> 
>      >     "OL" attribute can be used by the application to relate all 
> the
> 
>      >     overlay media streams enabling them to be added as overlay 
> on top of
> 
>      >     the immersive video.  The overlay grouping semantics is 
> required, if
> 
>      >     the media data is seperate and transported via different 
> protocols.
> 
>      >
> 
>      >
> 
>      >
> 
>      >
> 
>      > Please note that it may take a couple of minutes from the time 
> of submission
> 
>      > until the htmlized version and diff are available at 
> tools.ietf.org
> 
>      > <http://tools.ietf.org>.
> 
>      >
> 
>      > The IETF Secretariat
> 
>      >
> 
>      >
> 
>      > _______________________________________________
> 
>      > mmusic mailing list
> 
>      > [e-mail]
> 
>      > https://www.ietf.org/mailman/listinfo/mmusic
> 
>      >
> 
>      _______________________________________________
> 
>      mmusic mailing list
> 
>      [e-mail]
> 
>      https://www.ietf.org/mailman/listinfo/mmusic
> 

_______________________________________________
mmusic mailing list
[e-mail]
https://www.ietf.org/mailman/listinfo/mmusic