Re: [AVTCORE] Magnus Westerlund's Discuss on draft-ietf-payload-rtp-ttml-03: (with DISCUSS and COMMENT)

[Magnus Westerlund <[e-mail]>]

Hi James, 

Please see inline for response and comments. 


On Wed, 2019-10-16 at 15:17 +0000, James Sandford wrote:
> Thank you for the detailed review, Magnus. I've responded to your points in-
> line.
> 
> Regards,
> James
> 
> > ----------------------------------------------------------------------
> > DISCUSS:
> > ----------------------------------------------------------------------
> > 
> > James and WG,
> > 
> > I do have a couple of issues I want to have your feedback on if they should
> > be
> > corrected or not before proceeding to publication. Note they are for
> > discussion
> > and in cases where things have been discussed and there is consensus please
> > reference that so that I can take that into consideration when we resolve
> > these.
> > 
> > 1. Section 4.1:
> >        Timestamp:
> >        The RTP Timestamp encodes the time of the text in the packet.
> > 
> >        As timed text is a media that has duration, from a start time to an
> > end
> >        time, and the RTP timestmap is a single time tick in the chose clock
> >        resolution the above text is not clear. I would think the start time
> > of
> >        the document would be the most useful to include?
> 
> Suggested updated text for the Timestamp definition: 
> 
>     The RTP Timestamp encodes the start time of the document in User Data
> Words

Okay, and this corresponds to "E" mentioned below? 

> 
> >        I think the text in 4.2.1.2 combined with the above attempts to imply
> >        that the RTP timestamp will be the 0 reference for the time-
> > expression?
> > 
> >        I think this needs a bit more clarification. Not having detailed
> >        studied TTML2/1 I might be missing important details. But some more
> >        information how the document timebase:media time line connects to the
> >        RTP timestamp appears necessary.
> 
> Suggested updated text for the last sentence of Paragraph 2 of 4.2.1.2: 
> 
>     Computed TTML media times are offset relative to E in accordance with
> Section I.2 of [TTML2].
> 
> I'm hesitant to expand on the calculation of timing beyond that in this
> document. It is discussed at length in TTML2 which is included as a normative
> reference in this document. I'd like to avoid blurring the scope of this
> document into that of TTML2 if possible.

Okay, I am all fine with keeping that in the TTML2 specs. However, the point of
linking the timescales must be clear. And to me it wasn't clear that RTP
Timestamp value is equal to the instant E that is the basis for document. 

This also leads to another warning that likely need to be in the RTP payload
time. As the document "start time" is not strongly correlated with transmission
time of the RTP packet there should be a mention of that as the RTCP
measurements based on the RTP timestamp field will contain significant error
components. 

Section 5.2 of RFC 8088 says this: 

   RTP payload formats with a timestamp definition that results in no or
   little correlation between the media time instance and its
   transmission time cause the RTCP jitter calculation to become
   unusable due to the errors introduced on the sender side.  A common
   example is a payload format for a video codec where the RTP timestamp
   represents the capture time of the video frame, but frames are large
   enough that multiple RTP packets need to be sent for each frame
   spread across the framing interval.  It should be noted whether or
   not the payload format has this property.

> 
> > 2. A Discuss Discuss: As Timed Text is directly associated with one or more
> > video and audio streams and requires synchronization with these other media
> > streams to function correct. This leads to two questions.
> > 
> >        First of all is application/ttml+xml actually the right top-level
> > media
> >        type? If using SDP that forces one unless one have BUNDLE to use a
> >        different RTP session. Many media types having this type of
> > properties
> >        of being associated with some other media types have registered media
> >        types in all relevant top-level media types.
> 
> I think we need to be careful with assumptions here. Timed Text MAY be
> associated with Video and/or Audio streams but there is no requirement to do
> so. Just as there is no requirement for Video to be associated with Audio or
> visa-versa.
> 
> I'm cautious about opening a can of worms with regards to media types. But my
> personal opinion is that TTML is not video and it is not audio. It may be
> associated with them but it is not those. It therefore shouldn't have
> registered types for video, audio, etc. If the association of different top-
> level media types is currently difficult, that is an issue I believe should be
> addressed outside of the scope of this document.

Okay, if the needs exists additional media types can be registred. It is very
much a question of which RTP usage world have TTML2 being used. In a world that
supports multiple media types on the same RTP session this is not an issue. The
m= line or equivalent for declaring media sources will simply say that there is
one TTML2 media source using top type application. 

For the older style one media type per RTP session, the implication of
application/ttml2 is that one need to have an RTP session extra in parallel if
one don't already have an application/foo media type in the multi-media session.

I just want to ensure that people intended to use this don't throw a fit when
they realize that their system spec suddenly have to add another RTP session. 




> 
> >        Secondly, this payload format may need some references to mechanisms
> > in
> >        RTP and signalling that has the purpose of associating media streams?
> > I
> >        also assume that we have the interesting cases with localization that
> >        different languages have different time lines for the text and how
> > long
> >        it shows as there are different tranditions in different countries
> > and
> >        languages for how one makes subtitles.
> > 
> >        This may also point to the need for discussing the pick one out of n
> >        mechanism that a manifest may need.
> 
> I believe this is outside of the scope of this document in the same way it is
> for audio equivalents and other text formats.

Depends if the very basic mechanisms are sufficient. I do see that RTCP CNAME
work when all media sources from one endpoint shall be synchronized. And in more
advanced case Lipsynch signalling should work as long as all TTML2 tracks belong
to a single LS group and there are no overlapp. 

However, I think there are interesting cases that would occur if one have
multiple TTML2 media source, one for each language one provide and potentially
combined with multiple audio tracks for different dubbings. The question is if
one have a manifest description work for this case. I am uncertain if SDP can
actually properly express the relations between all these media sources. Maybe
someone else on the AVTCORE mailing list can answer this. 

> 
> > 3. Section 7.1:
> > 
> >        It may be appropriate to use the same Synchronization
> >   Source and Clock Rate as the related media.
> > 
> >        Using the same SSRC as another media stream in the same RTP Session
> > is
> >        no-no. If you meant to use multiple RTP sessions and associate them
> >        using the same SSRC in diffiernt, yes it works but is not
> > recommended.
> >        This points to the need for a clearer discussion of how to achieve
> >        linkage and the reasons for why same RTP timestamp may be useful or
> > not.
> 
> Thanks for spotting this. It should refer to the Reference Clock, not
> Synchronization Source.
> 
> > 4. Fragmentation:
> >        I think the fragmentation of an TTML document across multiple RTP
> >        payloads are a bit insufficiently described. I have the impression
> > that
> >        it is hard to do something more clever than to fill each RTP payload
> > to
> >        MTU limtiation, and send them out insequence. However, I think a firm
> >        requirement to apply RTP sequence number for a single document in
> >        consecutive numbers. Also the re-assebly process appear to have to
> >        parts for detecting what belongs together, same timestamp and last
> >        packet of document should have marker bit set. As a receiver can
> > loose
> >        the last packet in the previous document, still know that it has
> >        received everything for the following document. However, if the
> > losses
> >        are multiple, inspection of the re-assemblied document will be
> >        necessary to determine if the correct beginning is present. I have
> > the
> >        impression that a proper section discussing these matter of
> >        fragmentation and re-assembly are necessary for good interoperability
> >        and function.
> 
> This has been discussed in response to Adam Roach's Discuss.

Yes, Adam brought up one aspect that I didn't think about, the splitting of UTF
characters. My focus was on the transport aspects of having a clear description
of the behavior of senders and receivers when TTML documents gets split over
multiple RTP packets. 


> 
> > 
> > ----------------------------------------------------------------------
> > COMMENT:
> > ----------------------------------------------------------------------
> > 
> > A. Section 6.
> >        To my understanding the TTML document is basically not possible to
> >        encode better. A poor generator can create unnecessary verbose XML
> >        which could be shorter, but there are no possibility here to trade-
> > off
> >        media quality for lower bit-rate. I think that should be made more
> >        explicit in Section 6.
> 
> Again, this has been discussed in response to Adam's Discuss.
> 
> > B. Section 7.
> >        Wouldn't using 90kHz be the better default? 1kHz is the minimal from
> >        RTCP report that will work decently. However, if the timed text is
> >        primarily going to be synchronized with video 90k do ensure that
> >        (sub-)frame precise timing is possible to express. I don't see any
> > need
> >        raster line specific for time text so the SMPTE 27 MHz clock is not
> >        needed. And using non default for subtitling radio etc appears fine.
> 
> Could you justify a 90k default? Such a high rate isn't appropriate for most
> timed text use cases. As stated in my response to "2." above, there is no
> requirement for TTML to be used alongside video so defining a default rate
> based on video would be inappropriate. Just as we don't expect video to use a
> 48k rate or audio a 90k rate. The default for TTML should be appropriate to
> timed text and should not make assumptions about specific implementations.

Sure, leave one 1kHz clock in as default. Howver, if one synchronize to another
media or so, using the same time based has its benefits as it enables precise
synchronization between the two. For example to get the subtitles to synchronize
with the correct video frame. Otherwise I see that one frame errors could occur.
Not that they necessary are super visible to the user, but especially as scene
cuts if the subitle shows up on the last frame of the previous scene rather then
first of the new can be visible. Thus, I think you should consider a
recommendation for the cases where one like to synchronize the TTML with other
media to select matching clock frequencies for the RTP timestamp to avoid these
issues. 


> 
> > C. Repair operations and relation to documents. Based on basic properties of
> > TTML documents, I do think the repair operations should be highly targeting
> > single documents as there is likely seconds between documents, while the
> > fragments of a document will be sent in a rather short interval. That
> > recommendation would be good to include.
> 
> Also discussed in response to Adam's Discuss.

Fine

Cheers

Magnus Westerlund 


----------------------------------------------------------------------
Network Architecture & Protocols, Ericsson Research
----------------------------------------------------------------------
Ericsson AB                 | Phone  [phone]
Torshamnsgatan 23           | Mobile [phone]
SE-164 80 Stockholm, Sweden | mailto: [e-mail]
----------------------------------------------------------------------