[AVT] Re: Carrying SMPTE time-codes in RTP streams, discussion email

[Dave Singer <[e-mail]>]

At 4:01 PM -0800 2/13/05, lazzaro wrote:
>On Feb 13, 2005, at 3:45 PM, Dave Singer wrote:
>>>5.  Finally, I've specified a different representation of SMPTE time
>>>code values for the AC-3 payload format than you have. There's an
>>>extension defined in SMPTE 339M for applying time code values to audio
>>>frames, which may not be synchronous with their associated video frames.
>>>The key difference is that the 339M format adds a 'sample' field after
>>>the 'frames' field. This indicates which PCM sample in the coded audio
>>>frame the (video) frame number in the time stamp is associated with.
>>
>>That's important.  Thanks.  It needs thought.  If the RTP clock is 
>>sample-accurate, then I think I'm OK, as I am associating the 
>>time-code with an RTP timestamp, which may not be the timestamp of 
>>any given frame.
>
>Yes, this issue becomes harder if we end up supporting an RTP profile
>that includes a SMPTE time code in the RTP header.  We'll need some
>way to SMPTE stamp headers whose RTP timestamp does not coincide
>with an exact SMPTE timecode value, probably using something like
>the 'sample' field Brian describes.
>
>For example, as a rule, RTP MIDI RTP timestamps generated from a MIDI
>controller keyboard will not coincide with SMPTE timestamp positions ...

agreed.

here is the draft.  we can revise it later in the week, though I hope 
I have caught enough open issues to cover your questions and 
concerns.  I think I had better send this off to the IETF editors now!







Internet Engineering Task Force
INTERNET-DRAFT                                                 D. Singer
draft-singer-smpte-rtp-00                                 Apple Computer

                                                              Feb 13 2005
                                                     Expires: Aug 13 2005

              Associating SMPTE time-codes with RTP streams

IPR Notice

By submitting this Internet-Draft, I certify that any applicable patent
or other IPR claims of which I am aware have been disclosed, or will be
disclosed, and any of which I become aware will be disclosed, in
accordance with RFC 3668.


Status of This Memo

    This document is an Internet-Draft and is in full conformance with
    all provisions of Section 10 of RFC2026.

    Internet-Drafts are working documents of the Internet Engineering
    Task Force (IETF), its areas, and its working groups.  Note that
    other groups may also distribute working documents as Internet-
    Drafts.

    Internet-Drafts are draft documents valid for a maximum of six months
    and may be updated, replaced, or obsoleted by other documents at any
    time.  It is inappropriate to use Internet-Drafts as reference
    material or to cite them other than a "work in progress."

    The list of current Internet-Drafts can be accessed at
    http://www.ietf.org/1id-abstracts.html

    The list of Internet-Draft Shadow Directories can be accessed at
    http://www.ietf.org/shadow.html

    The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
    "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
    document are to be interpreted as described in RFC 2119.

Distribution of this document is unlimited.

Copyright Notice

    Copyright (C) The Internet Society (2005).  All Rights Reserved.




D. Singer                                                       [Page 1]





Internet Draft        draft-singer-smpte-rtp-00.doc        Feb 13 2005


Abstract

    This document describes a mechanism for associating SMPTE time-codes
    with media streams, in a way that is independent of the RTP payload
    format of the media stream itself.


1 Introduction

    First a brief background on SMPTE time-codes [SMPTE].

    SMPTE time-codes count frames.  There are two common forms of
    display:  either a simple counter, or what looks like a normal clock
    value (hh:mm:ss.frame).  When the frame rate is truly integer, then
    this can be a normal clock value, in that seconds tick by at the same
    rate as the seconds we know and love.

    However, NTSC video infamously runs slightly slower than 30
    frames/second.  Some people call it 29.97 (which isn't quite right)
    and some say that a frame takes 1001 ticks of a 30000 tick/second
    clock (which is closer).  Be that as it may, SMPTE time codes count
    30 of these frames and deem that to make a second.

    This causes a SMPTE time-code display to 'run slow' compared to real-
    time.  To ameliorate this, sometimes a format called drop-frame is
    used.  Some of the frame numbers are skipped, so that the counter
    periodically 'catches up' (so some time-code-seconds actually only
    have 28 or 29 frames in them).

    It is worth noting that in neither case is the SMPTE time-code an
    accurate clock;  in the first case, it runs slow, and in the second,
    the adjustments are abrupt and periodic - and still not quite
    accurate.  Hence in the rest of this document I try to be clear when
    referring to a second in a time-code as a 'time-code second'.

    However, SMPTE time-codes do run in real-time when used with systems
    with integral frames/second (e.g. film content at 24 frames/second,
    or PAL video).  The 'drift' issue is (I believe) unique to NTSC
    video.

2 Design Goals

    What we desire is a system that allows us to associate a SMPTE time-
    code with some media in an RTP [RTP] stream.  Since in RTP all media
    has a clock already, we can leverage that fact.  If we treat the
    media as having 'segments' of time in which the time-code is simply
    counting up, then the time-code anywhere within a segment can be
    calculated if you know:



D. Singer                                                       [Page 2]





Internet Draft        draft-singer-smpte-rtp-00.doc        Feb 13 2005


       1. the RTP timestamp of the start of the segment;
       2. the time-code of the start of the segment;
       3. the counting rate and other parameters of the time-code;
       4. the RTP timestamp where you want to know the time-code.

    My proposal is that we put periodic mappings between (1) and (2) into
    RTCP packets, and provide (3) the 'setup' information out-of-band,
    for example in SDP.  Then given a timestamp (4), we can calculate
    exactly what the time-code must be.

    The setup information includes:
        (the timescale of the RTP stream, already provided);
       the duration, in that timescale, of a single frame-count in the
       'frames' portion of the time-code
       the number of those frames that make a time-code-second
       the following booleans:
          is-NTSC-drop-frame:  should the usual 'left out numbers' of
          drop-frame be applied or not?
          wrap-at-24-hours:  should the hours portion wrap from 23 to 0,
          or keep counting up?
          allow-negative-time-codes:  are negative time-codes used in
          this stream?
          display-time-code-as-counter:  should the display be an integer
          frame-count, or hh:mm:ss.fr format?
          time-code-displayed:  is it intended that this time-code be
          displayed somehow?

    For example, if associated with a video track with the common time-
    scale of 90000, then frame-duration of 3003 and frames-per-tc-second
    of 30 would yield a 'normal' SMPTE time-code for NTSC video.
    Similarly values of 3750 and 24 yield a time-code for 24 fps film
    content, and so on.

    Now, we put into an RTCP APP packet (or a new RTCP packet), a mapping
    between an RTP time-stamp value and the time-code.  The RTP timestamp
    and the that time-code are 32 bits;  the time-code is either a signed
    counter value (if we're in counter format), or it is the format:
       hours(8) -- 0 to 255
       sign(1) -- 1 for negative, 0 for positive
       minutes(7) -- 0 to 59
       seconds(8) -- 0 to 59
       frames(8) -- 0 to (frames-per-tc-second - 1)

    This establishes the time-code for all RTP times greater than or
    equal to the one given, until a subsequent APP packet reestablishes
    the mapping.  It's unfortunate that the sign is in the middle, but
    that allows the hours to use the full range, and the minutes don't
    need to.



D. Singer                                                       [Page 3]





Internet Draft        draft-singer-smpte-rtp-00.doc        Feb 13 2005


    Note that the RTP time-stamp in the mapping may not match the time-
    stamp of any frame in the media stream.  For video, it normally
    would;  but a time-stamp transition may happen part-way through a
    decoded audio frame.  Since they share the same clock, the timing of
    that transition and the timing of the audio stream itself have the
    same accuracy.

3 Discussion

    This design has the advantage of introducing no new IP packets into
    the sessions, using low-bandwidth (vanishingly low in the case of
    streams with no discontinuities), and is independent of the design of
    the RTP packets themselves:  the RTP profile (including possibly
    encryption) and the RTP payload format.  SMPTE time-codes can be
    associated with any RTP stream, including those with existing payload
    formats.

    It might be argued that we could set the initial mapping also in the
    SDP, since RTCP packets might get lost.  But this means that the SDP
    now has to have knowledge of the RTP random offset, which is nasty;
    and if one puts this APP packet into all sender reports, it's
    probably good enough.  Then if you don't have time-codes, you don't
    have audio-video-sync either.

    This associates the time-code with a particular media stream.  An
    alternative would be to make it an RTP stream in its own right;  but
    the data rate is so low, this seems egregious.  And by packing it
    inline, we can do this backwards-compatible for gateways etc. that
    already handle dual-stream.

    The APP packets need not use the same RTP timestamp as the sender
    report in the same RTCP packet.  They can be sent 'ahead of need' if
    possible (e.g. for stored content, when the server can look-ahead) or
    just-in-time - send an RTCP immediately a discontinuity in the time-
    code is detected, and allow media-buffering in the client the chance
    to 'catch' the RTCP before the matching RTP packet is processed and
    displayed,

    If time-codes change more rapidly than RTCP packets are normally
    sent, then this might force a more rapid transmission.  However, it
    should be possible to send multiple mappings in one RTCP packet.

    There is no way in this draft to detect that an RTCP packet has been
    lost, and that a mapping may be being used outside its intended
    range.  The likelihood of this happening can be reduced, however, by
    permitting a pair of RTP times in the mapping, and defining that the
    mapping is only valid between those times.  This only works for
    stored media, when look-ahead is possible, of course.



D. Singer                                                       [Page 4]





Internet Draft        draft-singer-smpte-rtp-00.doc        Feb 13 2005


    This design has the advantage of being independent of both the RTP
    profile and the media-specific RTP payload design.  However, it does
    this at the expense of certainty.  It is also possible to imagine
    some payloads with explicit provision for this mapping.  However,
    such provision need not go as far as a full SMPTE time-code in every
    packet.  If the RTCP APP packets have a small 'time-code epoch'
    indicator, then that same epoch indicator can be in each packet.

    The current design assumes that clients will hold mappings until they
    are superseded, and that a client may need to buffer some number of
    upcoming mappings.  It may be necessary to introduce explicit
    statements about the amount of buffering needed.

    For trick modes, it may be desirable to signal that a given section
    of media has the time-code running in reverse;  this would require a
    new sign bit in the mapping record.

4 Security Considerations

    SMPTE time-codes are only informative and it is hard to see security
    considerations from associating them with media streams.

5 IANA Considerations

    None.

6 RFC Editor Considerations

    None.

7 Full Copyright Statement

    Copyright (C) The Internet Society (2005).  This document is subject
    to the rights, licenses and restrictions contained in BCP 78, and
    except as set forth therein, the authors retain all their rights.

    This document and the information contained herein are provided on an
    "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
    OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET
    ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED,
    INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
    INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
    WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

8 Intellectual Property Notice

    The IETF takes no position regarding the validity or scope of any
    intellectual property or other rights that might be claimed to



D. Singer                                                       [Page 5]





Internet Draft        draft-singer-smpte-rtp-00.doc        Feb 13 2005


    pertain to the implementation or use of the technology described in
    this document or the extent to which any license under such rights
    might or might not be available; neither does it represent that it
    has made any effort to identify any such rights.  Information on the
    IETF's procedures with respect to rights in standards-track and
    standards-related documentation can be found in BCP-11.  Copies of
    claims of rights made available for publication and any assurances of
    licenses to be made available, or the result of an attempt made to
    obtain a general license or permission for the use of such
    proprietary rights by implementers or users of this specification can
    be obtained from the IETF Secretariat.

    The IETF invites any interested party to bring to its attention any
    copyrights, patents or patent applications, or other proprietary
    rights which may cover technology that may be required to practice
    this standard.  Please address the information to the IETF Executive
    Director.


Acknowledgments

    Both Brian Link and John Lazzaro provided helpful comments on an
    initial draft (not all of which are yet addressed).




























D. Singer                                                       [Page 6]





Internet Draft        draft-singer-smpte-rtp-00.doc        Feb 13 2005


Authors' Contact Information
    David Singer
    Apple Computer, Inc.
    One Infinite Loop, MS:302-3MT
    Cupertino  CA 95014
    USA
    Email: [e-mail]
    Tel: [phone]


6. References
    [RTP]
    RFC3550, STD0064, RTP: A Transport Protocol for Real-Time
    Applications, H. Schulzrinne, S. Casner, R. Frederick, V. Jacobson,
    July 2003

    [SMPTE-12M]
    SMPTE 12M-1999, Television, Audio and Film - Time and Control  Code

Dates
                              Written: Feb 13 2005
                              Expires: Aug 13 2005





























D. Singer                                                       [Page 7]


-- 
David Singer
Apple Computer/QuickTime

_______________________________________________
Audio/Video Transport Working Group
[e-mail]
https://www1.ietf.org/mailman/listinfo/avt