[Cellar] Matroska Elements to support frame side data

Dave Rice <dave@dericed.com> Thu, 01 November 2018 14:03 UTC

Return-Path: <dave@dericed.com>
X-Original-To: cellar@ietfa.amsl.com
Delivered-To: cellar@ietfa.amsl.com
Received: from localhost (localhost []) by ietfa.amsl.com (Postfix) with ESMTP id D7A2F1277BB for <cellar@ietfa.amsl.com>; Thu, 1 Nov 2018 07:03:49 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.12
X-Spam-Status: No, score=-1.12 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, HTML_MESSAGE=0.001, SPF_NEUTRAL=0.779] autolearn=no autolearn_force=no
Received: from mail.ietf.org ([]) by localhost (ietfa.amsl.com []) (amavisd-new, port 10024) with ESMTP id ykuTcS5YosnE for <cellar@ietfa.amsl.com>; Thu, 1 Nov 2018 07:03:48 -0700 (PDT)
Received: from server172-3.web-hosting.com (server172-3.web-hosting.com []) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 5D158124D68 for <cellar@ietf.org>; Thu, 1 Nov 2018 07:03:48 -0700 (PDT)
Received: from [] (port=35663 helo=[]) by server172.web-hosting.com with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.91) (envelope-from <dave@dericed.com>) id 1gIDZY-0032d0-Oe for cellar@ietf.org; Thu, 01 Nov 2018 10:03:46 -0400
From: Dave Rice <dave@dericed.com>
Content-Type: multipart/alternative; boundary="Apple-Mail=_6C13AD63-EB5E-48AB-BDC0-61EF352CE3FD"
Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\))
Message-Id: <24ED459F-2375-4934-9156-E64BB1A8AC05@dericed.com>
Date: Thu, 1 Nov 2018 10:03:44 -0400
To: Cellar list <cellar@ietf.org>
X-Mailer: Apple Mail (2.3273)
X-OutGoing-Spam-Status: No, score=-2.6
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - server172.web-hosting.com
X-AntiAbuse: Original Domain - ietf.org
X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12]
X-AntiAbuse: Sender Address Domain - dericed.com
X-Get-Message-Sender-Via: server172.web-hosting.com: authenticated_id: dave@dericed.com
X-Authenticated-Sender: server172.web-hosting.com: dave@dericed.com
X-From-Rewrite: unmodified, already matched
Archived-At: <https://mailarchive.ietf.org/arch/msg/cellar/eI_hZ722KuDBdS2X4HQNFC3bMlg>
Subject: [Cellar] Matroska Elements to support frame side data
X-BeenThere: cellar@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Codec Encoding for LossLess Archiving and Realtime transmission <cellar.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/cellar>, <mailto:cellar-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/cellar/>
List-Post: <mailto:cellar@ietf.org>
List-Help: <mailto:cellar-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/cellar>, <mailto:cellar-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 01 Nov 2018 14:03:50 -0000

Hi all,

I’m looking for thoughts on pull request https://github.com/Matroska-Org/matroska-specification/pull/276 <https://github.com/Matroska-Org/matroska-specification/pull/276> for Matroska which addresses an issue described at https://github.com/Matroska-Org/matroska-specification/issues/270 <https://github.com/Matroska-Org/matroska-specification/issues/270>. This would add a new Master Element called “Metadata” within BlockGroup and the Metadata Element would contain a MetadataName which labels the content of either a MetadataString or MetadataBinary Element (this is a bit similar to the TagName/TagString/TagBinary set of elements). These elements would allow information to be associated directly with the contents of the Block.

I also considered Michael Niedermayer’s work in the nut specification at http://ffmpeg.org/~michael/nut.txt <http://ffmpeg.org/~michael/nut.txt> where a structure is defined to store side metadata per frame, see the section on "sm_data / side_data / meta_data”.

Here are a few use cases where I think this structure could be helpful:

=== timecode
We’ve made many starts and stops at defining how to store timecode in Matroska but haven’t found consensus or resolution on those attempts. With the Metadata Element I think a timecode value could be stored such as


The Metadata Element would here would be encoded as 0xD297D38874696D65636F6465D48B30313A32333A34353B3132 which adds 25 bytes per frame (about 2MB per hour for PAL).

Currently the decklink input device in libavdevice can pass along timecode as frame side metadata (see https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/0946c0ec177dc48ef0677f890aa42d95e667c417 <https://git.ffmpeg.org/gitweb/ffmpeg.git/commit/0946c0ec177dc48ef0677f890aa42d95e667c417>) but Matroska offers no method to store such side data.

=== DPX -> MKV -> DPX lossless remuxing
When converting a DPX image sequence to FFV1/MKV the image data is losslessly stored but the DPX headers contain lots of information specific to DPX which is lost. The RawCooked project has an approach where that DPX header data is stored in Matroska attachments and then recalled and used when RawCooked is reverting from Matroska back to DPX (so the resulting DPX bitstream is identical as the input, similar to how flac can do this with wav->flac->wav. With this proposal the DPX header data could be stored within the Metadata Elements of the block group, such as:


Comments welcome.
Dave Rice