Re: [Cellar] Matroska Elements to support frame side data

Steve Lhomme <slhomme@matroska.org> Sun, 11 November 2018 15:40 UTC

Return-Path: <slhomme@matroska.org>
X-Original-To: cellar@ietfa.amsl.com
Delivered-To: cellar@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5CD87128CB7 for <cellar@ietfa.amsl.com>; Sun, 11 Nov 2018 07:40:52 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.901
X-Spam-Level:
X-Spam-Status: No, score=-1.901 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, RCVD_IN_DNSWL_NONE=-0.0001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=matroska-org.20150623.gappssmtp.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id y-_WqQL1asiB for <cellar@ietfa.amsl.com>; Sun, 11 Nov 2018 07:40:49 -0800 (PST)
Received: from mail-pg1-x542.google.com (mail-pg1-x542.google.com [IPv6:2607:f8b0:4864:20::542]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 0C4A812785F for <cellar@ietf.org>; Sun, 11 Nov 2018 07:40:48 -0800 (PST)
Received: by mail-pg1-x542.google.com with SMTP id f8-v6so2908352pgq.5 for <cellar@ietf.org>; Sun, 11 Nov 2018 07:40:48 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=matroska-org.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=1YQFKK025+s9nTkgeTPPpVwirtXrnx1Yb945I7N/0J4=; b=eVlXFWuSCVGB4BAuDIwwetP1pDuPfGc2XwlTwAXNC/yl2gTi2/XTUZscZ0L5DmKGno I4RR58c0yMTD/suTvUlDC6Gk1JohgcCB53At/pD6KMvsEdB4eLGHaihKVvXBDu8nb0g4 SdYOg6QObRU767hebIQMnYH3IiKcV79UbV9JuMCc3BF3qMBiYzCp1Lcw8wOaWsd1ETBl KqIYBP1Q8TGNqHcBzQWGz3jg7wkfpLgZvXjCt0szFufRk42IRH59FVm4Q5ApS3/pbhIS cVIC1J+biPxYx58SGqRNTy+xj/ysOD5qrXx6D7yyugDqyTcY17DHDn3+nlTiFfhYNn+l ozuw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=1YQFKK025+s9nTkgeTPPpVwirtXrnx1Yb945I7N/0J4=; b=Nn9yn99FUlgzTOwhjmLRsONTJQDAyMIBMvlqXiwT/U7Tt6ggHsuaTdCUKSojkJp9Q5 CeQynN1jBTcnoHOl75HBhjX8cpINn/AFN++x60C61Ilu++ffWZCKOY3bFZibDz3WhK+M bmxPFtxHrHRWfR4sNzHCVoMofH9BDiVUjiRuRRetZk19mdP5Turku2jERQs4Lq1Vr8cE NSqiPuFKhs221m/dmiqG0WG2WtRH3D3gMWel+h/MD1LuAuChpxLoQoOTSMCKbOHvSrM2 WDj+KrvAi06BA82iDFtOb+EeuVfiDckmNTCLVOdIGy2m6EegamKrTCStu+LJZw5KwW7N RVEg==
X-Gm-Message-State: AGRZ1gKcCGMRiCeeRXuboU14O0SP3PeB9LhlUJH/zN6bVm5wzh6y5c+2 fzbVd7UkjhB7dbnkVgKg8y7JJtu71og3m+bdZQ8VQpWoZsGHFQ==
X-Google-Smtp-Source: AJdET5elxvUIS2MlKyIoWfJBFvj2ZjukrtLOP/mR9PHuY3xOfEMEOs1esExR+AUo/mjl05aPc0sXpy476l1ogBkbT5M=
X-Received: by 2002:aa7:818a:: with SMTP id g10-v6mr16622087pfi.153.1541950848307; Sun, 11 Nov 2018 07:40:48 -0800 (PST)
MIME-Version: 1.0
References: <24ED459F-2375-4934-9156-E64BB1A8AC05@dericed.com> <62624df3-77e8-3234-8496-30354570abee@noa-archive.com> <1E9938AC-B503-42A4-B1A9-43345F81B30F@dericed.com> <87r2fz7u01.fsf@bunkus.org>
In-Reply-To: <87r2fz7u01.fsf@bunkus.org>
From: Steve Lhomme <slhomme@matroska.org>
Date: Sun, 11 Nov 2018 16:40:36 +0100
Message-ID: <CAOXsMFJ-dEfn3k6xBwT1XQcKX_hvL1Om+0UVJGpscVgFDrB3xw@mail.gmail.com>
To: Moritz Bunkus <moritz=40bunkus.org@dmarc.ietf.org>
Cc: Dave Rice <dave@dericed.com>, Codec Encoding for LossLess Archiving and Realtime transmission <cellar@ietf.org>, Tobias Rapp <t.rapp@noa-archive.com>
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: <https://mailarchive.ietf.org/arch/msg/cellar/anjLkf1HyIkArWtI___YAF2maig>
Subject: Re: [Cellar] Matroska Elements to support frame side data
X-BeenThere: cellar@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Codec Encoding for LossLess Archiving and Realtime transmission <cellar.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/cellar>, <mailto:cellar-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/cellar/>
List-Post: <mailto:cellar@ietf.org>
List-Help: <mailto:cellar-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/cellar>, <mailto:cellar-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sun, 11 Nov 2018 15:40:53 -0000

I would go with the Track way as well. Primarily because storing a
string (which pretty much never changes) in each Block is a huge
waste.

Add extra data per Block is already supported using BlockAdditions.
There's already BlockAddID which correspond to Moritz' BlockMetadataID
and BlockAdditional which correspond to BlockMetadataString /
BlockMetadataBinary / BlockMetadataUInteger / BlockMetadataSInteger /
BlockMetadataFloat. It's like a codec where the CodecID defines how
the data in the binary blob should be interpreted.

The current system states that the additions are left to
interpretation to the codec. It was originally designed to hold the
lossless complement to lossy versions of Musepack. So in that case
it's really meant to be passed to the codec. I think we can expand
this system with keeping this default behaviour by default (albeit not
used anywhere) and have different ones on demand.
There's also an AlphaMode that also uses BlockAdditions to store the
alpha track. Which pretty much no info on how to do it....

As noted timecode may be a separate track (as originally intended) if
it is not related to the video frames (ie the timestamps doesn't
match).

This would look like this:
- Musepack lossless complement:
Segment\Tracks\TrackEntry\MaxBlockAdditionID: 1
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDValue: 1
(same as BlockAddID) (default)
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDName:
"complement" (default)
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDType: 0
(Codec Complement data) (default)
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAddID: 1
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAdditional:
lossless part interpreted by the codec

- Alpha layer:
Segment\Tracks\TrackEntry\MaxBlockAdditionID: 2
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDValue: 2
(same as BlockAddID)
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDName: "alpha"
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDType: 1
(Alpha layer data)
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAddID: 2
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAdditional:
alpha mask to apply on the video track

- RawCooked DPX data
Segment\Tracks\TrackEntry\MaxBlockAdditionID: 3
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDValue: 3
(same as BlockAddID)
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDName: "rawcooked"
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDType:
0x1234567 (rawcooked identifier)
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAddID: 3
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAdditional:
DPX data defined by RawCooked

- Timecode storing
Segment\Tracks\TrackEntry\MaxBlockAdditionID: 3
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDValue: 3
(same as BlockAddID)
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDName: "timecode"
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDType:
0x890ABCD (SMPTE TC identifier, can be another ID for different kind
of timecode)
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAddID: 3
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAdditional:
Timecode storage

That means the alpha mode would not be backward compatible with
existing files, because it requires non default values. But I don't
think anyone ever used this improperly defined feature.

The value of MaxBlockAdditionID is kept low on purpose. BlockAddID 1
was always for codec complement and thus 2 for the AlphaMode. But we
don't need to go much higher that than now that we have a mapping. If
there are Timecode AND Rawcooked it would be like this:
Segment\Tracks\TrackEntry\MaxBlockAdditionID: 4
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDValue: 3
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDName: "rawcooked"
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDType:
0x1234567 (rawcooked identifier)
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDValue: 4
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDName: "timecode"
Segment\Tracks\TrackEntry\BlockAdditionMapping\BlockAddIDType:
0x890ABCD (SMPTE TC identifier, can be another ID for different kind
of timecode)
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAddID: 3
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAdditional:
DPX data defined by RawCooked
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAddID: 4
Segment\Cluster\BlockGroup\BlockAdditions\BlockMore\BlockAdditional:
Timecode storage

Le lun. 5 nov. 2018 à 10:29, Moritz Bunkus
<moritz=40bunkus.org@dmarc.ietf.org>; a écrit :
>
> Hey,
>
> > In other thoughts on this suggestion, I think it could make it difficult
> > to easily understand if a file has a particular type of side data. For
> > instance if only a few Clusters somewhere in the Segment contain a
> > certain type of side data, it would require parsing every Cluster to know
> > what types of side data are available. This uncertainly wouldn’t be the
> > same issue if the side data was itself a Track.
>
> It's not entirely necessary to use a full track for side data. We can simply
> signal the presence of side data in the track headers and refer to it from
> the side data in the block groups. This would also mean we only have to
> store the string identifying the side data type once (in the track headers)
> instead of in each block.
>
> (I'll use "BlockMetadata" as the basis for all element names
> here. Initially I proposed "FrameMetadata", but "BlockMetadata" is fine
> with me, too.)
>
> For example:
>
> Tracks
> +- TrackEntry
>  +- TrackBlockMetadata (Master)
>   +- TrackBlockMetadataType (String, required)
>   +- TrackBlockMetadataID (Unsigned Integer, required)
>
> …
>
> Cluster
> +- BlockGroup
>  +- BlockMetadata (Master)
>   +- BlockMetadataID (Unsigned Integer, required, refers to existing
>      TrackBlockMetadataID in track headers)
>   +- BlockMetadataString (Unicode String, optional)
>   +- BlockMetadataBinary (Binary, optional)
>   +- BlockMetadataUInteger (Unsigned Integer, optional)
>   +- BlockMetadataSInteger (Signed Integer, optional)
>   +- BlockMetadataFloat (Float, optional)
>
> with the restriction that exactly one of (BlockMetadataString,
> BlockMetadataBinary, BlockMetadataUInteger, BlockMetadataSInteger,
> BlockMetadataFloat) must exist.
>
> Advantages as I see them:
>
> • Less overhead (no repeated string parsing required)
> • Quicker parsing (no repeated string parsing required)
> • Presence of meta data is known upfront
> • Not using a full-blown track for meta data would alleviate the need to
>   specify how all those track and block features (e.g. BlockDuration,
>   TrackDefaultDuration…) apply to a "meta data track".
>
> Kind regards,
> mosu
>
> _______________________________________________
> Cellar mailing list
> Cellar@ietf.org
> https://www.ietf.org/mailman/listinfo/cellar



-- 
Steve Lhomme
Matroska association Chairman