Re: [Cellar] On the multiplicity of Info elements

[Dave Rice <[e-mail]>]

> On Jan 14, 2016, at 12:54 PM, Sebastian G. <bastik> wrote:
> 
> 12.01.2016, 14:36 Moritz Bunkus:
>> Hey,
>> 
>>> From a sample set of 59,881 Matroska files uploaded to archive.org
>>> only 119 of them include CRC-32 Elements. Almost 0.2%! Additionally
>>> the webm Document Type lists the CRC-32 Element as unsupported. So I
>>> think it’s true that it’s rarely bothered with. The implementation of
>>> CRC-32 Elements does add many complications and perhaps demand has
>>> been too low to resolve them. Moritz may have comments here.
>> 
>> That's pretty much spot on. The CRC32 element was added comparatively
>> late; a lot of muxers had already been created by that time.
>> 
>> Most requests I've received where more concerned with error recovery
>> than with error detection. This is something that CRC32 wasn't meant
>> for; additionally CRC32 elements weren't meant to be written in the
>> cluster elements which make up the bulk of a Matroska file's data.
>> 
>> Yes, there is a feature request[1] for adding support in mkvmerge and
>> mkvinfo. It has been created in 2010, but except for those two people
>> mentioned in the bug I haven't really received any other requests for
>> such a functionality.
>> 
>> Kind regards,
>> mosu
>> 
>> [1] https://github.com/mbunkus/mkvtoolnix/issues/543
>> 
> 
> Hi,
> 
> to said ticket you added two entries [2][3] which mention a problem with
> verifying checksums.
> 
> Would that be much easier to handle if the CRC would be about the data
> that should be present in the file, but are omitted due to default
> values aren't written? Compared to creating a checksum of the data that
> actually get written and verifying said checksum.
> 
> I had a rather strong opinion towards including a CRC that covers the
> elements that are written. I'm just curious and if something can be
> solved elegantly rather than having to invest too much resources into
> it, it might be worth reconsidering. After all it would be defined
> within a proper specification.

The Matroska files with CRC Elements store a checksum of what data is stored in the Master-element rather than what is interpreted (via defaults).

For instance in the test file suite, test2.mkv has a CRC value in the EBML Header Element with a value of 0x67277B3A. The rest of the Master-element stores DocType, DocTypeVersion, and DocTypeReadVersion. The other EBML Header Elements of EBMLVersion, EBMLReadVersion, EBMLMaxIDLength, and EBMLMaxSizeLength are not stored but interpreted as the defined default values. The data as stored matched the checksum, not the data as interpreted.

I think switching the CRC to calculate interpreted data rather than stored data would present numerous challenges.

> Back then when I offered the bounty I expected matroska to be used a lot
> more often in the future by other people than me. My usecase for
> checksums was indeed just error recovery. However I am not against
> extending the potential usecases to have applications use it to detect
> errors.

A few use cases:

If I have a Matroska file in long-term storage and migrate it to new storage (hard drive to new hard drive, or LTO Tape, future format, etc), the CRC values can be verified to ensure the data is transferred authentically and that the data on the new form of storage.

Alternatively if the data is moved across a network the received file can be verified to ensure that it is still valid. This is possible with FLAC files as they have an MD5 in the header and CRCs per frame.

Also if a Matroska file in an archive needs to be changed. For instance to add contextual metadata or a supporting attachment, the file could be edited in place without affecting other Level 1 elements. After the edit the file could still be verified to ensure the Level 1 elements unaffected by the edit have maintained authenticity. A whole file checksum wouldn’t work in this case since the whole file is intended to be different before and after the change, but the parts that aren’t involved in the change can still be verified.

As discussed, CRC elements can allow authority in selecting one of an Identically-Recurring Element if there are many in a damaged file.

There’s also a use case with secure or legal recordings. The CameraV project, https://guardianproject.info/apps/camerav/, uses matroska and checksums in a way to allow verifiable recordings.

> [2] https://github.com/mbunkus/mkvtoolnix/issues/543#issuecomment-171036571
> [3] https://github.com/mbunkus/mkvtoolnix/issues/543#issuecomment-171422587
> 
> --
> Sebastian
> 
> _______________________________________________
> Cellar mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/cellar