Re: [art] Request for review of draft-diaz-lzip-07

[[e-mail]]

I'm OK with all of your points, except the following, which may or may
not be significant:

Antonio Diaz Diaz <[e-mail]> writes:
> Maybe I could change the phrase to something like:
>
> "Lzip uses a simplified form of the LZMA stream format to maximize 
> interoperability and provides an accurate and robust 3 factor integrity 
> checking."

I'd say "increase" or "improve" rather than "maximize".  I mean, are you
asserting it is not possible to improve interoperability any further?

>> This depends on exactly what is meant by data communications.
>
> I think I have used the same meaning as the RFCs for gzip, brotli, and zstd, 
> as all of them use a similar wording:

Hmmm.  I still don't like the usage, but clearly this isn't the place to
deal with my problem.

> Thanks. I have changed the phrase above to:
>
>     A lzip file consists of one or more independent "members" (compressed
>     data sets).
>
> Note that there is nothing in a gzip/lzip file but members. A file with no 
> members is an empty file, not a lzip file.

Actually, that's a tricky bit of semantics.  A compression scheme could
define the compression of an empty file to be an empty file.  But since
you've tightened up the wording, it's clear that lzip doesn't do that.

>>     Member size (8 bytes)
>>        Total size of the member, including header and trailer.  This
>>        field acts as a distributed index, allows the verification of
>>        stream integrity, and facilitates the safe recovery of undamaged
>>        members from multimember files.
>>
>> You might want to explain how this "facilitates the safe recovery of
>> undamaged members" -- I would assume that the recovery process is to
>> advance through the Lzip file looking for an ID, then seeing if decoding
>> can proceed successfully from that point.
>
> It basically allows to locate all the members efficiently (index), unless 
> some header or member size is corrupt, in which case it still helps in 
> validating the ID in the headers of the undamaged members.

Ugh, OK, it took me a while to realize that you can index the members of
an lzip file efficiently, but you have to start *at the end* and work
backward.

> Overflowing the data size field should not have bad effects in the decoding 
> itself. It will simply show a truncated decompressed size, just as "gzip -l" 
> has done until recently (but with a 32-bit data size field). Limiting the 
> "member size" field to 2 PiB is the easiest way of preventing the overflow, 
> but it can be prevented by other means, or not prevented at all.

Hmmm, it seems you can go one of two ways.  One is to be strict, where
the data size and member size must match in order for the member to be
valid.  That is what the current text specifies.  The other is to allow
overflow/truncation of one or both of the fields.  But you'd have to
change the text to make that valid, and you'd also need to check that
the reference implementation handles that correctly upon decoding.
(Note this could become a subtle point of incompatibility in decoding.)

My reflex is to have the strict requirement, but really there's little
or no loss of validation checking if the decoder only checks the lower
32 bits of the two quantities.  So there's no practical reason not to
change the text to the overflow/truncation version.

Dale