Re: [secdir] secdir review of draft-ietf-json-text-sequence-11

[Nico Williams <[e-mail]>]

On Thu, Dec 18, 2014 at 07:12:34AM -0500, Carl Wallace wrote:
> The suggestion is not to tolerate a missing <LF> but to not always add
> them in the text sequence encoder in the first place. The <LF> addition

They can be tolerated in the parser.  Logging applications that encode
sequences correctly might nonetheless be subject to truncation of
sequence elements (including RS and trailing LF bytes) while
incrementally encoding a sequence, followed by additional elements being
added later (again, O_APPEND...).

Aside: if we'd never considered truncation we also wouldn't have needed
RS bytes, just trailing LF bytes, and optional ones at that (which is
exactly what https://stedolan.github.io/jq implements and has for a long
time now).  The problem with truncation is that there was no way to
figure out how to resync a parser, therefore no way to make it tolerate
truncation, not without RS, which in turn makes RS-less sequences not so
useful for logging applications.  We could abandon this, but several
people expressed the need for this format only when this was addressed,
and I do believe using RS (or similar) is better than not.

> would essentially be an extension of the JSON text encoder (albeit
> implemented in the text sequence encoder).  There would be no ambiguity.

No extensions of the JSON text encoder can be had here, not even
notional ones.  We're not updating RFC7159, not even implicitly.

Off-the-shelf JSON text parsers and encoders must be sufficient to
implement JSON text sequence parsers and encoders.

> The parser would always leave it in. Where integrity mechanisms are used,

Whether the parser leaves it in or not is unimportant.  What's important
is that the parser uses it to detect truncation of JSON texts that can
be truncated and still be valid but _different_ from the original.

As noted, ambiguities result from this looseness about how the trailing
LFs are handled.  These are noted in the security considerations section
in the upcoming -12.  That should suffice.

> the auto-<LF> would need to be turned off and the source responsible for
> properly terminating inputs to the text sequence encoder.

Integrity mechanisms are out of scope in this document.

> Thinking more about stripping out the <LF>, that won’t always work either
> for the same reason you have the parser reject non-self-delimited texts
> that do not end with a <LF>. The text sequence encoder could terminate
> before the <LF> is written possibly leaving a <LF> added by the source
> exposed (and removed by a decoder). The parser really can’t know if the
> JSON encoder or JSON text sequence encoder added a trailing <LF>.

When the sequence parser knows that the encoder of a sequence wasn't
subject to truncation, then the parser can indeed require and remove the
trailing LF.  In other cases it can't.

As JSON texts can have arbitrary amounts of trailing whitespace,
integrity mechanisms are out of scope, and JSON texts have no canonical
form (and are subject to being changed when parsed then re-encoded
without meaningful modification to the parsed value), there's no need to
concern ourselves further with this matter.

The upcoming -12 notes the ambiguities and lack of integrity mechanism
support.  I propose no further changes regarding how and whether to
detect if a trailing LF was added by a sequence encoder or not, or
whether to pass it to the JSON text parser.

Nico
--