Re: [dmarc-ietf] indeterminisim of ARC-Seal b= value

[Peter Goldstein <[e-mail]>]

Brandon,

How do you validate that the signature verifies?  To do that you need an
entire ARC implementation that is packaged within the test suite (and the
ability for it to run as a service, or make it available in some way to
implementations being tested that are written in different languages).
That's the whole point.

The tests under discussion are validating that with assorted inputs
(original message, DNS configuration, etc.) an ARC implementation produces
a valid output message (correct ARC seal, ARC chain, etc.).  Essentially
we're looking at testing that:

SIGN(INPUTS) => OUTPUT MESSAGE

With the canonicalization being proposed, the OUTPUT MESSAGE becomes a
deterministic result, and verifying it becomes a matter of string
comparison.  This is the success criterion:

OUTPUT MESSAGE == EXPECTED MESSAGE

Without the canonicalization being proposed, it will be necessary to run
verification on the OUTPUT MESSAGE to assess whether the test gives the
expected results for a generic implementation.  In that case this is the
success criterion:

ARC_VALIDATION(OUTPUT MESSAGE, INPUTS) == EXPECTED_ARC_STATE(INPUTS)

We can't use the ARC validation implementation for the library under test,
because it may share bugs with the signing side.  Instead we'll need to
have a bundled ARC validaiton with the test suite.

Does that make sense?

Best,

Peter





On Mon, Mar 27, 2017 at 11:06 AM, Brandon Long <[e-mail]> wrote:

> What does validating the exact signature generated benefit us over just
> verifying that the signature verifies?
>
> I haven't looked at what the tests are for signing, though given the
> descriptions  for the verifying tests, I'm not sure it would be obvious.
> Can you summarize them?
>
> Brandon
>
> On Mar 26, 2017 9:26 PM, "Peter Goldstein" <[e-mail]> wrote:
>
>> John,
>>
>>
>>> I have to say I don't find that very persuasive.  It's not hard to write
>>> test code that checks for the hashes and fields in a signature header
>>> without regard to what order they're in.  I would rather write better test
>>> code than add fragile extra cruft to every ARC implementation.
>>
>>
>> I honestly find this response a little unpersuasive  :).  There are two
>> points I'd like to raise.
>>
>> First, the idea that locking down the format of the signature adds
>> 'fragile extra cruft' seems off to me.  On the contrary, the suggestions in
>> this thread reduce fragility - by requiring that the set of signed fields
>> (ARC Message Signatures, ARC Seals) meet a well-defined, strict, and
>> trivially verifiable format.  This is fairly standard practice in video,
>> security, etc. protocols - every field has a place, and large scale
>> re-ordering/re-formatting is not permitted unless there's a compelling
>> technical reason.
>>
>> I'm not sure why there's a perceived value in accommodating ambiguity
>> here - this is a brand new standard with a limited number of actual
>> implementers.  There's no random person not on this list implementing ARC
>> (especially given the standard isn't finalized).  And if there is, the
>> whole idea of providing a globally usable automated test suite is to ensure
>> that person can easily validate that their implementation will interoperate.
>>
>> Second, the premise that "it's not hard to write test code..." is simply
>> not true.  What would be required to actually write such code would be
>> either a) pick a preferred ordering/formatting for these tags, and only
>> have the tests pass for that ordering or b) include a complete ARC
>> verifying system in the test code.  The first is what OpenDKIM did for its
>> specs (see below).  The latter is possible, but seems ill-advised for a
>> test suite.
>>
>> To see that this is the case, it's useful to look at two earlier email
>> standards and their corresponding test suites.  SPF has a language-agnostic
>> test suite (http://www.openspf.org/Test_Suite) that makes it easy to
>> test implementations in any language.  I've done it myself, and it requires
>> minimal effort - https://github.com/ValiMail/
>> coppertone/tree/master/spec/open_spf.  I can tell you from personal
>> experience that the test suite was really helpful in ensuring the
>> implementation was correct in all cases.
>>
>> There is no corresponding test suite for DKIM, and the ambiguity in
>> tag/value ordering is the fundamental reason why.  The closest we come are
>> the tests for OpenDKIM.  If you examine the signing tests for OpenDKIM,
>> it's clear that they fail your "not hard" requirement.  I can write a
>> perfectly valid, legal DKIM signing implementation that will fail all of
>> OpenDKIM's signing tests, because OpenDKIM has a preferred ordering and
>> formatting for tags.
>>
>> The OpenDKIM signing tests all check that OpenDKIM produces a signature
>> header matching a constant string that embodies this preferred ordering.
>> Any implementation that uses a different ordering/formatting will obviously
>> fail.  Changing the tag ordering/formatting changes the hashed value in the
>> header, so it's not just a matter of parsing some tag/value pairs.  And so
>> short of a full DKIM verification implementation it's impossible to make a
>> generic test suite that accepts all legal implementations.
>>
>> Frankly, this is somewhat unfortunate.  But DKIM is a relatively simple
>> protocol.  ARC is more complicated, with multiple levels of signatures and
>> more complex state.  Given the substantially more complicated nature of
>> ARC, it's desirable to have a test suite that will pass any legal
>> implementation.  Interops are great, but I'd take a bulletproof test suite
>> over an interop any day of the week.
>>
>> I won't be at the IETF meeting this week, but I'd be happy to dive into
>> this in more detail over email or phone if it's helpful.
>>
>> Best,
>>
>> Peter
>>
>> On Sun, Mar 26, 2017 at 7:32 PM, John R Levine <[e-mail]> wrote:
>>
>>> It's even more important than it was with DKIM to have a test suite that
>>>> can verify signing behavior.  If we don't agree on any sort of
>>>> standard, a
>>>> test suite will need to select a preferred format for the ARC headers &
>>>> will fail all implementations that don't meet this form.  We've
>>>> discussed
>>>> this with Murray, and he agrees with this conclusion.
>>>>
>>>
>>> I have to say I don't find that very persuasive.  It's not hard to write
>>> test code that checks for the hashes and fields in a signature header
>>> without regard to what order they're in.  I would rather write better test
>>> code than add fragile extra cruft to every ARC implementation.
>>>
>>> As it happens, the IETF is meeting this week in Chicago, and I expect
>>> I'll see Murray in the next day or two so I can talk to him directly.
>>>
>>>
>>> R's,
>>> John
>>>
>>> _______________________________________________
>>> dmarc mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/dmarc
>>>
>>
>>
>>
>> --
>>
>>
>> [image: logo for sig file.png]
>>
>> Bringing Trust to Email
>>
>> Peter Goldstein | CTO & Co-Founder
>>
>> [e-mail]
>> +1.415.793.5783 <(415)%20793-5783>
>>
>> _______________________________________________
>> dmarc mailing list
>> [e-mail]
>> https://www.ietf.org/mailman/listinfo/dmarc
>>
>>


-- 


[image: logo for sig file.png]

Bringing Trust to Email

Peter Goldstein | CTO & Co-Founder

[e-mail]
+1.415.793.5783