Re: [dmarc-ietf] indeterminisim of ARC-Seal b= value

[Peter Goldstein <[e-mail]>]

Sure, generating the output message requires an ARC implementation.  Same
as generating a DKIM signature for validation requires an actual DKIM
implementation.  Or generating an SPF result for a complex situation
generally requires an actual SPF implementation.  Or generating an
ARC-signed input message for the verifying side of the test suite requires
some sort of ARC signing implementation.  But I wouldn't agree with the
inference that "we're testing against an unknown impl that we can't
validate is correct" in any of these cases.

The validity of the test suite derives from the fact that:

1. It was constructed from the RFC, and is built to cover a reasonably
comprehensive set of cases - more complete than any individual developer is
likely to consider.
2. While the output may have been generated with a particular
implementation, the test suite should be validated against a number of
independently developed implementations.  That ensures correctness, and
surfaces discrepancies which may represent ambiguities in the specification
and/or errors in the test suite or one or more of the implementations.
3. Once N implementations have validated, the N+1th implementation is
guaranteed a high degree of interoperability if it also passes the test
suite.
4. If new edge cases are discovered or evolved, interoperability can be
assured by adding them to the test suite and testing existing
implementations.  If there is ambiguity or disagreement in this
circumstance, this is an opportunity for the community to come up with the
agreed-upon right answer for behavior in this new case, and update the test
suite to reflect that.

None of this requires that we consider some unknown implementation that we
can't validate is correct.  The test suite becomes a statement of
consistency among the different implementers and their interpretations of
the RFC.  See the OpenSPF.org test suite for a good example of how this
works.

Having a fully deterministic specification makes this all a lot easier.

As I've stated previously, the alternative to the above is bundling a
reference ARC verifier with the test suite.  That means someone needs to
write such a reference verifier (or repurpose and repackage an existing
one).   In addition, the test suite needs to be updated to ensure the
reference implementation can be used as a service when developing ARC
implementations in any language.  That's more work, and it also makes the
use of the test suite much more complex.

I think tightening up some currently allowed ambiguity in the ARC
specification is a much simpler and much better solution.  I'm not sure why
there's such concern about canonicalizing the format and ordering of some
tag/value pairs.

Best,

Peter

On Mon, Mar 27, 2017 at 4:36 PM, Brandon Long <[e-mail]> wrote:

> Generating the tests did require an arc impl , however.  I can understand
> if this was simple enough to generate them by hand or with a couple lines
> of python, but instead we're testing against an unknown impl that we can't
> validate is correct.
>
> Brandon
>
>
>
> On Mar 27, 2017 12:02 PM, "Peter Goldstein" <[e-mail]> wrote:
>
> Brandon,
>
> How do you validate that the signature verifies?  To do that you need an
> entire ARC implementation that is packaged within the test suite (and the
> ability for it to run as a service, or make it available in some way to
> implementations being tested that are written in different languages).
> That's the whole point.
>
> The tests under discussion are validating that with assorted inputs
> (original message, DNS configuration, etc.) an ARC implementation produces
> a valid output message (correct ARC seal, ARC chain, etc.).  Essentially
> we're looking at testing that:
>
> SIGN(INPUTS) => OUTPUT MESSAGE
>
> With the canonicalization being proposed, the OUTPUT MESSAGE becomes a
> deterministic result, and verifying it becomes a matter of string
> comparison.  This is the success criterion:
>
> OUTPUT MESSAGE == EXPECTED MESSAGE
>
> Without the canonicalization being proposed, it will be necessary to run
> verification on the OUTPUT MESSAGE to assess whether the test gives the
> expected results for a generic implementation.  In that case this is the
> success criterion:
>
> ARC_VALIDATION(OUTPUT MESSAGE, INPUTS) == EXPECTED_ARC_STATE(INPUTS)
>
> We can't use the ARC validation implementation for the library under test,
> because it may share bugs with the signing side.  Instead we'll need to
> have a bundled ARC validaiton with the test suite.
>
> Does that make sense?
>
> Best,
>
> Peter
>
>
>
>
>
> On Mon, Mar 27, 2017 at 11:06 AM, Brandon Long <[e-mail]> wrote:
>
>> What does validating the exact signature generated benefit us over just
>> verifying that the signature verifies?
>>
>> I haven't looked at what the tests are for signing, though given the
>> descriptions  for the verifying tests, I'm not sure it would be obvious.
>> Can you summarize them?
>>
>> Brandon
>>
>> On Mar 26, 2017 9:26 PM, "Peter Goldstein" <[e-mail]> wrote:
>>
>>> John,
>>>
>>>
>>>> I have to say I don't find that very persuasive.  It's not hard to
>>>> write test code that checks for the hashes and fields in a signature header
>>>> without regard to what order they're in.  I would rather write better test
>>>> code than add fragile extra cruft to every ARC implementation.
>>>
>>>
>>> I honestly find this response a little unpersuasive  :).  There are two
>>> points I'd like to raise.
>>>
>>> First, the idea that locking down the format of the signature adds
>>> 'fragile extra cruft' seems off to me.  On the contrary, the suggestions in
>>> this thread reduce fragility - by requiring that the set of signed fields
>>> (ARC Message Signatures, ARC Seals) meet a well-defined, strict, and
>>> trivially verifiable format.  This is fairly standard practice in video,
>>> security, etc. protocols - every field has a place, and large scale
>>> re-ordering/re-formatting is not permitted unless there's a compelling
>>> technical reason.
>>>
>>> I'm not sure why there's a perceived value in accommodating ambiguity
>>> here - this is a brand new standard with a limited number of actual
>>> implementers.  There's no random person not on this list implementing ARC
>>> (especially given the standard isn't finalized).  And if there is, the
>>> whole idea of providing a globally usable automated test suite is to ensure
>>> that person can easily validate that their implementation will interoperate.
>>>
>>> Second, the premise that "it's not hard to write test code..." is simply
>>> not true.  What would be required to actually write such code would be
>>> either a) pick a preferred ordering/formatting for these tags, and only
>>> have the tests pass for that ordering or b) include a complete ARC
>>> verifying system in the test code.  The first is what OpenDKIM did for its
>>> specs (see below).  The latter is possible, but seems ill-advised for a
>>> test suite.
>>>
>>> To see that this is the case, it's useful to look at two earlier email
>>> standards and their corresponding test suites.  SPF has a language-agnostic
>>> test suite (http://www.openspf.org/Test_Suite) that makes it easy to
>>> test implementations in any language.  I've done it myself, and it requires
>>> minimal effort - https://github.com/ValiMail/
>>> coppertone/tree/master/spec/open_spf.  I can tell you from personal
>>> experience that the test suite was really helpful in ensuring the
>>> implementation was correct in all cases.
>>>
>>> There is no corresponding test suite for DKIM, and the ambiguity in
>>> tag/value ordering is the fundamental reason why.  The closest we come are
>>> the tests for OpenDKIM.  If you examine the signing tests for OpenDKIM,
>>> it's clear that they fail your "not hard" requirement.  I can write a
>>> perfectly valid, legal DKIM signing implementation that will fail all of
>>> OpenDKIM's signing tests, because OpenDKIM has a preferred ordering and
>>> formatting for tags.
>>>
>>> The OpenDKIM signing tests all check that OpenDKIM produces a signature
>>> header matching a constant string that embodies this preferred ordering.
>>> Any implementation that uses a different ordering/formatting will obviously
>>> fail.  Changing the tag ordering/formatting changes the hashed value in the
>>> header, so it's not just a matter of parsing some tag/value pairs.  And so
>>> short of a full DKIM verification implementation it's impossible to make a
>>> generic test suite that accepts all legal implementations.
>>>
>>> Frankly, this is somewhat unfortunate.  But DKIM is a relatively simple
>>> protocol.  ARC is more complicated, with multiple levels of signatures and
>>> more complex state.  Given the substantially more complicated nature of
>>> ARC, it's desirable to have a test suite that will pass any legal
>>> implementation.  Interops are great, but I'd take a bulletproof test suite
>>> over an interop any day of the week.
>>>
>>> I won't be at the IETF meeting this week, but I'd be happy to dive into
>>> this in more detail over email or phone if it's helpful.
>>>
>>> Best,
>>>
>>> Peter
>>>
>>> On Sun, Mar 26, 2017 at 7:32 PM, John R Levine <[e-mail]> wrote:
>>>
>>>> It's even more important than it was with DKIM to have a test suite that
>>>>> can verify signing behavior.  If we don't agree on any sort of
>>>>> standard, a
>>>>> test suite will need to select a preferred format for the ARC headers &
>>>>> will fail all implementations that don't meet this form.  We've
>>>>> discussed
>>>>> this with Murray, and he agrees with this conclusion.
>>>>>
>>>>
>>>> I have to say I don't find that very persuasive.  It's not hard to
>>>> write test code that checks for the hashes and fields in a signature header
>>>> without regard to what order they're in.  I would rather write better test
>>>> code than add fragile extra cruft to every ARC implementation.
>>>>
>>>> As it happens, the IETF is meeting this week in Chicago, and I expect
>>>> I'll see Murray in the next day or two so I can talk to him directly.
>>>>
>>>>
>>>> R's,
>>>> John
>>>>
>>>> _______________________________________________
>>>> dmarc mailing list
>>>> [e-mail]
>>>> https://www.ietf.org/mailman/listinfo/dmarc
>>>>
>>>
>>>
>>>
>>> --
>>>
>>>
>>> [image: logo for sig file.png]
>>>
>>> Bringing Trust to Email
>>>
>>> Peter Goldstein | CTO & Co-Founder
>>>
>>> [e-mail]
>>> +1.415.793.5783 <(415)%20793-5783>
>>>
>>> _______________________________________________
>>> dmarc mailing list
>>> [e-mail]
>>> https://www.ietf.org/mailman/listinfo/dmarc
>>>
>>>
>
>
> --
>
>
> [image: logo for sig file.png]
>
> Bringing Trust to Email
>
> Peter Goldstein | CTO & Co-Founder
>
> [e-mail]
> +1.415.793.5783 <(415)%20793-5783>
>
>
>


-- 


[image: logo for sig file.png]

Bringing Trust to Email

Peter Goldstein | CTO & Co-Founder

[e-mail]
+1.415.793.5783