Re: [dmarc-ietf] indeterminisim of ARC-Seal b= value

["Murray S. Kucherawy" <[e-mail]>]

On Sun, Mar 26, 2017 at 11:25 PM, Peter Goldstein <[e-mail]>
wrote:

> Second, the premise that "it's not hard to write test code..." is simply
> not true.  What would be required to actually write such code would be
> either a) pick a preferred ordering/formatting for these tags, and only
> have the tests pass for that ordering or b) include a complete ARC
> verifying system in the test code.  The first is what OpenDKIM did for its
> specs (see below).  The latter is possible, but seems ill-advised for a
> test suite.
>

OpenDKIM does both.  At some point when it was still "dkim-milter", when it
came time to write tests (because at one point there were none), I mainly
wanted to assure that commits made didn't cause any regressions;
correctness could be affirmed in other ways like at interop events or
through other implementers' auto-responders.  So I picked a tag ordering I
liked, generated a bunch of signatures with it, and then wrote tests that
did two things with them:

1) Confirmed that the signing code produces those specific signatures given
fixed inputs (timestamp, key, selector, domain, canonicalization, etc.).
This was inherently a tautology at the time I generated them of course, but
my goal was just to ensure no regressions were introduced.  I didn't know
for sure that the output was right, but I knew it would be consistent.

2) Since OpenDKIM conveniently includes a verifying capability, there are
some tests that instantiate the library to produce a signature, then
re-instantiate the library to verify it.  Obviously this doesn't prove that
its signatures are correct, but they do prove that its code is consistent
with itself, and again that changes made later don't break verifying.

None of this was done with an eye toward making my test suite applicable to
other implementations.  It's only for my own.  What we're talking about
here is different: coming up with tests that could be applied to any
implementation, not to a specific one, so long as they agree on fixed
syntax and inputs.

To see that this is the case, it's useful to look at two earlier email
> standards and their corresponding test suites.  SPF has a language-agnostic
> test suite (http://www.openspf.org/Test_Suite) that makes it easy to test
> implementations in any language.  I've done it myself, and it requires
> minimal effort - https://github.com/ValiMail/
> coppertone/tree/master/spec/open_spf.  I can tell you from personal
> experience that the test suite was really helpful in ensuring the
> implementation was correct in all cases.
>

An inherent difference between SPF and DKIM is that the order of the tokens
in SPF is significant.  In DKIM, reordering tags changes nothing about the
meaning of the header field, though obviously it alters the resulting
header hash.  In contrast, the order of tags in SPF is necessarily
specified.

-MSK