Re: [dmarc-ietf] DMARC2 & SPF Dependency Removal

[Douglas Foster <[e-mail]>]

This topic raised a question, at least in my mind, whether DKIM signing
algorithms are subject to random failures.   If random failures occur, they
could be blamed on either the sender algorithm or the receiver algorithm.
  The question can be assessed on incoming messages, using authentication
results, or on outgoing messages, using aggregate reports.

My environment has a single MX performing DKIM checking, with essentially
zero mailing list traffic.
I also have a single MTA performing DKIM signing.
The signing server is a different software implementation than the checking
server.

For outgoing messages, I see no evidence of random failures.   All of the
reported DKIM failures appear to have explanations.

For incoming messages, I defined a unique configuration as the tuple of (
Helo Domain, MailFrom domain, and From domain ).  Then I looked at
verification percentages for signed messages from each tuple.    The
results were interesting:

   - 92% of tuples, sending 89% of all signed messages, had 100%
   verification rates
   - 5% of tuples, sending 2% of all signed messages, had 0% verification
   rates
   - 3% of tuples, sending 9% of all signed messages, had a verification
   rate between these limits.
   - Of all messages with DKIM failures, 85% were authenticated using
   aligned SPF PASS.

These statistics are based on 100% of incoming messages, regardless of
subsequent disposition.

My conclusions:

   - SPF is a necessary supplement to DKIM.
   - DKIM can be reliable:   Most senders and receivers have 100% reliable
   DKIM implementations.
   - Senders with 100% failure rates appear to be playing games with DKIM.
    Some of these may already be on my block list.  The rest should  be
   reviewed to see if they should be added to my block list.
   - The 5% with inconsistent results need further investigation.   Perhaps
   a server farm has one server that is generating wrong signatures.

Doug Foster


On Tue, Jun 13, 2023 at 5:34 PM Tero Kivinen <[e-mail]> wrote:

> Barry Leiba writes:
> > > DKIM only: ~99.5%
> > > DKIM + SPF: ~100%
> > > SPF only: ~100%
> >
> > That's interesting and disturbing if it remains consistent.
>
> The statistics I have are quite different. The failure rate is much
> bigger both in DKIM and SPF.
>
> Following statistics is random subset of emails going through iki.fi
> system, from last 30 days, consisting bit less than 4 million emails.
> Iki.fi is email forwarding service, so about 90% of those emails will
> fail SPF checks after iki.fi sends them forward. DKIM will go through
> unmodified, and we do not modify normal messages (spam messages might
> get tagged as spam depending on the members configuration), so 85.75%
> of emails will still have valid DKIM signature after passing iki.
>
> We do graylisting of blacklisted ip-addresses, thus spammers that do
> not work around graylisting are not part of the statistics.
>
> There is significant amount of mailing lists going through iki, and
> quickly checking that 1.58% of emails going through has spf-errors,
> dkim signers or similar with domain name in form of list.domain or
> lists.domain, so that will cause some of the SPF and DKIM failures.
> Note, that this only counts cases where the domain name was used in
> the verification and printed in the logs i.e., only in error cases.
>
> As we are using ARC, and we add ARC-Authentication-Results header to
> all emails as first step when they come in, and I used those headers
> to generate these statistics.
>
> First some generic statistics:
>
>         Number of ARC-header levels
>         ===========================
>         95.61%  3811208 1
>         3.83%   152487  2
>         0.44%   17711   3
>         0.09%   3586    4
>         0.01%   460     5
>         0.01%   349     6
>         0.01%   207     7
>         0.00%   36      8
>         0.00%   15      9
>         0.00%   1       10
>
>         Mailer
>         ======================
>         91.96%  3665744 MTA-v4
>         8.04%   320315  MTA-v6
>         0.00%   1       MSA
>
> So 3.83% of emails already had one ARC header, and 0.56% had more than
> one arc header, with exactly one email having 10
> ARC-Authentication-Results headers. Most of the emails do not have ARC
> headers.
>
> 92% of traffic came in using IPv4..
>
> Then lets compare DKIM, SPF, DMARC and ARC results
>
>         DKIM summary results
>         =========================
>         85.75%  3417541 pass
>         13.11%  522367  none
>         1.12%   44604   fail
>         0.02%   893     temperror
>
>         SPF results
>         =========================
>         86.50%  3447577 pass
>         8.78%   349947  none
>         1.89%   75137   softfail
>         1.18%   46913   permerror
>         1.12%   44553   fail
>         0.49%   19536   neutral
>         0.05%   2037    temperror
>
>         DMARC results
>         =========================
>         62.82%  1243393 pass
>         30.99%  613478  none
>         6.05%   119800  fail
>         0.08%   1485    temperror
>         0.06%   1244    permerror
>
>         ARC results
>         =========================
>         91.66%  160268  pass
>         8.34%   14584   reject
>
> As you can see 85.75% of incoming email was already signed by DKIM,
> and 86.5% of emails had SPF records that passed. So they both have
> about same amount if usage coming in to our servers.
>
> The difference is that only 1.14% of emails had errors (fail, or
> temperror) in their DKIM signatures (most of those were because the
> email was from the mailing list that modified the body, but did not
> generate new DKIM header), compared to the 4.24% of emails having SPF
> failures (softfail, permerror, fail or temperror). Meaning there were
> much more emails that failed SPF than DKIM. Even if we ignore the
> softfails, we still have about double the emails failing (2.35%).
>
> Note, that the dmarc and arc statistics are not from all of the
> emails, it only includes those which actually had DMARC or ARC
> information. For dmarc this was about 50%, and for ARC it was only
> 4.3% of all emails.
>
> Here are some statistics abut the DKIM processing and the error cases.
> 76.75% had one DKIM signature, and over 20% had more than one
> signature. Here is number of DKIM signatures and their results, i.e.,
> 22.22% of emails had two DKIM signatures both passing, and 0.34% had
> one signature that passed, and another that failed etc:
>
>         DKIM results
>         =======================================
>         62.67%  2497633 pass
>         22.22%  885372  pass,pass
>         13.06%  520332  none
>         1.04%   41477   fail
>         0.34%   13353   pass,fail
>         0.19%   7506    none,pass
>         0.15%   5910    pass,none
>         0.07%   2635    fail,fail
>         0.06%   2235    pass,pass,pass
>         0.05%   2034    none,none
>         0.03%   1296    pass,pass,pass,pass
>         0.03%   1026    pass,pass,fail
>         0.03%   1002    fail,pass
>         0.02%   892     temperror
>         0.02%   631     pass,fail,fail
>         0.01%   583     pass,none,none
>         0.01%   369     fail,fail,fail
>         0.01%   356     fail,fail,pass
>         0.01%   335     pass,pass,none
>         0.00%   86      pass,fail,fail,fail
>         0.00%   69      none,fail
>         0.00%   67      pass,fail,pass
>         0.00%   48      pass,pass,fail,fail
>         0.00%   27      temperror,pass
>         0.00%   26      fail,fail,none
>         0.00%   22      pass,temperror
>         0.00%   15      pass,pass,none,none
>         0.00%   10      none,pass,pass
>         0.00%   9       fail,fail,fail,fail
>         0.00%   7       pass,fail,none
>         0.00%   7       none,fail,fail
>         0.00%   7       fail,fail,fail,fail,none
>         0.00%   4       pass,none,pass
>         0.00%   4       fail,none
>         0.00%   4       pass,fail,fail,fail,fail
>         0.00%   3       fail,pass,pass
>         0.00%   2       pass,pass,pass,pass,pass,pass
>         0.00%   2       pass,none,fail
>         0.00%   2       pass,pass,pass,fail
>         0.00%   2       none,fail,pass
>         0.00%   1       temperror,temperror
>         0.00%   1       pass,pass,pass,pass,fail
>         0.00%   1       fail,fail,temperror
>         0.00%   1       pass,temperror,pass
>         0.00%   1       none,none,none
>
> The none,none,none cases etc are where it had 3 DKIM signatures but it
> could not find any DKIM records from the DNS, and was not able to
> verify the signatures.
>
> And here are reasons why dkim signature checking failed. The Invalid
> DKIM record actually results the dkim result to be none, but other
> errors result to the final result to be fail. As you can see there is
> significant part where the body hash did not verify (most likely
> because this is coming from mailing list). This only includes those
> emails where there was no passing DKIM signature at all.
>
>         DKIM failures
>         ================================================================
>         36.34%  26619   invalid DKIM record
>         36.28%  26577   body hash did not verify
>         20.34%  14900   headers rsa verify failed
>         2.78%   2034    invalid DKIM record,invalid DKIM record
>         1.62%   1186    headers rsa verify failed,headers rsa verify
>                         failed
>         1.62%   1185    body hash did not verify,body hash did not
>                         verify
>         0.49%   360     body hash did not verify,body hash did not
>                         verify,body hash did not verify
>         0.30%   218     headers rsa verify failed,headers eddsa verify
>                         failed
>         0.09%   65      invalid DKIM record,body hash did not verify
>         0.05%   37      headers rsa verify failed,body hash did not
>                         verify
>         0.04%   26      body hash did not verify,body hash did not
>                         verify,invalid DKIM record
>         0.01%   9       headers eddsa verify failed,headers rsa verify
>                         failed
>         0.01%   9       body hash did not verify,body hash did not
>                         verify,body hash did not verify,body hash did
>                         not verify
>         0.01%   7       body hash did not verify,body hash did not
>                         verify,body hash did not verify,body hash did
>                         not verify,invalid DKIM record
>         0.01%   6       invalid DKIM record,body hash did not
>                         verify,body hash did not verify
>         0.01%   4       headers rsa verify failed,headers rsa verify
>                         failed,body hash did not verify
>         0.01%   4       invalid DKIM record,headers rsa verify failed
>         0.00%   3       headers rsa verify failed,headers rsa verify
>                         failed,headers rsa verify failed
>         0.00%   2       headers rsa verify failed,invalid DKIM record
>         0.00%   2       headers rsa verify failed,body hash did not
>                         verify,body hash did not verify
>         0.00%   2       body hash did not verify,invalid DKIM record
>         0.00%   1       invalid DKIM record,invalid DKIM
>                         record,invalid DKIM record
>         0.00%   1       body hash did not verify,headers rsa verify
>                         failed
>         0.00%   1       invalid DKIM record,headers rsa verify
>                         failed,headers rsa verify failed
>
> SPF failures show that it is not that big difference whether you use
> IPv4, or IPv6, as this matches the generic use of IP protocols for
> these incoming emails:
>
>         SPF failures
>         ==============================================================
>         92.71%  41307   MTA-v4: domain of x@y does not designate ipxxx
>                         as permitted sender
>         7.29%   3246    MTA-v6: domain of x@y does not designate ipxxx
>                         as permitted sender
>
> For DMARC failures there is quite a large number of those which do not
> have SPF or DKIM. I do not really known what I should interpret from
> those other errors for DMARC.
>
>         DMARC failures
>         ============================================================
>         52.53%  62925   No valid SPF, No valid DKIM
>         32.97%  39504   SPF not aligned (relaxed), DKIM not aligned
> (relaxed)
>         5.41%   6486    SPF not aligned (relaxed), No valid DKIM
>         3.49%   4186    No valid SPF
>         2.68%   3213    SPF not aligned (relaxed)
>         2.07%   2484    No valid SPF, DKIM not aligned (relaxed)
>         0.25%   297     SPF not aligned (strict), DKIM not aligned (strict)
>         0.21%   256     SPF not aligned (relaxed), DKIM not aligned
> (strict)
>         0.17%   207     SPF not aligned (strict)
>         0.09%   106     SPF not aligned (strict), No valid DKIM
>         0.08%   100     SPF not aligned (strict), DKIM not aligned
> (relaxed)
>         0.03%   36      No valid SPF, DKIM not aligned (strict)
>
> For ARC there is quite big number of signature check failures, I am
> not actually sure whether that is because there is no key to be found
> or what is the issue.
>
>         ARC failures
>         ===========================================================
>         80.36%  11720   "signature check failed: fail, {[1] =
> sig:xxx:reject}"
>         6.37%   929     "cv is fail on i=4"
>         6.31%   920     "cv is fail on i=2"
>         3.73%   544     "seal check failed: fail, {[1] = sig:xxx:reject}"
>         1.89%   275     "cv is none on i=2"
>         0.80%   116     "signature check failed: fail, {[1] =
>                         sig:xxx:dns request to xxx
>         0.52%   76      "cv is fail on i=3"
>         0.02%   3       "seal check failed: fail, {[1] = sig:xxx:dns
>                         request to xxx
>         0.01%   1       unknown
>
>
> Summary: Looking at the data there is much more SPF related failures
> than DKIM related failures, and as I said 90% of these emails WILL
> FAIL SPF checks when iki.fi will forward them to their final
> destination (only those that say +all or do not publish SPF record
> will survive), while the DKIM records still are correct.
>
> We have several cases where final email domain where the user asks us
> to forward his email is only using SPF, thus we simply ask them to
> switch to someone who does email properly and uses DKIM too...
>
> If the DMARCv2 would mandate support of DKIM and would get rid of the
> SPF checks completely then hopefully more people would actually start
> using DKIM also in the verification. It is quite widely already used
> in the generation of the messages.
>
> Of course this is selected data-set as if out user find out he can't
> use his iki.fi address for certain service as it does not do DKIM, and
> his/her final destination checks SPF, he/she will not use his iki.fi
> address in those places or he/she changes his email mailbox provider
> (which is easy to do if all your emails go through iki, you simply
> change the forward to go to your new address, and hour later
> all your emails go there :-)
> --
> [e-mail]
>
> _______________________________________________
> dmarc mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/dmarc
>