Re: [dmarc-ietf] DMARC2 & SPF Dependency Removal

[Barry Leiba <[e-mail]>]

Thanks for all this detail, Tero!  I will have to digest it and reply
further later.

Barry

On Tue, Jun 13, 2023 at 5:34 PM Tero Kivinen <[e-mail]> wrote:
>
> Barry Leiba writes:
> > > DKIM only: ~99.5%
> > > DKIM + SPF: ~100%
> > > SPF only: ~100%
> >
> > That's interesting and disturbing if it remains consistent.
>
> The statistics I have are quite different. The failure rate is much
> bigger both in DKIM and SPF.
>
> Following statistics is random subset of emails going through iki.fi
> system, from last 30 days, consisting bit less than 4 million emails.
> Iki.fi is email forwarding service, so about 90% of those emails will
> fail SPF checks after iki.fi sends them forward. DKIM will go through
> unmodified, and we do not modify normal messages (spam messages might
> get tagged as spam depending on the members configuration), so 85.75%
> of emails will still have valid DKIM signature after passing iki.
>
> We do graylisting of blacklisted ip-addresses, thus spammers that do
> not work around graylisting are not part of the statistics.
>
> There is significant amount of mailing lists going through iki, and
> quickly checking that 1.58% of emails going through has spf-errors,
> dkim signers or similar with domain name in form of list.domain or
> lists.domain, so that will cause some of the SPF and DKIM failures.
> Note, that this only counts cases where the domain name was used in
> the verification and printed in the logs i.e., only in error cases.
>
> As we are using ARC, and we add ARC-Authentication-Results header to
> all emails as first step when they come in, and I used those headers
> to generate these statistics.
>
> First some generic statistics:
>
>         Number of ARC-header levels
>         ===========================
>         95.61%  3811208 1
>         3.83%   152487  2
>         0.44%   17711   3
>         0.09%   3586    4
>         0.01%   460     5
>         0.01%   349     6
>         0.01%   207     7
>         0.00%   36      8
>         0.00%   15      9
>         0.00%   1       10
>
>         Mailer
>         ======================
>         91.96%  3665744 MTA-v4
>         8.04%   320315  MTA-v6
>         0.00%   1       MSA
>
> So 3.83% of emails already had one ARC header, and 0.56% had more than
> one arc header, with exactly one email having 10
> ARC-Authentication-Results headers. Most of the emails do not have ARC
> headers.
>
> 92% of traffic came in using IPv4..
>
> Then lets compare DKIM, SPF, DMARC and ARC results
>
>         DKIM summary results
>         =========================
>         85.75%  3417541 pass
>         13.11%  522367  none
>         1.12%   44604   fail
>         0.02%   893     temperror
>
>         SPF results
>         =========================
>         86.50%  3447577 pass
>         8.78%   349947  none
>         1.89%   75137   softfail
>         1.18%   46913   permerror
>         1.12%   44553   fail
>         0.49%   19536   neutral
>         0.05%   2037    temperror
>
>         DMARC results
>         =========================
>         62.82%  1243393 pass
>         30.99%  613478  none
>         6.05%   119800  fail
>         0.08%   1485    temperror
>         0.06%   1244    permerror
>
>         ARC results
>         =========================
>         91.66%  160268  pass
>         8.34%   14584   reject
>
> As you can see 85.75% of incoming email was already signed by DKIM,
> and 86.5% of emails had SPF records that passed. So they both have
> about same amount if usage coming in to our servers.
>
> The difference is that only 1.14% of emails had errors (fail, or
> temperror) in their DKIM signatures (most of those were because the
> email was from the mailing list that modified the body, but did not
> generate new DKIM header), compared to the 4.24% of emails having SPF
> failures (softfail, permerror, fail or temperror). Meaning there were
> much more emails that failed SPF than DKIM. Even if we ignore the
> softfails, we still have about double the emails failing (2.35%).
>
> Note, that the dmarc and arc statistics are not from all of the
> emails, it only includes those which actually had DMARC or ARC
> information. For dmarc this was about 50%, and for ARC it was only
> 4.3% of all emails.
>
> Here are some statistics abut the DKIM processing and the error cases.
> 76.75% had one DKIM signature, and over 20% had more than one
> signature. Here is number of DKIM signatures and their results, i.e.,
> 22.22% of emails had two DKIM signatures both passing, and 0.34% had
> one signature that passed, and another that failed etc:
>
>         DKIM results
>         =======================================
>         62.67%  2497633 pass
>         22.22%  885372  pass,pass
>         13.06%  520332  none
>         1.04%   41477   fail
>         0.34%   13353   pass,fail
>         0.19%   7506    none,pass
>         0.15%   5910    pass,none
>         0.07%   2635    fail,fail
>         0.06%   2235    pass,pass,pass
>         0.05%   2034    none,none
>         0.03%   1296    pass,pass,pass,pass
>         0.03%   1026    pass,pass,fail
>         0.03%   1002    fail,pass
>         0.02%   892     temperror
>         0.02%   631     pass,fail,fail
>         0.01%   583     pass,none,none
>         0.01%   369     fail,fail,fail
>         0.01%   356     fail,fail,pass
>         0.01%   335     pass,pass,none
>         0.00%   86      pass,fail,fail,fail
>         0.00%   69      none,fail
>         0.00%   67      pass,fail,pass
>         0.00%   48      pass,pass,fail,fail
>         0.00%   27      temperror,pass
>         0.00%   26      fail,fail,none
>         0.00%   22      pass,temperror
>         0.00%   15      pass,pass,none,none
>         0.00%   10      none,pass,pass
>         0.00%   9       fail,fail,fail,fail
>         0.00%   7       pass,fail,none
>         0.00%   7       none,fail,fail
>         0.00%   7       fail,fail,fail,fail,none
>         0.00%   4       pass,none,pass
>         0.00%   4       fail,none
>         0.00%   4       pass,fail,fail,fail,fail
>         0.00%   3       fail,pass,pass
>         0.00%   2       pass,pass,pass,pass,pass,pass
>         0.00%   2       pass,none,fail
>         0.00%   2       pass,pass,pass,fail
>         0.00%   2       none,fail,pass
>         0.00%   1       temperror,temperror
>         0.00%   1       pass,pass,pass,pass,fail
>         0.00%   1       fail,fail,temperror
>         0.00%   1       pass,temperror,pass
>         0.00%   1       none,none,none
>
> The none,none,none cases etc are where it had 3 DKIM signatures but it
> could not find any DKIM records from the DNS, and was not able to
> verify the signatures.
>
> And here are reasons why dkim signature checking failed. The Invalid
> DKIM record actually results the dkim result to be none, but other
> errors result to the final result to be fail. As you can see there is
> significant part where the body hash did not verify (most likely
> because this is coming from mailing list). This only includes those
> emails where there was no passing DKIM signature at all.
>
>         DKIM failures
>         ================================================================
>         36.34%  26619   invalid DKIM record
>         36.28%  26577   body hash did not verify
>         20.34%  14900   headers rsa verify failed
>         2.78%   2034    invalid DKIM record,invalid DKIM record
>         1.62%   1186    headers rsa verify failed,headers rsa verify
>                         failed
>         1.62%   1185    body hash did not verify,body hash did not
>                         verify
>         0.49%   360     body hash did not verify,body hash did not
>                         verify,body hash did not verify
>         0.30%   218     headers rsa verify failed,headers eddsa verify
>                         failed
>         0.09%   65      invalid DKIM record,body hash did not verify
>         0.05%   37      headers rsa verify failed,body hash did not
>                         verify
>         0.04%   26      body hash did not verify,body hash did not
>                         verify,invalid DKIM record
>         0.01%   9       headers eddsa verify failed,headers rsa verify
>                         failed
>         0.01%   9       body hash did not verify,body hash did not
>                         verify,body hash did not verify,body hash did
>                         not verify
>         0.01%   7       body hash did not verify,body hash did not
>                         verify,body hash did not verify,body hash did
>                         not verify,invalid DKIM record
>         0.01%   6       invalid DKIM record,body hash did not
>                         verify,body hash did not verify
>         0.01%   4       headers rsa verify failed,headers rsa verify
>                         failed,body hash did not verify
>         0.01%   4       invalid DKIM record,headers rsa verify failed
>         0.00%   3       headers rsa verify failed,headers rsa verify
>                         failed,headers rsa verify failed
>         0.00%   2       headers rsa verify failed,invalid DKIM record
>         0.00%   2       headers rsa verify failed,body hash did not
>                         verify,body hash did not verify
>         0.00%   2       body hash did not verify,invalid DKIM record
>         0.00%   1       invalid DKIM record,invalid DKIM
>                         record,invalid DKIM record
>         0.00%   1       body hash did not verify,headers rsa verify
>                         failed
>         0.00%   1       invalid DKIM record,headers rsa verify
>                         failed,headers rsa verify failed
>
> SPF failures show that it is not that big difference whether you use
> IPv4, or IPv6, as this matches the generic use of IP protocols for
> these incoming emails:
>
>         SPF failures
>         ==============================================================
>         92.71%  41307   MTA-v4: domain of x@y does not designate ipxxx
>                         as permitted sender
>         7.29%   3246    MTA-v6: domain of x@y does not designate ipxxx
>                         as permitted sender
>
> For DMARC failures there is quite a large number of those which do not
> have SPF or DKIM. I do not really known what I should interpret from
> those other errors for DMARC.
>
>         DMARC failures
>         ============================================================
>         52.53%  62925   No valid SPF, No valid DKIM
>         32.97%  39504   SPF not aligned (relaxed), DKIM not aligned (relaxed)
>         5.41%   6486    SPF not aligned (relaxed), No valid DKIM
>         3.49%   4186    No valid SPF
>         2.68%   3213    SPF not aligned (relaxed)
>         2.07%   2484    No valid SPF, DKIM not aligned (relaxed)
>         0.25%   297     SPF not aligned (strict), DKIM not aligned (strict)
>         0.21%   256     SPF not aligned (relaxed), DKIM not aligned (strict)
>         0.17%   207     SPF not aligned (strict)
>         0.09%   106     SPF not aligned (strict), No valid DKIM
>         0.08%   100     SPF not aligned (strict), DKIM not aligned (relaxed)
>         0.03%   36      No valid SPF, DKIM not aligned (strict)
>
> For ARC there is quite big number of signature check failures, I am
> not actually sure whether that is because there is no key to be found
> or what is the issue.
>
>         ARC failures
>         ===========================================================
>         80.36%  11720   "signature check failed: fail, {[1] = sig:xxx:reject}"
>         6.37%   929     "cv is fail on i=4"
>         6.31%   920     "cv is fail on i=2"
>         3.73%   544     "seal check failed: fail, {[1] = sig:xxx:reject}"
>         1.89%   275     "cv is none on i=2"
>         0.80%   116     "signature check failed: fail, {[1] =
>                         sig:xxx:dns request to xxx
>         0.52%   76      "cv is fail on i=3"
>         0.02%   3       "seal check failed: fail, {[1] = sig:xxx:dns
>                         request to xxx
>         0.01%   1       unknown
>
>
> Summary: Looking at the data there is much more SPF related failures
> than DKIM related failures, and as I said 90% of these emails WILL
> FAIL SPF checks when iki.fi will forward them to their final
> destination (only those that say +all or do not publish SPF record
> will survive), while the DKIM records still are correct.
>
> We have several cases where final email domain where the user asks us
> to forward his email is only using SPF, thus we simply ask them to
> switch to someone who does email properly and uses DKIM too...
>
> If the DMARCv2 would mandate support of DKIM and would get rid of the
> SPF checks completely then hopefully more people would actually start
> using DKIM also in the verification. It is quite widely already used
> in the generation of the messages.
>
> Of course this is selected data-set as if out user find out he can't
> use his iki.fi address for certain service as it does not do DKIM, and
> his/her final destination checks SPF, he/she will not use his iki.fi
> address in those places or he/she changes his email mailbox provider
> (which is easy to do if all your emails go through iki, you simply
> change the forward to go to your new address, and hour later
> all your emails go there :-)
> --
> [e-mail]