Re: [dmarc-ietf] DMARC2 & SPF Dependency Removal

[Tero Kivinen <[e-mail]>]

Barry Leiba writes:
> > DKIM only: ~99.5%
> > DKIM + SPF: ~100%
> > SPF only: ~100%
> 
> That's interesting and disturbing if it remains consistent.

The statistics I have are quite different. The failure rate is much
bigger both in DKIM and SPF.

Following statistics is random subset of emails going through iki.fi
system, from last 30 days, consisting bit less than 4 million emails.
Iki.fi is email forwarding service, so about 90% of those emails will
fail SPF checks after iki.fi sends them forward. DKIM will go through
unmodified, and we do not modify normal messages (spam messages might
get tagged as spam depending on the members configuration), so 85.75%
of emails will still have valid DKIM signature after passing iki.

We do graylisting of blacklisted ip-addresses, thus spammers that do
not work around graylisting are not part of the statistics.

There is significant amount of mailing lists going through iki, and
quickly checking that 1.58% of emails going through has spf-errors,
dkim signers or similar with domain name in form of list.domain or
lists.domain, so that will cause some of the SPF and DKIM failures.
Note, that this only counts cases where the domain name was used in
the verification and printed in the logs i.e., only in error cases.

As we are using ARC, and we add ARC-Authentication-Results header to
all emails as first step when they come in, and I used those headers
to generate these statistics.

First some generic statistics:

	Number of ARC-header levels
	===========================
	95.61%	3811208	1
	3.83%	152487	2
	0.44%	17711	3
	0.09%	3586	4
	0.01%	460	5
	0.01%	349	6
	0.01%	207	7
	0.00%	36	8
	0.00%	15	9
	0.00%	1	10

	Mailer
	======================
	91.96%	3665744	MTA-v4
	8.04%	320315	MTA-v6
	0.00%	1	MSA
		
So 3.83% of emails already had one ARC header, and 0.56% had more than
one arc header, with exactly one email having 10
ARC-Authentication-Results headers. Most of the emails do not have ARC
headers.

92% of traffic came in using IPv4..

Then lets compare DKIM, SPF, DMARC and ARC results

	DKIM summary results
	=========================
	85.75%	3417541	pass
	13.11%	522367	none
	1.12%	44604	fail
	0.02%	893	temperror

	SPF results
	=========================
	86.50%	3447577	pass
	8.78%	349947	none
	1.89%	75137	softfail
	1.18%	46913	permerror
	1.12%	44553	fail
	0.49%	19536	neutral
	0.05%	2037	temperror

	DMARC results
	=========================
	62.82%	1243393	pass
	30.99%	613478	none
	6.05%	119800	fail
	0.08%	1485	temperror
	0.06%	1244	permerror

	ARC results
	=========================
	91.66%	160268	pass
	8.34%	14584	reject
		
As you can see 85.75% of incoming email was already signed by DKIM,
and 86.5% of emails had SPF records that passed. So they both have
about same amount if usage coming in to our servers.

The difference is that only 1.14% of emails had errors (fail, or
temperror) in their DKIM signatures (most of those were because the
email was from the mailing list that modified the body, but did not
generate new DKIM header), compared to the 4.24% of emails having SPF
failures (softfail, permerror, fail or temperror). Meaning there were
much more emails that failed SPF than DKIM. Even if we ignore the
softfails, we still have about double the emails failing (2.35%).

Note, that the dmarc and arc statistics are not from all of the
emails, it only includes those which actually had DMARC or ARC
information. For dmarc this was about 50%, and for ARC it was only
4.3% of all emails. 

Here are some statistics abut the DKIM processing and the error cases.
76.75% had one DKIM signature, and over 20% had more than one
signature. Here is number of DKIM signatures and their results, i.e.,
22.22% of emails had two DKIM signatures both passing, and 0.34% had
one signature that passed, and another that failed etc:

	DKIM results
	=======================================
	62.67%	2497633	pass
	22.22%	885372	pass,pass
	13.06%	520332	none
	1.04%	41477	fail
	0.34%	13353	pass,fail
	0.19%	7506	none,pass
	0.15%	5910	pass,none
	0.07%	2635	fail,fail
	0.06%	2235	pass,pass,pass
	0.05%	2034	none,none
	0.03%	1296	pass,pass,pass,pass
	0.03%	1026	pass,pass,fail
	0.03%	1002	fail,pass
	0.02%	892	temperror
	0.02%	631	pass,fail,fail
	0.01%	583	pass,none,none
	0.01%	369	fail,fail,fail
	0.01%	356	fail,fail,pass
	0.01%	335	pass,pass,none
	0.00%	86	pass,fail,fail,fail
	0.00%	69	none,fail
	0.00%	67	pass,fail,pass
	0.00%	48	pass,pass,fail,fail
	0.00%	27	temperror,pass
	0.00%	26	fail,fail,none
	0.00%	22	pass,temperror
	0.00%	15	pass,pass,none,none
	0.00%	10	none,pass,pass
	0.00%	9	fail,fail,fail,fail
	0.00%	7	pass,fail,none
	0.00%	7	none,fail,fail
	0.00%	7	fail,fail,fail,fail,none
	0.00%	4	pass,none,pass
	0.00%	4	fail,none
	0.00%	4	pass,fail,fail,fail,fail
	0.00%	3	fail,pass,pass
	0.00%	2	pass,pass,pass,pass,pass,pass
	0.00%	2	pass,none,fail
	0.00%	2	pass,pass,pass,fail
	0.00%	2	none,fail,pass
	0.00%	1	temperror,temperror
	0.00%	1	pass,pass,pass,pass,fail
	0.00%	1	fail,fail,temperror
	0.00%	1	pass,temperror,pass
	0.00%	1	none,none,none

The none,none,none cases etc are where it had 3 DKIM signatures but it
could not find any DKIM records from the DNS, and was not able to
verify the signatures.

And here are reasons why dkim signature checking failed. The Invalid
DKIM record actually results the dkim result to be none, but other
errors result to the final result to be fail. As you can see there is
significant part where the body hash did not verify (most likely
because this is coming from mailing list). This only includes those
emails where there was no passing DKIM signature at all.

	DKIM failures
	================================================================
	36.34%	26619	invalid DKIM record
	36.28%	26577	body hash did not verify
	20.34%	14900	headers rsa verify failed
	2.78%	2034	invalid DKIM record,invalid DKIM record
	1.62%	1186	headers rsa verify failed,headers rsa verify
			failed 
	1.62%	1185	body hash did not verify,body hash did not
			verify 
	0.49%	360	body hash did not verify,body hash did not
			verify,body hash did not verify 
	0.30%	218	headers rsa verify failed,headers eddsa verify
			failed 
	0.09%	65	invalid DKIM record,body hash did not verify
	0.05%	37	headers rsa verify failed,body hash did not
			verify 
	0.04%	26	body hash did not verify,body hash did not
			verify,invalid DKIM record 
	0.01%	9	headers eddsa verify failed,headers rsa verify
			failed 
	0.01%	9	body hash did not verify,body hash did not
			verify,body hash did not verify,body hash did
 			not verify
	0.01%	7	body hash did not verify,body hash did not
			verify,body hash did not verify,body hash did
			not verify,invalid DKIM record 
	0.01%	6	invalid DKIM record,body hash did not
			verify,body hash did not verify 
	0.01%	4	headers rsa verify failed,headers rsa verify
			failed,body hash did not verify 
	0.01%	4	invalid DKIM record,headers rsa verify failed 
	0.00%	3	headers rsa verify failed,headers rsa verify
			failed,headers rsa verify failed 
	0.00%	2	headers rsa verify failed,invalid DKIM record 
	0.00%	2	headers rsa verify failed,body hash did not
			verify,body hash did not verify 
	0.00%	2	body hash did not verify,invalid DKIM record
	0.00%	1	invalid DKIM record,invalid DKIM
			record,invalid DKIM record 
	0.00%	1	body hash did not verify,headers rsa verify
			failed 
	0.00%	1	invalid DKIM record,headers rsa verify
			failed,headers rsa verify failed 

SPF failures show that it is not that big difference whether you use
IPv4, or IPv6, as this matches the generic use of IP protocols for
these incoming emails:

	SPF failures
	==============================================================
	92.71%	41307	MTA-v4: domain of x@y does not designate ipxxx
			as permitted sender 
	7.29%	3246	MTA-v6: domain of x@y does not designate ipxxx
			as permitted sender 

For DMARC failures there is quite a large number of those which do not
have SPF or DKIM. I do not really known what I should interpret from
those other errors for DMARC. 

	DMARC failures
	============================================================
	52.53%	62925	No valid SPF, No valid DKIM
	32.97%	39504	SPF not aligned (relaxed), DKIM not aligned (relaxed)
	5.41%	6486	SPF not aligned (relaxed), No valid DKIM
	3.49%	4186	No valid SPF
	2.68%	3213	SPF not aligned (relaxed)
	2.07%	2484	No valid SPF, DKIM not aligned (relaxed)
	0.25%	297	SPF not aligned (strict), DKIM not aligned (strict)
	0.21%	256	SPF not aligned (relaxed), DKIM not aligned (strict)
	0.17%	207	SPF not aligned (strict)
	0.09%	106	SPF not aligned (strict), No valid DKIM
	0.08%	100	SPF not aligned (strict), DKIM not aligned (relaxed)
	0.03%	36	No valid SPF, DKIM not aligned (strict)

For ARC there is quite big number of signature check failures, I am
not actually sure whether that is because there is no key to be found
or what is the issue. 

	ARC failures
	===========================================================
	80.36%	11720	"signature check failed: fail, {[1] = sig:xxx:reject}" 
	6.37%	929	"cv is fail on i=4"
	6.31%	920	"cv is fail on i=2"
	3.73%	544	"seal check failed: fail, {[1] = sig:xxx:reject}"
	1.89%	275	"cv is none on i=2"
	0.80%	116	"signature check failed: fail, {[1] =
			sig:xxx:dns request to xxx 
	0.52%	76	"cv is fail on i=3"
	0.02%	3	"seal check failed: fail, {[1] = sig:xxx:dns
			request to xxx 
	0.01%	1	unknown


Summary: Looking at the data there is much more SPF related failures
than DKIM related failures, and as I said 90% of these emails WILL
FAIL SPF checks when iki.fi will forward them to their final
destination (only those that say +all or do not publish SPF record
will survive), while the DKIM records still are correct.

We have several cases where final email domain where the user asks us
to forward his email is only using SPF, thus we simply ask them to
switch to someone who does email properly and uses DKIM too...

If the DMARCv2 would mandate support of DKIM and would get rid of the
SPF checks completely then hopefully more people would actually start
using DKIM also in the verification. It is quite widely already used
in the generation of the messages.

Of course this is selected data-set as if out user find out he can't
use his iki.fi address for certain service as it does not do DKIM, and
his/her final destination checks SPF, he/she will not use his iki.fi
address in those places or he/she changes his email mailbox provider
(which is easy to do if all your emails go through iki, you simply
change the forward to go to your new address, and hour later
all your emails go there :-)
-- 
[e-mail]