Re: [dmarc-ietf] Aggregate Reporting - "Not Evaluated" result

["Brotman, Alex" <[e-mail]>]

I’ve been going back and forth on this a bit.  On one side, I understand that we’d like to know when a receiving site does not evaluate both SPF and DKIM.  I also am not sure I know of any (sizable?) site which short-circuits evaluation after SPF.  Given how much time receivers talk about separation of streams and so on, I’d be surprised to see them knowingly discard data which could be used for data science-y things.

I receive a report without DKIM data from a specific site.  Today, I can’t know if that is a signing problem at the sender, a reporting failure/decision, a conscious choice for evaluation, a bug at the receiver, was stripped in transit, or something else.  I’m also not sure introducing a new “Not Evaluated” state solves/explains why it didn’t show in the reports.

--
Alex Brotman
Sr. Engineer, Anti-Abuse & Messaging Policy
Comcast

From: dmarc <[e-mail]> On Behalf Of Douglas Foster
Sent: Thursday, October 20, 2022 7:04 AM
To: IETF DMARC WG <[e-mail]>
Subject: Re: [dmarc-ietf] Aggregate Reporting - "Not Evaluated" result

My thinking has evolved during this discussion:

We should reject Incomplete Results
If an evaluator has decided to do incomplete evaluation, we have to consider the possibility that he may or may not collect enough information to enumerate what signatures were not evaluated.   So a signature result of "not evaluated" does not solve the whole problem, but does cause disaggregation.    A bit field indicating "incomplete results" could cover all types of incompleteness, and report recipients could decide whether to use the data or not.   But since we have aversion to incomplete results, the "must not report" approach both encourages complete results and provides upward compatibility for report receivers.

Scope
I am skeptical that our request for data about non-aligned signatures can justify the cost.   I have seen no defined strategy for integrating non-aligned signatures into the message evaluation process, so computing those results are pure waste to the evaluator.   Waste has real money costs and real opportunity costs.    Given that billions or trillions of messages are transmitted every day, the global cost of extra signature evaluations is really quite significant.    I am not freaking out about global warming, but it is on my radar.  The environmental impact of our decisions, when played out to Internet scale, are not trivial.   I would like some convincing that knowledge about non-aligned signatures is worth the non-trivial cost that we are asking evaluators, and the planet, to absorb.

Doug Foster

On Wed, Oct 19, 2022 at 8:48 PM Neil Anuskiewicz <[e-mail]<mailto:[e-mail]>> wrote:


> On Oct 19, 2022, at 5:42 PM, Neil Anuskiewicz <[e-mail]<mailto:[e-mail]>> wrote:
>
> 
>
>> On Oct 19, 2022, at 6:59 AM, Scott Kitterman <[e-mail]<mailto:[e-mail]>> wrote:
>>
>> 
>>
>>>> On October 19, 2022 12:44:16 PM UTC, Dotzero <[e-mail]<mailto:[e-mail]>> wrote:
>>> On Tue, Oct 18, 2022 at 11:18 PM Scott Kitterman <[e-mail]<mailto:[e-mail]>>
>>> wrote:
>>>
>>>>
>>>>
>>>> On October 18, 2022 10:16:44 PM UTC, Neil Anuskiewicz <
>>>> [e-mail]<mailto:[e-mail]>> wrote:
>>>>>
>>>>>
>>>>>> On Oct 2, 2022, at 11:01 AM, Douglas Foster <
>>>> [e-mail]<mailto:[e-mail]>> wrote:
>>>>>>
>>>>>> 
>>>>>> In many cases, an evaluator can determine a DMARC PASS result without
>>>> evaluating every available identifier.
>>>>>> If a message has SPF PASS with acceptable alignment, the evaluator has
>>>> no need to evaluate any DKIM signatures to know that the message produces
>>>> DMARC PASS.
>>>>> I think it’s critical to DMARC that receivers do things like evaluate and
>>>> report on DKIM whether or not SPF passes and is alignment. Without this, it
>>>> would make it harder for senders to notice and remediate gaps in their
>>>> authentication. Since there’s not a downside (that I know of), I’d say this
>>>> should be a MUST if at all possible.
>>>>
>>>>
>>>> What is the interoperability problem that happens if evaluators don't do
>>>> that?
>>>>
>>>> Scott K
>>>>
>>>
>>> Scott, What is the interoperability problem is evaluators didn't provide
>>> reports at all? Reporting isn't a "must" for interoperability but it
>>> certainly helps improve outcomes instead of senders flying blind.
>>
>> I read the email as suggesting a MUST for reporting both SPF and DKIM results if you report results at all, which would, I think lead to exactly the situation you're concerned about.  I'm skeptical of any kind of MUST around reporting since that's generally reserved for things that impact interoperability.  I do agree it should be encouraged.
>>
>> Mostly, at the moment, I'm trying to understand the proposed change and the rationale.
>
> I think the reactions were to the tone that that seemed to suggest that the importance of reporting was being downplayed. MUST is too strong and strongly encouraged is sufficient. The standards system relies on people making a good faith effort. To me, Doug’s comments came off as wanting to weaken the language which concerned me.
>
> Reporting is key for DMARC to work as a system so any hint of weakening that language or even could be interpreted as such caught my attention. I think Doug clarified his position as addressing specific cases not a weakening of the reporting language.
>
> DMARC is about the interests of the system but following the standard strengthens the system within which the sender or receiver operates. Even if one wasn’t interested in the health of system in and of itself, reporting benefits the admin as it increases security and reduces broken authentication. A *LOT* of Senders use reporting data as part of the process of fixing their own and third party senders they wish to allow or spoof, discovering errant shadow IT, etc.
>
> Reporting is or core importance for everyone if for no other reason than to avoid headaches. Thanks.

s/allow or spoof/allow to spoof/
_______________________________________________
dmarc mailing list
[e-mail]<mailto:[e-mail]>
https://www.ietf.org/mailman/listinfo/dmarc<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/dmarc__;!!CQl3mcHX2A!EsdF1EeMbORwM6q1BxYm9IsETTRVXfH8E62GtWR7mJvZ9jX2cB43dVfFr0WtGJmWfJuFPez7PgePL2wmqspAD63TEGtEypY$>