Re: [dmarc-ietf] Aggregate Reporting - "Not Evaluated" result

[Douglas Foster <[e-mail]>]

Although I said an evaluator could stop after SPF PASS, I don't think this
is a significant concern.   Savvy evaluators will understand that DKIM PASS
is a higher-certainty result than SPF.  I support requiring a compliant
implementation to always evaluate DKIM signatures, regardless of SPF result.

Part of the purpose of this topic is to reconcile a report owners' desire
for "complete" data with the algorithm that I proposed in Issue 2.  That
algorithm has multiple exit points, so either the algorithm has to
eliminate early exit or the reporting expectations need to change.   Here
is my justification for preserving the early exit logic:

Aligned DKIM PASS
When an aligned DKIM result is PASS, I don't see that the domain owner
needs any more data collection performed.   The verifiable DKIM scope ID
should tell him where the message originated, and the source IP and HELO
name tell the last place it landed before delivery.    I cannot see why a
domain owner would spend a lot of time trying to analyze success results,
unless his interests have transitioned from assuring identity to analyzing
marketing impacts.   We are not trying to provide marketing data, and any
successful use of DMARC for that purpose seems like an encroachment on the
privacy concerns that we want to minimize.

Duplicates:
If an aligned domain has multiple DKIM signatures and one passes, I suggest
that the PASS is the only one that needs to be reported.  If an aligned
domain has multiple DKIM signatures and none pass, I suggest that the first
one found (from the top) is the most important, because it is the last one
applied.   If duplicates are reported, disaggregation is increased, so why
report data that is not useful?

Failure
If no signature produces PASS, then all of the available signatures have
reportable results, which is the scenario that seems most important domain
owner interests.

Doug Foster




On Thu, Oct 20, 2022 at 9:17 AM Brotman, Alex <Alex_Brotman=
[e-mail]> wrote:

> I’ve been going back and forth on this a bit.  On one side, I understand
> that we’d like to know when a receiving site does not evaluate both SPF and
> DKIM.  I also am not sure I know of any (sizable?) site which
> short-circuits evaluation after SPF.  Given how much time receivers talk
> about separation of streams and so on, I’d be surprised to see them
> knowingly discard data which could be used for data science-y things.
>
>
>
> I receive a report without DKIM data from a specific site.  Today, I can’t
> know if that is a signing problem at the sender, a reporting
> failure/decision, a conscious choice for evaluation, a bug at the receiver,
> was stripped in transit, or something else.  I’m also not sure introducing
> a new “Not Evaluated” state solves/explains why it didn’t show in the
> reports.
>
>
>
> --
>
> Alex Brotman
>
> Sr. Engineer, Anti-Abuse & Messaging Policy
>
> Comcast
>
>
>
> *From:* dmarc <[e-mail]> *On Behalf Of * Douglas Foster
> *Sent:* Thursday, October 20, 2022 7:04 AM
> *To:* IETF DMARC WG <[e-mail]>
> *Subject:* Re: [dmarc-ietf] Aggregate Reporting - "Not Evaluated" result
>
>
>
> My thinking has evolved during this discussion:
>
>
>
> We should reject Incomplete Results
>
> If an evaluator has decided to do incomplete evaluation, we have to
> consider the possibility that he may or may not collect enough information
> to enumerate what signatures were not evaluated.   So a signature result of
> "not evaluated" does not solve the whole problem, but does cause
> disaggregation.    A bit field indicating "incomplete results" could cover
> all types of incompleteness, and report recipients could decide whether to
> use the data or not.   But since we have aversion to incomplete results,
> the "must not report" approach both encourages complete results and
> provides upward compatibility for report receivers.
>
>
>
> Scope
>
> I am skeptical that our request for data about non-aligned signatures can
> justify the cost.   I have seen no defined strategy for integrating
> non-aligned signatures into the message evaluation process, so computing
> those results are pure waste to the evaluator.   Waste has real money costs
> and real opportunity costs.    Given that billions or trillions of messages
> are transmitted every day, the global cost of extra signature evaluations
> is really quite significant.    I am not freaking out about global warming,
> but it is on my radar.  The environmental impact of our decisions, when
> played out to Internet scale, are not trivial.   I would like some
> convincing that knowledge about non-aligned signatures is worth the
> non-trivial cost that we are asking evaluators, and the planet, to absorb.
>
>
>
> Doug Foster
>
>
>
> On Wed, Oct 19, 2022 at 8:48 PM Neil Anuskiewicz <[e-mail]>
> wrote:
>
>
>
> > On Oct 19, 2022, at 5:42 PM, Neil Anuskiewicz <[e-mail]>
> wrote:
> >
> > 
> >
> >> On Oct 19, 2022, at 6:59 AM, Scott Kitterman <[e-mail]>
> wrote:
> >>
> >> 
> >>
> >>>> On October 19, 2022 12:44:16 PM UTC, Dotzero <[e-mail]>
> wrote:
> >>> On Tue, Oct 18, 2022 at 11:18 PM Scott Kitterman <[e-mail]
> >
> >>> wrote:
> >>>
> >>>>
> >>>>
> >>>> On October 18, 2022 10:16:44 PM UTC, Neil Anuskiewicz <
> >>>> [e-mail]> wrote:
> >>>>>
> >>>>>
> >>>>>> On Oct 2, 2022, at 11:01 AM, Douglas Foster <
> >>>> [e-mail]> wrote:
> >>>>>>
> >>>>>> 
> >>>>>> In many cases, an evaluator can determine a DMARC PASS result
> without
> >>>> evaluating every available identifier.
> >>>>>> If a message has SPF PASS with acceptable alignment, the evaluator
> has
> >>>> no need to evaluate any DKIM signatures to know that the message
> produces
> >>>> DMARC PASS.
> >>>>> I think it’s critical to DMARC that receivers do things like
> evaluate and
> >>>> report on DKIM whether or not SPF passes and is alignment. Without
> this, it
> >>>> would make it harder for senders to notice and remediate gaps in their
> >>>> authentication. Since there’s not a downside (that I know of), I’d
> say this
> >>>> should be a MUST if at all possible.
> >>>>
> >>>>
> >>>> What is the interoperability problem that happens if evaluators don't
> do
> >>>> that?
> >>>>
> >>>> Scott K
> >>>>
> >>>
> >>> Scott, What is the interoperability problem is evaluators didn't
> provide
> >>> reports at all? Reporting isn't a "must" for interoperability but it
> >>> certainly helps improve outcomes instead of senders flying blind.
> >>
> >> I read the email as suggesting a MUST for reporting both SPF and DKIM
> results if you report results at all, which would, I think lead to exactly
> the situation you're concerned about.  I'm skeptical of any kind of MUST
> around reporting since that's generally reserved for things that impact
> interoperability.  I do agree it should be encouraged.
> >>
> >> Mostly, at the moment, I'm trying to understand the proposed change and
> the rationale.
> >
> > I think the reactions were to the tone that that seemed to suggest that
> the importance of reporting was being downplayed. MUST is too strong and
> strongly encouraged is sufficient. The standards system relies on people
> making a good faith effort. To me, Doug’s comments came off as wanting to
> weaken the language which concerned me.
> >
> > Reporting is key for DMARC to work as a system so any hint of weakening
> that language or even could be interpreted as such caught my attention. I
> think Doug clarified his position as addressing specific cases not a
> weakening of the reporting language.
> >
> > DMARC is about the interests of the system but following the standard
> strengthens the system within which the sender or receiver operates. Even
> if one wasn’t interested in the health of system in and of itself,
> reporting benefits the admin as it increases security and reduces broken
> authentication. A *LOT* of Senders use reporting data as part of the
> process of fixing their own and third party senders they wish to allow or
> spoof, discovering errant shadow IT, etc.
> >
> > Reporting is or core importance for everyone if for no other reason than
> to avoid headaches. Thanks.
>
> s/allow or spoof/allow to spoof/
> _______________________________________________
> dmarc mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/dmarc
> <https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/dmarc__;!!CQl3mcHX2A!EsdF1EeMbORwM6q1BxYm9IsETTRVXfH8E62GtWR7mJvZ9jX2cB43dVfFr0WtGJmWfJuFPez7PgePL2wmqspAD63TEGtEypY$>
>
> _______________________________________________
> dmarc mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/dmarc
>