[ietf-822] Re: Fwd: [Ietf-dkim] Fwd: WG Action: Formed Mail Maintenance (mailmaint)

[John C Klensin <[e-mail]>]

(since we are talking about a group that has already been chartered,
copying that list in the hope that this discussion will at least help
inform its work)

--On Tuesday, May 14, 2024 23:06 -0400 Keith Moore
<[e-mail]> wrote:

>... 
> (It's my understanding that the traditional 2026 requirements for
> multiple implementations were intended to debug the specification,
> to make sure it's clear enough to allow independent implementations
> to interoperate.   It's not a sufficient test but it may still be
> a useful one.   I don't think the 2026 requirements were
> intended, or are sufficient, to assure "real world"
> interoperability.   Not that I think "real world"
> interoperability is a bad goal, just that it probably needs more
> work if that's what we're shooting for.)

Keith,

Someone should probably bring Scott Bradner in on this but my
understanding and recollection is that those requirements in 2026
were very much tied to the idea that a Proposed Standard could be
really just an idea for people to consider and that the notion of
independent and interoperable implementations was introduced with
Draft Standard.  The latter was the first spec that was intended to
really be taken seriously and it imposes an even stronger requirement
about different code bases.  The very first task that was assigned to
me when I became an AD involved explaining to a large vendor that
"proposed" meant just that -- that it was perfectly reasonable for
the IETF, based on experience with those implementations and some
field testing, to make significant changes between a Proposed
Standard of a spec and a Draft Standard version (or even another
Proposed Standard version).  If they had deployed something to users
based on claims of conformance to a Proposed Standard and we changed
things so that those claims were no longer meaningful, it was their
problem for ignoring those principles and assumptions, not ours for
making the change (especially if the change was made in the interest
of broader interoperability).  Reread the last paragraph of Section
4.1.1.

Perhaps unfortunately, those models never really worked.  As it
became progressively harder to get documents published at PS, I-Ds
became much more formal and permanent than 2026 and its predecessors
intended (note, e.g., the recent proposal to remove the "expiration"
notion entirely).  Remember when "remain unchanged after six months"
meant something rather close to "vanish" and when we acted as if we
believed the emphasized "Under no circumstances..." comment in
Section 2.2, and the effective prohibition on referencing an I-D (in
any way -- "work in progress" would not be associated with a file
name, URL, or equivalent--  at the end of that section?

For better or worse, things have changed.  In recognition of some
realities, we eliminated Draft Standard with RFC 6410.  I think part
of the hope was that doing so would cause more documents to advance
beyond PS directly to Internet Standard.  I have not tried counting,
but I think it would be safe to describe that idea/hope as not wildly
successful.  

>> If there aren't at least two implementers able to put together
>> proof  of concepts and have them interoperate with each other,
>> then you don't  have anything shown workable yet.  It can be as
>> simple as a  perl^Wpython script that spits out the wire format
>> and a parser that  consumes it.  Have two people write those,
>> send each other messages,  parse them. Viola,  you have yourself
>> a spec.

> Well, sort of.   Except, in my experience there's a huge gap
> between "proof of concept" and "specification".   And much of the
> "real world" lies in that gap.  Which means you don't get the
> "real world" experience from merely implementing "proof of concept"
> implementations.

Continuing with the comments above, my understanding of the PS/ DS /
Internet Standard boundary 30 years ago was such that they could have
been described as 

 * PS: "idea that the community believes it worth exploring
	further, described in enough detail to do that and to be
	confident that discussions would be about the same thing"
	(although the latter two were just hypotheses).
	
 * DS: proof of concept with documentation refined from
	experience discussing the PS version.
	
 * Standard: Demonstrated to work in the real world and with
	documentation good enough to allow production-quality,
	independent, and interoperable implementation without having
	to consult within the IETF for details.

Again, at least IMO, never worked.  But those aspirations seem to
still drive our way of talking about things.

> I'll elaborate.   If you're considering adding some feature to
> your email MTA or MUA, and you don't actually test that feature
> implemented in your MTA or MUA products, with a significant
> population of real users, you're not getting benefit of real world
> experience.
> 
> And most IETF specifications, even those that have made it to Full
> Standard, don't have the benefit of such real-world
> interoperability tests.    I'd argue that most IETF
> specifications aren't complete enough to allow an application
> that's written to the specification to function well in the "real
> world".
> 
> "real world" includes security that works.  "real world" includes
> the ability to effectively manage the application at scale.  "real
> world" means the application holds up under full load and then
> some.   "real world" means that legitimate emails get delivered
> despite firewalls, middleboxes, spam filters, etc.
> 
> So in summary: I want IETF applications, including email, that are
> written to specification, to function well in the "real world".
> Right now, I'd argue, they do a poor job of it.   I'm all for
> improving that situation.

I think I agree with all of the above at least until I start asking
hard questions about how to do things in practice.

> Is the best place to put this test prior to IESG submittal?   (I
> don't think it's sufficient.)  Does a "proof of concept"
> interoperability test help to further this goal?   If so, why?
> (Again, I don't think it's sufficient, and I'm not even sure
> whether it helps much without considerable additional testing
> beyond that.)
> 
> But in my mind, there's really no argument about interoperability
> testing being a good idea, I just wonder about particulars like
> timing and thoroughness.

Indeed.  And about what our aspirations, rather than just the
vocabulary we use, actually are.

best,
   john