[ietf-822] Re: Fwd: [Ietf-dkim] Fwd: WG Action: Formed Mail Maintenance (mailmaint)

[Keith Moore <[e-mail]>]

On 5/14/24 07:51, Bron Gondwana wrote:

>> Offhand, I think it's unreasonable to expect parties to commit to
>> implement a standard before they've even seen a near-complete draft of
>> it, so I'd like to see this provision go away.
>
> I'd like to see it become more widely distributed.  You discover a LOT 
> more when actually implementing something than when a bunch of smart 
> people sit down and think really hard without actually real-worlding 
> it.  I have so many cases I've designed something that seemed really 
> nice only to see it fall over under real world conditions, and I 
> consider myself at least medium-smart.

I agree with the latter two statements and my experience reflects 
yours.   I certainly want to encourage timely implementation of IETF 
specifications before they are finalized.
The question in my minds are: (a) when should such "real-worlding" be 
done? and (b) does the kind of implementation being required in this 
WG's charter amount to "real-worlding"?

(It's my understanding that the traditional 2026 requirements for 
multiple implementations were intended to debug the specification, to 
make sure it's clear enough to allow independent implementations to 
interoperate.   It's not a sufficient test but it may still be a useful 
one.   I don't think the 2026 requirements were intended, or are 
sufficient, to assure "real world" interoperability.   Not that I think 
"real world" interoperability is a bad goal, just that it probably needs 
more work if that's what we're shooting for.)

> If there aren't at least two implementers able to put together proof 
> of concepts and have them interoperate with each other, then you don't 
> have anything shown workable yet.  It can be as simple as a 
> perl^Wpython script that spits out the wire format and a parser that 
> consumes it.  Have two people write those, send each other messages, 
> parse them. Viola,  you have yourself a spec.
Well, sort of.   Except, in my experience there's a huge gap between 
"proof of concept" and "specification".   And much of the "real world" 
lies in that gap.  Which means you don't get the "real world" experience 
from merely implementing "proof of concept" implementations.

I'll elaborate.   If you're considering adding some feature to your 
email MTA or MUA, and you don't actually test that feature implemented 
in your MTA or MUA products, with a significant population of real 
users, you're not getting benefit of real world experience.

And most IETF specifications, even those that have made it to Full 
Standard, don't have the benefit of such real-world interoperability 
tests.    I'd argue that most IETF specifications aren't complete enough 
to allow an application that's written to the specification to function 
well in the "real world".

"real world" includes security that works.  "real world" includes the 
ability to effectively manage the application at scale.  "real world" 
means the application holds up under full load and then some.   "real 
world" means that legitimate emails get delivered despite firewalls, 
middleboxes, spam filters, etc.

So in summary: I want IETF applications, including email, that are 
written to specification, to function well in the "real world". Right 
now, I'd argue, they do a poor job of it.   I'm all for improving that 
situation.
Is the best place to put this test prior to IESG submittal?   (I don't 
think it's sufficient.)  Does a "proof of concept" interoperability test 
help to further this goal?   If so, why? (Again, I don't think it's 
sufficient, and I'm not even sure whether it helps much without 
considerable additional testing beyond that.)

But in my mind, there's really no argument about interoperability 
testing being a good idea, I just wonder about particulars like timing 
and thoroughness.

Keith