Re: [apps-discuss] Comments on Malformed Message BCP draft

[Keith Moore <[e-mail]>]

On Apr 16, 2011, at 4:27 AM, Nathaniel Borenstein wrote:

> Although all my basic instincts are in agreement with Keith's position -- the only way you will ever persuade message senders to comply more fully with the standards is to reject their non-compliant messages -- my recent experience has turned me 180 degrees and convinced me that this position is hopelessly idealistic.  
> 
> As some of you know, I have only recently returned to work at a company (Mimecast) whose primary business includes the maintenance and operation of a large-scale MTA.  The situation has changed a lot in the last ten (not to say twenty) years, and in many ways not for the better.  Today's commercial reality is a race to the bottom that more or less guarantees that MTAs will do precisely what Keith says they should not do, which means, to my mind, that it behooves us to document our collective wisdom on the least harmful and most consistent way to do it.
> 
> A single example should make this clear:  Nearly every business now receives its mail via some third party email security mechanism -- perhaps an appliance, perhaps locally installed code, and increasingly a cloud-based service.  Competition among these third parties is fierce, and inevitably standards compliance is far less of a driver of business decisions than customer satisfaction.  Imagine, then, that company X, which enforces the standards very strictly, as Keith advocates, manages to "steal" a customer from company Y, which does not.  All of a sudden -- and I assure you this really happens -- a host of slightly-misformatted but customer-desired messages which were delivered by company Y will start getting blocked by company X.  The customer, inevitably, will scream bloody murder.  They won't care that company X is doing a better job of enforcing the standards, they will only care that company X is -- wrongly, in their eyes -- blocking desirable streams of messages that the previous vendor delivered with no visible problem.  The generalization is clear:  the financial and business incentive is to deliver every message that you can possibly figure out how to deliver, so that your service doesn't appear "inferior" to customers who don't give a rat's rump about the details of standards compliance.

This is why I think the checking for syntax and either correct or rejection of malformed messages needs to be done on the sender's side, by the MSA.

But there's also the opposite argument - say that handling of incoming messages moves from company X to company Y, and malformed messages that were blocked by company X are now passed by company Y.  As a result, the client company receiving such messages, now finds itself subject to drastically increased attacks that use malformed messages (and UA susceptibility to them) as a vector.   Now the client will scream bloody murder because Y did not block such messages.

Both scenarios argue for more uniform handling of messages, and/or more strict adherence to *822/MIME syntax. so there will be less variation between how different MTAs/providers handle mail.     (Though the mail screen/delivery providers might want to distinguish themselves from one another by how strict/tolerant/clever they are, thus increasing the variation between providers.)

But yes, the desire for uniform handling might argue for defining a standard syntax verifier (with some recommendations about correction, where appropriate) and encouraging MSAs to implement it.   The desire for more uniform handling of messages might also argue for trying to move such verification/correction to MSAs and having downstream MTAs treat such messages as opaque, so that there will be fewer opportunities for messages to be subjected to variable amounts of verification/correction.  

(Honestly, I have long thought that traditional SMTP store-and-forward these days is more trouble than it's worth, and what we need is to reduce the number of hops that a message endures as it goes from sender to recipient.  To this end, I'd support some sort of pass-through extension to SMTP to allow SMTP to continue to be used as a means to let messages get through firewalls, but without actually doing store-and-forward when not necessary.)

> I'm not sure why this came as such a surprise to me, as it is actually just another instance of Postel's Law.  Being liberal in what we accept means, in this case, accepting and delivering any message where we believe, with a high level of confidence, that the sender's intentions are clear despite its standards violations.

Postel's Law was great in ARPAnet days, but I have long wondered how well it scales to a network with billions of users and thousands or tens of thousands of implementations.

But I don't really have any problem in principle with correcting messages where "the sender's intentions are clear".  In practice, I wonder how many cases exist for which code can reliably detect that "the sender's intentions are clear".   For example, a missing Date header field - sure the MSA or downstream MTA can supply one, but it doesn't really know the date/time at which the sender caused the message to be sent.

> I can tell you that nothing the IETF says is likely to have any effect on whether or not Mimecast tries to "fix" and deliver such messages -- we do and we will, as do nearly all of our competitors, because although we care what the IETF says, we have no choice but to care what our paying customers say even more.  However, if the IETF offered guidance on the least harmful way to do this, the odds are good that we would follow it.  And I think it would be better if vendors who felt the need to "fix" messages would at least be mutually consistent in how they fix them.
> 
> One (probably controversial) idea that might reduce our collective angst would be to specify that, when an MTA fixes such a message, it also generates an explanatory/warning message back to the sender or the sender's postmaster (perhaps limited to 1 message per day per malformation type).  That way we might not be exacerbating existing problems quite as much as Keith and other (myself included) fear.  And the senders of misformatted messages might eventually fix their code, if only to shut up the annoying warning messages.  -- Nathaniel

Well, if there were a standard set of tests to apply to outgoing mail, and these tests were implemented by MSAs, those MSAs could presumably make such data available to sending UAs and/or their users.    And yes, hopefully MUA vendors would make sure that messages generated by their MUAs passed such tests.  That would be, IMO, the best of all possible outcomes.

Keith