Re: Yahoo breaks every mailing list in the world including the IETF's

[Eric Dynamic <[e-mail]>]

Responding to John Levine, Brian Carpenter, Sabahattin Gucukoglu and S Moonesamy:

The Problem

I have not read the DMARC protocol in detail, nor looked at any software provided for implementing it.
What I have done is looked at the DNS record DMARC uses to "advertise" a site's DMARC policy,
and the commentary surrounding that.

>From what I can tell, DMARC is not a very clever protocol.
What it appears to do, relative to a sending site X, is say

"Sending site X uses SPF or DKIM or both or neither, in order for you the recipient to verify whether X
actually sent the email you're currently inspecting."

If that is ALL that DMARC does, I would suggest DMARC is a worthless protocol and that it should be
suppressed and rescinded as an IETF-recognized protocol.

Why? If a receiving site cares to combat spam, it will be using SPF and DKIM already; there is no
point for an extra protocol on top to say simply "yes, I the sender did implement SPF and/or DKIM,
so /you should/ proceed to use those methods to validate email claiming to come from my domain."
Namely, DMARC is simply redundant bureaucratic overhead; the ISP would use these verification
methods without having to be reminded to do so. And if the receiving ISP DOESN'T implement SPF
or DKIM, then DMARC doesn't refer to anything the recipient CAN do. So DMARC is effectively useless
make-work.

I did note on the DMARC home page glowing self-reviews of how many corporations were protecting
how much email using DMARC - but again, it WASN'T DMARC doing that - it was SPF and DKIM.

Given that the primary adopters of DMARC seem to be large, for-profit corporations, and given e.g.
Yahoo's policy of excusing dumping MILLIONS of valid emails with the excuse of "DMARC policy
violations", DMARC appears to be a Corporate Trojan Horse and an excuse to start tampering with
the public's email at will, with no accountability.

I run mailing lists for the Berkeley (California) Parent-Teacher's Association (PTA.) So Yahoo is
essentially saying that no Yahoo users can get email sent to such lists BY Yahoo users. This of course
makes no sense. Moreover, the issue has been outstanding for long enough that Yahoo should by now
have REMEDIED the situation. But this seems to much to expect: Corporate Accountability, in an age
where these corporations think they are "important" for having millions of subscribers.

There were calls to complain to Yahoo (these are steadfastly ignored) and to boycott Yahoo (these are
also ignored by a public on automatic pilot, bought out by the notion of "free" accounts of all kinds.)
We are entering a stage in the Corporatization of America where it's time to pay attention and fight back,
or these Corporations will WIN and effectively replace the IETF with their own "common" (malfeasant)
practices. It has been suggested that Yahoo, Google, et. al. are busy interfering in open-source-run
mailing list for the express purpose of making all independent mailing list operators look bad.
The intended effect is "If Joe Smith, ISP, can't make his mailing lists work then maybe we should switch
to using a Google/Yahoo list" - which is only to reward the very outfits that are trying to commit this
piracy in the first place.

The problem is solvable - at a cost

I have rescued my own mailing lists from this corporate malfeasance - at the cost of having to send
a unique email separately to each and every mailing list subscriber. This burdens my server and it
burdens the Internet (but do the Corporations care? No.) it surely must violate some "best practices"
RFCs published by the IETF.

I have provided a description of what I did to solve the issues to a few fellow ISPs; I include these remarks below.

I would suggest people start complaining to the EFF (Electronic Frontier Foundation.) This is something which is supposed to be
their "back yard", that they would address, but they ignored my initial complaints to them. I think if they start hearing from the
public at large they will realize this is non-trivial and something very worth their attention.

I think the IETF needs to issue an emergency policy statement that condemns the interference these corporations are causing with our Free Speech.
(to repeat:) It is almost certain that what Yahoo, Google, AOL, et. al. are doing violates a number of "best practices" RFCs that the IETF has published.
It is multiplying Internet traffic loads for no valid purpose.

the fix-it advice:
===========================================================================================================

What I did to bypass the Google/Yahoo/AOL issues was simple enough.

1. install the latest mailman (2.1.18_1) - that is the version that resulted from the bug I reported to them. So if the provider already has 2.1.18 they still need to insure they have the LATEST version of 2.1.18.
2. As suggested to me by Mark Sapiro of lists.org, the two lines suggested to be included in Mailman/mm_cfg.py:

VERP_DELIVERY_INTERVAL = 1
SMTP_MAX_RCPTS = 1

3. However, doing that did not have the desired/intended effect of forcing one email per list recipient, so I did this additional hack to Mailman/Handlers/SMTPDirect.py:

def process(mlist, msg, msgdata):
    recips = msgdata.get('recips')
    if not recips:
        # Nobody to deliver to!
        return
    # Calculate the non-VERP envelope sender.
    envsender = msgdata.get('envsender')
    if envsender is None:
        if mlist:
            envsender = mlist.GetBouncesEmail()
        else:
            envsender = Utils.get_site_email(extra='bounces')
    # Time to split up the recipient list.  If we're personalizing or VERPing
    # then each chunk will have exactly one recipient.  We'll then hand craft
    # an envelope sender and stitch a message together in memory for each one
    # separately.  If we're not VERPing, then we'll chunkify based on
    # SMTP_MAX_RCPTS.  Note that most MTAs have a limit on the number of
    # recipients they'll swallow in a single transaction.
#<>ecsd    deliveryfunc = None
    deliveryfunc = verpdeliver

On line 115, I manually forced the delivery function to be "verpdeliver" instead of "None". The two affected lines in bold. This had the desired
effect of forcing one email per list recipient (without having to make sendmail behave that way for everyone on the entire server.)

4. In the list configs, as recommended by Mark Sapiro of lists.org again, the choice under both "General" and "Privacy Options -> Sender Filters"
    is "Munge From" in the new fields referring to DMARC;

5. But again, since those settings did not seem to disguise a Yahoo sender from being seen and rejected by Yahoo, I simply told the list YES
    in the field "Hide the sender of a message, replacing it with the list address (Removes From, Sender and Reply-To fields)" under General options.

#5 begs the question whether any of the DMARC (mailman) stuff was necessary at all, insofar as doing #5 would hide the "a yahoo user sent this email"
from Yahoo, apparently regardless the DMARC stuff new to mailman 2.1.18. It suggests that just doing #5 is enough to get past DMARC on ANY version
of mailman, but I leave such testing to others.

===

WHAT GETS FIXED?

1. Forcing the list to send one email per user per list fixes:
* google's "guess which domains we host" shellgame policy that would randomly block emails sent to @gmail and google-hosted domains simultaneously.
* yahoo's "too many recipients so up yours" policy that effectively killed emails sent to many yahoo users at once: and Yahoo will not say what their limit is.

2. Doing the "anonymous list" change fixes:
* yahoo's and presumably AOL's DMARC policy

Of course, sending one email per list member means having to try to parallelize sendmail's queue processing to overcome the inefficiency.
The complaint (that lists.org did not seem to care about as long as they had a workaround no matter the inefficiency, and that the EFF effectively
ignored with protests about how "busy" they were, as if this was not important) is that: that these jerk corporations are thumbing their noses at
IETF best practices by forcing a great deal more traffic onto the Internet for no valid reason that they can demonstrate if such a demonstration is
demanded of them. The sendmail default on "MAX RCPTS" is 100. I see no reason to limit that number. Yahoo, some years ago, tried to impose a
limit of 6 on that number and finally relented, I'm guessing, when too many people told them how stupid they were being. But they are back to
that practice again, and this time they will not tell you what the limit is (and I haven't bothered trying to establish it empirically.)
The Google "shellgame" is that Google feels free to reject inbound email, when that email is addressed both to a @gmail user and to a user inside
a private domain the recipient is having Google host, at the same time. That is, suppose

foo.com in MX something.google.com

If I send an email
To: [e-mail],[e-mail]

I will get a Google bounce message like this (the two examples do not use same domain names):

... while talking to aspmx.l.google.com.:
[e-mail]... Deferred: 451-4.3.0 Multiple destination domains per transaction is unsupported.  Please try again. td10si7147932pac.181 - gsmtp

and neither recipient gets the email. This of course is beyond any simple ability of a mailing list program to fix -
UNLESS you limit the outbound email to one email per recipient; although there is NO VALID REASON for Google to impose that restriction.

This general practice of "let's mess with independent mailing lists and see how long it takes people to start using Google/Yahoo mailing lists
to get away from failures WE IMPOSED ON PEOPLE" has to stop: these corporations have to be publicly put on the carpet for being at best STUPID
in their "presumed" antispam practices - unless they are being deliberately and calculatedly malfeasant. I suspect the truth is in-between.
On Google's part they have to know they are wrecking mailing lists; I think Yahoo's admin is just moron-managed.

I intend to post a notice to the various lists I manage asking the Yahoo and Google users to complain EN MASSE,
and I intend also to suggest that people should look forward to abandoning these "free mail services" if they refuse to behave themselves.
However, American Human Nature is so craven and bought-out by now that most people would scoff at the idea that (gasp) they might
consider paying a pittance per year to have their own email address within their own domain, untampered with, unsnooped upon, and
unbranded by some third-party jerk.

I can play Google off against Yahoo, by telling the Yahoo users to threaten to switch to Google if Yahoo won't behave; but it's no great
solution if they actually do so. Another recommendation to people is to NOT HAVE Google host their email domains, but once people hear
"free" they ignore reason, logic and everything else. How sad it is how cheaply people are bought out against their own interests nowadays.
===========================================================================================================

Solving the SPAM problem once and "for all time", the right way

The genuine solution to the problem of worldwide spam and 98% of all email flows being "buy this garbage" solicitations and "please infect
yourself" virus-carrying messages is to QUIT USING MICROSOFT SOFTWARE connected to the Internet. Every quarter without fail, cert.org
releases a list of "latest detected exploits" containing an average of a half-dozen mentions of Microsoft (base system) exploits.
In contrast, Linux/Unix have had TWO issues affecting the base system in the last FIFTEEN YEARS (both the telnet daemon, I think, and
telnet for login is deprecated anyway.) Get Microsoft software off the Internet and all the "bot-nets" will be killed within two months,
worldwide problem SOLVED.

-----

-ecsd (Eric Dynamic)
Sysad for Transbay.net / Transpacific.net / UCTelecommunications.com
the polemic tone reflects my irritation, sorry. :)