Re: IETF Mail List Failure

Alexa Morris <amorris@amsl.com> Sat, 19 May 2018 03:03 UTC

Return-Path: <amorris@amsl.com>
X-Original-To: ietf@ietfa.amsl.com
Delivered-To: ietf@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 7CE2112E89E for <ietf@ietfa.amsl.com>; Fri, 18 May 2018 20:03:51 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -4.2
X-Spam-Level:
X-Spam-Status: No, score=-4.2 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yrD3VOkdlqUf for <ietf@ietfa.amsl.com>; Fri, 18 May 2018 20:03:50 -0700 (PDT)
Received: from mail.amsl.com (c8a.amsl.com [4.31.198.40]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 864E2126CD8 for <ietf@ietf.org>; Fri, 18 May 2018 20:03:50 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1]) by c8a.amsl.com (Postfix) with ESMTP id 3E9261C00B6 for <ietf@ietf.org>; Fri, 18 May 2018 20:03:41 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
Received: from c8a.amsl.com ([127.0.0.1]) by localhost (c8a.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id PJplMuPaq-0C for <ietf@ietf.org>; Fri, 18 May 2018 20:03:41 -0700 (PDT)
Received: from [IPv6:2601:647:4200:a088:6918:6652:6349:b866] (unknown [IPv6:2601:647:4200:a088:6918:6652:6349:b866]) by c8a.amsl.com (Postfix) with ESMTPSA id 085B61C00B5 for <ietf@ietf.org>; Fri, 18 May 2018 20:03:41 -0700 (PDT)
From: Alexa Morris <amorris@amsl.com>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (Mac OS X Mail 11.3 \(3445.6.18\))
Subject: Re: IETF Mail List Failure
Date: Fri, 18 May 2018 20:03:49 -0700
References: <152661185460.1578.3189815605270968957.idtracker@ietfa.amsl.com> <m2603lieru.wl-randy@psg.com>
To: IETF <ietf@ietf.org>
In-Reply-To: <m2603lieru.wl-randy@psg.com>
Message-Id: <542AF553-F655-4588-A5DE-E067B6449ECD@amsl.com>
X-Mailer: Apple Mail (2.3445.6.18)
Archived-At: <https://mailarchive.ietf.org/arch/msg/ietf/PYdRyF6ENyijdTb5jNXiQJZBFOI>
X-BeenThere: ietf@ietf.org
X-Mailman-Version: 2.1.22
Precedence: list
List-Id: IETF-Discussion <ietf.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/ietf>, <mailto:ietf-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ietf/>
List-Post: <mailto:ietf@ietf.org>
List-Help: <mailto:ietf-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ietf>, <mailto:ietf-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 19 May 2018 03:03:52 -0000

I am relieved to report that the issue was repaired earlier today and DMARC rewriting is back online (and has been for several hours). I know that many of you are interested in hearing the technical details, and I will make sure that we share that information shortly. 

Regards,
Alexa

> On May 18, 2018, at 12:50 AM, Randy Bush <randy@psg.com> wrote:
> 
>> On 17 May, from 17:48 UTC (13:48 EDT) to 23:48 UTC (18:49 EDT),
>> outbound email from the IETF email lists failed to send. Mail sent to
>> the lists went into the archives, but was not received by list
>> subscribers.
>> 
>> A problem in the DMARC-rewriting system caused outbound mail delivery
>> to fail silently, without alarm or error message. The rewriting system
>> has been taken offline, which restored outbound email service.  The
>> lists are back up.
>> 
>> Unfortunately, because the mail was discarded rather than queued, if
>> you sent mail to any IETF list today during the outage window, we ask
>> you to please resend your message.
>> 
>> We apologize for this issue and are working with the Tools Team to
>> correct the system before it is put back online again.
> 
> thanks, alexa.  these things happen, and i won't even demand a refund :)
> sympathies for the pain.
> 
> < micro-management = off >
> 
> i really appreciate technical details.  there are still a few operators
> left in the ietf community, and we try to learn from our oopsies.  a
> deeper post mortem would even be welcome as many of us are also trying
> to deal with dmarc hell.  you run a medium scale mail operation and any
> lessons you share are appreciated.
> 
> randy
>