Re: [Sidrops] ARIN RPKI Repository issue - Wednesday 24 October 2018

Christopher Morrow <christopher.morrow@gmail.com> Thu, 25 October 2018 15:32 UTC

Return-Path: <christopher.morrow@gmail.com>
X-Original-To: sidrops@ietfa.amsl.com
Delivered-To: sidrops@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 51F2E130E7C for <sidrops@ietfa.amsl.com>; Thu, 25 Oct 2018 08:32:17 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.998
X-Spam-Level:
X-Spam-Status: No, score=-1.998 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2_k87pRX9yDo for <sidrops@ietfa.amsl.com>; Thu, 25 Oct 2018 08:32:14 -0700 (PDT)
Received: from mail-it1-x135.google.com (mail-it1-x135.google.com [IPv6:2607:f8b0:4864:20::135]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 5F425130E76 for <sidrops@ietf.org>; Thu, 25 Oct 2018 08:32:14 -0700 (PDT)
Received: by mail-it1-x135.google.com with SMTP id 74-v6so2142536itw.1 for <sidrops@ietf.org>; Thu, 25 Oct 2018 08:32:14 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=9u+dpR8xdm8gR5BbnxYr7uze1y9tlHtu9dygZtTtfeQ=; b=U00tPSZYN8r7tryy7YjWTO8lkw3f4Tp6/eQZzxiI3iEHiecL9P0ngC0cF6peYgwnhi iFzgkkYaph7IQa8+lpSo0/+ZiRNVNMHSIJZXbNYff54zdh5aKwjg0zLWxVjAnY+wHgg1 BuPuWVCKy/HNO6w6tJHtt5O3mGOnHGMjPQRhwbD/TU5N5GilMZezg8bbB0K4SbsywK7Z IcuGEaYyXiFfxsLX0dFh5LmE7fcGpzi6HHa3vUkPm4O4w/TM6xtsln43WqRohBAYDjp4 vN+ZJSpjf8d2CQAA1p/WR3EtOcq2MIHA9TGjK470w0O9UxS95IL32IDrCIpe3Ei7q5M7 7LZA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=9u+dpR8xdm8gR5BbnxYr7uze1y9tlHtu9dygZtTtfeQ=; b=RGjn65QvzwFiVyIk9M9T7C8/W1njmA+nXttzey3sKCIoDKSawW/RUuV2HFHcdmpulZ q6vO91+1fs2p+Cqv4H2slqPChWq16gW/NeD+XCCnflPkRz/34+SAcJN8pqzHraKYNDMi pXZl5UvqEu7hpcJfbmlGAnpSsn1ibYkfgi1Uhy/SJXpJX00qA/OLzk4VnBSiGuhXL6eZ FxDYs+E5b+YVkEB4JzJVCy9GHju4CT39RtPEFB0pSdEolkA7oePceZC4rFlbfpzm6pWI GVnCRcknLqd28MztjvxcWkORXifHvz2G7yGFVePDFuy5dqxTnkebrITGwoh4LXQSfMxK ibpA==
X-Gm-Message-State: AGRZ1gJuLbn2JNfx01oW8tKGoi95riq6ks73WJz1Mb7gyWfZw/PmxI/q /lzBYqZtoFCINK/EC7AUnKWouQN0+SMA0490H7Q=
X-Google-Smtp-Source: AJdET5cFp91SbgQG94RwGF/qVIvd4H8B6cRD1OZGTFSeSdPdyWa98Fp2CrMOi0A2PlAqyd3XFyjGzukXrzDISO/yN0o=
X-Received: by 2002:a24:5cb:: with SMTP id 194-v6mr1318799itl.139.1540481533371; Thu, 25 Oct 2018 08:32:13 -0700 (PDT)
MIME-Version: 1.0
References: <15B30322-2CE0-4E0B-A54A-86A2743B73AF@arin.net>
In-Reply-To: <15B30322-2CE0-4E0B-A54A-86A2743B73AF@arin.net>
From: Christopher Morrow <christopher.morrow@gmail.com>
Date: Thu, 25 Oct 2018 11:32:01 -0400
Message-ID: <CAL9jLaZT3G3CrD8vRLok8K2nH6-uV13wAuZEbc-DyG=T1gcOUg@mail.gmail.com>
To: John Curran <jcurran@arin.net>
Cc: sidrops@ietf.org
Content-Type: multipart/alternative; boundary="000000000000947e7b05790f4e09"
Archived-At: <https://mailarchive.ietf.org/arch/msg/sidrops/rcnoRybzPdVGrxBtuHccpq7PBlA>
Subject: Re: [Sidrops] ARIN RPKI Repository issue - Wednesday 24 October 2018
X-BeenThere: sidrops@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: A list for the SIDR Operations WG <sidrops.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/sidrops>, <mailto:sidrops-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/sidrops/>
List-Post: <mailto:sidrops@ietf.org>
List-Help: <mailto:sidrops-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/sidrops>, <mailto:sidrops-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 25 Oct 2018 15:32:17 -0000

hey!
1) thanks for the info/update/transparency
2) it'd be cool to know: "hsm like FOO don't like getting packets over BAR
kb, be aware if you operate like this"
3) I'm glad we're finding these (and writing test code, and monitoring)
while we are still young in this process

Also, it's nice to occasionally see the sausage making.


On Thu, Oct 25, 2018 at 3:37 AM John Curran <jcurran@arin.net> wrote:

> <https://www.arin.net/announcements/2018/20181024_update.html>
>
> ARIN experienced an RPKI Repository issue yesterday afternoon (Wednesday
> 24 October 2018) which resulted in the number of objects in the repository
> dropping from its normal hundreds of objects to only 7 objects.  The root
> cause for the failure was too many ROAs for a resource cert that led to an
> untested situation whereby we exceeded the MTU for the HSM and an invalid
> manifest in repository..  It appeared that some validators stop processing
> after the invalid manifest, thus resulting in a greatly reduced count of
> objects published.
>
> This condition occurred for much of day, and was resolved via a workaround
> at approximately 22:00 ET.   The ARIN RPKI Repository appears normal on
> RPKI monitors at this time, and while we have some work to do to durably
> address the issue for long-term, we not expect any recurrence of this
> event.
>
> FYI,
> /John
>
> John Curran
> President and CEO
> American Registry for Internet Numbers
>
>
>
> _______________________________________________
> Sidrops mailing list
> Sidrops@ietf.org
> https://www.ietf.org/mailman/listinfo/sidrops
>