Re: [Tools-implementation] Revisiting whether we should continue using Docker as we currently do.

Glen <glen@amsl.com> Wed, 09 September 2020 17:51 UTC

Return-Path: <glen@amsl.com>
X-Original-To: tools-implementation@ietfa.amsl.com
Delivered-To: tools-implementation@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 117B13A0BDC for <tools-implementation@ietfa.amsl.com>; Wed, 9 Sep 2020 10:51:51 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -102.858
X-Spam-Level:
X-Spam-Status: No, score=-102.858 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, NICE_REPLY_A=-0.948, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, USER_IN_WELCOMELIST=-0.01, USER_IN_WHITELIST=-100] autolearn=ham autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id gPEkiIQgR2Sd for <tools-implementation@ietfa.amsl.com>; Wed, 9 Sep 2020 10:51:49 -0700 (PDT)
Received: from mail.amsl.com (c8a.amsl.com [4.31.198.40]) (using TLSv1.2 with cipher ADH-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 50C6E3A0C47 for <tools-implementation@ietf.org>; Wed, 9 Sep 2020 10:51:49 -0700 (PDT)
Received: from mail.amsl.com (localhost [127.0.0.1]) by c8a.amsl.com (Postfix) with ESMTPS id 23B7E3C13D4 for <tools-implementation@ietf.org>; Wed, 9 Sep 2020 10:51:28 -0700 (PDT)
Received: from [192.168.86.10] (173-8-133-94-SFBA.hfc.comcastbusiness.net [173.8.133.94]) by c8a.amsl.com (Postfix) with ESMTPSA id EE2FF3C13D3 for <tools-implementation@ietf.org>; Wed, 9 Sep 2020 10:51:27 -0700 (PDT)
To: tools-implementation@ietf.org
References: <3ae28788-898a-de72-22b6-b0f036d1b23a@nostrum.com>
From: Glen <glen@amsl.com>
Organization: AMS
Message-ID: <49b0f12f-48a8-f810-2570-1d7f4c4b6342@amsl.com>
Date: Wed, 09 Sep 2020 10:51:47 -0700
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.2.1
MIME-Version: 1.0
In-Reply-To: <3ae28788-898a-de72-22b6-b0f036d1b23a@nostrum.com>
Content-Type: text/plain; charset="utf-8"; format="flowed"
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Archived-At: <https://mailarchive.ietf.org/arch/msg/tools-implementation/J0lXAtEoFQ53mpk9KOKz9B2O86k>
Subject: Re: [Tools-implementation] Revisiting whether we should continue using Docker as we currently do.
X-BeenThere: tools-implementation@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Tools Implementation <tools-implementation.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tools-implementation>, <mailto:tools-implementation-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tools-implementation/>
List-Post: <mailto:tools-implementation@ietf.org>
List-Help: <mailto:tools-implementation-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tools-implementation>, <mailto:tools-implementation-request@ietf.org?subject=subscribe>
X-List-Received-Date: Wed, 09 Sep 2020 17:51:59 -0000

My full support and agreement here.  To clarify:  I suggested *not* 
doing it only because I didn't want to be the cause of heartburn or the 
loud voice whining.  :-)  But these arguments are persuasive, and I 
agree with everything said here, both in terms of present *and* future 
planning.

Glen

On 9/9/2020 09:21, Robert Sparks wrote:
> A few weeks ago, shortly after the yc.o team's work managed to crash the 
> host they were working on through changing settings in a container, I 
> proposed that we unroll the things we currently have in containers on 
> production. At the time, Glen suggested we not do that. I'd like to ask 
> the question again - I think we may have more to consider.
> 
> 1. I remain uneasy about docker's implementation on OpenSuse. The 
> container crash above, the issues we've run into with containers locking 
> (and sometimes causing processes talking to them like apache) to hang 
> are suspicious. That we've not been able to pin down what's really going 
> on suggests to me the issue is in a place we can't really look, inside 
> docker's interstitial networking or filesystem abstraction code perhaps.
> 
> 2. Many of the containers we have (and in particular the one for the 
> website) really need to be designed differently if they are going to 
> remain deployed as containers. The amount of file-system mapping they do 
> is not what the docker architects expect as a normal use-case. Mapping 
> sockets in the way we do is also likely not something they focus on 
> testing.
> 
> 3. Docker is making Glen uncomfortable and the benefit for him 
> (operationally) of the containerization is not proportional to the extra 
> problems it is bringing.
> 
> So I again suggest that we unroll for the production deploys, at least 
> for now. I think we can unroll everything at this point, but there might 
> still be a hitch in unrolling the trac instances. Henrik - could you 
> remind me what our thinking was with respect to those?
> 
> I do plan to keep up the pressure to have containerized versions of 
> these services - this isn't a call to abandon Docker - but I suggest we 
> need to change how we're currently using it.
> 
> RjS
> 
>