[ietf-smtp] Characteristics of Isolated (or mostly-isolated) industrial IP Networks

Keith Moore <moore@network-heretics.com> Sat, 04 January 2020 07:26 UTC

Return-Path: <moore@network-heretics.com>
X-Original-To: ietf-smtp@ietfa.amsl.com
Delivered-To: ietf-smtp@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C953412011F for <ietf-smtp@ietfa.amsl.com>; Fri, 3 Jan 2020 23:26:59 -0800 (PST)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.597
X-Spam-Level:
X-Spam-Status: No, score=-2.597 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=0.001, MIME_QP_LONG_LINE=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_NONE=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=messagingengine.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vzlbV_vmww5I for <ietf-smtp@ietfa.amsl.com>; Fri, 3 Jan 2020 23:26:57 -0800 (PST)
Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 83931120073 for <ietf-smtp@ietf.org>; Fri, 3 Jan 2020 23:26:57 -0800 (PST)
Received: from compute6.internal (compute6.nyi.internal [10.202.2.46]) by mailout.west.internal (Postfix) with ESMTP id CAEC75DE; Sat, 4 Jan 2020 02:26:55 -0500 (EST)
Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Sat, 04 Jan 2020 02:26:55 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:message-id:mime-version:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=0kvViH nqm6o6lalx5WaPWenxf9LsV2BMDBQprMcKR4o=; b=fgmZdazGYjObm6w1lB+xSV R9rRvJvg+kr0GTa5uDsFR6m9ut6Xyh12ENwHhVzslqor2AJSekoARGre/BLiVd4C I1LE9emGo2bFlY6nv0KVw7CscaKe8002A6n5Mw+3XhCY8FH+wVZkDXDY73XKokP+ pLyqPTmFHzGWCA1a4bCnSNnSdvsi+KMrvYbGNRn/XP5Qc1MBVqDabYK6YJ60Rz/W Tnv9HRpBzHFeXNjHCh9KzZVWLEq8094jliW27OTA0Aq+EBzRRqNITWOPTquk0wLA pJQLO7+LhueHKxgINdCarylN4YYovJHDzgSghByHG8uACfTzhAg4bqlDOGrIDoFQ ==
X-ME-Sender: <xms:Pz4QXrpUAAEW4Dgy0Q9835ePNabPCxKF6yKr8IVOk7-n7wDwqDY7BA>
X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedufedrvdeggedguddtkecutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurheptgfghfgguffkfffvofesrgejmh erhhdtjeenucfhrhhomhepmfgvihhthhcuofhoohhrvgcuoehmohhorhgvsehnvghtfiho rhhkqdhhvghrvghtihgtshdrtghomheqnecukfhppeelledrvddtfedrfedvrddutdenuc frrghrrghmpehmrghilhhfrhhomhepmhhoohhrvgesnhgvthifohhrkhdqhhgvrhgvthhi tghsrdgtohhmnecuvehluhhsthgvrhfuihiivgeptd
X-ME-Proxy: <xmx:Pz4QXoMPb1NZoiG0MiqKbYTpnTy9ChxvWwkyaHR99sX-OlaTfnQWmw> <xmx:Pz4QXr0hG7ylQJAWajTS7xCkbhrOFk1Tz08KXHtoDb1RIer4ySXHzw> <xmx:Pz4QXvDgpYWJh3ZyJ4FlL_LRJJV58_WSxZvvBuQKZlhEMpgaVB-Yjw> <xmx:Pz4QXk_blG5OJvtPBqtf2J013MS6pum0EX-tYG5anc1fz3I7JIG8iA>
Received: from [30.64.126.197] (ip-99-203-32-10.pools.cgn.spcsdns.net [99.203.32.10]) by mail.messagingengine.com (Postfix) with ESMTPA id 978E930600A8; Sat, 4 Jan 2020 02:26:54 -0500 (EST)
Content-Type: multipart/alternative; boundary="Apple-Mail-CE3328A7-F129-4F33-8BF1-07755F114A91"
Content-Transfer-Encoding: 7bit
From: Keith Moore <moore@network-heretics.com>
Mime-Version: 1.0 (1.0)
Message-Id: <92D1347D-9993-41F8-902B-0C9EDC79AD7D@network-heretics.com>
Date: Sat, 04 Jan 2020 02:26:53 -0500
To: ietf-smtp@ietf.org
X-Mailer: iPhone Mail (17C54)
Archived-At: <https://mailarchive.ietf.org/arch/msg/ietf-smtp/VZb94K2TmmQckyasR7HmaM99J1Y>
Subject: [ietf-smtp] Characteristics of Isolated (or mostly-isolated) industrial IP Networks
X-BeenThere: ietf-smtp@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "Discussion of issues related to Simple Mail Transfer Protocol \(SMTP\) \[RFC 821, RFC 2821, RFC 5321\]" <ietf-smtp.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/ietf-smtp>, <mailto:ietf-smtp-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/ietf-smtp/>
List-Post: <mailto:ietf-smtp@ietf.org>
List-Help: <mailto:ietf-smtp-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/ietf-smtp>, <mailto:ietf-smtp-request@ietf.org?subject=subscribe>
X-List-Received-Date: Sat, 04 Jan 2020 07:27:00 -0000


This is an attempt to summarize my observations about (mostly-)isolated networks, and also about some dubious assumptions that I've seen some equipment developers make about security requirements on such networks.

Most of these observations and assumptions aren't at all specific to email, but several of them might inform email recommendations.

Sorry for material duplicated from other posts; I thought it useful to try to collect everything in one place.

1. Scope

I'm only considering IP networks that are disconnected or mostly disconnected, and devices on such networks which use some IP protocols, including IETF application layer protocols.   While these networks may have no external IP connectivity at all, they may indirectly interact with hosts on private IP networks and/or the public Internet, through proxies or application-level gateways.

What I've seen is mostly (broadly speaking) "industrial" networks, used in laboratories, manufacturing, energy production, and energy delivery.   Some of these networks are used to monitor and occasionally control production equipment in factories, some used to monitor and/or control handling of hazardous materials, some used in critical infrastructure, some where explosive materials are present with consequent risks to human lives.   I've also worked with production networks In non-industrial settings that have similar characteristics to networks described here.

Even equipment used only for "monitoring" may still present risks if the equipment is compromised, if the monitoring is employed in a control feedback loop, and/or because compromised monitoring equipment can be used to create diversions from attacks made elsewhere.

What I've seen is a small sample.   I can't claim that it's representative, but I've seen the same patterns in so many different contexts that I suspect they're very common.   And I understand why many of these things are the way they are.   They are generally not easy problems to fix.    I expect wide variation       given that every industry has different regulations, certification requirements, and standard practices including safety practices.   It would be worthwhile to do a comprehensive survey across a variety of industries.

My biggest concerns for these networks are: 

(a) security.   I'm not only concerned about the potential for such networks and devices hosted on such networks to be attacked via external network connections; I'm also concerned about the potential for such networks and devices to be attacked locally, e.g. by attacks over WiFi, by unauthorized devices plugged into local Ethernet, compromise of authorized devices via physical access, and compromise of authorized devices on the isolated network by devices that also connect to other networks.

(b) design choices and recommendations in Internet protocol specifications that don't take into account the particular characteristics of these networks.



2. The Networks

- Common layer 2 media are Ethernet, 802.11, and "EtherNet/IP" (perhaps the most confusing name for any protocol on Earth).    There are a few other variants of Ethernet media but adapted for industrial use, for instance, using a different kind of media access control than collision detection.   There are "industrial wireless" networks that mimic Ethernet but over radio.    I've seen some interest in using IP over mesh networks, but haven't personally seen this deployed.

- Generally they are running IPv4 using RFC1918 address space.   I have seen a tiny amount of interest in IPv6.   I suspect most operators of such networks will prefer IPv4 out of familiarity and because IPv4 addresses are easier to transcribe.   Given that the networks are usually isolated at the IP layer anyway, use of IPv4 with RFC 1918 addresses doesn't present much of a problem.

- Static IP address assignment is common.   DHCP might not be available.

- Often there is no external IP connectivity, though some of the same equipment used on such isolated networks will also be used on other networks that do have external IP connectivity.   (I mean this in several ways: one is that the same design and implementation of a device may be expected to function well on both an isolated network and a more conventional network; another is that the same device may be connected to conventional and otherwise-isolated networks at the same time via multiple network interfaces; another is that it's common for portable devices such as phones and laptops to migrate from one environment to another and connect to such isolated networks when needed (even if this violates safety or security regulations).   Sometimes there is intermittent IP connectivity, say, used for firmware update.

- DNS may not be supported or even wanted.   I've been told that reliance on DNS is a risk in a manufacturing environment that provides significant potential for costly disruption with no benefit.   There may be no name lookup mechanism at all, not even host files.   (I don't know how much mDNS is used, but suspect that the same considerations apply; in my limited experience it wasn't reliable enough to depend on.)   I have seen a few devices that supported NetBios lookup so that PCs could find them by name.

- There may be some interaction with external networks via proxies or application-level gateways.   (sometimes these are called "firewalls" even if they never forward IP packets.)

- Zero threat analysis and handwaving security arguments seem to be the norm.    I should not give specific examples.

- Generally, these networks are not managed or operated by networking professionals.   Sometimes there are defensible reasons for that.   Sometimes the network is just a few hosts on WiFi or Ethernet, some of them are in very remote areas,  some are very disruption-sensitive.  



3. Devices typically used on such networks

- Most interaction on such networks is between "appliance" devices that primarily aren't designed to interact with humans.   They may have no human input or output devices at all (e.g. no keyboard, no display) or support only very limited interactions with humans (a few buttons, a numeric keypad, maybe a tiny display or touchscreen).    That's why they don't need DNS; the devices generally communicate mostly with other devices that are pre-configured and there might not even be a way to type letters in.    There's some pressure to keep configuration settings to a bare minimum, not only because of input difficulties but also because of customer support issues.

- I've seen many such devices with configuration interfaces and status monitoring based on HTTP, ssh, bare tcp/telnet (sometimes with no authentication at all), or occasionally rs232.

- I've seen several such devices that were configurable to send email reports or notices of exceptional conditions.

- It seems to be common to bring a laptop into such an environment as needed to configure devices, without dedicating a computer for that purpose.  Of course PCs still need firmware updates which are most easily accomplished by connecting them to the public internet.

- I've seen wide variation in the ways that such devices can have their firmware updated, if such capability even exists.   Sometimes you can insert a USB stick, sometimes you replace the device's SD card, sometimes you can push a firmware update to the device from a web browser.   Not only can it usually not be automated, an update can be a cumbersome process and each device is different.

- It's increasingly common to have such devices be based on Linux or similar open source platforms (much as we've seen for internet appliances used in other environments), but without much thought given to physical security, secure firmware update, supply chain issues, back doors, removable media, JTAG probes, etc.   In my experience it's often trivially easy to alter such a device's functionality, often with no physical modification needed.

- I've seen far too many devices for which every serial number had the same root or "back door" password.  Even if the device doesn't provide a shell login from a network interface, there may still be a serial "console" interface.

- The problem of configuring and updating such devices, coupled with the problem of managing passwords in such an environment (these passwords may rarely need to be used by humans), is such that devices may be (re)configurable or updated with no credentials at all - just using the app provided by the vendor.



4. Implications

These networks remind me of IP networks from the 1980s.    For instance, password sniffing is often trivial because either no encryption is used, or the encryption is TLS but with self-signed certificates.   (There's an increasing bias against use of self-signed certificates in applications, which makes sense on the public Internet and most private IP networks.   But the same browsers are used on these isolated networks as are used on the public Internet.)

TLS, as normally used, doesn't work terribly well in such environments because they don't use host names and there's no good way to rotate keys or update trusted certificates. 

Even HTTP digest authentication doesn't work very well because the protocol specification doesn't really support all the features needed (how do you logoff in a way that works for all browsers?  how do you timeout a login session?) and the user interface is so poor. 

There's a widespread assumption that connected devices need on-demand firmware updates  in order to minimize the threat from zero-day exploits, but firmware update is especially difficult in these environments.

Since DNS is not generally available and devices are generally not assigned DNS names, and it can be difficult even to explicitly configure some of them,  it's not reasonable to expect these devices to use DNS names, say, in email.