[bess] Re: [EXTERNAL] [RTG-DIR]Rtgdir early review of draft-ietf-bess-evpn-bfd-07

Alexander Vainshtein <Alexander.Vainshtein@rbbn.com> Tue, 17 September 2024 12:29 UTC

Return-Path: <alexander.vainshtein@rbbn.com>
X-Original-To: bess@ietfa.amsl.com
Delivered-To: bess@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 5B48BC14F699 for <bess@ietfa.amsl.com>; Tue, 17 Sep 2024 05:29:05 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.104
X-Spam-Level:
X-Spam-Status: No, score=-2.104 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=unavailable autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (1024-bit key) header.d=rbbn.com
Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ncdj4EPFth7L for <bess@ietfa.amsl.com>; Tue, 17 Sep 2024 05:29:01 -0700 (PDT)
Received: from usb-smtp-delivery-110.mimecast.com (usb-smtp-delivery-110.mimecast.com [170.10.151.110]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-256) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 3D5B7C14F694 for <bess@ietf.org>; Tue, 17 Sep 2024 05:29:01 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rbbn.com; s=mimecast20230413; t=1726576140; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=3C3Zntw2xuqEOTtXf2pWIxH3dHH9EGSuMF5MEgGt7S4=; b=FioXCVoaoVN/PY7ueCGGLRrujTrQte3L2ypBHzi8M1CBaqOoDIREFWbBdHBkCZ0U+DaApU 8LJ9t7w/vBH0w4O9Vvq1OFDCgwESXDd3Fvb2r5BdMyhurFrh9ShjyLD1zCqplbP2F+x/H6 3qv1NSR7mLyEK+nuHkVYxc4yw8yWJUY=
Received: from CY4PR05CU001.outbound.protection.outlook.com (mail-westcentralusazlp17010002.outbound.protection.outlook.com [40.93.6.2]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id usb-mta-41-fmMWVeFvNauHqCN6ywrIMw-2; Tue, 17 Sep 2024 05:27:28 -0700
X-MC-Unique: fmMWVeFvNauHqCN6ywrIMw-2
Received: from PH0PR03MB6300.namprd03.prod.outlook.com (2603:10b6:510:e2::5) by PH7PR03MB7268.namprd03.prod.outlook.com (2603:10b6:510:24f::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7962.24; Tue, 17 Sep 2024 12:27:25 +0000
Received: from PH0PR03MB6300.namprd03.prod.outlook.com ([fe80::a48b:db16:775a:4a16]) by PH0PR03MB6300.namprd03.prod.outlook.com ([fe80::a48b:db16:775a:4a16%7]) with mapi id 15.20.7962.022; Tue, 17 Sep 2024 12:27:24 +0000
From: Alexander Vainshtein <Alexander.Vainshtein@rbbn.com>
To: Mohamed Boucadair <mohamed.boucadair@orange.com>
Thread-Topic: [EXTERNAL] [RTG-DIR]Rtgdir early review of draft-ietf-bess-evpn-bfd-07
Thread-Index: AQHbCEi/EEq0r4/e406Fp8TnASv977Jb24Aw
Date: Tue, 17 Sep 2024 12:27:24 +0000
Message-ID: <PH0PR03MB63000A6128F35CBE1273C452F6612@PH0PR03MB6300.namprd03.prod.outlook.com>
References: <172649857459.4021334.16064172944993408610@dt-datatracker-68b7b78cf9-q8rsp>
In-Reply-To: <172649857459.4021334.16064172944993408610@dt-datatracker-68b7b78cf9-q8rsp>
Accept-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-ms-publictraffictype: Email
x-ms-traffictypediagnostic: PH0PR03MB6300:EE_|PH7PR03MB7268:EE_
x-ms-office365-filtering-correlation-id: 6d7344b9-dce3-40a7-1435-08dcd7141603
x-ms-exchange-senderadcheck: 1
x-ms-exchange-antispam-relay: 0
x-microsoft-antispam: BCL:0;ARA:13230040|376014|366016|1800799024|38070700018
x-microsoft-antispam-message-info: RDXflGDgeZvOTVGFXg745L95IYakQ1PRcDCyAbpr9emoa7TqallMy1C59VbDbovo9ZTr3n5XSRtIRaERKppoXtYkQyHYBNg1VhBdq10uDododqxFJ3ZJxs9aMd0cLAt4VaC8qj976iADpFd9j5zQLvSXPMCdZ1xRbzrL/GFQ0NdExllq45kCJhvV5+W7O6eQ2346Bph7F/rqwsHSKEokmQhO8rp+bpKVK4OYWxcOz0FbhkaBkwn7rvh+xqKmmZw+1Ks67qkoATF9Q3dr9X+iQrQ87hJnZ291pW8nwXWBUKaKnF7uE1PjhDWu8maFgXG7vUIOTnpv/Of1+ccJpKVgPUQgPGexDH9iSFJ3+UdLkMNykOaf8KFiqUGLx+rNVHZ8Mgp1s866AnRnCDkF08nJfT6E4cyHlBGWzfRYsXsS1fugFm7PMRpKuMzo2g6DwEMWYbfWomDiJNE8GU8Tq/iuMMKdfc8VqVIK3ClcJ7rYXvr73CGSH+DCJF4V/f9p66fNsgxxVTnsdikZFwE9lg4TZarwrYznmtmoPF3oXUQIyGn/75DoSsua4VMhC9YaFBMxS6Kc1sJpDuWr1XkQjF6DjxKoAGaEHf9owKq3vQ6+lncx842HRcT+BjirUXenugMalOmNN+eaoOJqmc2R+DHSzXBO6TCwrH6ydA0HOVw9rZhDLOuadBLFjVoPpjYMqAdGmOxEaPi2Gu11iiC2590Cfwhlng0v7zBv6guudIBgz00p+x3WGSTCG3EDdaRmDc6a/qVb3gcZ+3DIGjC158kb7kTHCEC9QoNwLL34X1syr8hW9QM9y/gJU7s66MzpNDa64+NFFRtQlTMNT/hudRF4GTyIkpHJjeU+CBlpAu2FOaySZcQ6sza6uTv9lNwhTXVB1DQ9eLLLsHY+c/Pf09U7mHUh0vzchM52lX1nwSs5nc1M7E/QXJHSQJl2bG4dqt86cyXzn0f/fSjqD7UhHgUb/tZrWvRH2cyU0a/VpVlQtpYW9rMaZxLd6Uf1cj74WG2Q79SE9h/Hj03yQYo+tvmAxsNV4d48WzVgu2Kvo5aLuLg3eTn1gbOZdkPb0/Hyvzwu1x9b3Wdf8HvzgSY32rBWM50PEOjQGqr544391yFkVqW9NO5MMpy+CP/1x0kaJIM77MM58YKmod2+Ba29j6IYX2F/bOvhcegTWqeoxd2tef9Xp88W24aaizJnImh9s++O6BBg2bdOivXQagGjDSMLib/66CmkBlQcii6u1wHBq/PrJuPSbn6Rx3VWHHpztiUffK43lRADbRtbFQSmHj5OD+zHkeUjrd6P52vLPiUnYh1Drh88/frRPdb3Cpn6ZiLIusWugCz56xVKVUS1z32Zhv9/VqbczP+UsUfbCEOBquM=
x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:PH0PR03MB6300.namprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(366016)(1800799024)(38070700018);DIR:OUT;SFP:1102
x-ms-exchange-antispam-messagedata-chunkcount: 1
x-ms-exchange-antispam-messagedata-0: Q77PcNZHCaUxDUSR/fVy93BtJkLm4xcwRtC5vElOTPkifJ7nKgwq2/qICLSwDQEFTmTWpPnNEeiJN5FRRp4ERAu2YBIEysfjCi6rFNQt44w4Px1iTg9U07IqbX2g7ApJ85p9GhSbGb1IbPVeP47sfJLz91q5APsjHsEIkR0Gd14mqMP/jqKmnkIsXtQNAteT0kbcKlmf93Qfy897QvlhlptN93DgOr6q2S7tUV1I20mZ4/+RTmUJkTSM4s9542LTaNFi8LjCH7+kHHrHz4VJ6L73+vgwautV+5lWZvM49SN4u7NWputkHCjmmyN5pm2YVVLPfMSRsdZzEYVohWy8FxqlDfv5H+cmU4SXoV5hF/dygBNnXz0sCk3O7LPUHPU8i2BfcLHmHibaHwTBTS9OZzzBan9+0ghlFWh/OzhakBX2zqnIFtzsUbuiki74w5BdbEXkIm3K72WFSlAkGZ8Gbkj+/LFrAss7b9aGa+O9O+N+kTQPrjHmbrnMb1NFXnfrnGDntHuzpt3Ym6Gtvy35EuOwIrUy0WvoxlnmhgLP1T3FBaUvxtPf6eV2Qw+g69dwNL6P6fBnsXHtxpIheH3HsVIljLXIBYxRCQz/Z2pGn1ZEoVd9DMt+43c+enCXnVyHehtH2CCYntbat/8w/PYExo8i3BdUmKO3gHBgFkt6FG6AqjZb3DOpUtlJGpMNxgu/hifWi0zoSarUKM969v6xbij9jUnDvGAHJHl/7K5tY4TxWOnKV/saW+7kg8ynm7L7xDJFizwtXLtmiHheftP/DeVAfGCCWTN7ubsTeZsm3X4DYDKkpXZJtu6RU8p9BVqr9JhX0GuDToS0Gl+CHLY4t2XllgFRyWJO8Ha4yT9b422vCuFImHzlWp2fCluZw2sx+Y21C6B6mEcIBAQAw1jU4qfn9AafFJBS6rwyPCHnk55Va8WjkGLsb5EN+KR6gtHL9FDwWiKpFrH9oJv+s677g6K+NgZ8RWUZy6YDzHUL6c3Tg7l3yMLdKcdgNgglEs+SPKaFwErgmAo4mA15Dc/dS2KMu2pAx4f8eZVuTPisp0sGmAZMhMrL02Cu93nBvPfe692X4Bw25xt7Lqt5M/jiQET/ajKUPAp2+PSNUO0KO8RB6eSt5hKGSyCr057Z5WbwvW9i9n8lZ8hPYAqwqOMJen+liOoNojHH0UG9uS579xaDac5WsvKUxTlSSjXwK9WMuB4ccV6mbBqIKcpHIiT+4mIj9oWCLaCF3dOvEx1Va741B58U+HuYV1x6BPnL8jJk/8u25UbbpjnDeZD3CFh1CqSJM3rC7dLCVx+fx+zqe+u1bCVzSfx3/VUtCEDGD644/Eg8jDOnF0DbbaanKJFxo57U25pa4jPJlqjc3b5dHa8Fz8iFnxCxpkaVjLhNyOBeonfWK4fJFiQKBqlaePjnss3t6Zwy/Jp1VZpb1/T1HTh1H1ZRUBU4+R3YYV1sV4f9C00PYOhSf4CEEIoqQIn5IbtIGG/jRqMPs8mR9fWQK3nEh3vLLRUpBmoJYeroxXJJ95+xtR1aEz7YvzHer/DVcNEHVsJSo3L2tBhK0X1cmhZ/+cLKHZkqhc7BnIadUIOJ
MIME-Version: 1.0
X-OriginatorOrg: rbbn.com
X-MS-Exchange-CrossTenant-AuthAs: Internal
X-MS-Exchange-CrossTenant-AuthSource: PH0PR03MB6300.namprd03.prod.outlook.com
X-MS-Exchange-CrossTenant-Network-Message-Id: 6d7344b9-dce3-40a7-1435-08dcd7141603
X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Sep 2024 12:27:24.8316 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: 29a671dc-ed7e-4a54-b1e5-8da1eb495dc3
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-CrossTenant-userprincipalname: jhOMsQVDuEDUFMMBfUMEd8Xe004CwgW8XqN6SiH5DJhakMQeoe9HsA07kiirJbT+GtkqSV4FefkJNqm6A+gvlA==
X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR03MB7268
X-Mimecast-Spam-Score: 0
X-Mimecast-Originator: rbbn.com
Content-Language: en-US
Content-Type: multipart/alternative; boundary="_000_PH0PR03MB63000A6128F35CBE1273C452F6612PH0PR03MB6300namp_"
Message-ID-Hash: NY3DBYL3VJQTS4ZONEPU5XGW6RMQLEVO
X-Message-ID-Hash: NY3DBYL3VJQTS4ZONEPU5XGW6RMQLEVO
X-MailFrom: alexander.vainshtein@rbbn.com
X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-bess.ietf.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header
CC: "bess@ietf.org" <bess@ietf.org>, "draft-ietf-bess-evpn-bfd.all@ietf.org" <draft-ietf-bess-evpn-bfd.all@ietf.org>, "rtg-dir@ietf.org" <rtg-dir@ietf.org>
X-Mailman-Version: 3.3.9rc4
Precedence: list
Subject: [bess] Re: [EXTERNAL] [RTG-DIR]Rtgdir early review of draft-ietf-bess-evpn-bfd-07
List-Id: BGP-Enabled ServiceS working group discussion list <bess.ietf.org>
Archived-At: <https://mailarchive.ietf.org/arch/msg/bess/eDBGJ5HeLztjPeInYLAe3uebZkw>
List-Archive: <https://mailarchive.ietf.org/arch/browse/bess>
List-Help: <mailto:bess-request@ietf.org?subject=help>
List-Owner: <mailto:bess-owner@ietf.org>
List-Post: <mailto:bess@ietf.org>
List-Subscribe: <mailto:bess-join@ietf.org>
List-Unsubscribe: <mailto:bess-leave@ietf.org>

Mohammed,
Lots of thanks for the review.

I have posted my concerns about the draft in question<https://mailarchive.ietf.org/arch/msg/bess/lXi2BB6Fn95UW3bbJc0tL1mgiKQ/> some time ago, and they are mainly orthogonal to the issues you raise.

However, there is one important point that you are raising and that overlaps, to some extent, with some of my comments.

You have written that you have failed finding “EVPN network layer” in 7432 or 7432bis, and your guess is that the authors may refer to the definition in Section 2.1 of RFC 9062.

But I think that the real question here should be whether EVPN network layer exists at all, and, if yes, whether it could be monitored using BFD.

Quoting from Section 9.2.1 of RFC 7432 (the relevant text is highlighted):

   A PE may advertise the same single EVPN label for all MAC addresses
   in a given MAC-VRF.  This label assignment is referred to as a per
   MAC-VRF label assignment.  Alternatively, a PE may advertise a unique
   EVPN label per <MAC-VRF, Ethernet tag> combination.  This label
   assignment is referred to as a per <MAC-VRF, Ethernet tag> label
   assignment.  As a third option, a PE may advertise a unique EVPN
   label per <ESI, Ethernet tag> combination.  This label assignment is
   referred to as a per <ESI, Ethernet tag> label assignment.  As a
   fourth option, a PE may advertise a unique EVPN label per MAC
   address.  This label assignment is referred to as a per MAC label
   assignment.  All of these label assignment methods have their
   trade-offs.  The choice of a particular label assignment methodology
   is purely local to the PE that originates the route.

This is definition is re-phrased (without any change in the semantics)  in Section 9.2.1 of 7432bis<https://datatracker.ietf.org/doc/html/draft-ietf-bess-rfc7432bis-10#section-9.2.1> as following:

The choice of a particular label assignment methodology is purely local to the PE that originates the route :¶<https://datatracker.ietf.org/doc/html/draft-ietf-bess-rfc7432bis-10#section-9.2.1-8>
·       A PE may advertise the same single EVPN label for all MAC addresses in a given MAC-VRF. This label assignment is referred to as a per MAC-VRF label assignment.
·       Alternatively, a PE may advertise a unique EVPN label per <MAC-VRF, Ethernet tag> combination. This label assignment is referred to as a per <MAC-VRF, Ethernet tag> label assignment.
·       As a third option, a PE may advertise a unique EVPN label per <ESI, Ethernet tag> combination. This label assignment is referred to as a per <ESI, Ethernet tag> label assignment.
·       As a fourth option, a PE may advertise a unique EVPN label per MAC address. This label assignment is referred to as a per MAC label assignment.
All of these label assignment methods have their trade‑offs. An assignment per MAC-VRF label requires the least number of EVPN labels but requires a MAC lookup in addition to an MPLS lookup on an egress PE for forwarding. On the other hand, a unique label per <ESI, Ethernet tag> or a unique label per MAC allows an egress PE to forward a packet that it receives from another PE to the connected CE, after looking up only the MPLS labels without having to perform a MAC lookup. This includes the capability to perform appropriate VLAN ID translation on egress to the CE.

In both cases 4 (four) different options for allocating labels carried in the Label1 field of the NLRI of EVPN Type 2 routes are listed, and 7432bis explains that each of these options has its own trade-offs.

At the same time, Section 2.3 EVPN Network Layer OAM” of RFC 9062 says:
EVPN Network OAM is visible to the PE nodes only. This OAM layer is analogous to Virtual Circuit Connectivity Verification (VCCV) [RFC5085<https://datatracker.ietf.org/doc/html/rfc5085>] in the case of VPLS/VPWS. It provides mechanisms to check the correct operation of the data plane as well as a mechanism to verify the data plane against the control plane. This includes the ability to perform fault detection and diagnostics on:¶<https://datatracker.ietf.org/doc/html/rfc9062#section-2.3-1>
·       the MP2P tunnels used for the transport of unicast traffic between PEs. EVPN allows for three different models of unicast label assignment: label per EVI, label per <ESI, Ethernet Tag>, and label per MAC address. In all three models, the label is bound to an EVPN Unicast Forwarding Equivalence Class (FEC). EVPN Network OAM MUST provide mechanisms to check the operation of the data plane and verify that operation against the control plane view.

This text is slightly inconsistent with the text in 7432/7432bis (one of the four options of the latter is missing in the former). But in any case, the “EVPN network layer” in the specific PE may be associated not just with a specific MAC-VRF (or with a specific BD within a MAC-VRF) but with a specific NAC-VRF, locally attached Ethernet Segment} pair or even with a specific <MAC-VRF, locally learned MAC address> pair.

And this raises a question about the number of EVPN BFD sessions that could be required to monitor such EVPN Network layer.

Hope these notes will be useful.

Regards,
Sasha

From: Mohamed Boucadair via Datatracker <noreply@ietf.org>
Sent: Monday, September 16, 2024 5:56 PM
To: rtg-dir@ietf.org
Cc: bess@ietf.org; draft-ietf-bess-evpn-bfd.all@ietf.org
Subject: [EXTERNAL] [RTG-DIR]Rtgdir early review of draft-ietf-bess-evpn-bfd-07

Reviewer: Mohamed Boucadair
Review result: Has Issues

Hi authors,

Thanks for the effort put into this document.

Overall, the document reads well. The specification leverages existing
specifications with exceptions called out it in the document. This approach
seems reasonable, but there are some issues that need to be fixed. These are
highlighted in the detailed review (see below). A subset of them are
highlighted hereafter:

# Better position the document: For example, I failed to find this "network
layer" defined in RFC7432 or 7432bis. I think that you are referring to the
layering in 2.1 of 9062. For example, you can consider adding a sentence in the
introduction about 2.1 of 9062 to position the layer you are considering.

# 7432 or 7432bis: Any reason why the bis is cited explicitly here? Are there
parts of the spec that are not applicable to 7432? I don't think so, but it is
better clarify this in the doc rather than leaving the readers guess.

# "future versions of this document" vs "other documents": The document says in
several places that "It is intended to address this in future versions of this
document.". The intended scope should be clarified.

# Internal inconsistency:

## The document refers to TBD3 and cites Section 8, but there is no such
definition in the IANA section ## The document cites "dedicated unicast MAC"
and "dedicated multicast MAC" but these are not defined in the document.

## RFC 9026

Previous sections state that 9026 is not mandatory and other mechanisms can be
used. However, Section This text seems to assume that it is always used:

"It also contains a BFD Discriminator
Attribute [RFC9026] with BFD Mode TDB4 giving the BFD discriminator
that will be used by the tail.
"

(note that s/TDB4/TBD2)

# Redundant requirements: For example, the document says

" The mechanisms specified in BFD for MPLS LSPs [RFC5884] [RFC7726] and
BFD for VXLAN [RFC8971] are, except as otherwise provided herein,
applied to test loss of continuity for unicast EVPN traffic.
"
but then

" Once the BFD session is UP, the ends of the BFD session MUST NOT
change the local discriminator values of the BFD Control packets they
generate, unless they first bring down the session as specified in
[RFC5884].
"

the intended behavior vs "local discriminator values" here is redundant with
this part in Section 7 of 5884:

"Note that once the BFD session for the MPLS LSP is UP, either end of the BFD
session MUST NOT change the source IP address and the local discriminator
values of the BFD Control packets it generates, unless it first brings down the
session."

No?

# Detailed review can be found here, fwiw:

* pdf:
https://github.com/boucadair/IETF-Drafts-Reviews/blob/master/2024/draft-ietf-bess-evpn-bfd-07-rev%20Med.pdf<https://github.com/boucadair/IETF-Drafts-Reviews/blob/master/2024/draft-ietf-bess-evpn-bfd-07-rev%20Med.pdf>
* doc:
https://github.com/boucadair/IETF-Drafts-Reviews/blob/master/2024/draft-ietf-bess-evpn-bfd-07-rev%20Med.doc<https://github.com/boucadair/IETF-Drafts-Reviews/blob/master/2024/draft-ietf-bess-evpn-bfd-07-rev%20Med.doc>

Feel free to grab whatever you think useful.

Hope this helps.

Cheers,
Med

Disclaimer

This e-mail together with any attachments may contain information of Ribbon Communications Inc. and its Affiliates that is confidential and/or proprietary for the sole use of the intended recipient. Any review, disclosure, reliance or distribution by others or forwarding without express permission is strictly prohibited. If you are not the intended recipient, please notify the sender immediately and then delete all copies, including any attachments.