Re: [tcpm] Seeking WG opinions on ACKing ACKs with good cause (was: Possible error in accurate-ecn)

Martin Duke <martin.h.duke@gmail.com> Tue, 16 March 2021 19:04 UTC

Return-Path: <martin.h.duke@gmail.com>
X-Original-To: tcpm@ietfa.amsl.com
Delivered-To: tcpm@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 9AAB93A0890 for <tcpm@ietfa.amsl.com>; Tue, 16 Mar 2021 12:04:15 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -2.097
X-Spam-Level:
X-Spam-Status: No, score=-2.097 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=ham autolearn_force=no
Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NuDyJUWTv3wl for <tcpm@ietfa.amsl.com>; Tue, 16 Mar 2021 12:04:12 -0700 (PDT)
Received: from mail-il1-x135.google.com (mail-il1-x135.google.com [IPv6:2607:f8b0:4864:20::135]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id CDFEF3A0883 for <tcpm@ietf.org>; Tue, 16 Mar 2021 12:04:11 -0700 (PDT)
Received: by mail-il1-x135.google.com with SMTP id d5so13735231iln.6 for <tcpm@ietf.org>; Tue, 16 Mar 2021 12:04:11 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=DqcdACaXfUEwdIaBJ8UX+O5br6PFoFlcUAbmrmaWBvk=; b=dFGPCU7EqhXWw/n33JnwvOeV/kQN8Kzit6BBbm8qeomjuD0br+pwQ4jZIPx81ReSZa Q39XtpOZ5L/g4wWGUjumug9stwoqK8TjSQQmRPlhm8WcHIMWzp+26/bZgWrTLQNdP/uB Z9znKk7qc7/q0uBDcODQBjsYWMruBTsu7yJ9ZbttEj2/YAeVGz+WVLH78DWhF2rc3F+F n2dcKUuh8jFlckXYJNWnuv1mH7hBh+X+Ws6b9xm1eKho4gmTOAqth8KN2lKNf48z9Nw0 ZtLlvsRhq1ZU/dkauX91wQco34dGSYr4t2gGcqbWZelWTmTD4GHlR1HoJFnKJ2YeDsyp 6pHQ==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=DqcdACaXfUEwdIaBJ8UX+O5br6PFoFlcUAbmrmaWBvk=; b=ezhJ7yFFGiEg9ekEb9FxjzCzklKWG7JKWsFYh5TFjcmGOLd8ocVZY7pIMrocJ9EOFa Sz95cLSgZDu+u2jwxJRpC8aWJ10S4e5uHvGV/8iHiVM04jce1RN2RcEX3nyKmAZ/9Pfd drcqaQDbLqnofB8Tk3BhE1hUG/lOerBafZsrow9uDtVDRURNMCyAOozwkpDnvMBj3L8g tjzV3HqPJTlNY8+mjkdMYU/iNRvbaE0vMx5Je80wPWWcUNhSbmzKsHMq3khFn3VYWn5Z fO4EPWflxmfuNOFDwQbRkJt+5XegUMdA7vTs/zLMpoWRx+CIt1vNmlsd9gBd+Fv0HRMi 2vIA==
X-Gm-Message-State: AOAM531rkQjbSMwcqJnTXOIoeKOp+QTiU8EilFupORFhvSPgUTbSnkZX HfexdhYv7eBew1ZHO7lru0K7ipjKKVv5wnCuuUs=
X-Google-Smtp-Source: ABdhPJz7eDFesMaif2CUK+lQeYfYacTKmPNiy47CGb7FbWovtvThKM89ichY5uBhCFG7teWDRw+0Mcoum+rE5geafvg=
X-Received: by 2002:a92:c709:: with SMTP id a9mr4837888ilp.272.1615921451015; Tue, 16 Mar 2021 12:04:11 -0700 (PDT)
MIME-Version: 1.0
References: <47df9b8b-515e-d40d-3473-599b0a3e3876@bobbriscoe.net> <6031BE2B-4D33-426F-BA17-DDF15CF821DE@kuehlewind.net> <060c8bd8-d64b-3e46-7874-742e35e6d114@bobbriscoe.net> <221e58f3-ada0-c880-db72-d98af84fedb8@gmx.at> <bd6ab65d-ccd5-9fa9-58be-6d9fea4af870@bobbriscoe.net> <CAAK044QgF4pz5Wamnxkobthou5ac4_LBxh8=nBYWyOxQUtcW-Q@mail.gmail.com> <8151fdef-ae78-80f3-adfc-d40db878ac8e@gmx.at> <CAAK044RhdAYexcGRj_XDkdY_o6JqB0DDo1X0H2AeFkRcsb0i4A@mail.gmail.com> <48c5910d-5340-acd6-8fd9-fff1b7758310@bobbriscoe.net> <CAM4esxTiw7_es60DDK2E1wa3-c1nUD2W_Rf7Fhw5u0qJ0bFQpg@mail.gmail.com> <8275e3ff-24af-7f0c-d251-867673503741@gmx.at> <631a3f3d-52e1-68c0-8b5f-ce41b30d2e7f@bobbriscoe.net> <48BF06D1-F362-450F-9F89-6DAC6E96B1AC@kuehlewind.net>
In-Reply-To: <48BF06D1-F362-450F-9F89-6DAC6E96B1AC@kuehlewind.net>
From: Martin Duke <martin.h.duke@gmail.com>
Date: Tue, 16 Mar 2021 12:04:00 -0700
Message-ID: <CAM4esxRc-rsh=M=v-brgy+Z+8wnL9x6gXiteFy7W+tV2uOetkg@mail.gmail.com>
To: Mirja Kuehlewind <ietf@kuehlewind.net>
Cc: Bob Briscoe <ietf@bobbriscoe.net>, "Scheffenegger, Richard" <rs.ietf@gmx.at>, tcpm IETF list <tcpm@ietf.org>, Yoshifumi Nishada <nishida@sfc.wide.ad.jp>
Content-Type: multipart/alternative; boundary="00000000000012a3d205bdac0804"
Archived-At: <https://mailarchive.ietf.org/arch/msg/tcpm/-URfmwc5hemnHJCwztYfE8o1cbk>
Subject: Re: [tcpm] Seeking WG opinions on ACKing ACKs with good cause (was: Possible error in accurate-ecn)
X-BeenThere: tcpm@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: TCP Maintenance and Minor Extensions Working Group <tcpm.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/tcpm>, <mailto:tcpm-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/tcpm/>
List-Post: <mailto:tcpm@ietf.org>
List-Help: <mailto:tcpm-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/tcpm>, <mailto:tcpm-request@ietf.org?subject=subscribe>
X-List-Received-Date: Tue, 16 Mar 2021 19:04:16 -0000

Mirja,

I agree there's little harm with classic ECN -- other than the spurious
retransmission, any CE signal in an RTT or a Fast Retransmit will have a
similar effect on cwnd.

But for L4S, 3 CE marks are likely to have a quite different impact than a
packet loss.

This case can occur when data is outstanding, which is why the 5681
language is useful in mitigating it.

As I said in my first email, these differences are all quite small IMO.

On Tue, Mar 16, 2021 at 9:50 AM Mirja Kuehlewind <ietf@kuehlewind.net>
wrote:

> Hi all,
>
> Actually I think there is no problem if a ACK on an ACK is misinterpreted
> as dup ACK. What happens if a dupACK is detected? The cwnd is decreased and
> lost data is retransmitted. However, the cwnd was either already decreased
> with the first ACK and would not be decreased again for the next RTT (in
> case of Reno-like cc), or should be decreased anyway because the ACK
> carries a new CE feedback. I guess we could note that one should not
> decrease twice because if the information in the same ACK (being dup and CE
> counter). However, there will be no retransmission because there is no
> outstanding non-acked data given the received ack'ed only an ACK and no new
> data. So in summary, there is no problem here with dupACK generated in
> response to a pure CE-marked ACK that needs to be addressed in any way.
>
> Mirja
>
>
>
> > On 15. Mar 2021, at 23:56, Bob Briscoe <ietf@bobbriscoe.net> wrote:
> >
> > RIchard, Martin, 2 brief responses inline tagged [BB]...
> >
> > On 13/03/2021 10:29, Scheffenegger, Richard wrote:
> >> Hi Martin,
> >>
> >> > To me, the real complexity here is marking ACKs in the first place,
> >> > which forces you to do a bit of accounting about the number of acks
> >> > you've sent.
> >>
> >> I don't quite follow;
> >
> > [BB] I think Martin is not disagreeing with your assessment. He was just
> pointing out that the knock-on effects of ACKing pure ACKs originally stems
> from allowing pure ACKs to be ECN capable in the first place. He's not
> saying that's bad. Just it's the root cause of the new phenomenon we're
> seeing here.
> >
> >> Given that both AccECN endpoints are also using
> >> ECN++, all control packets (ACKs without any data) can be sent as ECT.
> >> In order for these packet to use the same Q as data packets, it is
> >> beneficial to the sender of those, set ECT on them even.
> >>
> >> The network is then free to set a CE mark - should the AQM deem that
> >> necessary - on these control packets.
> >>
> >> On the reflector, the CE.packet counter is simply incremented for each
> >> incoming packet with the CE codepoint - no special treatment is
> >> necessary for that aspect to work.
> >>
> >> Thus the same rules apply to CE-marked data segments, as to any control
> >> packets received.
> >>
> >> > to include "does not increase the ACE counter" as another criterion.
> >> > Thus TCP treats this as it would a pure window update, which is
> >> > functionally similar. This should solve any spurious retransmission
> >> issues.
> >>
> >> Unfortnuately, a true duplicate ack can just as well become CE-marked -
> >> and IMHO this is a more probably scenario, than the misinterpretation of
> >> a flight of CE-carrying ACKs to trigger a spurious retransmission.
> >
> > [BB] I was going to say roughly the same as Richard in response to
> Martin here.
> >
> > Just because some apparent DupACKs with an increased CE count are not
> DupACKs, does not imply that a true DupACK cannot have increase CE count.
> >
> > That's why we need other tests, like lack of SACK when negotiated, or
> timestamp evidence.
> >
> >
> > Bob
> >
> >>
> >> As Bob's slide shows, the risk only exists under certain pathological
> >> conditions:
> >>
> >> a) the window from A->B must have been at least 3*n*(1/mark probability
> >> for pure ACKs) in size
> >> b) right afterward, the data direction has to change, with B becoming
> >> the sender
> >> c) B must have no other means (like SACK, Timestamp) to differentiate,
> >> if the incoming ACKs (carrying new CE infomation, but not advancing
> >> snd.una) have been triggered by a gap/reordering in the data segements
> >> delivered to A
> >>
> >> So, we effectively talk about a non-SACK, non-Timestamp, but AccECN and
> >> ECN++ enabled flow, possibly encountering a pathological network
> >> situation (sudden high CE marking probabilty), swapping the data
> >> directions at that specific moment also.
> >>
> >> What would AccECN loose by NOT sending ACKs carrying new CE-state back:
> >>
> >> o) no timely update of the (currently quiescent) endpoint about changed
> >> network congestion state - likely to influence the cwnd before the next
> >> transmission and NOT bursting into a congested network (with all the
> >> dire effects - loss, RTT/RTO delay for loss recovery, ...)
> >> o) more complex implementation (special case for CE-marks on ACKs
> >> without data)
> >> o) Sender can not assume to get full information about the networks
> >> congestion state unless data is transmitted
> >> o) Delayed (untimely) CC reaction once data is being transmitted, if the
> >> ACE counter wrap detection triggers (or complete removal of this
> >> conservative CC functionality) [the ACK for new data will have the new
> >> CE.packet counter state, indicating that some time between the prior
> >> transmit from B, and now, there was network congestion. But no
> >> infomation when it exactly happend.
> >>
> >>
> >> Besides: As pure ACKs may be lost at any moment, the above effects can
> >> impact an AccECN sender anyway, even if the receiver is properly
> >> reflecting, in a timely manner, most up-to-date network congestion
> >> state. But by removing the (simple) Ack-for-new-CE-marks capability from
> >> the receiver, we forego any chance to improve the status quo in the
> future.
> >>
> >> If an implementer does want to make the choice of deliberately
> >> surpressing these ACKs, an AccECN sender has to deal with it anyway.
> >>
> >> However, at the "cost" of a typically very minor increase in pure ACKs
> >> sent 1/(n*1/(ACK-CE-mark probability)), AccECN can allow new CC to
> >> remain up-to-date while no data transmissions are currently happening.
> >>
> >>
> >> Richard
> >>
> >> Am 12.03.2021 um 22:37 schrieb Martin Duke:
> >>> Hi Bob,
> >>>
> >>> Although I don't think the difference between these alternatives is all
> >>> that large, I believe I would go with (B) -- allow acks of acks at a
> >>> rate less than 100%. I suppose that in some corner cases it will
> prevent
> >>> drops out of a shallow queue, which isn't a big deal, but why not do
> it?
> >>> To me, the real complexity here is marking ACKs in the first place,
> >>> which forces you to do a bit of accounting about the number of acks
> >>> you've sent.
> >>>
> >>> It would be good to also add language to extend the RFC5681 definition
> >>> of "Duplicate ACK"
> >>>
> >>> "
> >>>
> >>>   DUPLICATE ACKNOWLEDGMENT: An acknowledgment is considered a
> >>>        "duplicate" in the following algorithms when (a) the receiver of
> >>>        the ACK has outstanding data, (b) the incoming acknowledgment
> >>>        carries no data, (c) the SYN and FIN bits are both off, (d) the
> >>>        acknowledgment number is equal to the greatest acknowledgment
> >>>        received on the given connection (TCP.UNA from [RFC793 <
> https://www.rfc-editor.org/rfc/rfc793>] and (e)
> >>>        the advertised window in the incoming acknowledgment equals the
> >>>        advertised window in the last incoming acknowledgment.
> >>>
> >>>        Alternatively, a TCP that utilizes selective acknowledgments
> >>>        (SACKs) [RFC2018,RFC2883 <
> https://www.rfc-editor.org/rfc/rfc2883>] can leverage the SACK
> information to
> >>>        determine when an incoming ACK is a "duplicate" (e.g., if the
> ACK
> >>>        contains previously unknown SACK information).
> >>>
> >>> "
> >>>
> >>> to include "does not increase the ACE counter" as another criterion.
> >>> Thus TCP treats this as it would a pure window update, which is
> >>> functionally similar. This should solve any spurious retransmission
> issues.
> >>>
> >>> On Fri, Mar 12, 2021 at 2:54 AM Bob Briscoe <ietf@bobbriscoe.net
> >>> <mailto:ietf@bobbriscoe.net>> wrote:
> >>>
> >>>     Yoshi, tcpm list,
> >>>
> >>>     As promised we set up a small design team on the single issue of
> >>>     occasionally ACKing pure ACKs if they carry new info (ECN marking
> at
> >>>     the IP layer). The design team has two solutions and everyone would
> >>>     be prepared to accept either, but preferences differ. So we're
> >>>     seeking wider opinions (more opinions obviously won't narrow the
> >>>     choices, but at least the WG can then make a decision informed by
> >>>     those who care).
> >>>
> >>>     To understand the question, you can either read the email below. Or
> >>>     the 3 slides for the tcpm meeting are briefer, and include
> pictures:
> >>>
> https://datatracker.ietf.org/meeting/110/materials/slides-110-tcpm-draft-ietf-tcpm-accurate-ecn-00#page=6
> >>> <
> https://datatracker.ietf.org/meeting/110/materials/slides-110-tcpm-draft-ietf-tcpm-accurate-ecn-00#page=6
> >
> >>>
> >>>     Here's the current draft text in question (see here for context
> >>>
> https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-14#section-3.2.2.5
> >>> <
> https://tools.ietf.org/html/draft-ietf-tcpm-accurate-ecn-14#section-3.2.2.5
> >
> >>>     ):
> >>>
> >>>
> >>>                 3.2.2.5.1 Data Receiver Safety Procedures
> >>>
> >>>         An AccECN Data Receiver:
> >>>
> >>>         o  SHOULD immediately send an ACK whenever a data packet
> marked CE
> >>>            arrives after the previous packet was not CE.
> >>>
> >>>         o  MUST immediately send an ACK once 'n' CE marks have arrived
> since
> >>>            the previous ACK, where 'n' SHOULD be 2 and MUST be in the
> range 2
> >>>            to 6 inclusive.
> >>>
> >>>     Only the second bullet is in question. Here is the proposed diff
> for
> >>>     each alternative, then we explain:
> >>>
> >>>     Alternative A
> >>>
> >>>         o  MUST immediately send an ACK once 'n' CE marks have arrived
> since
> >>>     -     the previous ACK,
> >>>     +     the previous ACK and there is outstanding data to
> acknowledge,
> >>>            where 'n' SHOULD be 2 and MUST be in the range 2 to 6
> inclusive.
> >>>
> >>>
> >>>     Alternative B
> >>>
> >>>         o  MUST immediately send an ACK once 'n' CE marks have arrived
> since
> >>>     -     the previous ACK, where 'n' SHOULD be 2 and MUST be in the
> range 2
> >>>     +     the previous ACK, where 'n' SHOULD be 3 and MUST be in the
> range 3
> >>>            to 6 inclusive.
> >>>
> >>>
> >>>     Extra guidance text would be required in each case too (see the
> end).
> >>>
> >>>     Background:
> >>>     AccECN is a change to the TCP wire protocol that requires the
> packet
> >>>     count of congestion feedback to include any congestion experienced
> >>>     (CE) arriving on Pure ACKs (amongst other things). AccECN doesn't
> >>>     require Pure ACKs to be ECN-capable, but allows for them to be.
> >>>     Similarly, AccECN doesn't require any congestion response to CE on
> >>>     pure ACKs, but having the feedback information there allows a
> >>>     response to be added with a one-ended update, if desired/necessary.
> >>>     Basically the data receiver is a 'dumb reflector'.
> >>>
> >>>     The above two bullets were designed to ensure that an ACK is
> >>>     triggered a) on the first sign of congestion, and b) frequently
> >>>     enough for the count of CE markings to be fed back using the 3-bit
> >>>     ACE field before it wraps, even if an occasional pure ACK is lost.
> >>>
> >>>     We then realized that the wording could require an ACK to be
> >>>     triggered in response to a CE-marked pure ACK. The circumstance
> when
> >>>     this could occur would be when peer X sends a volley of data to Y
> >>>     then stops, and the path back from Y to X is congested (probably by
> >>>     other flow(s) so that many of the ACKs are CE-marked. The second
> >>>     bullet above under alternative (B) would require X to ACK every
> >>>     'n-th' CE-marked pure ACK. However, if Y immediately started
> sending
> >>>     a volley of data to X, Y could misinterpret those ACKs (of ACKs)
> >>>     from X as DupACKs.
> >>>
> >>>     There are two ways to deal with this:
> >>>     A) Some of us prefer to completely prevent ACKs on pure ACKs, on
> the
> >>>     basis that they do not want to risk sometimes generating more ACKs
> today
> >>>     B) Others want to ensure that these rules will cause pure ACKs to
> be
> >>>     ACKed when the amount of CE on the ACKs merits it. But sparingly
> and
> >>>     strongly damping any ACK ping-pong.
> >>>
> >>>     There are complexity arguments on both sides.
> >>>
> >>>     B more complex:
> >>>          extra (non-mandatory) 'if' condition for lack of SACK options
> >>>     on a pure ACK (to decide it's not a DupACK).
> >>>     B less complex:
> >>>          consistent handling of CE marking whether on pure ACKs or data
> >>>     (which would probably remove an 'if' condition).
> >>>
> >>>     A more complex:
> >>>          CE markings on a string of pure ACKs can build up without
> >>>     feeding them back, until released by a data packet (if ever).
> >>>          More code at the other end to deal with the resulting risk of
> >>>     many wraps of the ACE field (or ignore?).
> >>>     A less complex:
> >>>          less different from current TCP.
> >>>
> >>>     Extra guidance text would be necessary in either case.
> >>>
> >>>     * Alt A) would need text on handling the risk of many ACE wraps
> >>>          (to be written).
> >>>
> >>>     * Alt B) would need something like the following changes:
> >>>
> >>>          For the avoidance of doubt, the above change-triggered ACK
> >>>     mechanism
> >>>          is deliberately worded to solely apply to data packets, and to
> >>>     ignore
> >>>          the arrival of a control packet with no payload, because it is
> >>>     -  important that TCP does not acknowledge pure ACKs.  The change-
> >>>     +  important that TCP does not acknowledge pure ACKs which convey
> no
> >>>     new
> >>>     +  state information to the sender. The change-
> >>>          triggered ACK approach can lead to some additional ACKs but it
> >>>     feeds
> >>>          back the timing and the order in which ECN marks are received
> with
> >>>          minimal additional complexity.  If only CE marks are
> >>>     infrequent, or
> >>>          there are multiple marks in a row, the additional load will be
> >>>     low.
> >>>          Other marking patterns could increase the load significantly.
> >>>     +
> >>>     +  Providing feedback on the congestion state of the return channel
> >>>     +  after a sender has ceased transmitting more data helps inform
> the
> >>>     +  clients TCP congestion controller about the state of the return
> >>>     path.
> >>>     +  Should the role of data sender and receiver subsequently
> change, the
> >>>     +  new sender has more up to date knowledge of the network state,
> >>>     +  preventing transmissions of inappropriate size at that moment.
> >>>
> >>>          Even though the first bullet is stated as a "SHOULD", it is
> >>>     important
> >>>          for a transition to immediately trigger an ACK if at all
> >>>     possible, so
> >>>          that the Data Sender can rely on change-triggered ACKs to
> detect
> >>>          queue growth as soon as possible, e.g. at the start of a flow.
> >>>     This
> >>>          requirement can only be relaxed if certain offload hardware
> needed
> >>>          for high performance cannot support change-triggered ACKs
> >>>     (although
> >>>          high performance protocols such as DCTCP already successfully
> use
> >>>          change-triggered ACKs).  One possible compromise would be for
> the
> >>>          receiver to heuristically detect whether the sender is in
> >>>     slow-start,
> >>>          then to implement change-triggered ACKs while the sender is in
> >>>     slow-
> >>>          start, and offload otherwise.
> >>>
> >>>     +   The second bullet creates a possible case where an AccECN
> >>>     implementation
> >>>     +   could sometimes ACK pure ACKs, which in turn might be mistaken
> for
> >>>     +   duplicate ACKs (in scenarios where TCP peers take turns to send
> >>>     +   sets of data packets). To prevent spurious transmissions in
> such
> >>>     +   circumstances, if SACK has been negotiated, an implementation
> could
> >>>     +   optionally assume that an ACK is not a Duplicate ACK if it has
> >>>     no SACK option,
> >>>     +   which would indicate it was an ACK of an  ACK. Alternatively it
> >>>     could use
> >>>     +   timestamp options to rule out DupACKs.
> >>>
> >>>
> >>>
> >>>
> >>>     Bob
> >>>
> >>>     --
> >>> ________________________________________________________________
> >>>     Bob Briscoehttp://bobbriscoe.net/ <http://bobbriscoe.net/>
> >>>
> >>>     _______________________________________________
> >>>     tcpm mailing list
> >>>     tcpm@ietf.org <mailto:tcpm@ietf.org>
> >>>     https://www.ietf.org/mailman/listinfo/tcpm
> >>>     <https://www.ietf.org/mailman/listinfo/tcpm>
> >>>
> >>>
> >>> _______________________________________________
> >>> tcpm mailing list
> >>> tcpm@ietf.org
> >>> https://www.ietf.org/mailman/listinfo/tcpm
> >>>
> >
> > --
> > ________________________________________________________________
> > Bob Briscoe                               http://bobbriscoe.net/
> >
> >
>
>