Re: [6lo] Benjamin Kaduk's Discuss on draft-ietf-6lo-fragment-recovery-13: (with DISCUSS and COMMENT)

["Pascal Thubert (pthubert)" <[e-mail]>]

Hello Again Benjamin,

Continuing with the COMMENT piece of your really appreciated deep review below: 

And as usual please find the diffs, and since this impacts both drafts we get:

https://www.ietf.org/rfcdiff?url2=draft-ietf-6lo-minimal-fragment-14
https://www.ietf.org/rfcdiff?url2=draft-ietf-6lo-fragment-recovery-15 
 
Please find the discussion below

> ----------------------------------------------------------------------
> COMMENT:
> ----------------------------------------------------------------------
> 
> It might be worth mentioning that whereas the RFC 4944 fragmentation
> procedures uses different dispatch types for the first fragment and subsequent
> fragments, we have no need to do so (the first fragment is identified by
> sequence number zero).

I agree with your statement; that's because with RFC 4944 the first and next fragments have a different format to save a few bits in the first fragment. 

Added the least sentence in:
"
   "LLN Minimal Fragment Forwarding" [FRAG-FWD] specifies the generic
   behavior that all FF techniques including this specification follow,
   and presents the associated caveats.  In particular, the routing
   information is fully indicated in the first fragment, which is always
   forwarded first.  With this specification, the first fragment is
   identified by a Sequence of 0 as opposed to a dispatch type in
   [RFC4944].  
"



> 
> I have a question about the "stickiness" of per-fragment ECN markings.
> That is, suppose that fragment 1 is marked as having experienced congestion
> when it arrives at the recipient.  The sender requests an ACK after fragment 4,
> and the recipient duly sets the E bit in the ack.
> Suppose that fragments 5 through 9 then arrive with no congestion indicated,
> and fragment 9 requests and ACK.  Does this new ACK still have to set the E bit
> since, as discussed in Section 5.2, "at least one of the acknowledged fragments
> was received with an Explicit Congestion Notification"?  My (very very non-
> expert) recollection is that we tend to say that any given ECN marking should
> produce at most one "congestion present" response on the return path.


I experimented with FECNs and BECNs at the times of frame relay. I found that the number of times the bit was set was unreliable, could get many in a row not really because of the receiving end point but because the congested node sets it a lot. 
I expect the 6LoWPAN network to look more like FR than IP. 
Bottom line is that it is for the most part on the shoulder of the sender and that is out of scope (clarified with Mirja's DISCUSS).

To your point I suggest adding in the next section " 4.3.  Flow Control "
"
   An intermediate node that experiences a congestion MAY set the ECN
   bit in a fragment, and the reassembling endpoint echoes the ECN
   bit at most once at the next opportunity to acknowledge back.

"
In section 6, the normative text becomes:
"

   The RFRAG Acknowledgment carries an ECN indication for flow control
   (see Appendix C).  The reassembling endpoint of a fragment with the
   'E' (ECN) flag set MUST echo that information at most once by setting
   the 'E' (ECN) flag in the next RFRAG Acknowledgment.

"

For my FR implementation I ended up applying at most one action every RTT (that was an IBM FRTE / FRSE).

> 
> Section 1
> 
>    LLNs.  One critical issue with this original design is that routing
>    an IPv6 [RFC8200] packet across a route-over mesh requires
>    reassembling the full packet at each hop, which may cause latency
>    along a path and an overall buffer bloat in the network.  The "6TiSCH
> 
> I think I encountered at least one good reference for this statement during my
> reading this week (posibly just -minimal-fragment, but maybe something
> referenced from there?); it might be nice to help with cross-linking.

The 2 next sentences are just those references : )  (see 4.6.3.  Fragment Forwarding  of https://tools.ietf.org/html/draft-ietf-6tisch-architecture-28 which may be your memory).

Changed the text to read like:

"

"Transmission of IPv6 Packets over IEEE 802.15.4 Networks" [RFC4944]
   defines the original 6LoWPAN datagram fragmentation mechanism for
   LLNs.  One critical issue with this original design is that routing
   an IPv6 [RFC8200] packet across a route-over mesh requires the
   reassembly of the packet at each hop.  The "6TiSCH Architecture"
   [I-D.ietf-6tisch-architecture] indicates that this may cause latency
   along a path and impact critical resources such as memory and
   battery; to alleviate those undesirable effects it recommends using a
   fragment forwarding (FF) technique .

   "LLN Minimal Fragment Forwarding" [FRAG-FWD] specifies the generic
   behavior that all FF techniques including this specification follow,
   and presents the associated caveats.

"

> 
>    That problem is exacerbated when forwarding fragments over multiple
>    hops since a loss at an intermediate hop will not be discovered by
>    either the source or the destination, and the source will keep on
>    sending fragments, wasting even more resources in the network and
>    possibly contributing to the condition that caused the loss to no
>    avail since the datagram cannot arrive in its entirety.  [...]
> 
> nit: I'm not sure that "to no avail" is placed properly.

Restructured as follows:

"
  That problem is exacerbated when forwarding fragments over multiple
   hops since a loss at an intermediate hop will not be discovered by
   either the source or the destination endpoints, and the source will
   keep on sending fragments, wasting even more resources in the network
   since the datagram cannot arrive in its entirety, and possibly
   contributing to the condition that caused the loss.

"


> 
> Section 2.2
> 
>    "LLN Minimal Fragment Forwarding" [I-D.ietf-6lo-minimal-fragment]
>    introduces the generic concept of a Virtual Reassembly Buffer (VRB)
> 
> It's perhaps debatable whether -minimal-fragment vs. lwig-vrb really
> "introduces" the concept -- draft-ietf-6lo-minimal-fragment-12 doesn't really
> say very much about it, to me.

True. Reassembly Buffer is an old concept, see https://tools.ietf.org/html/draft-ietf-intarea-frag-fragile-17#section-3.1 
=> s/ introduces/discusses /


> 
>    and specifies behaviours and caveats that are common to a large
>    family of FF techniques including this, which fully inherits from
>    that specification.
> 
> nit: I'd suggest rewording "including this" for clarity, perhaps "including the
> mechanism specified by this document".

Done; also a bit of restructuring, giving:

"

   In this document, readers will encounter terms and concepts that are
   discussed in "IPv6 over Low-Power Wireless Personal Area Networks
   (6LoWPANs): Overview, Assumptions, Problem Statement, and Goals"
   [RFC4919], "Transmission of IPv6 Packets over IEEE 802.15.4 Networks"
   [RFC4944], and "Problem Statement and Requirements for IPv6 over
   Low-Power Wireless Personal Area Network (6LoWPAN) Routing"
   [RFC6606].

   "LLN Minimal Fragment Forwarding" [FRAG-FWD] discusses the generic
   concept of a Virtual Reassembly Buffer (VRB) and specifies behaviors
   and caveats that are common to a large family of FF techniques
   including the mechanism specified by this document, which fully
   inherits from that specification.  It also defines terms used in this
   document: 6LoWPAN endpoints, Compressed Form, Datagram_Tag,
   Datagram_Size, and Fragment_Offset.

"


> 
> Section 2.3
> 
>    6LoWPAN endpoints:  The LLN nodes in charge of generating or
>       expanding a 6LoWPAN header from/to a full IPv6 packet.  The
>       6LoWPAN endpoints are the points where fragmentation and
>       reassembly take place.
> 
> [-minimal-fragment also defines this, and many of the other terms we define
> here, with minimal-but-nonzero variation.  Given that we already have a
> normative dependency on that document, perhaps we should pull definitions
> from it as well, instead of repeating them?  If not, I note that on -minimal-
> fragment, I commented:
> nit: I think they are only "the [only] points" where fragmentation/reassembly
> happen if the entire newtork is using fragment forwarding; in other cases some
> intermediate nodes will also reassemble and (re)fragment.]

Yes, done already for another review. 

For the nit, I realize that I missed your point at the time. 

For the purpose of FF, the intermediate nodes you mention become the fragmentation endpoints, which are the points we care about here. I need to change both specs to use "6LoWPAN FF endpoints" as opposed to "6LoWPAN endpoints".

In minimal:
"

   6LoWPAN Fragment Forwarding endpoints:  The 6FF endpoints are the
      first and last nodes in an unbroken string of 6LoWPAN fragment
      forwarding nodes.  They are also the only points where the
      fragmentation and reassembly operations take place.
"
 And later
"
   We use Figure 1 to illustrate 6LoWPAN fragmentation.  We assume node
   A forwards a packet to node B, possibly as part of a multi-hop route
   between 6LoWPAN Fragment Forwarding endpoints which may be neither A
   nor B, though 6LoWPAN may compress the IP header better when they are
   both the 6FF and the 6LoWPAN compression endpoints.

"


> 
> Section 4.1
> 
>    This specification cannot allow this operation since fragments are
>    recovered end-to-end based on a sequence number.  This means that the
> 
> I suggest s/this operation/such a refragmentation operation/.

Done

> 
>    fragments that contain a 6LoWPAN-compressed header MUST have enough
>    slack to enable a less efficient compression in the next hops that
>    still fits in one MAC frame.  For instance, if the IID of the source
> 
> (This assumes a homogeneous MAC mesh, an assumption that might be worth
> making explicit.

Well in fact the assumption is that the MIN MTU is known, since the fragmentation can adapt to that. 
4.1 becomes:

"
4.1.  Slack in the First Fragment

   [FRAG-FWD] allows for refragmenting in intermediate nodes, meaning
   that some bytes from a given fragment may be left in the VRB to be
   added to the next fragment.  The need for more space in the outgoing
   fragment than was needed for the incoming fragment arises when the
   6LoWPAN Header Compression is not as efficient on the outgoing link
   or the Link MTU is reduced.

   This specification cannot allow such a refragmentation operation
   since the fragments are recovered end-to-end based on a sequence
   number.  The Fragment_Size MUST be tailored to fit the minimal MTU
   along the path, and the first fragment that contains a 6LoWPAN-
   compressed header MUST have enough slack to enable a less efficient
   compression in the next hops to still fits within the Link MTU.  If
   the fragmenting endpoint is also the 6LoWPAN compression endpoint, it
   will elide the IID of the source IPv6 address if it matches the Link-
   Layer address [RFC6282].  In a network with a consistent MTU, it MUST
   compute the Fragment_Size as if the MTU was 8 bytes less, so the next
   hop can expand the IID within the same fragment.
"


But the MTU discovery is out of scope.

Changed a bit section 6.2
"
   This specification does not provide a method to discover the number
   of hops or the minimal value of MTU along those hops.  In a typical
   case, the MTU is constant and the same across the network.  But
   should the minimal MTU along the path decrease,
"


> 
>    IPv6 address is elided by the originator, then it MUST compute the
>    Fragment_Size as if the MTU was 8 bytes less.  This way, the next hop
>    can restore the source IID to the first fragment without impacting
>    the second fragment.
> 
> Is it worth noting that this to large extent obviates the benefits of header
> compression?

On the next hops it does, yes. It is a  generic thing for all 6FF techniques. 
It would be better to say it in minimal I guess. Text in minimal section 5 becomes:

"
   Fragment forwarding obviates some of the benefits of the 6LoWPAN
   header compression [RFC6282] in intermediate hops.  In return, the
   memory used to store the packet is distributed along the path, which
   limits the buffer bloat effect.  Multiple fragments may progress
   simultaneously along the network as long as they do not interfere.
"

> 
> Section 4.2
> 
>    This specification introduces a concept of an inter-frame gap, which
>    is a configurable interval of time between transmissions to the same
> 
> I think draft-ietf-6lo-minimal-fragment also talks about an "inter-frame gap",
> though in less detail.

Yes, already changed for another review. Now reads as

"

[FRAG-FWD] requires that a configurable interval of time is inserted
   between transmissions to the same next hop and in particular between
   fragments of a same datagram.  In the case of half duplex interfaces,
   this inter-frame gap ensures that the next hop is done forwarding the
   previous frame and is capable of receiving the next one.

"




> Section 4.3
> 
>    The compression of the Hop Limit, of the source and destination
>    addresses in the IPv6 Header, and of the Routing Header may change en
>    route in a Route-Over mesh LLN.  If the size of the first fragment is
>    modified, then the intermediate node MUST adapt the Datagram_Size to
>    reflect that difference.
> 
> Hmm, if this really only affects the first fragment, then maybe Section
> 4.1 could be more clear that the need for extra "slack" in practice only applies
> to the first fragment, as well.

Yes, see the revised section 4.1 above


> 
>    The intermediate node MUST also save the difference of Datagram_Size
>    of the first fragment in the VRB and add it to the Datagram_Size and
>    to the Fragment_Offset of all the subsequent fragments for that
>    datagram.
> 
> The RFRAG header does not carry the Datagram_Size, so I think this needs
> rewording (we do, however, need to store the updated size to detect the final
> fragment).
> 

It does, actually, well hidden in the Fragment_Offset of the first fragment. 
Clearly you found that later in the review.

> Section 5
> 
>    This specification provides a technique that is derived from MPLS to
>    forward individual fragments across a 6LoWPAN route-over mesh without
>    reassembly at each hop.  The Datagram_Tag is used as a label; it is
>    locally unique to the node that owns the source MAC address of the
>    fragment, so together the MAC address and the label can identify the
>    fragment globally.  A node may build the Datagram_Tag in its own
> 
> (More "globally unique" with implied "lifetime of the packet".)

Yes, moved it up:

"

   A 6LoWPAN Fragment Forwarding [FRAG-FWD] technique derived from MPLS
   enables the forwarding of individual fragments across a 6LoWPAN
   route-over mesh without reassembly at each hop.  The Datagram_Tag is
   used as a label; it is locally unique to the node that owns the
   source Link-Layer address of the fragment, so together the Link-Layer
   address and the label can identify the fragment globally within the
   lifetime of the datagram.  A node may build the Datagram_Tag in its
   own locally-significant way, as long as the chosen Datagram_Tag stays
   unique to the particular datagram for its lifetime.
"

> 
>    locally-significant way, as long as the chosen Datagram_Tag stays
>    unique to the particular datagram for the lifetime of that datagram.
> 
> [okay, less implied and more explicit here than it was above :) ]
> 
indeed

> Section 5.1
> 
> We discuss the header format, but I don't think we actually say that "you put
> the fragment payload after the header" :)
> 
>                              1                   2                   3
>         0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
>                                        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>                                        |1 1 1 0 1 0 0|E|  Datagram_Tag |
>        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>        |X| Sequence|   Fragment_Size   |       Fragment_Offset         |
>        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> 
> I strongly suggest a stronger indicator that the field marked "Fragment_Offset"
> is also used to carry the Datagram_Size (for sequence number zero).

Reading your review I can see why. So why not adapt 4.4 as follows:

"

4.4.  Modifying the First Fragment

   The compression of the Hop Limit, of the source and destination
   addresses in the IPv6 Header, and of the Routing Header may change en
   route in a Route-Over mesh LLN.  If the size of the first fragment is
   modified, then the intermediate node MUST adapt the Datagram_Size,
   encoded in the Fragment_Size field, to reflect that difference.

   The intermediate node MUST also save the difference of Datagram_Size
   of the first fragment in the VRB and add it to the Fragment_Offset of
   all the subsequent fragments that it forwards for that datagram.

"

> 
>    The format of the fragment header is shown in Figure 1.  It is the
>    same for all fragments.  The format has a length and an offset, as
> 
> (Similarly, this could mention the slight exception to "all fragments".)

Shuffing a bit the text and adding the end of this sentence:
"
   The format of the fragment header is shown in Figure 1.  It is the
   same for all fragments though the Fragment_Offset is overloaded.

"


>    order in which the fragments are received.  This enables out-of-
>    sequence subfragmenting, e.g., a fragment seq. 5 that is retried end-
>    to-end as smaller fragments seq. 5, 13 and 14 due to a change of MTU
>    along the path between the 6LoWPAN endpoints.
> 
> Per the Discuss, reusing a sequence number to mean a different fragment
> (here, a smaller one) seems pretty risky.  I recall that one of the big advantages
> of QUIC over TCP is that we don't reuse sequence numbers for retransmissions
> in QUIC, to get better clarity about what exactly is lost vs. ack'd; reusing a
> sequence number to mean something with a different *size* seems
> qualitatively worse (riskier) to me.

Yes, that was changed for the discuss and published already.




> 
>    There is no requirement on the receiver to check for contiguity of
>    the received fragments.  The sender knows that the datagram is fully
>    received when the acknowledged fragments cover the whole datagram.
> 
> But what if there's ambiguity about which version of a fragment (identified by
> sequence number) is ack'd?

There's no more with the change you suggested, since there is no reuse.
There was none either id the reuse was for a smaller fragment but that's a thing of the past.


> Also, I'm confused by "no requirement" to check for contiguity; presumably the
> receiver also is going to check that it has all the data in the datagram
> somehow...


Yes, I meant that the sequence in the fragment matches a perfect tiling of the compressed packet

What about

"

   There is no requirement on the reassembling endpoint to check that
   the received fragments are consecutive and non-overlapping.

"




>    The first fragment is recognized by a Sequence of 0; it carries its
>    Fragment_Size and the Datagram_Size of the compressed packet before
>    it is fragmented, whereas the other fragments carry their
>    Fragment_Size and Fragment_Offset.  The last fragment for a datagram
>    is recognized when its Fragment_Offset and its Fragment_Size add up
>    to the Datagram_Size.
> 
> I suggest "add up to the stored Datagram_Size of the packet identified by the
> sender and Datagram_Tag".

Done

"

   The first fragment is recognized by a Sequence of 0; it carries its
   Fragment_Size and the Datagram_Size of the compressed packet before
   it is fragmented, whereas the other fragments carry their
   Fragment_Size and Fragment_Offset.  The last fragment for a datagram
   is recognized when its Fragment_Offset and its Fragment_Size add up
   to the stored Datagram_Size of the packet identified by the sender
   Link-Layer address and the Datagram_Tag.

"


> 
>    E:  1 bit; Explicit Congestion Notification; the "E" flag is reset by
>       the source of the fragment and set by intermediate routers to
>       signal that this fragment experienced congestion along its path.
> 
> nit: I suggest s/reset/cleared/.

done


> 
>    Datagram_Tag:  8 bits; an identifier of the datagram that is locally
>       unique to the sender.
> 
> RFC 4944 had a 16-bit tag field; I would expect some discussion of why the
> shorter tag is acceptable now.  I guess that's in the security considerations,
> which is okay (albeit not where I would have first looked).

Text was added with Eric' s review deeper in the spec, and I see you found it.

> 
>    Sequence:  5-bit unsigned integer; the sequence number of the
>       fragment in the acknowledgement bitmap.  Fragments are numbered
>       [0..N] where N is in [0..31].  A Sequence of 0 indicates the first
> 
> This implies that a one-fragment packet is allowed (N of 0).  Wouldn't you just
> not use the fragmentation framing for the case where the packet fits in a single
> frame?

Each bit counts when the MTU is 80 to 100 octets.
You'd do that if you want an ack from the edge of the 6LoPWAN network.
But a full packet that fails on path A can be be rerouted, so the L2 ack usually suffices.


> 
>       *  if a VRB already exists and is not broken, the fragment is to
>          be forwarded along the associated Label Switched Path (LSP) as
>          described in Section 6.1.2, but regardless of the value of the
>          Sequence field;
> 
> I'm not sure I'm parsing "but regardless of the value of the Sequence field"
> properly; if the intent is to say that it is forwarded without checking the value
> of the Sequence field, I would remove the "but".

picked

> Also, we seem to be intending "is not broken" to match up with "the next hop
> is still reachable" in the preceding paragraph; I'd suggest harmonizing the
> wording between the two instances.

done

> 
>       *  else, if the Sequence is 0, then the fragment is to be routed
>          as described in Section 6.1.1, but no state is conserved
>          afterwards.  In that case, the session if it exists is aborted
>          and the packet is also forwarded in an attempt to clean up the
>          next hops along the path indicated by the IPv6 header (possibly
>          including a routing header).
> 
> Do we want an "*  else (the Sequence is nonzero and either no VRB exists or
> the next hop is unavailable), the fragment is discarded and an abort RFRAG-
> ACK is sent back [...]" to replace the last paragraph of the section?

Much nicer, thanks!

"

      *  else (the Sequence is nonzero and either no VRB exists or the
         next hop is unavailable), the fragment cannot be forwarded or
         routed; the fragment is discarded and an abort RFRAG-ACK is
         sent back to the source as described in Section 6.1.2.
"


 
> Section 5.2
> 
>    This specification also defines a 4-octet RFRAG Acknowledgment bitmap
>    that is used by the reassembling endpoint to confirm selectively the
>    reception of individual fragments.  A given offset in the bitmap maps
>    one-to-one with a given sequence number and indicates which fragment
>    is acknowledged as follows:
> 
> [Except it *doesn't* indicate that, since the sequence number is not a unique
> fragment identifier if the fragment size can change!]

Point taken and applied : )



> 
>         0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
>                                        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>                                        |1 1 1 0 1 0 1|E|  Datagram_Tag |
>        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
>        |          RFRAG Acknowledgment Bitmap (32 bits)                |
>        +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> 
> I think we need to explicitly say here that the Datagram_Tag, in a reversal of
> the previous usage, is scoped to the link-layer *recipient* of the RFRAG_ACK
> frame.

It is indeed. Added:
"
   Datagram_Tag:  8 bits; an identifier of the datagram that is locally
      unique to the Link-Layer recipient.

"

> 
> Section 6
> 
>    that was the originator of the fragments.  To achieve this, each hop
>    that performed an MPLS-like operation on fragments reverses that
>    operation for the RFRAG_ACK by sending a frame from the next hop to
>    the previous hop as known by its MAC address in the VRB.  The
> 
> I'm not 100% sure that the previous work in this space (e.g.,
> draft-ietf-6lo-minimal-fragment) framed things such that there was a
> requirement to be able to perform this "reverse lookup".  It might be worth

Agreed

> calling this out as an additional requirement over the prevoius fragment-
> forwarding methodologies...
> (We do seem to do so in Section 6.1.1, though I am not sure if it merits an
> earlier mention in the document.)


Modifying 2.2:

"

   Quoting the "Multiprotocol Label Switching (MPLS) Architecture"
   [RFC3031]: with MPLS, 'packets are "labeled" before they are
   forwarded' along a Label Switched Path (LSP).  At subsequent hops,
   there is no further analysis of the packet's network layer header.
   Rather, the label is used as an index into a table which specifies
   the next hop, and a new label".  [FRAG-FWD] leverages MPLS to forward
   fragments that actually do not have a network layer header, since the
   fragmentation occurs below IP, and this specification makes it
   reversible so the reverse path can be followed as well. 

"

> 
>    limiting the number of outstanding fragments, which are the fragments
>    that have been sent but for which reception or loss was not
>    positively confirmed by the reassembling endpoint.  The maximum
> 
> (Do we have a way for *loss* to be "positively confirmed"?  Appendix C notes
> that we infer loss from timeout without acknowledgment.)
 
If a particular network guarantees the order of frames then yes. But we do not mandate that.
Otherwise, time out, though it is not really positive. Not sure what text to change though.

> 
>    The Ack-Request ('X') set in an RFRAG marks the end of a window.
>    This flag MUST be set on the last fragment if the sender wishes to
>    protect the datagram, and it MAY be set in any intermediate fragment
>    for the purpose of flow control.
> 
> Is this usage of "protect" conventional or should it get clarified somewhere?

Clarified as

"

   The Ack-Request ('X') set in an RFRAG marks the end of a window.
   This flag MUST be set on the last fragment if the fragmenting
   endpoint wishes to perform an automatic repeat request (ARQ) process
   for the datagram, and it MAY be set in any intermediate fragment for
   the purpose of flow control.

"

 
>    This automatic repeat request (ARQ) process MUST be protected by a
>    Retransmission Time Out (RTO) timer, and the fragment that carries
> 
> nit: I'm not sure that "this" is appropriate yet, since we haven't really talked
> about "repeat"ing or retransmission yet.  (Perhaps also give a bit more
> discussion about what this ARQ process specifically entails, too.)

With the above it's now done.

> 
>    of times (see Section 7.1).  Upon exhaustion of the retries the
>    sender may either abort the transmission of the datagram or retry the
>    datagram from the first fragment with an 'X' flag set in order to
>    reestablish a path and discover which fragments were received over
>    the old path in the acknowledgment bitmap.  When the sender of the
> 
> I'm not sure I'm understanding properly what's meant by "retry the datagram
> from the first fragment with an 'X' flag set".  My best guess is that it's saying
> "re-send the fragment with sequence number 0, and set the 'X' flag in this new
> transmission", in which case I'd suggest s/from the first fragment with an 'X'
> flag set/from the first fragment, setting the 'X' flag in the retransmitted
> fragment,/.

Not from frag 0; the idea is to resent Frag 0 with ack request. That forms a new path, and enables the other end point to ack what it has received. From there, the remainder can be set over the new path.

Per a prior review it was reworded to
"
Upon exhaustion of the retries the
   fragmenting endpoint may either abort the transmission of the
   datagram or resend the first fragment with an 'X' flag set in order
   to establish a new path for the datagram and obtain the list of
   fragments that were received over the old path in the acknowledgment
   bitmap.

"

> 
>    The receiver MAY issue unsolicited acknowledgments.  An unsolicited
>    acknowledgment signals to the sender endpoint that it can resume
>    sending if it had reached its maximum number of outstanding
>    fragments.  Another use is to inform the sender that the reassembling
> 
> How would the sender get into a situation where it has saturated its transmit
> window but not requested an ACK?  Is this just the case where the fragment
> requesting an ACK gets lost and thus we have to rely on some timeout
> (whether at the sender or the receiver) to recover?  It seems like we may need
> to define a symbolic constant for the "unsolicited ACK" timer after no
> fragments are received, and give some guidance on how to determine it.

"if" was meant to say " in case ", replacing. This can happen. E.g., the window may be smaller than the capacity of the network (related to the nb of hops). 

 (even more after Mirja's review) I do not want to dig into flow control and this is borderline. I just wanted to leave the door open for future mechanisms that are out of scope and that would do it.


> 
>    The RFRAG Acknowledgment can optionally carry an ECN indication for
>    flow control (see Appendix C).  The receiver of a fragment with the
>    'E' (ECN) flag set MUST echo that information by setting the 'E'
>    (ECN) flag in the next RFRAG Acknowledgment.
> 
> I'm not sure I'd use the word "optionally" here -- IIUC the idea is to make it
> mandatory to implement the ECN behavior, and the ECN indication is only
> "optional" in that it is sometimes set, but when it is set and when it is not set
> are mandatory parts of the protocol, not subject to implementation discretion.
> If that's correct, then I'd just say "The RFRAG Acknowledgment carries and ECN
> indication for flow control".

Done, a slight update from what we did for 14
"

   The RFRAG Acknowledgment carries an ECN indication for flow control
   (see Appendix C).  The reassembling endpoint of a fragment with the
   'E' (ECN) flag set MUST echo that information by setting the 'E'
   (ECN) flag in the next RFRAG Acknowledgment

"

>    round-trip delay in the network.  As the timer runs, the receiving
>    endpoint absorbs the fragments that were still in flight for that
>    datagram without creating a new state.  The receiving endpoint aborts
>    the communication if it keeps going on beyond the duration of the
>    timer.
> 
> (This "absorbs" the behavior is to catch retransmissions, right?  Might be worth
> saying so, though also might not.) I'd suggest expanding "if it keeps going",
> perhaps as "if fragments with matching source and tag continue to be received
> after the timer expires".

"
   When enough fragments are received to cover the whole datagram, the
   reassembling endpoint reconstructs the packet, passes it to the upper
   layer, sends an RFRAG Acknowledgment on the reverse path with a FULL
   bitmap, and arms a short timer, e.g., on the order of an average
   round-trip delay in the network.  The FULL bitmap is used as opposed
   to a bitmap that acknowledges only the received fragments to let the
   intermediate nodes know that the datagram is fully received.  As the
   timer runs, the reassembling endpoint absorbs the fragments that were
   still in flight for that datagram without creating a new state,
   acknowledging the ones that that bear an Ack-Request with an FRAG
   Acknowledgment and the FULL bitmap.  The reassembling endpoint aborts
   the communication if fragments with matching source and Datagram-Tag
   continue to be received after the timer expires.



"



> 
>    Fragments are sent in a round-robin fashion: the sender sends all the
>    fragments for a first time before it retries any lost fragment; lost
>    fragments are retried in sequence, oldest first.  This mechanism
>    enables the receiver to acknowledge fragments that were delayed in
>    the network before they are retried.
> 
> Is this round-robin within a given "window" or within the entire packet?

Entire packet is more efficient and intended here;

"

   Fragments are sent in a round-robin fashion: the fragmenting endpoint
   sends all the fragments of the datagram for a first time before it
   retries any lost fragment; lost fragments are retried in sequence,
   oldest first through the whole datagram.  This mechanism enables the
   reassembling endpoint to acknowledge fragments that were delayed in
   the network before they are retried.

"

> 
>    When a single frequency is used by contiguous hops, the sender should
> 
> nit: "radio frequency"?  Retransmissions have a frequency, too (though several
> orders of magnitude different!).

Radio : ) added.


> 
> Section 6.1
> 
>    label, which is swapped in each hop.  All fragments follow the same
>    path and fragments are delivered in the order in which they are sent.
> 
> Where do we make a normative requirement for intermediate nodes to retain
> order in transmitting fragments?

we do not but I'm not even sure that this sentence meant that. Removed.
Changed the beginning as:


"
   This specification extends the Virtual Reassembly Buffer (VRB)
   technique to forward fragments with no intermediate reconstruction of
   the entire packet.  It inherits operations like Datagram_Tag
   switching and using a timer to clean the VRB once the traffic ceases.
   The first fragment carries the IP header and creates a path from the
   fragmenting endpoint to the reassembling endpoint that all the other
   fragments follow.

"

> 
> Section 6.1.1
> 
>    In Route-Over mode, the source and destination MAC addresses in a
> 
> It feels surprising to mention Route-Over mode and not have a corresponding
> mention of Mesh-Under.

In mesh under, fragments are always forwarded since they are inside a frame and the frame is what gets forwarded. This draft serves a route-over-only problem. Recently we have changed the abstract to:

"
   This draft updates RFC 4944 with a simple protocol to recover
   individual fragments across a route-over mesh network, with a minimal
   flow control to protect the network against bloat.

"


> 
>    (and unique) for that source MAC address.  Upon receiving the first
>    fragment (i.e., with a Sequence of zero), an intermediate router
>    creates a VRB and the associated LSP state for the tuple (source MAC
>    address, Datagram_Tag) and the fragment is forwarded along the IPv6
>    route that matches the destination IPv6 address in the IPv6 header as
>    prescribed by [I-D.ietf-6lo-minimal-fragment], where the receiving
>    endpoint allocates a reassembly buffer.
> 
> nit: please check the comma usage here (the last one looks misplaced, to my
> eye).
 Removed the comma, changed the last words, which were misleading since the reassemble may not be the IPv6 destination.
"
   Upon receiving the first fragment (i.e., with a Sequence of 0), an
   intermediate router creates a VRB and the associated LSP state
   indexed by the incoming interface, the previous-hop Link-Layer
   address, and the Datagram_Tag, and forwards the fragment along the
   IPv6 route that matches the destination IPv6 address in the IPv6
   header until it reaches the reassembling endpoint, as prescribed by
   [FRAG-FWD].  The LSP state enables to match the next incoming
   fragments of a datagram to the abstract forwarding information (next
   interface, source and next-hop Link-Layer addresses, swapped
   Datagram_Tag). 
"

>    Datagram_Tag.  This reverse LSP state also points at the VRB and
>    enables matching the (next MAC address, swapped_Datagram_Tag) found
>    in an RFRAG Acknowledgment to the tuple (previous MAC address,
> 
> nit(?): doesn't this "reverse lookup" only depend on the
> swapped_Datagram_Tag?  I don't think "next MAC address" is needed.

The trick is that the node may have more than one interface and more than one MAC address on that interface (think privacy MAC). The whole text is written with the assumption of a single interface because that's usually the case in 6LoWPAN but at least there could be more than one MAC sued as source by this node in parallel for difference datagrams. A node must remember which MAC addresses it uses for a datagram and that's part of the key used as index for the rfrag ack. I revisited the text a bit.

"
  In addition, the router also forms a reverse LSP state indexed by the
   interface to the next hop, the Link-Layer address the router uses as
   source for that datagram, and the swapped Datagram_Tag.  This reverse
   LSP state enables matching the (next interface, destination Link-
   Layer address, swapped_Datagram_Tag) found in an RFRAG Acknowledgment
   to the abstract forwarding information (previous interface, previous
   Link-Layer address, Datagram_Tag) used to forward the Fragment
   Acknowledgment (RFRAG-ACK) back to the fragmenting endpoint.

"


> 
>    The first fragment may be received a second time, indicating that it
>    did not reach the destination and was retried.  In that case, it
>    SHOULD follow the same path as the first occurrence.  It is up to
>    sending endpoint to determine whether to abort a transmission and
>    then retry it from scratch, which may build an entirely new path.
> 
> This seems to be alluding to the text in Section 6 about "or retry the datagram
> from the first fragment with an 'X' flag set in order to reestablish a path",
> however, my reading of that text is that the same datagram tag value is used in
> the retry, in which case the retransmission of the first fragment seems like it
> would be indistinguishable from any other retransmission, to the intermediate
> node.  If a new tag was to be allocated, then we would not be able to get
> information about fragments received over the old path as Section 6 discusses.
> So it seems like this recommendation here ("use the same
> path") is at odds with the implied recommendation in the earlier text ("use a
> potentially different path"), and it would be worth some discussion to resolve
> the apparent disparity.
> 

It is indeed. Let me remove the whole paragraph.



> Section 6.1.2
> 
>    [I-D.ietf-6lo-minimal-fragment] indicates that the receiving endpoint
>    stores "the actual packet data from the fragments received so far, in
>    a form that makes it possible to detect when the whole packet has
>    been received and can be processed or forwarded".  How this is
>    computed is implementation specific but relies on receiving all the
>    bytes up to the Datagram_Size indicated in the first fragment.  An
>    implementation may receive overlapping fragments as the result of
>    retries after an MTU change.
> 
> This text leaves me feeling unsatisfied, for a couple reasons:
> (1) I have to think to infer that "receiving endpoint" means "the intermediate
> router receiving the fragments" and not the ultimate (IPv6) destination

There was wrong stuff that was cleaned with your review above. In particular:

- The term "receiving endpoint" is gone. I'm using "reassembling endpoint" throughput and the endpoints are the 6FF endpoints not the 6LoWPAN compression or the IPv6 endpoints. 
- That guy may not be the node that undoes the 6loWPAN compression but a node in between as you noticed. This makes sense with RFC 8138 that forwards in the compressed form.
- The final IPv6 destination might be even farther, far, far away in the internet.



> (2) overlapping fragments that contain different data can be a security risk,
> and we don't discuss that here (whether directly or by reference).

We do now. Minimal frag section 7:

"
   *  Overlapping fragment attacks are possible with 6LoWPAN fragments
      but there is no known firewall operation that would work on
      6LoWPAN fragments at the time of this writing, so the exposure is
      limited.  An implementation of a firewall SHOULD NOT forward
      fragments but instead should recompose the IP packet, check it in
      the uncompressed form, and then forward it again as fragments if
      necessary.  Overlapping fragments are acceptable as long as they
      contain the same payload.  The firewall MUST drop the whole packet
      if overlapping fragments are encountered that result in different
      data at the same offset.

"
And this spec has

"
An implementation
   may receive overlapping fragments as the result of retries after an
   MTU change.
"
and
"
There is no requirement on the reassembling endpoint to check that
   the received fragments are consecutive and non-overlapping.
"
We do not mandate checking because it is more code and the attack is difficult "
                  since the
      fragmentation happens within the LLN, meaning that the intruder
      should already be inside to perform the attack.
"



> Section 6.2
> 
>    Upon receipt of an RFRAG-ACK, the router looks up a reverse LSP
>    indexed by the tuple (MAC address, Datagram_Tag), which are
> 
> (As above, is the MAC address needed here?)

You'd be correct yes.

Still we want a MAC address but the destination MAC of the RFRAG ACK not the source, and that's in case this node uses more than one MAC e.g., to reuse the tag address space.
Also we need to introduce the interface. This is a change that spans to minimal-fragments in fact, I'll publish a new one as well.

Rewording that piece to
"

   Upon receipt of an RFRAG-ACK, the router looks up a reverse LSP
   indexed by the interface and destination Link-Layer address of the
   received frame and the received Datagram_Tag in the RFRAG-ACK.  If it
   is found, the router forwards the fragment using the associated VRB
   as prescribed by [FRAG-FWD], but using the reverse LSP so that the
   RFRAG-ACK flows back to the fragmenting endpoint.

"

> 
>    Either way, if the RFRAG-ACK indicates that the fragment was entirely
>    received (FULL bitmap), it arms a short timer, and upon timeout, the
>    VRB and all the associated state are destroyed.  Until the timer
> 
> Is there anything that could cancel this timer (e.g., an RFRAG-ACK that with
> non-FULL bitmap)?

I cannot see that? Logically that one would have been sent before and received after. 
Once the FULL is achieved the fragmenting node knows that the reassembling endpoint is done.


> 
>    This specification does not provide a method to discover the number
>    of hops or the minimal value of MTU along those hops.  But should the
>    minimal MTU decrease, it is possible to retry a long fragment (say
>    Sequence of 5) with first a shorter fragment of the same Sequence (5
>    again) and then one or more other fragments with a Sequence that was
>    not used before (e.g., 13 and 14).  Note that Path MTU Discovery is
>    out of scope for this document.
> 
> [another mention of sequence number reuse]
> 

The text is now (since -14)
"
But
   should the minimal MTU along the path decrease, it is possible to
   retry a long fragment (say Sequence of 5) with several shorter
   fragments with a Sequence that was not used before (e.g., 13 and 14).
   Fragment 5 is marked as abandoned and will not be retried anymore.
"


> Section 6.3
> 
>    A reset is signaled on the forward path with a pseudo fragment that
>    has the Fragment_Offset, Sequence, and Fragment_Size all set to 0,
>    and no data.
> 
> The text in Section 5.1 indicates that an abort is characterized by
> Fragment_Offset of zero.  I don't think that it has a requirement to set the
> other fields to zero in order to be considered an abort.

True:

"

   A reset is signaled on the forward path with a pseudo fragment that
   has the Fragment_Offset set to 0.  The sender of a reset SHOULD also
   set the Sequence and Fragment_Size field to 0.

"



>    When the sender or a router on the way decides that a packet should
> 
> nit: I don't think we've used "on the way" previously in this sense (and do use
> "on the path").  Though, it looks like we do use "on the way"
> again a few paragraphs later.

on the path looks better

> 
>    The other way around, the receiver might need to abort the process of
>    a fragmented packet for internal reasons, for instance if it is out
>    of reassembly buffers, already uses all 256 possible values of the
>    Datagram_Tag, or if it keeps receiving fragments beyond a reasonable
>    time while it considers that this packet is already fully reassembled
>    and was passed to the upper layer.  In that case, the receiver SHOULD
> 
> nit: s/process/processing/?

done

> 
> Section 7
> 
> We can probably inherit discussion of integrity and confidentiality (non-
> )protection from -minimal-fragment, assuming my recommended changes
> there are made.  It remains the case, though, that intermediate nodes can
> always modify the payload and disrupt/deny service, and I, for one, would not
> turn down another reminder of that.

Done some in 14

"

  This document specifies an instantiation of a 6FF technique and
   inherits from the generic description in [FRAG-FWD].  The
   considerations in the Security Section of [FRAG-FWD] equally apply to
   this document.

   In addition to the threats detailed therein, an attacker that is on-
   path can prematurely end the transmission of a datagram by sending a
   RFRAG Acknowledgment to the fragmenting endpoint.  It can also cause
   extra transmissions of fragments by resetting bits in the RFRAG
   Acknowledgment bitmap, and of RFRAG Acknowledgments by forcing the
   Ack-Request bit in fragments that it forwards.  As indicated in
   [FRAG-FWD], Secure joining and the Link-Layer security are REQUIRED
   to protect against those attacks.
"

Do we want more?

> 
>    This specification extends "On Forwarding 6LoWPAN Fragments over a
>    Multihop IPv6 Network" [I-D.ietf-6lo-minimal-fragment] and requires
>    the same parameters in the receiver and on intermediate nodes.  There
>    is no new parameter as echoing ECN is always on.  These parameters
>    typically include the reassembly time-out at the receiver and an
>    inactivity clean-up timer on the intermediate nodes, and the number
>    of messages that can be processed in parallel in all nodes.
> 
> Do we have a new parameter for the "timer for a receiver node to send an
> unsolicited ack if no additional fragments are received"?
> 

I'd rather not if that's OK. Too close to the flow control that we stay away from.


>    The configuration settings introduced by this specification only
>    apply to the sender, which is in full control of the transmission.
> 
> (Similarly, this is "full control" modulo acks.)

The acks are not commands, they are information : ) The  fragmenting endpoint (sender) is the boss.


> 
> Section 7.1
> 
>    OptWindowSize:  The OptWindowSize is the value for the Window_Size
>       that the sender should use to start with.  It is greater than or
>       equal to MinWindowSize.  It is less than or equal to
>       MaxWindowSize.  The Window_Size should be maintained below the
>       number of hops in the path of the fragment to avoid stacking
>       fragments at the bottleneck on the path.  If an inter-frame gap is
>       used to avoid interference between fragments then the Window_Size
>       should be at most on the order of the estimation of the trip time
>       divided by the inter-frame gap.
> 
> Let's do some dimensional analysis here: the window size is measured in
> number of frames, the trip time is measured in time, and the inter-frame gap is
> discussed above as an "amount of time" (though whether this is an absolute
> time value or a count of transmit windows is perhaps subject to
> interpretation).  The ratio of trip time deivided by inter-frame gap, then, is
> dimensionless, which we then try to interpret as a number of frames.  That's
> probably okay, though mathematics pedants might want some additional
> clarifying discussion.

Me who thought that the physics exams were behind!

We transmit at most 1 frame per interframe gap. So the number of interframe gaps per RTT gives us the max number of frames that can be sent by RTT, that is till an ack comes in. So we fill the network with the number given, any more than that is a potential addition to the bloat but to no added throughput.

The typo in divided was fixed too...

> 
>    An implementation may be capable of performing flow control based on
>    ECN; see in Appendix C.  This is controlled by the following
>    parameter:
> 
>    UseECN:  Indicates whether the sender should react to ECN.  The
>       sender may react to ECN by varying the Window_Size between
>       MinWindowSize and MaxWindowSize, varying the Fragment_Size between
>       MinFragmentSize and MaxFragmentSize, and/or by increasing the
>       inter-frame gap.
> 
> Please forgive the naivite, but shouldn't this parameter merely control
> *how* an implementation responds to ECN markings, not *whether or not it
> should respond at all* (since obviously it should respond in some fashion)?

The spec provides either do nothing or reset the window to one and live with it till the end of the datagram. The goal is that other specs provide a better flow control.

"

   UseECN:  Indicates whether the fragmenting endpoint should react to
      ECN.  The fragmenting endpoint may react to ECN by varying the
      Window_Size between MinWindowSize and MaxWindowSize, varying the
      Fragment_Size between MinFragmentSize and MaxFragmentSize, and/or
      by increasing or reducing the inter-frame gap.  With this
      specification, if UseECN is set and a fragmenting endpoint detects
      a congestion, it resets the Window_Size to 1 till the end of the
      datagram, whereas if UseECN is reset, the endpoint does not react
      to congestion.  Future specifications may provide additional
      parameters and capabilities.

"

> 
> Section 11
> 
> I'm not seeing why RFC 6554 (as referenced in one location only) is a normative
> reference.

fixed

> 
> Section 12
> 
> Given that "Readers are expected to be familiar with all the terms and
> concepts that are discussed in "IPv6 over Low-Power Wireless Personal Area
> Networks (6LoWPANs): Overview, Assumptions, Problem Statement, and
> Goals" [RFC4919]", RFC 4919 seems like it is more properly categorized as
> Normative.  Similarly for RFC 6606.
> 

OK, will be the usual downref

> I don't have strong feelings about RFC 8200's status, though I suppose one
> could argue that one needs to parse the network header in order to route the
> initial fragment.

Sure, moved 8200 to normative

> 
> Appendix A
> 
>    Considering that RFC 4944 defines an MTU is 1280 bytes and that in
>    most incarnations (but 802.15.4g) a IEEE Std. 802.15.4 frame can
>    limit the MAC payload to as few as 74 bytes, a packet might be
> 
> I'm not sure I'm parsing this properly, but I think "except" is more approriate
> than "but" in the parenthetical.
> 

Applied

> Appendix C
> 
>    From the standpoint of a source 6LoWPAN endpoint, an outstanding
>    fragment is a fragment that was sent but for which no explicit
>    acknowledgment was received yet.  This means that the fragment might
> 
> Is there also a timer that has to have not expired yet?
> 

Added " and are still covered by the ARQ timer "

Whao, that was a lot!

Again and again, so many thanks!

Pascal