Re: [core] Benjamin Kaduk's Discuss on draft-ietf-core-new-block-11: (with DISCUSS and COMMENT)

Benjamin Kaduk <kaduk@mit.edu> Thu, 13 May 2021 21:52 UTC

Return-Path: <kaduk@mit.edu>
X-Original-To: core@ietfa.amsl.com
Delivered-To: core@ietfa.amsl.com
Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id 93F3C3A1584; Thu, 13 May 2021 14:52:41 -0700 (PDT)
X-Virus-Scanned: amavisd-new at amsl.com
X-Spam-Flag: NO
X-Spam-Score: -1.499
X-Spam-Level:
X-Spam-Status: No, score=-1.499 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, KHOP_HELO_FCRDNS=0.398, SPF_HELO_NONE=0.001, SPF_NONE=0.001, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no
Received: from mail.ietf.org ([4.31.198.44]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3lDxwETKPQhK; Thu, 13 May 2021 14:52:36 -0700 (PDT)
Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id 1E3733A1588; Thu, 13 May 2021 14:52:30 -0700 (PDT)
Received: from kduck.mit.edu ([24.16.140.251]) (authenticated bits=56) (User authenticated as kaduk@ATHENA.MIT.EDU) by outgoing.mit.edu (8.14.7/8.12.4) with ESMTP id 14DLqM0d022229 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 13 May 2021 17:52:26 -0400
Date: Thu, 13 May 2021 14:52:21 -0700
From: Benjamin Kaduk <kaduk@mit.edu>
To: mohamed.boucadair@orange.com
Cc: The IESG <iesg@ietf.org>, "draft-ietf-core-new-block@ietf.org" <draft-ietf-core-new-block@ietf.org>, "core-chairs@ietf.org" <core-chairs@ietf.org>, "core@ietf.org" <core@ietf.org>, "marco.tiloca@ri.se" <marco.tiloca@ri.se>
Message-ID: <20210513215221.GH79563@kduck.mit.edu>
References: <162026630680.17506.6477675472375470197@ietfa.amsl.com> <12589_1620322520_609428D8_12589_262_1_787AE7BB302AE849A7480A190F8B933035377DD2@OPEXCAUBMA2.corporate.adroot.infra.ftgroup>
MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-1
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <12589_1620322520_609428D8_12589_262_1_787AE7BB302AE849A7480A190F8B933035377DD2@OPEXCAUBMA2.corporate.adroot.infra.ftgroup>
Archived-At: <https://mailarchive.ietf.org/arch/msg/core/MWpNPKO0-WRMnhThW0ioo-ajgT8>
Subject: Re: [core] Benjamin Kaduk's Discuss on draft-ietf-core-new-block-11: (with DISCUSS and COMMENT)
X-BeenThere: core@ietf.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: "Constrained RESTful Environments \(CoRE\) Working Group list" <core.ietf.org>
List-Unsubscribe: <https://www.ietf.org/mailman/options/core>, <mailto:core-request@ietf.org?subject=unsubscribe>
List-Archive: <https://mailarchive.ietf.org/arch/browse/core/>
List-Post: <mailto:core@ietf.org>
List-Help: <mailto:core-request@ietf.org?subject=help>
List-Subscribe: <https://www.ietf.org/mailman/listinfo/core>, <mailto:core-request@ietf.org?subject=subscribe>
X-List-Received-Date: Thu, 13 May 2021 21:52:42 -0000

Hi Med,

On Thu, May 06, 2021 at 05:35:19PM +0000, mohamed.boucadair@orange.com wrote:
> Hi Ben, 
> 
> Thank you for the review. All the changes can be tracked at: https://tinyurl.com/new-block-latest. 

(Apparently I waited long enough to be able to just use rfcdiff on the
published -12; PR #26 posted with a couple nits from that diff.)

> Please see inline. 
> 
> Cheers,
> Jon & Med
> 
> > -----Message d'origine-----
> > De : Benjamin Kaduk via Datatracker [mailto:noreply@ietf.org]
> > Envoyé : jeudi 6 mai 2021 03:58
> > À : The IESG <iesg@ietf.org>
> > Cc : draft-ietf-core-new-block@ietf.org; core-chairs@ietf.org;
> > core@ietf.org; marco.tiloca@ri.se; marco.tiloca@ri.se
> > Objet : Benjamin Kaduk's Discuss on draft-ietf-core-new-block-11:
> > (with DISCUSS and COMMENT)
> > 
> > Benjamin Kaduk has entered the following ballot position for
> > draft-ietf-core-new-block-11: Discuss
> > 
> > When responding, please keep the subject line intact and reply to all
> > email addresses included in the To and CC lines. (Feel free to cut
> > this introductory paragraph, however.)
> > 
> > 
> > Please refer to https://www.ietf.org/iesg/statement/discuss-
> > criteria.html
> > for more information about DISCUSS and COMMENT positions.
> > 
> > 
> > The document, along with other ballot positions, can be found here:
> > https://datatracker.ietf.org/doc/draft-ietf-core-new-block/
> > 
> > 
> > 
> > ---------------------------------------------------------------------
> > -
> > DISCUSS:
> > ---------------------------------------------------------------------
> > -
> > 
> > I have a concern about the MAX_PAYLOADS congestion-control parameter.
> > In Section 7.2 it is stated that both endpoints only SHOULD have the
> > same value.  I don't see how this can be anything less than MUST,
> > given that we attribute semantics to whether NUM modulo MAX_PAYLOADS
> > is zero or non-zero in the processing of the Q-Block2 option.  If the
> > endpoints disagree on the value of MAX_PAYLOADS they will disagree on
> > the semantics of Q-Block2 -- how can that be interoperable?
> > (Being able to negotiate the value does not seem inherently
> > problematic, but since it is relevant for protocol semantics it seems
> > like the value must be identical on both endpoints.) This seems
> > especially important to have clarity on given that the current
> > specification allows for MAX_PAYLOADS to be decreased at runtime in
> > response to congestion feedback over a 24-hour period, with no
> > synchronization between peers provided ("Note that the CoAP peer will
> > not know about the MAX_PAYLOADS change until it is reconfigured".)
> > 
> > 
> 
> Implementation testing with MAX_PAYLOAD being different in the peers showed things worked badly (unnecessary timeouts/recovery) but continued to work. 
> 
> If the peers were negotiating and installing a new common value (application layer), there was a brief period of instability.

I still think there's a fundamental mismatch between "this is a parameter
that is part of the protocol and part of request/response semantics" and
"this is a parameter that each endpoint is allowed to vary unilaterally".
This holds even if we attempt to set things up so in the former case the
behavior degrades somewhat gracefully, with one direction transmitting as
expected and the other only making slow progress.

If this was just a local parameter for how to batch and pace transmissions,
that would be fine, and letting it vary (even if not recommended) between
peers would also be fine.

But the -12 currently has text like (§4.4):

   In a request for any block number, the M bit unset indicates the
   request is just for that block.  If the M bit is set, this has
   different meanings based on the NUM value:

   NUM is zero:  This is a request for the entire body.

   'NUM modulo MAX_PAYLOADS' is zero, while NUM is not zero:  This is
      used to confirm that the current MAX_PAYLOADS_SET (the latest
      block having block number NUM-1) has been successfully received
      and that, upon receipt of this request, the server can continue to
      send the next MAX_PAYLOADS_SET (the first block having block
      number NUM).  This is the 'Continue' Q-Block-2 and conceptually
      has the same usage (i.e., continue sending the next set of data)
      as the use of 2.31 (Continue) for Q-Block1.

   Any other value of NUM:  This is a request for that block and for all
      of the remaining blocks in the current MAX_PAYLOADS_SET.

That is, the semantics of a request are supposed to depend on the value of
MAX_PAYLOADS.  The semantics of what the sender sent can disagree with the
semantics of what the receiver interprets, if there is skew in this value.

Now, maybe it happens that things work out okay if the client and server
have different MAX_PAYLOADS values, in that setting the M bit does not have
to be tied to a specific MAX_PAYLOADS_SET and rather means "give me a chunk
of up to your max-transmit-window of payloads", and the receiver's value
determines whether the "up to" means a full window's worth or not, but
that's not what we say it means.  For Proposed Standards, I think we need
to be accurate about what the semantics actually are, not just what they
are in the ideal case, if the MAX_PAYLOADS value is allowed to vary between
peers.  And if skew is allowed, we need to say that behavior degrades in
case of skew and how, but that progress will still be made.

(Even if the semantics actually are "give me a chunk of up to your
max-window", things can still end up behaving badly if the skew is quite
significant -- if I think I'm asking for a window of up to 10 blocks but I
get back 100, I may not be able to handle the full response.)


> Please note that this is a value that is unlikely to change: 1500 byte packets and MAX_PAYLOADS(10) gives a set of 15,000 bytes. With an up to NON_TIMEOUT (2 secs) delay, average data rates are of the order of 60kbps (1500 * 8 * 10 / 2), so providing the congested network can handle 60Kbps, all will be OK. If higher values, then the 'Continue' will come back quicker than the NON_TIMEOUT.

If it's unlikely to change, then what's wrong with making it a fixed
parameter (subject to change by out-of-band agreement)?

> 
> > ---------------------------------------------------------------------
> > -
> > COMMENT:
> > ---------------------------------------------------------------------
> > -
> > 
> > I made some editorial suggestions in a github pull request at
> > https://github.com/core-wg/new-block/pull/21 .  It seems that there
> > are now some merge conflicts; I cannot promise to have availability
> > to try to resolve them particularly quickly, but I can do so
> > "eventually" if needed.
> > 
> 
> Fixed the conflict. Went with many of your suggestions. Thanks.
> 
> > Section 1
> > 
> >    There is a requirement for these blocks of data to be transmitted
> > at
> >    higher rates under network conditions where there may be
> > asymmetrical
> >    transient packet loss (i.e., responses may get dropped).  An
> > example
> >    is when a network is subject to a Distributed Denial of Service
> >    (DDoS) attack and there is a need for DDoS mitigation agents
> > relying
> >    upon CoAP to communicate with each other (e.g.,
> >    [RFC8782][I-D.ietf-dots-telemetry]).  As a reminder, [RFC7959]
> > 
> > I suppose the RFC Editor will do the right thing about referencing
> > 8782 vs 8782bis ... and I am overdue in following up on the IETF LC
> > results for the latter :(
> > 
> > Section 2
> > 
> > We currently introduce the concept of MAX_PAYLOADs by implicit use in
> > a few places before it is actually given a proper definition.  I
> > wonder if mentioning here that it is used to group a batch of blocks
> > would help the reader.
> > 
> 
> Added the following in Section 2:
> 
> NEW:
>    MAX_PAYLOADS refers to the maximum number of blocks that can be sent
>    by an endpoint
> 
> > Section 3
> > 
> >    o  They support sending an entire body using Non-confirmable (NON)
> >       messages without requiring a response from the peer.
> > 
> > I put this change in my github PR, but repeating here for visibility
> > since I am making an assumption: I propose adding "intermediate" for
> > "without requiring an intermediate response from the peer".  My
> > understanding is that a final message indicaing successful receiption
> > is still used (or a selective ack in case of loss), so the contrast
> > to RFC
> > 7959 is in the (lack of) need for intermediate responses for each
> > block.
> > 
> 
> OK.
> 
> >    o  Mixing of NON and CON during requests/responses using Q-Block
> > is
> >       not supported.
> > 
> > There is perhaps subtle differences across "not supported",
> > "forbidden", and "not defined in this specification".  Do we perhaps
> > actually mean "forbidden"?
> 
> It is not allowed as per:  
> 
>    The transmission of all the blocks of a single body over an
>    unreliable transport MUST either all be Confirmable or all be Non-
>    confirmable.  This is meant to simplify the congestion control
>    procedure.
> 
> Saying "not supported" is OK in Section 3, though. 
> 
> > 
> > Section 3.2
> > 
> >    (DOTS) that cannot use CON responses to handle potential packet
> > loss
> >    and that support application-specific mechanisms to assess whether
> >    the remote peer is able to handle the messages sent by a CoAP
> >    endpoint (e.g., DOTS heartbeats in Section 4.7 of [RFC8782]).
> > 
> > Can we get greater clarity on what "able to handle" is intended to
> > mean?
> > I can't tell if it's anywhere between "the transport is able to
> > deliver message bodies" and "the software stack implements and
> > enables a particular feature".
> 
> The issues is to check that we are not overloading the remote peer. I made this change: 
> 
> s/able to handle/is not overloaded and is thus able to process

Thanks.

> > 
> > Section 4.1
> > 
> >    When the Content-Format Option is present together with the Q-
> > Block1
> >    or Q-Block2 Option, the option applies to the body not to the
> > payload
> >    (i.e., it must be the same for all payloads of the same body).
> > 
> > Do we have a normative requirement somewhere that the recipient track
> > and compare the content-format values across blocks?  If not, should
> > we?
> 
> No text is included because rfc7959#section-2.9.2 (which defines 4.08 and cited in the spec) already covers that check. 
> 
> Likewise, the client side is also covers in RFC7959: 
> 
>    If blocks of a
>    response body arrive with different Content-Format Options, it is up
>    to the client how to handle this error (it will typically abort any
>    ongoing block-wise transfer). 

Okay, thanks for the pointer (that part of 7959 is not where I would have
started looking, so I appreciate the help).

> As a reminder, we do say: 
> 
>    The deviations from Block1 and Block2 Options are specified in
>    Section 4.
> 
> > 
> >    Q-Block2 Option is useful with GET, POST, PUT, FETCH, PATCH, and
> >    iPATCH requests and their payload-bearing responses (2.01, 2.02,
> >    2.03, 2.04, and 2.05) (Section 5.5 of [RFC7252]).
> > 
> > Do we need an "e.g." in front of the list, to account for the
> > potential future registration of new payload-bearing response codes?
> 
> Agree. Please note that this is the reason why we are using "or similar" in other sections. 
> 
> > 
> >    If Q-Block1 Option is present in a request or Q-Block2 Option in a
> >    response (i.e., in that message to the payload of which it
> > pertains),
> > 
> > Can we reword this parenthetical in a less convoluted way?  I'm not
> > even sure I'm parsing it properly.
> 
> Changed to: 
> 
> "If the Q-Block1 Option is present in a request or the Q-Block2 Option is returned in a response,..."

Thank you!

> > 
> >    [RFC7252]).  To reliably get a rejection message, it is therefore
> >    REQUIRED that clients use a Confirmable message for determining
> >    support for Q-Block1 and Q-Block2 Options.
> > 
> > (I know that some other discussion happened on this mechanism, but I
> > forget if there are already plans to add a clarification that this is
> > only needed once per peer within a given set of exchanges.)
> 
> We have added the following to address some other comments :
> 
> " ...apart from the
>    initial Q-Block functionality negotiation. "
>    ^^^^^^^^
>  
> > 
> >    The Q-Block2 Option is repeatable when requesting retransmission
> > of
> >    missing blocks, but not otherwise.  Except that case, any request
> >    carrying multiple Q-Block1 (or Q-Block2) Options MUST be handled
> >    following the procedure specified in Section 5.4.5 of [RFC7252].
> > 
> > Since these are critical options, the referenced procedures involve
> > rejecting the message, right?  Is that important enough to note
> > directly?
> 
> Section 5.4.5 of 7252 includes a pointer to 5.4.1. I don't think a change is needed.  
> 
> > 
> >    Note that if Q-Block1 or Q-Block2 Options are included in a packet
> > as
> >    Inner options, Block1 or Block2 Options MUST NOT be included as
> > Inner
> >    options.  Similarly there MUST NOT be a mix of Q-Block and Block
> > for
> >    the Outer options.  [...]
> > 
> > (Hopefully a silly question, but do we make the analogous prohibition
> > against combining Q-Block and regular Block for non-OSCORE cases
> > anywhere?  I thought we did, but now I can't find it...)
> 
> The outer wrapper covers this in our opinion: if there is no OSCORE option in the outer wrapper, then there is no inner data. 
> 
> Also, we do already have: 
> 
>    If the new options are not supported by a peer, then
>    transmissions can fall back to using Block1 and Block2 Options
> 
> That's said we can be more explicit.
> 
> > 
> > Section 4.3
> > 
> >    being transferred.  The Request-Tag is opaque, the server still
> >    treats it as opaque but the client MUST ensure that it is unique
> > for
> >    every different body of transmitted data.
> > 
> > (nit) the structure of this sentence seems off, to me.  I may just
> > want a comma after "server still treats it as opaque", but looking
> > more closely I might rewrite to more like "The Request-Tag is opaque
> > to the server, but the client MUST ensure that it is unique for every
> > different request body being transmitted".
> > 
> 
> OK
> 
> >       Implementation Note: It is suggested that the client treats the
> >       Request-Tag as an unsigned integer of 8 bytes in length.  An
> >       implementation may want to consider limiting this to 4 bytes to
> >       reduce packet overhead size.  The initial Request-Tag value
> > should
> >       be randomly generated and then subsequently incremented by the
> >       client whenever a new body of data is being transmitted between
> >       peers.
> > 
> > In the vein of draft-gont-numeric-ids-sec-considerations, is the
> > increment necessarily 1 or can there be gaps?  Similarly, the risk of
> > information disclosure (via side channel) is reduced if the initial
> > random value is generated anew for each connection.  This is maybe
> > implied by the current text but could be stated more clearly.
> 
> We don't say "monotonically incremented" or specify an increment unit. So, yes there can be random gaps assuming the resultant values must not clash though.
> 
> Note that the initial value is randomly generated:
>  
>       reduce packet overhead size.  The initial Request-Tag value should
>       be randomly generated and then subsequently incremented by the
>          ^^^^^^^^ 
>       client whenever a new body of data is being transmitted between
>       peers.
> 
> > 
> >    The client MUST send the payloads with the block numbers
> > increasing,
> >    starting from zero, until the body is complete (subject to any
> >    congestion control (Section 7)).  Any missing payloads requested
> > by
> >    the server must in addition be separately transmitted with
> > increasing
> >    block numbers.
> > 
> > When I first read this, I thought that the block numbers of
> > retransmissions needed to continue to increase in the same sequence
> > as the original transmission, i.e., retransmitted blocks are assigned
> > new block numbers.  The examples do not bear this out (and it seems
> > like it would be complicated to specify clearly), so I suggest
> > rephrasing to "in order of increasing block number".
> 
> ACK.
> 
> > 
> >       If the FETCH request includes the Observe Option, then the
> > server
> >       MUST use the same token as used for the initial response for
> >       returning any Observe triggered responses so that the client
> > can
> >       match them up.
> > 
> >       The client should then release all of the tokens used for this
> >       body unless a resource is being observed.
> > 
> > If a resource is being observed, should the client release all the
> > other tokens (than the one used for the initial response)?
> 
> There is no reason why the client cannot release all the other tokens.
> 
> > 
> > Also, is the "initial response" the first response for the blockwise
> > transfer (which might be a 2.31 or 4.08 for NON requests), or the
> > first one with response code 2.05?
> 
> The "initial all data received response" - i.e. 2.01 or 2.05.

I'd suggest adding that clarification.

> > 
> >    2.31 (Continue)
> > 
> >       This Response Code can be used to indicate that all of the
> > blocks
> >       up to and including the Q-Block1 Option block NUM (all having
> > the
> >       M bit set) have been successfully received.  The token used
> > MUST
> >       be one of the tokens that were received in a request for this
> >       block-wise exchange.  However, it is desirable to provide the
> > one
> >       used in the last received request.
> > 
> > Can the client release any tokens upon receipt of such a response?
> 
> Yes.  If going with the Implementation Note #6., the lower 32 bits tracker must not be released.

Since we mention releasing tokens for other response codes, it would feel
natural to mention it here as well, to me.

> > 
> >    4.02 (Bad Option)
> > 
> >       This Response Code MUST be returned for a Confirmable request
> > if
> >       the server does not support the Q-Block Options.  Note that a
> >       reset message must be sent in case of Non-confirmable request.
> > 
> > Reset only needs to be sent if the server is not ignoring the request
> > entirely, though, right?
> 
> The request will be rejected. Please see: 
> 
>    A Non-confirmable
>    message with an unrecognized critical option is either rejected with
>    a Reset message or just silently ignored (Sections 5.4.1 and 4.3 of
>    [RFC7252]).

I don't think I understand.  "either rejected ... or just silently ignored"
says that "just silently ignore" is a valid way to handle such a message.
So, if "silently ignore" is a valid option for the server, I don't see how
we can say "must be sent" without deviating from RFC 7252.

> Hence the Q-Block availability check must be a CON.
> 
> > 
> > 
> > %%%
> > The following few comments are interrelated:
> > 
> >       This Response Code returned with Content-Type "application/
> >       missing-blocks+cbor-seq" indicates that some of the payloads
> > are
> >       missing and need to be resent.  The client then retransmits the
> >       missing payloads using the same Request-Tag, Size1 and Q-Block1
> > to
> >       specify the block NUM, SZX, and M bit as appropriate.
> > 
> > The new 'M' bit is "as appropriate" for the new flight of messages,
> > or as was sent initially?  (The examples in §10.x suggest "as was
> > sent
> > initially".)
> > 
> >       The Request-Tag value to use is determined by taking the token
> > in
> >       the 4.08 (Request Entity Incomplete) response, locating the
> >       matching client request, and then using its Request-Tag.
> > 
> > The "value to use" here seems to be indicating the value to use in
> > the retransmitted request...
> 
> Yes. 
> 
> > 
> >       The token used MUST be one of the tokens that were received in
> > a
> >       request for this block-wise exchange.  However, it is desirable
> > to
> >       provide the one used in the last received request.  See Section
> > 5
> >       for further information.
> > 
> > ... but here the "token used" seems to be indicating the token to be
> > used in constructing the response that has response code 4.08.
> 
> This is to token to be used to send the response code: s/token user/token to use
> 
> > 
> > If my understanding is correct, we really should have more clarity on
> > which value is "used" for which message.
> > 
> > Additionally, in the last quoted paragraph we refer to Section 5 for
> > further information, which includes a SHOULD-level requirement to
> > "provide the [token] used in the last received request".  It is very
> > surprising to have the normative requirements for behavior split
> > across sections in this manner.  (Or was the intent that Section 5
> > also use the "desirable" wording?) %%%
> 
> "desirable" is more appropriate. Fixed. 
> 
> > 
> > Section 4.4
> > 
> >    The ETag is opaque, the client still treats it as opaque but the
> >    server MUST ensure that it is unique for every different body of
> >    transmitted data.
> > 
> > [analogous comment as for Request-Tag]
> 
> OK
> 
> > 
> >       Implementation Note: It is suggested that the server treats the
> >       ETag as an unsigned integer of 8 bytes in length.  An
> >       implementation may want to consider limiting this to 4 bytes to
> >       reduce packet overhead size.  The initial ETag value should be
> >       randomly generated and then subsequently incremented by the
> > server
> >       whenever a new body of data is being transmitted between peers.
> > 
> > [analogous comment as for Request-Tag]
> 
> Noted. 
> 
> > 
> >    The client SHOULD wait for up to NON_RECEIVE_TIMEOUT (Section 7.2)
> >    after the last received payload for NON payloads before issuing a
> >    GET, POST, PUT, FETCH, PATCH, or iPATCH request that contains one
> > or
> >    more Q-Block2 Options that define the missing blocks with the M
> > bit
> >    unset.  The client MAY set the M bit to request this and later
> > blocks
> >    from this MAX_PAYLOADS set.  Further considerations related to the
> >    transmission timing for missing requests are discussed in
> >    Section 7.2.
> > 
> > Does the MAY grant permission to send with M bit set prior to
> > NON_RECEIVE_TIMEOUT, or just permission to send with M bit set in
> > addition to with M bit unset (but still after the timeout)?
> 
> If the missing blocks are a complete set from last received block, then the MAY allows the M bit to be set indicating 'the rest' as opposed for defining each one that is missing in the request.  It was not intended as being an alternative to "The client SHOULD wait".
> 
> > 
> >    For Confirmable responses, the client continues to acknowledge
> > each
> >    packet.  Typically, the server acknowledges the initial request
> > using
> >    an ACK with the payload, and then sends the subsequent payloads as
> >    CON responses.  The server will detect failure to send a packet,
> > but
> >    the client can issue, after a MAX_TRANSMIT_SPAN delay, a separate
> >    GET, POST, PUT, FETCH, PATCH, or iPATCH for any missing blocks as
> >    needed.
> > 
> > Starting out with "for confirmable responses" implies that we're
> > going to separately cover non-confirmable responses later, or at some
> > point transition to statements of general applicability (to both
> > confirmable and non-confirmable responses).  Where does that happen?
> 
> It is in this section. The text you quoted is what is specific to CON responses. 

I would have preferred a more explicit signal that the subsequent
paragraphs apply to both the confirmable and non-confirmable cases, but the
only option I'm coming up with at the moment ("for both confirmable and
non-confirmable responses" to start the subsequent paragraph") is not
great.

> > 
> >    A client SHOULD maintain a partial body (missing payloads) for up
> > to
> >    NON_PARTIAL_TIMEOUT (Section 7.2) or as defined by the Max-Age
> > Option
> >    (or its default of 60 seconds (Section 5.6.1 of [RFC7252])),
> >    whichever is the less.  On release of the partial body, the client
> >    should then release all of the tokens used for this body unless a
> >    resource is being observed.
> > 
> > [as above, can the client release any subset of tokens in the case of
> > observe?]
> 
> Yes. 
> 
> > 
> >    It is RECOMMENDED that the server maintains a cached copy of the
> > body
> >    when using the Q-Block2 Option to facilitate retransmission of any
> >    missing payloads.
> > 
> > It's surprising to write that the client SHOULD but it is RECOMMENDED
> > that the server cache, when those two requirements keywords have an
> > equivalent strength per BCP 14.  Can't we used consistent terminology
> > for the same requirement level?
> 
> Sure.  
> 
> > 
> >    If the server detects part way through a body transfer that the
> >    resource data has changed and the server is not maintaining a
> > cached
> >    copy of the old data, then the transmission is terminated.  Any
> >    subsequent missing block requests MUST be responded to using the
> >    latest ETag and Size2 Option values with the updated data.
> > 
> > This sounds like the server starts responding "in the middle" of the
> > new representation, so the client would need to go back and re-
> > request the initial parts, possibly across multiple groups of
> > MAX_PAYLOADS blocks.
> 
> This is only needed if the server does not cache a response and the resource changes mid-flight. The Etag + possible SIZE2 change indicate the resource has hanged and needs to be re-requested from the start (block 0).
> 
> > It seems like this requirement for client behavior should be more
> > clearly documented somewhere.  We do go on to talk about the client
> > removing the stale partial body, but not about completing the new
> > body.
> > 
> > Section 4.5
> > 
> >    For a response that uses Q-Block2, the Observe value MUST be the
> > same
> >    for all the payloads of the same body.  This is different from
> > Block2
> >    usage where the Observe value is only present in the first block
> >    (Section 3.4 of [RFC7959]).  This includes payloads transmitted
> >    following receipt of the 'Continue' Q-Block2 Option (Section 4.4)
> > by
> >    the server.  If a missing payload is requested by a client, then
> > both
> >    the request and response MUST NOT include the Observe Option.
> > 
> > (side note?) It seems very surprising to omit Observe from only
> > retransmitted payloads but keep it in all initial payload
> > transmissions.
> 
> This is different from Block2 in the (first and observe triggered additional) response is that block#0 (and other blocks) may not arrive, so the Observe Option value needs to be in all the packets (including a subsequent MAX_PAYLOADS chunk as the first MAX_PAYLOADS set may not have arrived).  Once the client has at least one packet with Observe, then the requesting of missing packets does not need the Observe) as Block2 does not need it when asking for the next packet.

Ah, thanks for the explanation.

> > 
> > Section 4.6
> > 
> >    The Size1 or Size2 option values MUST exactly represent the size
> > of
> >    the data on the body so that any missing data can easily be
> >    determined.
> > 
> > Is this MUST duplicating the behavior already specified by RFC 7959?
> 
> No, 7959 discusses "size estimates" or "size indication". 
> 
> > 
> > Section 5
> > 
> >    The data payload of the 4.08 (Request Entity Incomplete) response
> > is
> >    encoded as a CBOR Sequence [RFC8742].  It comprises of one or more
> > 
> > I think we want some qualifying text that reaffirms that the behavior
> > being described is applicable only to the application/missing-
> > blocks+cbor-seq content-type case, possibly by having the previous
> > discussion state that "this section defines the behavior and
> > semantics for 4.08 responses using the new content-type."
> 
> Why is this needed?

The data payload of the 4.08 response as specified in RFC 7959 is not a
CBOR sequence.  The current wording and paragraph structure doesn't provide
a clear connection between the new content-type (as mentioned in the first
sentence of the section) and this text, so a reader might misread this text
(which has no normative keywords) as describing the preexisting situation
for 4.08 responses, and that is not correct.  Even just adding the word
"new" ("data payload of the new 4.08 (Request Entity Incomplete) response")
would help clarify that the description is of the new behavior, not the
preexisting behavior.

> > 
> >    The Concise Data Definition Language [RFC8610] (and see Section
> > 4.1
> >    [RFC8742]) for the data describing these missing blocks is as
> >    follows:
> > 
> > (Should we mention that this is only informational and that the prose
> > description is normative, in line with RFC 8610 being only an
> > informative reference?)
> 
> I'm not sure this is needed. What is authoritative is the CBOR sequence. 

In my experience, it's quite common for documents to specifically say
whether the protocol-description-language version or the prose version is
authoritative in case of conflict.  Hopefully there is no conflict, but to
have a clear and unambiguous specification, we have to say which version to
use if there is a conflict.

> > 
> >          ; A notional array, the elements of which are to be used
> >          ; in a CBOR Sequence:
> > 
> > (nit) Is there a reason to use a different wording than the
> > referenced example from RFC 8742?
> 
> No. 
> 
> > 
> > Section 6
> > 
> >    Implementation Note:  By using 8-byte tokens, it is possible to
> >       easily minimize the number of tokens that have to be tracked by
> >       clients, by keeping the bottom 32 bits the same for the same
> > body
> >       and the upper 32 bits containing the current body's request
> > number
> >       (incrementing every request, including every re-transmit).
> > This
> >       allows the client to be alleviated from keeping all the per-
> >       request-state, e.g., in Section 3 of [RFC8974].
> > 
> > If we're going to introduce structure into a nominally opaque
> > identifier, we need to discuss the consequences of that in the
> > security considerations.  draft-gont-numeric-ids-sec-considerations
> > has some guidance in this regard.
> 
> It is all opaque to the server, the client is just using it to make sure his requests are unique. If one can access the token, it can access more critical information. I'm not sure there is much to say as the messages are not supposed to be sent on clear. Tampering of the token is thus very difficult given we are not using NoSec. 

It's only NOT RECOMMENDED to use NoSec, which means we still have to
accurately document the security considerations if NoSec is used.

> > 
> > Section 7.1
> > 
> >    Congestion control for CON requests and responses is specified in
> >    Section 4.7 of [RFC7252].  For faster transmission rates, NSTART
> > will
> >    need to be increased from 1.  However, the other CON congestion
> >    control parameters will need to be tuned to cover this change.
> > [...]
> > 
> > I thought there had been some discussion in a different AD's ballot
> > thread on this text, but I can't find it now.  I'm happy to defer to
> > the previous discussion if I'm not just imagining it.
> > Anyways, I might suggest phrasing this as "if faster transmission
> > rates are needed, NSTART will need to be increased from 1".
> 
> Changed to "In order to benefit from faster "
> 
> > 
> >    It is implementation specific as to whether there should be any
> >    further requests for missing data as there will have been
> > significant
> >    transmission failure as individual payloads will have failed after
> >    MAX_TRANSMIT_SPAN.
> > 
> > (editorial) I don't think I can successfully parse this sentence.
> > There may be a few missing words, and splitting into multiple
> > sentences would likely help as well.
> 
> Will update it. 
> 
> > 
> > Section 7.2
> > 
> >    NON_RECEIVE_TIMEOUT is the initial maximum time to wait for a
> > missing
> >    payload before requesting retransmission for the first time.
> > Every
> >    time the missing payload is re-requested, the time to wait value
> >    doubles.  The time to wait is calculated as:
> > 
> > Thank you for being very clear about the exponential backoff
> > procedure
> > :)
> > 
> >    payloads to prevent the client unnecessarily delaying.  If not all
> > of
> >    the MAX_PAYLOADS payloads were received, the server SHOULD delay
> > for
> >    NON_RECEIVE_TIMEOUT (exponentially scaled based on the repeat
> > request
> >    count for a payload) before sending the 4.08 (Request Entity
> >    Incomplete) Response Code for the missing payload(s).  If this is
> > a
> >    repeat for the 2.31 (Continue) response, the server SHOULD send a
> >    4.08 (Request Entity Incomplete) response detailing the missing
> >    payloads after the block number that would have been indicated in
> > the
> >    2.31 (Continue).  [...]
> > 
> > I don't understand what "if this is a repeat for the 2.31 (Continue)
> > response" is intended to mean.
> 
> Does this help?
> 
> OLD
>    If this is a
>    repeat for the 2.31 (Continue) response, the server SHOULD send a
>    4.08 (Request Entity Incomplete) response detailing the missing
>    payloads after the block number that would have been indicated in the
>    2.31 (Continue).
> 
> NEW
>    
>    If all of the
>    MAX_PAYLOADS were received and a 2.31 (continue) had been sent, but
>    no more payloads were received for NON_RECEIVE_TIMEOUT (exponentially
>    scaled), the server SHOULD send a 4.08 (Request Entity Incomplete)
>    response detailing the missing payloads after the block number that
>    was indicated in the sent 2.31 (Continue).

Yes; thank you!

> > 
> >    The client does not need to acknowledge the receipt of the entire
> >    body.
> > 
> > Does that mean that the last group of response blocks will always be
> > retransmitted NON_MAX_RETRANSMIT times?
> 
> No. The server won't retransmit if not request is received from the client.   

Okay.  I assume I got confused about this at some point.

> > 
> > Section 10
> > 
> >             QB1: Q-Block1 Option values NUM/More/SZX
> >             QB2: Q-Block2 Option values NUM/More/SZX
> > 
> > What's depicted in the figure seems to be the actual block size, and
> > not the three-bit SZX value.
> 
> It is actual size. Clarified in the text. 
> 
> > 
> > Section 10.1.3
> > 
> > Should we indicate somehow in Figure 6 that the 4.08 responses use
> > the new content-format?
> > 
> > Also, is there any value in indicating that there might be a race
> > between the client continuing to send the next set of payloads and
> > the initial 4.08 response?
> 
> Added a pointer to the section where 4.08 is defined. 
> 
> > 
> > Section 10.2.3
> > 
> > I don't understand why the NON_RECEIVE_TIMEOUT (client) triggers --
> > shouldn't the delivery of the 11th block indicate that the server
> > thinks it sent a full MAX_PAYLOADS group and thus a selective ACK,
> > after perhaps just a modest reordering delay?
> 
> Fair question for the first the NON_RECEIVE_TIMEOUT.  We handle this for Q-Block1 (Figure 6).  We do have:
> 
>    It is likely that the server will start transmitting the next set of
>    MAX_PAYLOADS payloads before the client times out on waiting for the
>    last of the previous MAX_PAYLOADS payloads.  Upon receipt of the
>    first payload from the new set of MAX_PAYLOADS payloads, the client
>    SHOULD send a request indicating any missing payloads from any
>    previous set of MAX_PAYLOADS payloads.  Upon receipt of such request,
>    the server SHOULD send the missing payloads before continuing to send
>    the remainder of the MAX_PAYLOADS payloads and then go into another
>    NON_TIMEOUT delay prior to sending the next set of payloads.
> 
> OLD:
>        [[NON_TIMEOUT (server) delay expires]]
>           |     [[Server sends next set of payloads]]
>           |<---------+ NON 2.05 M:0xab T:0xf0 O:1236 ET=23 QB2:10/0/1024
>           |   ...    |
>        [[NON_RECEIVE_TIMEOUT (client) delay expires]]
>           |     [[Client realizes blocks are missing and asks for the
>           |       missing ones in one go]]
>           +--------->| NON GET /path M:0x04 T:0xf3 QB2:1/0/1024\
>           |          |                             QB2:9/0/1024
>           |     X<---+ NON 2.05 M:0xac T:0xf3 ET=23 QB2:1/1/1024 
> 
> NEW:
>        [[NON_TIMEOUT (server) delay expires]]
>           |     [[Server sends next set of payloads]]
>           |<---------+ NON 2.05 M:0xab T:0xf0 O:1236 ET=23 QB2:10/0/1024
>           |     [[On seeing a payload from the next set of payloads, 
>           |      Client realizes blocks are missing and asks for the
>           |       missing ones in one go]]
>           +--------->| NON GET /path M:0x04 T:0xf3 QB2:1/0/1024\
>           |          |                             QB2:9/0/1024
>           |     X<---+ NON 2.05 M:0xac T:0xf3 ET=23 QB2:1/1/1024

Thanks.

> 
> > 
> > Section 10.3.2
> > 
> >    [[MAX_PAYLOADS has been reached]]
> >       |     [[MAX_PAYLOADS blocks acknowledged by client using
> >       |       'Continue' Q-Block2]]
> >       +--------->| NON FETCH /path M:0x3b T:0xab QB2:10/1/1024
> >       |<---------+ NON 2.05 M:0x8b T:0xaa O:1334 ET=21 QB2:10/0/1024
> > 
> > Shouldn't the server switch to using T:0xab now?
> 
> This is a 'Continue' as per:
> 
>    The server SHOULD recognize the 'Continue' Q-Block2 request as a
>    continue request and just continue the transmission of the body
>    (including Observe Option, if appropriate for an unsolicited
>    response) rather than as a request for the remaining missing blocks.
> 
> So token does not change. We can add it the text. 

Thanks for adding the text.

> > 
> >       +--------->| NON FETCH /path M:0x3c T:0xac QB2:10/1/1024
> >       |<---------+ NON 2.05 M:0x96 T:0xaa O:1335 ET=22 QB2:10/0/1024
> > 
> > and 0xac here?
> 
> Idem as previous one. 
> 
> > 
> > Section 10.3.3
> > 
> >           |<---------+ NON 2.05 M:0xa6 T:0xc6 ET=23 QB2:3/0/1024
> >           |   ...    |
> >        [[NON_RECEIVE_TIMEOUT (client) delay expires]]
> > 
> > Why does the client time out here (at least with the full
> > NON_RECEIVE_TIMEOUT); the final-message indication seems like it
> > would allow for an ~immediate response (delayed only for some
> > reordering threshold)?
> > 
> 
> We need to observe a delay here in case there is any packet arrival re-ordering. We decided at the time to not have another yet extra timer to cover this. Please note that this is an optimization and even with the current design, the performance is much better than the legacy block-wise.  

Okay.

Thanks for the updates in the -12 and all the responses here; I did not
comment on the ones that were clear, but appreciate them nonetheless.

-Ben