From nobody Wed May 10 19:00:22 2023 Return-Path: X-Original-To: ippm@ietfa.amsl.com Delivered-To: ippm@ietfa.amsl.com Received: from localhost (localhost [127.0.0.1]) by ietfa.amsl.com (Postfix) with ESMTP id C27F2C1F65D2 for ; Wed, 10 May 2023 19:00:21 -0700 (PDT) X-Virus-Scanned: amavisd-new at amsl.com X-Spam-Flag: NO X-Spam-Score: -4.397 X-Spam-Level: X-Spam-Status: No, score=-4.397 tagged_above=-999 required=5 tests=[BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_ZEN_BLOCKED_OPENDNS=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, URIBL_DBL_BLOCKED_OPENDNS=0.001, URIBL_ZEN_BLOCKED_OPENDNS=0.001] autolearn=ham autolearn_force=no Authentication-Results: ietfa.amsl.com (amavisd-new); dkim=pass (2048-bit key) header.d=apple.com Received: from mail.ietf.org ([50.223.129.194]) by localhost (ietfa.amsl.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 631GXfEHZpn7 for ; Wed, 10 May 2023 19:00:18 -0700 (PDT) Received: from ma-mailsvcp-mx-lapp02.apple.com (ma-mailsvcp-mx-lapp02.apple.com [17.32.222.23]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by ietfa.amsl.com (Postfix) with ESMTPS id EA7FFC1F65C1 for ; Wed, 10 May 2023 19:00:17 -0700 (PDT) Received: from rn-mailsvcp-mta-lapp02.rno.apple.com (rn-mailsvcp-mta-lapp02.rno.apple.com [10.225.203.150]) by ma-mailsvcp-mx-lapp02.apple.com (Oracle Communications Messaging Server 8.1.0.22.20230228 64bit (built Feb 28 2023)) with ESMTPS id <0RUH00QIA1KGPH30@ma-mailsvcp-mx-lapp02.apple.com> for ippm@ietf.org; Wed, 10 May 2023 19:00:17 -0700 (PDT) X-Proofpoint-ORIG-GUID: z4xayUDIaJTIBD7SFLrbFWvelZoxEFrT X-Proofpoint-GUID: z4xayUDIaJTIBD7SFLrbFWvelZoxEFrT X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.573, 18.0.942 definitions=2023-05-10_04:2023-05-05, 2023-05-10 signatures=0 X-Proofpoint-Spam-Details: rule=interactive_user_notspam policy=interactive_user score=0 adultscore=0 spamscore=0 bulkscore=0 malwarescore=0 mlxlogscore=999 suspectscore=0 phishscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2304280000 definitions=main-2305100144 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=RB4k55WZ1ocALcW2gBhQuKCqBUkSWGzQcGbLqN2TRdc=; b=Q0RaWLgofRWCgT/ocbBR4x+JuUg+mlF9SubvdawG8Q6YDIRyCALk757xsmpt//i/gK86 GWQ+H9M7hMxt4F0miSBTKjbK/XCVAPr5j6ib1r1Hf7yh0hp2VSLB2zpyJ2FQZf2w6wsY pqUZICN4UlfbM0hr6bBqBK6yl+7znCmW0OPNYq3/z3EiJAiltEOf0GRS+Cqz493oAxcV T6TL1ykpvvNA5gffdibzQ0xEOpz4xZFxUfDmqbcqDxCB2L1H4smh/nYJHR9oerXMB4ci j2+w6VZ/C0gi+OZQJ3oGf+iskIjbhTheP6H3pY0YXrgaoJeAADTJdgqNu/sOshSQ3wqP nQ== Received: from rn-mailsvcp-mmp-lapp04.rno.apple.com (rn-mailsvcp-mmp-lapp04.rno.apple.com [17.179.253.17]) by rn-mailsvcp-mta-lapp02.rno.apple.com (Oracle Communications Messaging Server 8.1.0.22.20230228 64bit (built Feb 28 2023)) with ESMTPS id <0RUH00XWH1KG3BH0@rn-mailsvcp-mta-lapp02.rno.apple.com>; Wed, 10 May 2023 19:00:16 -0700 (PDT) Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp04.rno.apple.com by rn-mailsvcp-mmp-lapp04.rno.apple.com (Oracle Communications Messaging Server 8.1.0.22.20230228 64bit (built Feb 28 2023)) id <0RUH009001C48400@rn-mailsvcp-mmp-lapp04.rno.apple.com>; Wed, 10 May 2023 19:00:16 -0700 (PDT) X-Va-A: X-Va-T-CD: 7c41845d202abaa5133654699ccc487a X-Va-E-CD: 40482a9204de29ea59e7976fd75ec058 X-Va-R-CD: 8295a2a75553fc1869ef1ce2ff6d2036 X-Va-ID: 5fb461c1-1525-4584-a7c9-6a95ab1eb790 X-Va-CD: 0 X-V-A: X-V-T-CD: 7c41845d202abaa5133654699ccc487a X-V-E-CD: 40482a9204de29ea59e7976fd75ec058 X-V-R-CD: 8295a2a75553fc1869ef1ce2ff6d2036 X-V-ID: dbefa822-65ad-4d01-81df-3699f9e5c2d8 X-V-CD: 0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.573, 18.0.942 definitions=2023-05-10_04:2023-05-05, 2023-05-10 signatures=0 Received: from smtpclient.apple (unknown [17.234.39.68]) by rn-mailsvcp-mmp-lapp04.rno.apple.com (Oracle Communications Messaging Server 8.1.0.22.20230228 64bit (built Feb 28 2023)) with ESMTPSA id <0RUH00UG11KEYS00@rn-mailsvcp-mmp-lapp04.rno.apple.com>; Wed, 10 May 2023 19:00:15 -0700 (PDT) From: Tommy Pauly Message-id: <3103CBFB-5112-4FAC-A2F0-5209F52AB288@apple.com> Content-type: multipart/alternative; boundary="Apple-Mail=_A05AF089-05E6-4312-961B-17570F299D10" MIME-version: 1.0 (Mac OS X Mail 16.0 \(3755.100.3\)) Date: Wed, 10 May 2023 17:46:20 -0700 In-reply-to: Cc: IETF IPPM WG , QUIC WG To: Lucas Pardue References: X-Mailer: Apple Mail (2.3755.100.3) Archived-At: Subject: Re: [ippm] QUIC concerns relating to draft-ietf-ippm-explicit-flow-measurements X-BeenThere: ippm@ietf.org X-Mailman-Version: 2.1.39 Precedence: list List-Id: IETF IP Performance Metrics Working Group List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 May 2023 02:00:21 -0000 --Apple-Mail=_A05AF089-05E6-4312-961B-17570F299D10 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 The document is describing how bits *could* be used, not actually = suggesting reserving specific ones at this point. This same feedback did come up in IESG review, and I believe that there = will be an update to remove details that refer to specific bit ranges to = avoid this danger. I=E2=80=99ll let the authors chime in more =3D) Tommy > On May 10, 2023, at 2:41=E2=80=AFPM, Lucas Pardue = wrote: >=20 > Hi IPPM, >=20 > For the purpose of this email, I'm speaking as an individual. I'm = cross posting to the QUIC mailing list for visibility of others that = might also be interested. The TL;DR is the draft -03 contains text that = indicates how visible bits of QUIC packets could be used and that raises = several concerns that I think need to be addressed before this document = can progress further. >=20 > draft-ietf-ippm-explicit-flow-measurements is fairly far progressed = through its IETF journey. To drastically summarise it, it's about the = various forms of marking bits in packets that can be used for = performance measurements. And that requires collaboration between client = and server in order to expose some information that on-path observers = might use. >=20 > This might seem familiar to the QUIC WG. We have a spin bit formally = defined in version 1. And we last heard, explicitly in QUIC, about the = concepts presented in draft-ietf-ippm-explicit-flow-measurements in a = thread in 2021. The work was adopted to IPPM and progressed there and = that is all good. However, I've not been able to track this draft as = closely as I would have liked and I suspect some other QUIC stakeholders = may have missed it too. =20 >=20 > Other folks have raised some concerns during last call already and I'm = generally in agreement with them. However, for clarity of discussion = consider this a new thread that has some overlap and some non-overlap. >=20 > The major point that I think remains unaddressed by this document is = how bits in QUIC packets would _actually_ get used in a coordinated and = interoperable manner. The QUIC invariants draft, Section 5.2 [1] states = that short header packets consists of 7 Version-Specific Bits. Per 9000 = Section 17.3.1 [1], QUIC version 1 defines the 1-RTT short header packet = with the 7 bits allocated as so >=20 > Fixed Bit (1) =3D 1, > Spin Bit (1), > Reserved Bits (2), > Key Phase (1), > Packet Number Length (2), > Where the Reserved Bits have the requirements: >=20 > _The value included prior to protection MUST be set to 0. An endpoint = MUST treat receipt of a packet that has a non-zero value for these bits, = after removing both packet and header protection, as a connection error = of type PROTOCOL_VIOLATION._ >=20 > The Intro draft-ietf-ippm-explicit-flow-measurements states that: >=20 > _this document proposes adding a small number of dedicated measurement = bits to the clear portion of the transport protocol headers. These bits = can be added to an unencrypted portion of a transport-layer header, e.g. = UDP surplus space (see [UDP-OPTIONS = ] and [UDP-SURPLUS = ]) or reserved bits in a QUIC v1 header, as = already done with the latency Spin bit (see [QUIC-TRANSPORT = ])._ >=20 > This is problematic because you can't just change those Reserved Bits = at will and hope that independent implementations will interoperate. = Similarly you can't just substitute the Spin Bit as proposed in Section = 6. This really makes me question how this has been implemented and = tested to date because, to make it function properly and operate safely = on the Internet, you'd either need to have defined new versions of QUIC = or defined an extension. For an example of the latter, see RFC 9287 [3], = which describes a mechanism for repurposing the Fixed Bit. Yes, I = understand some people have trialled these out in a limited fashion but = I'm talking about how would an operator that is watching millions of = QUIC flows from heterogenous clients and servers know which ones are = using measurements bits and which aren't? >=20 > At the point that client and server need to negotiate a version or an = extension, there is now a coordination challenge with these on-path = observers. The QUIC manageability spec has some great text detailing the = considerations that apply here, this draft is sorely lacking a reference = to Section 3 of RFC 9312 [4], and also lacking any commentary on dealing = with those considerations here. (but maybe it shouldn't, see end of = email). >=20 > In addition to these concerns, I also find the protocol ossification = considerations in Section 5 to be awkward. It's not clear who the onus = is on to do anything because it talks about protocol designers and then = says implementations could decide to do something. It mentions "Latency = Spin bit greasing" in RFC 9000 but that term is not used by that name. = I'm going to presume you're alluding to Section 17.4 [5]. If so there = are two important separate aspects, disabling the spin bit on every 1 in = 16 paths or connection IDs, and randomizing values in the spin bit. The = reason this is awkward is because there are many different permutations = of bits described in the specification and operators need to somehow = guess i) what permutation is being used ii) if the endpoints have it = enabled and iii) how randomization affects analysis. >=20 > Finally, the draft consistently cites other QUIC documents (great!) = but lacks deep linking into the specific sections where you really want = the reader to go and focus. That places a lot of expectation on the = reader to read the right thing and do it. It would be helpful to add = more-specific links whereever possible. >=20 > All in all, these combination of concerns lead me to believe the = document in its current state is potentially unsafe, because it = endangers people picking it up literally and starting to write or read = bits without due coordination or feature detection. The confusion arises = because the intro implies this is a proposal to change bits but then = backs off that commitment and uses "coulds" etc. I think we need to be = crystal clear. Speak about some bits in the abstract and mention the few = protocol-specific concerns or related examples that each protocol = already has defined. However, remove the implications that we have well = thought out how to get these deployed for QUIC and punt that to future = work by explicitly stating so. I think this takes the form of removing = section 6, rewriting section 5 to speak about the higher-level problems = and possible approaches, and removing any straggling sentences about = reusing existing or reserved bits. >=20 > Cheers > Lucas >=20 > [1] - https://datatracker.ietf.org/doc/html/rfc8999#section-5.2 > [2] - https://datatracker.ietf.org/doc/html/rfc9000#section-17.3.1 > [3] - https://datatracker.ietf.org/doc/html/rfc9287.html > [4] - https://datatracker.ietf.org/doc/html/rfc9312#section-3 > [5] - https://datatracker.ietf.org/doc/html/rfc9000#section-17.4 --Apple-Mail=_A05AF089-05E6-4312-961B-17570F299D10 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 The document = is describing how bits *could* be used, not actually suggesting = reserving specific ones at this point.

This same = feedback did come up in IESG review, and I believe that there will be an = update to remove details that refer to specific bit ranges to avoid this = danger.

I=E2=80=99ll let the authors chime in = more =3D)

Tommy

On May 10, 2023, at 2:41=E2=80=AFPM, Lucas Pardue = <lucaspardue.24.7@gmail.com> wrote:

Hi = IPPM,

For the purpose of this email, I'm = speaking as an individual. I'm cross posting to the QUIC mailing list = for visibility of others that might also be interested. The TL;DR is the = draft -03 contains text that indicates how visible bits of QUIC packets = could be used and that raises several concerns that I think need to be = addressed before this document can progress = further.

draft-ietf-ippm-explicit-flow-measurements is fairly far = progressed through its IETF journey. To drastically summarise it, it's = about the various forms of marking bits in packets that can be used for = performance measurements. And that requires collaboration between client = and server in order to expose some information that on-path observers = might use.

This might seem familiar to the QUIC = WG. We have a spin bit formally defined in version 1. And we last heard, = explicitly in QUIC, about the concepts presented in = draft-ietf-ippm-explicit-flow-measurements in a thread in 2021. The work = was adopted to IPPM and progressed there and that is all good. However, = I've not been able to track this draft as closely as I would have liked = and I suspect some other QUIC stakeholders may have missed it too.  =

Other folks have raised some concerns = during last call already and I'm generally in agreement with them. = However, for clarity of discussion consider this a new thread that has = some overlap and some non-overlap.

The major point = that I think remains unaddressed by this document is how bits in QUIC = packets would _actually_ get used in a coordinated and interoperable = manner. The QUIC invariants draft, Section 5.2 [1] states that short = header packets consists of 7 Version-Specific Bits. Per 9000 Section = 17.3.1 [1], QUIC version 1 defines the 1-RTT short header packet with = the 7 bits allocated as so

  Fixed Bit (1) =3D 1,
  Spin Bit (1),
  Reserved Bits (2),
  Key Phase (1),
  Packet Number Length (2),
Where the Reserved Bits have = the requirements:

_The value included prior to = protection MUST be set to 0. An = endpoint MUST treat receipt of a packet that has a non-zero value for these bits, after = removing both packet and header protection, as a connection error of type PROTOCOL_VIOLATION._

The Intro = draft-ietf-ippm-explicit-flow-measurements states = that:

_this document proposes adding a small = number of dedicated measurement bits to the clear portion of the transport protocol headers. These bits can be = added to an unencrypted portion of a transport-layer header, e.g. UDP surplus = space (see [UDP-OPTIONS] and [UDP-SURPLUS]) or reserved bits in a QUIC v1 = header, as already done with the latency Spin bit (see [QUIC-TRANSPORT])._

This = is problematic because you can't just change those Reserved Bits at will = and hope that independent implementations will interoperate. Similarly = you can't just substitute the Spin Bit as proposed in Section 6. This = really makes me question how this has been implemented and tested to = date because, to make it function properly and operate safely on the = Internet, you'd either need to have defined new versions of QUIC or = defined an extension. For an example of the latter, see RFC 9287 [3], = which describes a mechanism for repurposing the Fixed Bit. Yes, I = understand some people have trialled these out in a limited fashion but = I'm talking about how would an operator that is watching millions of = QUIC flows from heterogenous clients and servers know which ones are = using measurements bits and which = aren't?

At the point that client and server = need to negotiate a version or an extension, there is now a coordination = challenge with these on-path observers. The QUIC manageability spec has = some great text detailing the considerations that apply here, this draft = is sorely lacking a reference to Section 3 of RFC 9312 [4], and also = lacking any commentary on dealing with those considerations here. (but = maybe it shouldn't, see end of email).

In = addition to these concerns, I also find the protocol ossification = considerations in Section 5 to be awkward. It's not clear who the onus = is on to do anything because it talks about protocol designers and then = says implementations could decide to do something. It mentions "Latency = Spin bit greasing" in RFC 9000 but that term is not used by that name. = I'm going to presume you're alluding to Section 17.4 [5]. If so there = are two important separate aspects, disabling the spin bit on every 1 in = 16 paths or connection IDs, and randomizing values in the spin bit. The = reason this is awkward is because there are many different permutations = of bits described in the specification and operators need to somehow = guess i) what permutation is being used ii) if the endpoints have it = enabled and iii) how randomization affects = analysis.

Finally, the draft consistently cites = other QUIC documents (great!) but lacks deep linking into the specific = sections where you really want the reader to go and focus. That places a = lot of expectation on the reader to read the right thing and do it. It = would be helpful to add more-specific links whereever = possible.

All in all, these combination of = concerns lead me to believe the document in its current state is = potentially unsafe, because it endangers people picking it up literally = and starting to write or read bits without due coordination or feature = detection. The confusion arises because the intro implies this is a = proposal to change bits but then backs off that commitment and uses = "coulds" etc. I think we need to be crystal clear. Speak about some bits = in the abstract and mention the few protocol-specific concerns or = related examples that each protocol already has defined. However, remove = the implications that we have well thought out how to get these deployed = for QUIC and punt that to future work by explicitly stating so. I think = this takes the form of removing section 6, rewriting section 5 to speak = about the higher-level problems and possible approaches, and removing = any straggling sentences about reusing existing or reserved = bits.

Cheers
Lucas

= --Apple-Mail=_A05AF089-05E6-4312-961B-17570F299D10--