Re: [IPv6] draft-ietf-6man-hbh-processing WGLC comments

[Tom Herbert <[e-mail]>]

On Fri, Aug 25, 2023 at 1:52 AM Toerless Eckert <[e-mail]> wrote:
>
> Inline.
>
> Btw: Is there a github ? Would make it easier to address multiple smaller
> textual points individually and discuss them.
>
> On Wed, Aug 23, 2023 at 02:10:11PM +0100, Gorry Fairhurst wrote:
> > On 22/08/2023 18:32, Toerless Eckert wrote:
> > > To follow up on my generic comments,
> > Thanks, the comments were helpful, and the new revision will respond to
> > these, and will be better for this insight.
> > > a few more generic concerns:
> > >
> > > 1. I am worried that by constraining the document to HBH, we will have to duplicate
> > >     probably all of the work for routing headers as well. It would be preferrable to
> > >     see how much more work it would be to make the document address both
> > >     headers that can be processed by routers along the path (HBH and routing).
> > >
> > >     Conceptually i think it should be easy, but textually given how the draft
> > >     refers to so much detailled rfc8200 text, it might lead to equal amount of
> > >     duplication of text to fix *sigh*.
> >
> > This could be a much bigger question.
> >
> > - First, I agree there are things in common - and I think key parts can be
> > carried forward to any work on the routing header.
>
> Well, if you do not want to take this on now, how about cloning and working
> (at a slower pace on that second draft).
>
> But multiple draft will just create risk of more undesirable discrepanices.
>
> Maybe worst existing discrepancy: draft-raviolli-intarea-trusted-domain-srv6,
>
> If we had a doc like your HBH doc for RH. Would that have avoided the problems
> that draft attempts to address ? Would the authors of that draft still think it's
> the safer, easier solution ?
>
> Aka: How can we best avoid discrepancies.
>
> IMHO: I would not want to see the process rushed at this point...
>
> > > 2. As seen from the other threads, i think we want to move into a direction where
> > >     we remove the hurdles from applications to send EH (HBH/routing). But we then
> > >     need to make sure we can feel safe with that recommendations and not be beaten
> > >     up by opeators to open floodgates of attacks.
> > >     The draft does a good base level of asking for MUST configure (to process),
> > >     but i wonder if the granularity that is specified (aka: pretty much none)
> > >     is sufficient. For example, i would think that one at least wants to have
> > >     ACLs to permit/deny individual option header processing, and extended ACLs
> > >     at that (aka: only for TCP or UDP traffic, or based on source/dest combinations).
> > >
> > >     (and no, i am not paid for by Tom to create pain with ACLs that FAST would
> > >      solve/remove, but pretty much those ACLs would be the cost of doing EH business
> > >      without FAST. And igoring the cost by not specifying the need for them would just backfire).
> >
> > From my perspective, I could see configuration to allow/deny forwarding
> > packets with a HBH EH and for processing individual HBH Options, and I hope
> > the draft says that. Is there a need for specific text?
>
> See above reply re. the risks at a domain edge and how well you want to harden it.
>
> Practically speaking i think we need to think about the actual policies required
> and see how to best apply them to requirements text. The one we have is very
> hand waiving.
>
> Aka: It seems we want interface level filters If we want to make an argument that
> something like draft-raviolli-intarea-trusted-domain-srv6 was not needed
> (assuming similarily successfully deployed, but intradomain intended HBH header
>  with similar attack options for traffic from interdomain).
>
> But we also need router - wide filter (which i think is what you may have implied,
> but the text is not specific about the context of the policy).
> We may also want to have src/dst filters at the edge or hop-by-hop to be able
> to e.g.: distinguish intradomain from transit traffic (based on ACL recognition
> of intradomain traffic as opposed to put it onto edge interface).
>
> Wrt to the discuss re. which headers can be used in conjunction with differnt
> transport protocols. I can see a lot of interaction between transport protocol
> and HBH, so it would be very prudent to mandate support for extended ACL
> (5-tuple filter).
>
> Wrt. reaction: I think the draft ignores the problem, that ignoring of option
> headers/options in existing routers may not work at the desired rate (not
> to say linerate as i have issues with that, see prior response from me).

Toerless,

I don't think this is a problem. RFC8200 states that routers may
completely ignore the HbH Options header; that is there are no
mandatory options that must be processed. So if a router can't keep up
at the desired rate, then it can just ignore HBH entirely.

>
> In this case it seems routers should be able to discard and ICMP reply to such
> headers. Aka: Force senders to not insert such headers/options - or else traffic
> will not make it through.

A router ignoring HbH is MUCH PREFERABLE to routers dropping packets
with HbH options.

>
> Of course, we wouldn't want such future routers, but we MUST be able to support
> such existing routers.
>
> And for example, we have a lot of history with such discarding when packet has
> something routers do not like. At least we should have a SHOULD raise ICMP
> errors so apps can react faster.
>
Yes, that's covered by RFC8883. Of course in reality, ICMP has about
as much luck making it through the Internet as HBH EH!

> And then there is diagnostics, aka: counters for what type of extension header/option
> was received, and what policy reaction it had.  Without that, introduction of
> new HBH will just fail due to frustrated operators.

But again, routers can just ignore any new HbH.

Tom

>
> I know, we traditionally where very weak on such policy/operations aspects in our
> core protocol specifications and hoped for vendors to somehow to it, and when
> we where luckyl we had some MIB 20 years later for it. But that also contributed
> IMHO heavily to the demise of most advanced technologies we proposed, because
> they could not well be operationalized.
>
> Cheers
>     Toerless
>
> > Gorry
> >
> > > Cheers
> > >      toerless
> > >
> > > On Sun, Jul 30, 2023 at 07:47:59PM +0200, Toerless Eckert wrote:
> > > > Hi Gory, Bob, *:
> > > >
> > > > Thanks a lot for the document, and sorry that i can jump in only so late
> > > > in the process. I have a couple of NIT/MINOR comments, but primarily two
> > > > MAYOR concerns, and those are that i think the document is way too negative
> > > > about HbH headr going forward, by not including simple, but mandatory
> > > > forwarding plane requirements to ensure new work can not cause issues.
> > > >
> > > > In one case, this is about new router-alert spec, which i think are not
> > > > more problematic than other control plane, if just the right requirements
> > > > are met (which this document should well specify),
> > > >
> > > > In the other case, it is higher than "full-line-rate" perfomance of HbH
> > > > packets, which also can be made to work perfectly fine in the face of
> > > > attacks by just having appropriate protection against attacks (such as
> > > > rate-limiters).
> > > >
> > > > We have all the same issues in similar form in other forwarding/control-plane,
> > > > and we know how to overcome them. But the work to overcome problems usually
> > > > happens when stuff is widely deployed only. Thats why seemingly nobody
> > > > thinks about applying the same principles to HbH - because we do so little
> > > > with it.
> > > >
> > > > Cheers
> > > >      Toerless
> > > >
> > > > 1. NIT:
> > > >
> > > > Suggest to change:
> > > >
> > > >    When the IPv6 Specification was updated and published in July 2017 as
> > > >    [RFC8200], the procedures relating to Hop-by-Hop options were as
> > > >    follows:
> > > >
> > > > to:
> > > >
> > > >    When the IPv6 Specification was updated and published in July 2017,
> > > >    the procedures relating to Hop-by-Hop options where specified as
> > > >    follows (from [RFC8200], section 4):
> > > >
> > > > Reason: It was not clear to me that the text was really 1:1 copied
> > > > from RFC8200, and i think quotations should really be marked very
> > > > explicitly. Also, because i didn't read it immediately as a citation,
> > > > i bothered in suggesting a NIT for that text, which i guess now is
> > > > inapproprite.
> > > >
> > > > The same note may apply to other citations in the text, aka: please
> > > > refer to them with section and ensure it's similarily clear it is a citation.
> > > >
> > > > 2. NIT:
> > > >
> > > > This one (see above) you'll likely want to skip given how its in
> > > > an rfc8200 citation.
> > > >
> > > > suggest to rewrite:
> > > >
> > > >    by any node along a packet's delivery
> > > >    path, until the packet reaches the node (or each of the set of
> > > >    nodes, in the case of multicast) identified in the Destination
> > > >    Address field of the IPv6 header
> > > >
> > > > to:
> > > >
> > > >    by any node along a packet's delivery
> > > >    path, until the packet reaches the node
> > > >    identified in the Destination Address field of the IPv6 header,
> > > >    or each of the set of nodes, in the case of multicast.
> > > >
> > > > Reason: In SSM, RFC4607, the set of nodes is identified by
> > > > the combination of source and destination address, not only
> > > > by the destination address. I think the original sentence
> > > > excluded that interpretation, and this rewrite allows it
> > > > (so as not having to elaborate more about these intricacies).
> > > >
> > > > 3. MAYOR
> > > >
> > > > Section:
> > > > Any option that can be used to force packets into the router's
> > > > Control Plane can be exploited as a denial of service attack on a
> > > > transit router by saturating the resources needed for router
> > > > management protocols ...
> > > >
> > > > IMHO, this is inappropriate.
> > > >
> > > > Just because something needs to go to control plane does not mean
> > > > that it would hurt other control plane. HbH header processing would
> > > > not be different from any other control plane activity in this
> > > > respect.
> > > >
> > > > For every control plane activity, you need to have some appropriate
> > > > share of control plane resources depending on its functions and the
> > > > desired performance, and you need to provide protection against
> > > > that control plane function to exceed these shares, for example when
> > > > it is being DoS'ed.
> > > >
> > > > Routers can do as little as round-robin for CPU across control plane
> > > > functions, as well as per-function rate-limiters on packets, just as
> > > > a simple example.
> > > >
> > > > Of course it is inappropriate to implement a HbH function in
> > > > control-plane if its meant to operate at line rate, but if a HbH
> > > > header function is defined to operate at the same performance
> > > > level as other control plane, it is perfectly valid. For example
> > > > router-alert for PGM or RSVP where logically correct. Just our
> > > > specifications where and are bad, and implementations where (and are)
> > > > likely even worse. But providing better specs would at least
> > > > make it right from the IETF side (and put blame clearly on bad
> > > > implementations).
> > > >
> > > > 3. MAYOR
> > > >
> > > > Section:
> > > >
> > > > "In a nutshell, the IP Router Alert Option does not provide a
> > > > convenient universal mechanism to accurately and reliably
> > > > distinguish between IP Router Alert packets of interest and
> > > > unwanted IP Router Alert packets..."
> > > >
> > > > That text does not make a lot of sense, because you can IMHO
> > > > downtalk any function we have specified by prepending it with
> > > > "convenient, universal, accurately and reliably" - and find some
> > > > issue.
> > > >
> > > > The actual issue at the point of writing rfc6398 was that some router
> > > > OS that was back then widely used, and AFAIK very likely the reference
> > > > for Francois' experiences was punting without any rate-limiters or without
> > > > any fair scheduling afterwards router-alert packets for protocols
> > > > that where not even configured to be used on the router. So all
> > > > those packets simply got slow-path forwarded, which was even
> > > > slower than the actual router-alert processing would have been.
> > > >
> > > > PGM was the core example of this (for me), Francois' experience was
> > > > likely RSVP. Router that had PGM processing enabled
> > > > did actually work well with the production level of PGM messages,
> > > > but those that did not have it enabled (maybe they did not even
> > > > support it) suffered from those punted PGM packets. And of course
> > > > the simple solution would have been to simply punt router alert
> > > > packets only for those router alerts whose protocols where enabled
> > > > on the router.
> > > >
> > > > My takeway from this situation was that router alert already had
> > > >   a really useful (enough) way to distinguish packets by IP protocol
> > > > ID, but because the router alert specification (to this day) does
> > > > not outline that you MUST NOT punt when the protocol is not
> > > > supported, and that you MUST ensure rate-limiting and limiting
> > > > CPU ressources (like you need to do for non onpath control plane
> > > > functions), we then got these lousy implementations. Which killed
> > > > good protocols in whole industries.
> > > >
> > > > If you are interested to revisit this text and the conclusions to draw
> > > > from these experiences, i'm happy to help, but i don't want to volunteer
> > > > unwanted text now.
> > > >
> > > > 3. MINOR
> > > >
> > > > "The share of traffic containing more than one EH however, is very small."
> > > >
> > > > Such a data point from the past is no useful indication for prior
> > > > experiences. The conclusion to support at least one EH is still
> > > > useful, but IMHO only in conjunction with a somewhat better
> > > > logic:
> > > >
> > > > - It is unclear how man EH a forwader can support.
> > > > - If it is possible to always put the most important EH
> > > >    first, and ensure that all routers that do support this EH
> > > >    will also always act on it if it is the first EH, then hosts
> > > >    can at least realiably get support for their most important EH.
> > > > - And once there is more deployment of EH where at least the first
> > > >    EH will work, there is more motivation for deployments
> > > >    to make more EH work, if/when there is sufficient need.
> > > >
> > > > 4. MAYOR
> > > >
> > > > "Transmission and Processing of IPv6 Extension Headers" [RFC7045]
> > > > ...
> > > >
> > > > Unfortunately, RFC7045 does also not capture the spec/implementation
> > > > issues relating to slow-path EH processing outlined above.
> > > >
> > > > 5. MINOR
> > > >
> > > > In 5.1 the text says:
> > > >
> > > > Routers SHOULD process the Hop-by-Hop Options header.  If a router
> > > >
> > > > In 5.1.1, the rfc8200 text is repeated:
> > > >
> > > > Options header if explicitly configured to do so
> > > >
> > > > It seems this draft wants to keep that rfc8200 text intact and just
> > > > extend it, but that text seems to conflict with the 5.1 text.
> > > >
> > > > E.g.: the 5.1 text could say "Routers SHOULD process the HbH Options header
> > > > if explicitly configured to do so".
> > > >
> > > > And/or be explicit about what is the default. To me, the rfc8200 text
> > > > reads as if the default could be to not process (which would be safe,
> > > > and i'd implement it that way). But the 5.1 text also reads like default.
> > > > Else it should better we written as "SHOULD be able to process...".
> > > >
> > > > 6. MINOR
> > > >
> > > > configuration could control that normal processing skips any or all
> > > > of the Hop-by-Hop options carried in the Hop-by-Hop Options header.
> > > > A router that does not process the contents of the Hop-by-Hop Options
> > > > header does not therefore process the identifiers of individual
> > > > Option Types to perform any specified action.
> > > >
> > > > So, what is the semantic difference between "any or all" and "all" ?
> > > > I was confused to think that "any or all" could also mean that
> > > > specific Options could be enabled/disabled, but the following text would
> > > > then be inconsistent (it only talks about "all" IMHO).
> > > >
> > > > In any case, IMHO it wold make perfect sense to explicity enable
> > > > each Option separately to be on the safe side (and enabling all could
> > > > of course also be an option).
> > > >
> > > > 6. MINOR
> > > >
> > > > 01 - MAY discard the packet. Nodes should not rely on routers
> > > >       dropping these unrecognized Option Types.
> > > >
> > > > change to:
> > > >
> > > > 01 - SHOULD discard the packet. Nodes MUST NOT not rely on routers
> > > >       dropping these unrecognized Option Types.
> > > >
> > > > Aka: keep the original intention, but just make it easier to implement,
> > > > but also ensure that apps will deal with not getting the desired drop.
> > > >
> > > > Otherwise, if you want to keep your text, you need better explanation
> > > > why the radical shift in behavior, and why no normative requirement
> > > > against the application to deal with it.
> > > >
> > > > 7. MAYOR
> > > >
> > > > The 10 and 11 text really needs to be changed to allow sending back
> > > > ICMP replies ofr multicast packets only upon explicit configuration
> > > > that ideally should also be tied to source (and group) ACL so that
> > > > it only is issued for source addresses known to not be faked. Aka:
> > > > ICMP replies to a multicast packet is an amplification attack,
> > > > and the source address may be faked (agreeable more difficult
> > > > with clasical RPF forwarding multicast protocols, but i wouldn't
> > > > want to guarantee all this self protection when we have e.g.: BIER,
> > > > or whatever else may come along), so it can be a great DDoS attack
> > > > vector.
> > > >
> > > > Othewise, for the unicast case, i think the text should be like i
> > > > proposed for 01, e.g.: SHOULD, and also add the MUST NOT rely on
> > > > the dropping...
> > > >
> > > > 8. MAYOR
> > > >
> > > > I reject the directons of this sections because as laid out in before,
> > > > fundamentally the needs to make a control plane function work against
> > > > attacks is not fundamentally different from when using onpath signalling
> > > > to using explicit addressing (e.g.: host stack address as destination).
> > > > In both cases, you do need to protect yourself against too many packets,
> > > > and you need to ensure that the control plane service does not
> > > > consume more CPU than feasible compared to othre functions.
> > > >
> > > > RSVP or PGM (not defined for IPv6, but equally applicable) are
> > > > perfectly good signaling mechanism and highly valueable, and where
> > > > they failed (as explained above), that only happened because the spec
> > > > (router alertt) did not specify MANDATORY requirements to MUST NOT
> > > > punt/process/cause-performance-issues of any protocols not supported
> > > > or enabled.
> > > >
> > > > Aka: I really would like to see router-alert be specified a a forward
> > > > going, full supported option, but specifying in this text all the
> > > > requirements not only for implementations in general, but also
> > > > for any router-alert specifications to be safe.
> > > >
> > > > I think the second part of the section already has good starting points,
> > > > and i'd be happy to help improving the text so it really is as bullet
> > > > proof for implementation and deployment of route-alerts as we know.
> > > >
> > > > One simple example is that MUST support
> > > > line-rate ignoring of router-alert options that they do not support nor
> > > > have enabled. If an implementation can not do that, it MUST support
> > > > limiting processing of router-alert to only trusted sources, known
> > > > to send router-alert only at supportable (but not attack) levels,
> > > > such as via source IPv6 address filtering of processing of router-alert.
> > > >
> > > > And of course MUST NOT enable router-alert processing by default,
> > > > unless the software deployed has not been upgraded to support this documents
> > > > behavior but prior RFCs (aka: for running software you might need to
> > > > explicitly enable the new behavior).
> > > >
> > > > 9. NIT
> > > >
> > > > The split in text between 5.1 and 5.2.2 is confusing. I think these
> > > > are better merged.
> > > >
> > > > I also think the text should talk about "supported and enabled", aka:
> > > > if we really want to be safe, we need operators to explicitly enable
> > > > hbh header functions.
> > > > .
> > > >
> > > > 10. MAYOR
> > > >
> > > > Section 6:
> > > >
> > > > The whole logic of "full forwarding rate" is highly questionable and would
> > > > severely limit the ability to build useful on-path processing if left
> > > > in the text unchanged.
> > > >
> > > > COnsider for example OAM style HbH - just because lots of people would
> > > > think its useful (well, users. But lets assume enlightened operators
> > > > would allow them to be enabled... which i think will not happen because
> > > > operators do like to keep secrets. But thats a different issu).
> > > >
> > > > Eample HbH header packets with e.g.: some OAM might take as much forwarding
> > > > plane resources as e.g.: 10 "normal" packets. But that would be fine,
> > > > bcause the spec for it could say for example that one should use
> > > > e.g.: only a maximum of 1 in 100 packets with the OAM header. Which wold
> > > > create an overall overhead of 10% performance more. Or maybe converted,
> > > > a 10% higher average packet sized that the router can process at linerate.
> > > > Aka: perfectly acceptable. Because the operator for example knows that
> > > > his packet size mix is well below linerate mix.
> > > >
> > > > What is of course for an Internet use of such a HbH feature significant is
> > > > that the router MUST be able to rate-limit processing of the option to
> > > > said < 1 % of packets processed. and if someone does a (D)DoS attack with
> > > > packets using such "expensive" options, then the router would still be
> > > > safe and not need more than the expected 10% more resources.
> > > >
> > > > If on the other hand, a HbH option is meant to be used in controlled networks,
> > > > then the cost could even be higher. FOr example, IP multicast often
> > > > douples, if not triples the average packet size routers can provide linerate
> > > > forwarding for. Just because its more lookup and replication also may cost
> > > > more (per replica). E.g.: 256 bit (S,G) lookup instead of 128 bit D lookup.
> > > > But that is not a problem, because in controlled networks you do want to
> > > > sped the router resources for the value. And you have more tools at disposal
> > > > to ensure routers are not overrun.
> > > >
> > > > I think that all the cool new HbH would start out in controlled networks,
> > > > so it is important to allows also similiarily "expensive" hBh features
> > > > as we've had success with other forwarding features in the past (e.g.:
> > > > ip multicast).
> > > >
> > > > Would be happy to help think of more appropriate text, e.g. as above, something
> > > > like:
> > > >
> > > > If a HbH option requires more router performance than "full forwarding rate",
> > > > routers MUST be able to support rate-limiting the  processing of packets with
> > > > such an option so that if configured with an appropriate maximum rate,
> > > > distributed denial of service attacks via larger rates of such packets than
> > > > supportable by the router will not cause issues in the routers deployment
> > > > and performance.
> > > >
> > > > I'll stop here with review for the moment because i would like to first see
> > > > where this document wants to go with the MAYOR points raised before checking
> > > > further details.
> > > >
> > > > Cheers
> > > >      Toerless
> > > >
> > > > In-Reply-To: <5840.1690652533@localhost>
> > > > On Sat, Jul 29, 2023 at 01:42:13PM -0400, Michael Richardson wrote:
> > > > > Ron Bonica<rbonica=[e-mail]>  wrote:
> > > > >      > The Internet is a loosely confederated group of "limited domains". Each
> > > > >      > limited domain maintains its own forwarding and routing policy. HBH
> > > > >      > processing, as it is defined in RFC 8200, is OK within a limited
> > > > >      > domain. It provides yet another tool for deployment of that limited
> > > > >      > domain's forwarding and policies.
> > > > >
> > > > > It's an interesting view.
> > > > >
> > > > > So, the "unlimited Internet" is just a path across a bunch of limited
> > > > > domains, where the sender does not know which limited domains will be used.
> > > > > I think that is a useful change from how we used to think about the Internet,
> > > > > because it might be that we know something about the nearby limited domain,
> > > > > and something else about the receivers limited domain.   It's the unknown
> > > > > part in the middle that gets in our way.
> > > > >
> > > > > An interesting thing about Ben Schwartz and Co's RISAV proposal is that it
> > > > > gives a way for those two limited domains to communicate directly.
> > > > >
> > > > >      > Deployment of the HBH across limited domains may not be a very good
> > > > >      > idea, because it provides a tool for one limited domain to influence
> > > > >      > the forwarding and routing polices of another. Can you think of a
> > > > >      > single HBH option for which the above statement is not true? (The
> > > > >      > question is not rhetorical.)
> > > > >
> > > > > If the HBH didn't influence things, then we wouldn't bother!
> > > > > So by definition, it's true.
> > > > > But, you are missing a key phrase in this:
> > > > >
> > > > >       it provides a tool for one limited domain to influence
> > > > >       the forwarding and routing polices of another IN AN UNAUTHORIZED WAY
> > > > >
> > > > >
> > > > > --
> > > > > Michael Richardson<[e-mail]>    . o O ( IPv6 IøT consulting )
> > > > >             Sandelman Software Works Inc, Ottawa and Worldwide
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > > > --------------------------------------------------------------------
> > > > > IETF IPv6 working group mailing list
> > > > > [e-mail]
> > > > > Administrative Requests:https://www.ietf.org/mailman/listinfo/ipv6
> > > > > --------------------------------------------------------------------
> > > >
> > > > --
> > > > ---
> > > > [e-mail]
> > > >
>
> --
> ---
> [e-mail]
>
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [e-mail]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------