Re: [IPv6] draft-ietf-6man-hbh-processing WGLC comments

[Toerless Eckert <[e-mail]>]

Inline.

Btw: Is there a github ? Would make it easier to address multiple smaller
textual points individually and discuss them.

On Wed, Aug 23, 2023 at 02:10:11PM +0100, Gorry Fairhurst wrote:
> On 22/08/2023 18:32, Toerless Eckert wrote:
> > To follow up on my generic comments,
> Thanks, the comments were helpful, and the new revision will respond to
> these, and will be better for this insight.
> > a few more generic concerns:
> > 
> > 1. I am worried that by constraining the document to HBH, we will have to duplicate
> >     probably all of the work for routing headers as well. It would be preferrable to
> >     see how much more work it would be to make the document address both
> >     headers that can be processed by routers along the path (HBH and routing).
> > 
> >     Conceptually i think it should be easy, but textually given how the draft
> >     refers to so much detailled rfc8200 text, it might lead to equal amount of
> >     duplication of text to fix *sigh*.
> 
> This could be a much bigger question.
> 
> - First, I agree there are things in common - and I think key parts can be
> carried forward to any work on the routing header.

Well, if you do not want to take this on now, how about cloning and working
(at a slower pace on that second draft).

But multiple draft will just create risk of more undesirable discrepanices.

Maybe worst existing discrepancy: draft-raviolli-intarea-trusted-domain-srv6,

If we had a doc like your HBH doc for RH. Would that have avoided the problems
that draft attempts to address ? Would the authors of that draft still think it's
the safer, easier solution ?

Aka: How can we best avoid discrepancies.

IMHO: I would not want to see the process rushed at this point...

> > 2. As seen from the other threads, i think we want to move into a direction where
> >     we remove the hurdles from applications to send EH (HBH/routing). But we then
> >     need to make sure we can feel safe with that recommendations and not be beaten
> >     up by opeators to open floodgates of attacks.
> >     The draft does a good base level of asking for MUST configure (to process),
> >     but i wonder if the granularity that is specified (aka: pretty much none)
> >     is sufficient. For example, i would think that one at least wants to have
> >     ACLs to permit/deny individual option header processing, and extended ACLs
> >     at that (aka: only for TCP or UDP traffic, or based on source/dest combinations).
> > 
> >     (and no, i am not paid for by Tom to create pain with ACLs that FAST would
> >      solve/remove, but pretty much those ACLs would be the cost of doing EH business
> >      without FAST. And igoring the cost by not specifying the need for them would just backfire).
> 
> From my perspective, I could see configuration to allow/deny forwarding
> packets with a HBH EH and for processing individual HBH Options, and I hope
> the draft says that. Is there a need for specific text?

See above reply re. the risks at a domain edge and how well you want to harden it.

Practically speaking i think we need to think about the actual policies required
and see how to best apply them to requirements text. The one we have is very
hand waiving.

Aka: It seems we want interface level filters If we want to make an argument that
something like draft-raviolli-intarea-trusted-domain-srv6 was not needed
(assuming similarily successfully deployed, but intradomain intended HBH header
 with similar attack options for traffic from interdomain).

But we also need router - wide filter (which i think is what you may have implied,
but the text is not specific about the context of the policy).
We may also want to have src/dst filters at the edge or hop-by-hop to be able
to e.g.: distinguish intradomain from transit traffic (based on ACL recognition
of intradomain traffic as opposed to put it onto edge interface).

Wrt to the discuss re. which headers can be used in conjunction with differnt
transport protocols. I can see a lot of interaction between transport protocol
and HBH, so it would be very prudent to mandate support for extended ACL
(5-tuple filter).

Wrt. reaction: I think the draft ignores the problem, that ignoring of option
headers/options in existing routers may not work at the desired rate (not
to say linerate as i have issues with that, see prior response from me).

In this case it seems routers should be able to discard and ICMP reply to such
headers. Aka: Force senders to not insert such headers/options - or else traffic
will not make it through.

Of course, we wouldn't want such future routers, but we MUST be able to support
such existing routers.

And for example, we have a lot of history with such discarding when packet has
something routers do not like. At least we should have a SHOULD raise ICMP
errors so apps can react faster.

And then there is diagnostics, aka: counters for what type of extension header/option
was received, and what policy reaction it had.  Without that, introduction of
new HBH will just fail due to frustrated operators.

I know, we traditionally where very weak on such policy/operations aspects in our
core protocol specifications and hoped for vendors to somehow to it, and when
we where luckyl we had some MIB 20 years later for it. But that also contributed
IMHO heavily to the demise of most advanced technologies we proposed, because
they could not well be operationalized.

Cheers
    Toerless

> Gorry
> 
> > Cheers
> >      toerless
> > 
> > On Sun, Jul 30, 2023 at 07:47:59PM +0200, Toerless Eckert wrote:
> > > Hi Gory, Bob, *:
> > > 
> > > Thanks a lot for the document, and sorry that i can jump in only so late
> > > in the process. I have a couple of NIT/MINOR comments, but primarily two
> > > MAYOR concerns, and those are that i think the document is way too negative
> > > about HbH headr going forward, by not including simple, but mandatory
> > > forwarding plane requirements to ensure new work can not cause issues.
> > > 
> > > In one case, this is about new router-alert spec, which i think are not
> > > more problematic than other control plane, if just the right requirements
> > > are met (which this document should well specify),
> > > 
> > > In the other case, it is higher than "full-line-rate" perfomance of HbH
> > > packets, which also can be made to work perfectly fine in the face of
> > > attacks by just having appropriate protection against attacks (such as
> > > rate-limiters).
> > > 
> > > We have all the same issues in similar form in other forwarding/control-plane,
> > > and we know how to overcome them. But the work to overcome problems usually
> > > happens when stuff is widely deployed only. Thats why seemingly nobody
> > > thinks about applying the same principles to HbH - because we do so little
> > > with it.
> > > 
> > > Cheers
> > >      Toerless
> > > 
> > > 1. NIT:
> > > 
> > > Suggest to change:
> > > 
> > >    When the IPv6 Specification was updated and published in July 2017 as
> > >    [RFC8200], the procedures relating to Hop-by-Hop options were as
> > >    follows:
> > > 
> > > to:
> > > 
> > >    When the IPv6 Specification was updated and published in July 2017,
> > >    the procedures relating to Hop-by-Hop options where specified as
> > >    follows (from [RFC8200], section 4):
> > > 
> > > Reason: It was not clear to me that the text was really 1:1 copied
> > > from RFC8200, and i think quotations should really be marked very
> > > explicitly. Also, because i didn't read it immediately as a citation,
> > > i bothered in suggesting a NIT for that text, which i guess now is
> > > inapproprite.
> > > 
> > > The same note may apply to other citations in the text, aka: please
> > > refer to them with section and ensure it's similarily clear it is a citation.
> > > 
> > > 2. NIT:
> > > 
> > > This one (see above) you'll likely want to skip given how its in
> > > an rfc8200 citation.
> > > 
> > > suggest to rewrite:
> > > 
> > >    by any node along a packet's delivery
> > >    path, until the packet reaches the node (or each of the set of
> > >    nodes, in the case of multicast) identified in the Destination
> > >    Address field of the IPv6 header
> > > 
> > > to:
> > > 
> > >    by any node along a packet's delivery
> > >    path, until the packet reaches the node
> > >    identified in the Destination Address field of the IPv6 header,
> > >    or each of the set of nodes, in the case of multicast.
> > > 
> > > Reason: In SSM, RFC4607, the set of nodes is identified by
> > > the combination of source and destination address, not only
> > > by the destination address. I think the original sentence
> > > excluded that interpretation, and this rewrite allows it
> > > (so as not having to elaborate more about these intricacies).
> > > 
> > > 3. MAYOR
> > > 
> > > Section:
> > > Any option that can be used to force packets into the router's
> > > Control Plane can be exploited as a denial of service attack on a
> > > transit router by saturating the resources needed for router
> > > management protocols ...
> > > 
> > > IMHO, this is inappropriate.
> > > 
> > > Just because something needs to go to control plane does not mean
> > > that it would hurt other control plane. HbH header processing would
> > > not be different from any other control plane activity in this
> > > respect.
> > > 
> > > For every control plane activity, you need to have some appropriate
> > > share of control plane resources depending on its functions and the
> > > desired performance, and you need to provide protection against
> > > that control plane function to exceed these shares, for example when
> > > it is being DoS'ed.
> > > 
> > > Routers can do as little as round-robin for CPU across control plane
> > > functions, as well as per-function rate-limiters on packets, just as
> > > a simple example.
> > > 
> > > Of course it is inappropriate to implement a HbH function in
> > > control-plane if its meant to operate at line rate, but if a HbH
> > > header function is defined to operate at the same performance
> > > level as other control plane, it is perfectly valid. For example
> > > router-alert for PGM or RSVP where logically correct. Just our
> > > specifications where and are bad, and implementations where (and are)
> > > likely even worse. But providing better specs would at least
> > > make it right from the IETF side (and put blame clearly on bad
> > > implementations).
> > > 
> > > 3. MAYOR
> > > 
> > > Section:
> > > 
> > > "In a nutshell, the IP Router Alert Option does not provide a
> > > convenient universal mechanism to accurately and reliably
> > > distinguish between IP Router Alert packets of interest and
> > > unwanted IP Router Alert packets..."
> > > 
> > > That text does not make a lot of sense, because you can IMHO
> > > downtalk any function we have specified by prepending it with
> > > "convenient, universal, accurately and reliably" - and find some
> > > issue.
> > > 
> > > The actual issue at the point of writing rfc6398 was that some router
> > > OS that was back then widely used, and AFAIK very likely the reference
> > > for Francois' experiences was punting without any rate-limiters or without
> > > any fair scheduling afterwards router-alert packets for protocols
> > > that where not even configured to be used on the router. So all
> > > those packets simply got slow-path forwarded, which was even
> > > slower than the actual router-alert processing would have been.
> > > 
> > > PGM was the core example of this (for me), Francois' experience was
> > > likely RSVP. Router that had PGM processing enabled
> > > did actually work well with the production level of PGM messages,
> > > but those that did not have it enabled (maybe they did not even
> > > support it) suffered from those punted PGM packets. And of course
> > > the simple solution would have been to simply punt router alert
> > > packets only for those router alerts whose protocols where enabled
> > > on the router.
> > > 
> > > My takeway from this situation was that router alert already had
> > >   a really useful (enough) way to distinguish packets by IP protocol
> > > ID, but because the router alert specification (to this day) does
> > > not outline that you MUST NOT punt when the protocol is not
> > > supported, and that you MUST ensure rate-limiting and limiting
> > > CPU ressources (like you need to do for non onpath control plane
> > > functions), we then got these lousy implementations. Which killed
> > > good protocols in whole industries.
> > > 
> > > If you are interested to revisit this text and the conclusions to draw
> > > from these experiences, i'm happy to help, but i don't want to volunteer
> > > unwanted text now.
> > > 
> > > 3. MINOR
> > > 
> > > "The share of traffic containing more than one EH however, is very small."
> > > 
> > > Such a data point from the past is no useful indication for prior
> > > experiences. The conclusion to support at least one EH is still
> > > useful, but IMHO only in conjunction with a somewhat better
> > > logic:
> > > 
> > > - It is unclear how man EH a forwader can support.
> > > - If it is possible to always put the most important EH
> > >    first, and ensure that all routers that do support this EH
> > >    will also always act on it if it is the first EH, then hosts
> > >    can at least realiably get support for their most important EH.
> > > - And once there is more deployment of EH where at least the first
> > >    EH will work, there is more motivation for deployments
> > >    to make more EH work, if/when there is sufficient need.
> > > 
> > > 4. MAYOR
> > > 
> > > "Transmission and Processing of IPv6 Extension Headers" [RFC7045]
> > > ...
> > > 
> > > Unfortunately, RFC7045 does also not capture the spec/implementation
> > > issues relating to slow-path EH processing outlined above.
> > > 
> > > 5. MINOR
> > > 
> > > In 5.1 the text says:
> > > 
> > > Routers SHOULD process the Hop-by-Hop Options header.  If a router
> > > 
> > > In 5.1.1, the rfc8200 text is repeated:
> > > 
> > > Options header if explicitly configured to do so
> > > 
> > > It seems this draft wants to keep that rfc8200 text intact and just
> > > extend it, but that text seems to conflict with the 5.1 text.
> > > 
> > > E.g.: the 5.1 text could say "Routers SHOULD process the HbH Options header
> > > if explicitly configured to do so".
> > > 
> > > And/or be explicit about what is the default. To me, the rfc8200 text
> > > reads as if the default could be to not process (which would be safe,
> > > and i'd implement it that way). But the 5.1 text also reads like default.
> > > Else it should better we written as "SHOULD be able to process...".
> > > 
> > > 6. MINOR
> > > 
> > > configuration could control that normal processing skips any or all
> > > of the Hop-by-Hop options carried in the Hop-by-Hop Options header.
> > > A router that does not process the contents of the Hop-by-Hop Options
> > > header does not therefore process the identifiers of individual
> > > Option Types to perform any specified action.
> > > 
> > > So, what is the semantic difference between "any or all" and "all" ?
> > > I was confused to think that "any or all" could also mean that
> > > specific Options could be enabled/disabled, but the following text would
> > > then be inconsistent (it only talks about "all" IMHO).
> > > 
> > > In any case, IMHO it wold make perfect sense to explicity enable
> > > each Option separately to be on the safe side (and enabling all could
> > > of course also be an option).
> > > 
> > > 6. MINOR
> > > 
> > > 01 - MAY discard the packet. Nodes should not rely on routers
> > >       dropping these unrecognized Option Types.
> > > 
> > > change to:
> > > 
> > > 01 - SHOULD discard the packet. Nodes MUST NOT not rely on routers
> > >       dropping these unrecognized Option Types.
> > > 
> > > Aka: keep the original intention, but just make it easier to implement,
> > > but also ensure that apps will deal with not getting the desired drop.
> > > 
> > > Otherwise, if you want to keep your text, you need better explanation
> > > why the radical shift in behavior, and why no normative requirement
> > > against the application to deal with it.
> > > 
> > > 7. MAYOR
> > > 
> > > The 10 and 11 text really needs to be changed to allow sending back
> > > ICMP replies ofr multicast packets only upon explicit configuration
> > > that ideally should also be tied to source (and group) ACL so that
> > > it only is issued for source addresses known to not be faked. Aka:
> > > ICMP replies to a multicast packet is an amplification attack,
> > > and the source address may be faked (agreeable more difficult
> > > with clasical RPF forwarding multicast protocols, but i wouldn't
> > > want to guarantee all this self protection when we have e.g.: BIER,
> > > or whatever else may come along), so it can be a great DDoS attack
> > > vector.
> > > 
> > > Othewise, for the unicast case, i think the text should be like i
> > > proposed for 01, e.g.: SHOULD, and also add the MUST NOT rely on
> > > the dropping...
> > > 
> > > 8. MAYOR
> > > 
> > > I reject the directons of this sections because as laid out in before,
> > > fundamentally the needs to make a control plane function work against
> > > attacks is not fundamentally different from when using onpath signalling
> > > to using explicit addressing (e.g.: host stack address as destination).
> > > In both cases, you do need to protect yourself against too many packets,
> > > and you need to ensure that the control plane service does not
> > > consume more CPU than feasible compared to othre functions.
> > > 
> > > RSVP or PGM (not defined for IPv6, but equally applicable) are
> > > perfectly good signaling mechanism and highly valueable, and where
> > > they failed (as explained above), that only happened because the spec
> > > (router alertt) did not specify MANDATORY requirements to MUST NOT
> > > punt/process/cause-performance-issues of any protocols not supported
> > > or enabled.
> > > 
> > > Aka: I really would like to see router-alert be specified a a forward
> > > going, full supported option, but specifying in this text all the
> > > requirements not only for implementations in general, but also
> > > for any router-alert specifications to be safe.
> > > 
> > > I think the second part of the section already has good starting points,
> > > and i'd be happy to help improving the text so it really is as bullet
> > > proof for implementation and deployment of route-alerts as we know.
> > > 
> > > One simple example is that MUST support
> > > line-rate ignoring of router-alert options that they do not support nor
> > > have enabled. If an implementation can not do that, it MUST support
> > > limiting processing of router-alert to only trusted sources, known
> > > to send router-alert only at supportable (but not attack) levels,
> > > such as via source IPv6 address filtering of processing of router-alert.
> > > 
> > > And of course MUST NOT enable router-alert processing by default,
> > > unless the software deployed has not been upgraded to support this documents
> > > behavior but prior RFCs (aka: for running software you might need to
> > > explicitly enable the new behavior).
> > > 
> > > 9. NIT
> > > 
> > > The split in text between 5.1 and 5.2.2 is confusing. I think these
> > > are better merged.
> > > 
> > > I also think the text should talk about "supported and enabled", aka:
> > > if we really want to be safe, we need operators to explicitly enable
> > > hbh header functions.
> > > .
> > > 
> > > 10. MAYOR
> > > 
> > > Section 6:
> > > 
> > > The whole logic of "full forwarding rate" is highly questionable and would
> > > severely limit the ability to build useful on-path processing if left
> > > in the text unchanged.
> > > 
> > > COnsider for example OAM style HbH - just because lots of people would
> > > think its useful (well, users. But lets assume enlightened operators
> > > would allow them to be enabled... which i think will not happen because
> > > operators do like to keep secrets. But thats a different issu).
> > > 
> > > Eample HbH header packets with e.g.: some OAM might take as much forwarding
> > > plane resources as e.g.: 10 "normal" packets. But that would be fine,
> > > bcause the spec for it could say for example that one should use
> > > e.g.: only a maximum of 1 in 100 packets with the OAM header. Which wold
> > > create an overall overhead of 10% performance more. Or maybe converted,
> > > a 10% higher average packet sized that the router can process at linerate.
> > > Aka: perfectly acceptable. Because the operator for example knows that
> > > his packet size mix is well below linerate mix.
> > > 
> > > What is of course for an Internet use of such a HbH feature significant is
> > > that the router MUST be able to rate-limit processing of the option to
> > > said < 1 % of packets processed. and if someone does a (D)DoS attack with
> > > packets using such "expensive" options, then the router would still be
> > > safe and not need more than the expected 10% more resources.
> > > 
> > > If on the other hand, a HbH option is meant to be used in controlled networks,
> > > then the cost could even be higher. FOr example, IP multicast often
> > > douples, if not triples the average packet size routers can provide linerate
> > > forwarding for. Just because its more lookup and replication also may cost
> > > more (per replica). E.g.: 256 bit (S,G) lookup instead of 128 bit D lookup.
> > > But that is not a problem, because in controlled networks you do want to
> > > sped the router resources for the value. And you have more tools at disposal
> > > to ensure routers are not overrun.
> > > 
> > > I think that all the cool new HbH would start out in controlled networks,
> > > so it is important to allows also similiarily "expensive" hBh features
> > > as we've had success with other forwarding features in the past (e.g.:
> > > ip multicast).
> > > 
> > > Would be happy to help think of more appropriate text, e.g. as above, something
> > > like:
> > > 
> > > If a HbH option requires more router performance than "full forwarding rate",
> > > routers MUST be able to support rate-limiting the  processing of packets with
> > > such an option so that if configured with an appropriate maximum rate,
> > > distributed denial of service attacks via larger rates of such packets than
> > > supportable by the router will not cause issues in the routers deployment
> > > and performance.
> > > 
> > > I'll stop here with review for the moment because i would like to first see
> > > where this document wants to go with the MAYOR points raised before checking
> > > further details.
> > > 
> > > Cheers
> > >      Toerless
> > > 
> > > In-Reply-To: <5840.1690652533@localhost>
> > > On Sat, Jul 29, 2023 at 01:42:13PM -0400, Michael Richardson wrote:
> > > > Ron Bonica<rbonica=[e-mail]>  wrote:
> > > >      > The Internet is a loosely confederated group of "limited domains". Each
> > > >      > limited domain maintains its own forwarding and routing policy. HBH
> > > >      > processing, as it is defined in RFC 8200, is OK within a limited
> > > >      > domain. It provides yet another tool for deployment of that limited
> > > >      > domain's forwarding and policies.
> > > > 
> > > > It's an interesting view.
> > > > 
> > > > So, the "unlimited Internet" is just a path across a bunch of limited
> > > > domains, where the sender does not know which limited domains will be used.
> > > > I think that is a useful change from how we used to think about the Internet,
> > > > because it might be that we know something about the nearby limited domain,
> > > > and something else about the receivers limited domain.   It's the unknown
> > > > part in the middle that gets in our way.
> > > > 
> > > > An interesting thing about Ben Schwartz and Co's RISAV proposal is that it
> > > > gives a way for those two limited domains to communicate directly.
> > > > 
> > > >      > Deployment of the HBH across limited domains may not be a very good
> > > >      > idea, because it provides a tool for one limited domain to influence
> > > >      > the forwarding and routing polices of another. Can you think of a
> > > >      > single HBH option for which the above statement is not true? (The
> > > >      > question is not rhetorical.)
> > > > 
> > > > If the HBH didn't influence things, then we wouldn't bother!
> > > > So by definition, it's true.
> > > > But, you are missing a key phrase in this:
> > > > 
> > > >       it provides a tool for one limited domain to influence
> > > >       the forwarding and routing polices of another IN AN UNAUTHORIZED WAY
> > > > 
> > > > 
> > > > -- 
> > > > Michael Richardson<[e-mail]>    . o O ( IPv6 IøT consulting )
> > > >             Sandelman Software Works Inc, Ottawa and Worldwide
> > > > 
> > > > 
> > > > 
> > > > 
> > > 
> > > 
> > > > --------------------------------------------------------------------
> > > > IETF IPv6 working group mailing list
> > > > [e-mail]
> > > > Administrative Requests:https://www.ietf.org/mailman/listinfo/ipv6
> > > > --------------------------------------------------------------------
> > > 
> > > -- 
> > > ---
> > > [e-mail]
> > > 

-- 
---
[e-mail]