Re: [Idr] WG Last Call on Extened Message Support

[<[e-mail]>]

Susan,

We now agree that the problem can happen with existing BGP attributes. Good. So the argument to postpone the error handling discussion for future specification defining large attributes does not hold anymore.

Coming back to the draft,

Old/
   A BGP announcement will, in the normal case, propagate throughout the
   BGP speaking Internet; and there will undoubtedly be BGP speakers
   which do not have the Extended Message capability.  Therefore,
   putting an attribute which can not be decomposed to 4096 octets or
   less in an Extended Message is a likely path to routing failure.


1)      As a minor comment, the error is not limited to an (one single) attribute (type) which size is bigger than 4096 octets. The error happens when the Total Path Attribute Length (for all attributes) is bigger than 4096 octets.

2)      As a major comment, I am asking for the specification to specify how it handles that failure, which is not the result of a possible bug, but created by design by the solution. It will happen.

Your proposition to "raise an alarm" is required but not sufficient as we still need to specify what we do with this "big" route. One option could be "treat as withdraw", another option suggested by Robert is to propagate the NLRI with subset of attributes plus an indication of the failure ("a short attribute BGP_PIPE_TOO_THIN"). There may be others options.

To trigger the discussion on the resolution, I would propose the following text "One BGP Route too large to be propagated to a BGP receiver not supporting BGP Extended Message MUST be handled, over this BGP session, using the Treat-as-withdraw approach as defined in [RFC7606]. The BGP speaker MAY, beforehand, remove some BGP attributes which are eligible to the Attribute discard approach [RFC7606]."

3)      Once error handling is specified, the draft will be able to discuss

a.       the operational considerations for this issue (e.g. The one from RFC7606 could be referenced as-is, or used as inspiration)

b.      the security considerations (as a remote attacker would be able to craft a large BGP route that will be withdrawn (assuming "treat-as-withdraw") somewhere in some AS, possibly creating routing inconsistencies within the AS)

c.       the proposed mitigations (e.g., an operator should enable support for Extended messages on all IBGP sessions first, before enabling it on an EBGP session; If not possible, an operator could define a policy to filter BGP attributes from EBGP sessions, typically based on their type and size;....)

more inline [Bruno2]


From: Susan Hares [mailto:[e-mail]]
Sent: Thursday, February 14, 2019 6:20 PM
To: DECRAENE Bruno TGI/OLN
Cc: [e-mail]; 'Jakob Heitz (jheitz)'; 'John Scudder'; 'Alvaro Retana'
Subject: RE: [Idr] WG Last Call on Extened Message Support

Bruno:

Thank you for careful considerations on the use case.   I think we have translation issues between our "English" language again so I have tried to be precise in the detailed comments.  I've summarized my comments as well to help us take the next step.

Sue

Summary
----
Good catch on the 1000 large BGP communities blowing up the message size!  This is a case where a code path could get caught with program error.  The error handling (section 6.1 in RFC4271 per section 5 in the extended specification) IMHO handles the case where the attribute is longer than the message.   Please let me know if I've missed something.

[Bruno2] I don't think that section 6.1 of RFC4271 covers this. The problem is not a Message Header error handling as a proper BGP implementation would not sent the message. (not to mention with inconsistent size)

As I understand it,  you are asking this specification to be expanded to add notes on policies on what implementation should do to correctly combine RFC4271 length sizes with extended length sizes.   The implementers and operations people commented earlier that this should not be in the protocol specification other than as a brief footnote.

[Bruno2]An error handling must exist even in the absence of SP policies. The argument related to previous discussions does not hold IMHO as I've raised a new point (this happens with existing BGP attributes, not in a hypothetical new attribute)

I agree there is a need for such standardization as BCPs for particular use cases (large communities, BESS (EVPN or centralized controller), SPRING, BGP-LS information (see Daylight comments yesterday), SD-WANs (rtgwg), and SPRING.   I think the chairs of the WGs utilizing should consider if there is need for standardization for a minimal set of policy for message sizes for specific use cases.   I will be glad to send request to these WG or to host something in IDR if there is policy that can be used across all use cases.

[Bruno2] I don't think that Error handling is to be considered as optional or for future consideration.

To provide a slight expansion to point people in this direction, how about this change to paragraph in section 4:

Old/
   A BGP announcement will, in the normal case, propagate throughout the
   BGP speaking Internet; and there will undoubtedly be BGP speakers
   which do not have the Extended Message capability.  Therefore,
   putting an attribute which can not be decomposed to 4096 octets or
   less in an Extended Message is a likely path to routing failure.
/
New/
   A BGP announcement will, in the normal case, propagate throughout the
   BGP speaking Internet; and there will undoubtedly be BGP speakers
   which do not have the Extended Message capability.  Therefore,
   putting an attribute which cannot be decomposed to 4096 octets or
   less in an Extended Message is a likely path to routing failure.
   One example of a valid attribute which cannot be precisely decomposed if it
   sent in Extended message is a large community attribute with 1000 communities.
   (12,000 bytes).
/
[PS - in American English "Can not" is normally concatenated "cannot".]

[Bruno2] Thanks for the proposition, but this does not address my concern.


From: [e-mail] [mailto:[e-mail]]
Sent: Thursday, February 14, 2019 3:29 AM
To: Susan Hares
Cc: [e-mail]; 'Jakob Heitz (jheitz)'
Subject: RE: [Idr] WG Last Call on Extened Message Support

Sue,

Thanks for the reply. Please see inline [Bruno]

From: Susan Hares [mailto:[e-mail]]
Sent: Wednesday, February 13, 2019 8:03 PM
To: DECRAENE Bruno TGI/OLN
Cc: [e-mail]; 'Jakob Heitz (jheitz)'
Subject: RE: [Idr] WG Last Call on Extened Message Support

Bruno:

I'd like to answer some of the questions you sent Randy based on comments (public and private) the chairs and I as the shepherd received on this draft in our last WG LCs.

Several of the comments you questioned were put in to answer other questions.   This gets into a catch-22... problem - where you put changes in for one person/take out the changes out for another.  In academic circles this is called the reviewer #2 problem (reviewer 1 insists changes go in, reviewer 2 insists changes go out) during publication.  Therefore, I'm I am going to answer most of your questions where this is the case.

I believe that Randy has new version with the changes I've indicated.  I'm hoping he'll just post that version.  AFAILK, we do have 2 compliant implementations.

Sue



<shepherd hat on>
This draft's technology provides efficient processing for some application by allow a huge BGP message size.  Whether "chunking" BGP to 4096 byte messages or packing large amounts of data into 65,000 byte message is more effective for a particular application is something only an implementer of a particular BGP implementation  can decide.   This draft simply gives freedom for applications to not be restrict to an artificially small message size.  The market place will choose whether this technology is used or not.

As Jakob pointed out, changing a highly tuned BGP implementation which "chunks" BGP data into 4096 byte messages may involve substantial rework of the implementation.


(snip)
From: Idr <[e-mail]> On Behalf Of [e-mail]
Sent: 29 January 2019 13:33
To: Susan Hares <[e-mail]>
Cc: [e-mail]
Subject: Re: [Idr] WG Last Call on Extened Message Support

Hi WG,

>Please find below some comments.
>As of today, I don't believe this specification is ready to be progressed to IESG/RFC, especially for a document >updating RFC 4271 (core BGP spec).

> The WG chairs intend to forward this draft to the IESG with the current level of implementation.

>https://trac.ietf.org/trac/idr/wiki/draft-ietf-idr-bgp-extended->implementations<https://trac.ietf.org/trac/idr/wiki/draft-ietf-idr-bgp-extended-%3eimplementations> says : 5a

Does not send Extended Message capability

Yes

Yes

Yes


>I may be misunderstanding the implementation report, but my reading of the above is that none of the reported >implementations sends the capability hence no implementation supports draft-ietf-idr-bgp-extended-messages..
>Here this document is updating RFC 4271, so it is not a minor extension for a niche use case. So I don't see the >arguments for not requiring the IDR's usual two interoperable implementations.

>In our mail thread, we discovered that the implementation report should have indicate
>full support.   I'm waiting for Olivier to update based on the last point, but my understanding is that
All implementations have full support.

[Bruno] ok, solved
>----
>§ 1
>" As BGP is extended to support newer AFI/SAFIs and
>   newer capabilities (e.g., [I-D.ietf-sidr-bgpsec-protocol<https://tools.ietf.org/html/draft-ietf-idr-bgp-extended-messages-27#ref-I-D.ietf-sidr-bgpsec-protocol>]), there is
>   a need to extend the maximum message size beyond 4096 octets.  "
>
> https://tools.ietf.org/html/draft-ietf-idr-bgp-extended-messages-27#section-1


>[I-D.ietf-sidr-bgpsec-protocol<https://tools.ietf.org/html/draft-ietf-idr-bgp-extended-messages-27#ref-I-D.ietf-sidr-bgpsec-protocol> is now RFC 8205 (thanks for updating the reference). It has removed the normative/any >reference to draft-ietf-idr-bgp-extended-messages. So presumably BGP Sec does not need draft-ietf-idr-bgp->extended-messages. Can we have an update on this?  Can the introduction of draft-ietf-idr-bgp-extended-messages >be updated to introduce on the real reasons/needs?

SIDR gave up waiting on IDR's work and took a reduced size of encryption.   The security people were very sad about the reduction.  How is this working in practice? WG sidrops can answer this question.

The real problem is there is finite message space.
[Bruno] So too bad the draft does not solve the real problem ;-) (as message space is still limited)

[Sue]: The message is still finite (smile),  but I hope 65,635 (smile) is probably enough for 10 years.

There are two ways to send lots of BGP routes and/or attributes - to "chunk" these into small message sizes or place these into a large message.    Some applications may always want to "chunk" into message sizes smaller than 4096.   Some application make work more efficiently utilizing large message size.  As the potential number of NLRI and BGP attributes grows, new applications may consider large message sizes to a more reasonable way to exchange amounts of BGP data.

The decision on how to send BGP messages over the wire is a peer-by-peer decision in a network.   Each BGP peer is in charge of reformatting the BGP message it sends to the remote BGP peer.   The BGP capabilities in this draft represent an agreement between two BGP peers.  These are facts of basic BGP implementations
[Bruno] Let's take the example of a BGP Route Reflector having a mixed of clients, some supporting Extend Message (EM) and some not.
>From an EM client, it receives a BGP update containing one (single) NLRI with an Total Path Attribute Length bigger than  4090 byte (e.g. having 1000 large communitiess)
Can you explain me how it behaves to send this NLRI to an IBGP peer not supporting EM?
[Sue]:  Gladly...
A BGP route reflector (RR-A) is peering with each of its BGP clients.   It agrees with each of its clients (BGP-C1, BGPC2, BGPC3) whether the BGP Extended messages is accepted or not (based on capability announcement).
Let us assume that BGP-C1 does not support extended messages, and  BGP-C2 and BGP-C3 do support extended messages.

If the policy in RR accepts a total path length of 1000 large communities with  a single NLRI, it may send the BGP attribute via extended communities to BGP-C2 and BGP-C2.
The policy in the RR must specify whether:

a)      the "too long" attribute path length must be dropped, or

b)      the BGP attribute path can be truncated.

This is a policy issue within the BGP peer and the network operation staff.  Each operator may choose to write this policy differently.
[Bruno2] So we agree that a new behavior is REQUIRED. The draft does not mention this. The ability to change the behavior by policy does not remove the need for specifying the error handling.

Rather than specify the numerous cases in the protocol specification,  the previous discussions (private and public) asked us to trim this information out of the protocol specification.
[Bruno2] Can't be have a general rule which is good enough to handle the issue? Previous discussions seemed based on the fact the error condition would happen for future attribute. This is not the case.

If you believe this is worthy of a use case draft for the sake of BESS, LSVR, SPRING, Grow and others, I will happily WG chair sponsor a draft that can be reviewed by all these WGs.
[Bruno2] I believe that this is worthy in this specification.


- so I do not believe adding these facts to Randy's draft is useful.  In fact, in the previous rounds of review this point was made.

Your question (and perhaps Jakob's) is whether we have reached a time/place where the "chunking" approach will fail on a peer-by-peer basis?   This draft does not make this statement.

The point of this draft is to enable more efficient processing by increasing the message size for BGP peers that send a lot of data (e.g. BGP-LS) .   The choice to "chunk" or stream in large BGP message is an implementation choice for a particular BGP application.

----
§4

§3 says "A peer which does not advertise this capability MUST NOT send BGP
   Extended Messages, and BGP Extended Messages MUST NOT be sent to it."
[I believe randy accepted this change with the following text in version 4 [upcoming below]]/.

--------------

Fine. Text in §4 should probably be aligned with the above .e.g.

OLD: A BGP speaker
   MAY send Extended Messages to its peer only if it has received the
   Extended Message Capability from that peer.

NEW:
A BGP speaker
   MAY send Extended Messages to its peer only if it has sent and received the
   Extended Message Capability to and from that peer.


----

>"   Applications generating information which might be encapsulated
>   within BGP messages MUST limit the size of their payload to take the
>   maximum message size into account."
>
>I don't see what new behavior is been defined here. If there is none, I would suggest to remove this sentence

Bruno - people asked to put this text in as an operational warning.  At this point, I'm going to say unless there is harm it should remain.
[Bruno] OK.

----
>   A BGP announcement will, in the normal case, propagate throughout the
>   BGP speaking Internet; and there will undoubtedly be BGP speakers
>   which do not have the Extended Message capability.  Therefore,
>   putting an attribute which can not be decomposed to 4096 octets or
>   less in an Extended Message is a likely path to routing failure.
>
>The issue is not specific to attributes bigger than 4096 octets, but to BGP message whose length is bigger than >4096, irrespective of the size of each attribute.
>Please elaborate on what you mean by
>
>"an attribute which cannot be decomposed to 4096 octets"

An attribute which cannot be decomposed to 4096 is an attribute with length 6000 inside of a BGP message which requires the entire attribute to be included.   No such attribute exists today.   If a small BGP peers create a "killer application" that uses a 6000 byte attribute, the rest of the Internet does not have to use it.   However, the person defining that attribute must carefully specify it must go with Extended BGP messages.


[Bruno] I don't think that I agree with this. E.g. I don't see where RFCs 1997 or 8092 [(large)community] limits the size of their attribute. Hence I don't see why you could not have, today (with Extended Message support) , attributes larger than 4096 octets. Most probably same status for the BGPSec attribute since they wanted for more space.
[Sue]: You make a good point about the possibility of large communities not limiting the size.  It is worthy of a change to both the large communities and a footnote in Randy's specification.  How about the addition of a single sentence to point people in the right direction.

Old/
   A BGP announcement will, in the normal case, propagate throughout the
   BGP speaking Internet; and there will undoubtedly be BGP speakers
   which do not have the Extended Message capability.  Therefore,
   putting an attribute which can not be decomposed to 4096 octets or
   less in an Extended Message is a likely path to routing failure.
/
New/
   A BGP announcement will, in the normal case, propagate throughout the
   BGP speaking Internet; and there will undoubtedly be BGP speakers
   which do not have the Extended Message capability.  Therefore,
   putting an attribute which cannot be decomposed to 4096 octets or
   less in an Extended Message is a likely path to routing failure.
   One example of a valid attribute which cannot be precisely decomposed if it
   sent in Extended message is a large community attribute with 1000 communities.
   (12,000 bytes).
/



This text was added during an earlier discussion where the error cases of a large BGP attribute was given.  The originator of the common could not provide the exact description, but we agreed to put this  "warning label" into the draft.   This is one of the cases where adding/removing text gets to be problematic.

>---
>"   It is RECOMMENDED that BGP protocol developers and implementers are
>   conservative in their application and use of Extended Messages."
>
>What does this mean exactly? That they don't use this extension?
>That they don't use this extension unless XX_TO BE SPECIFIED_XX?

This warning was text that other participants asked to be added.

It is a common sense warning.  Extended messages are enabled on a peer by peer basis as all BGP capabilities are.  If you run into a Peer that does not support the larger sizes, the BGP messages will need to be reformatted to the smaller size.   Running immediately to the Extended messages as the default could cause problems.

[Bruno] I think that the sentence could be elaborated. Indicating the problem, how the specification partially address it, and how the specification does not address it.
In "Running immediately to Extended Message" I don't think they that the issue is with "immediately". Whenever you start the transition, your likely to run into problems.


[Sue]: IMHO specification specifies how to encode the bytes on the wire and the appropriate error handling of the messages.   If I have missed something in this case, please let me know.
[Bruno2] IMHO specifications need to handle error conditions that they create by design.

The specification does not specify the necessary policies implementation should have in order to decide what to do in the case of mixed network.   The clear indication was this type was not appropriate for the protocol specification.

Earlier discussion indicated that these is policy and mechanism specific.  It is worthy of "use case or application notes" written for specific applications (BESS, SPRING, GROW), but not in the specification.  Again, if it is significant enough concern for operators and implementers - it would be worthy to have combined use case document besides this point.    I'm happy to sponsor cross-area work on this use case.

---
  Future protocol specifications will need to describe how to handle
   peers which can only accommodate 4096 octet messages.

>Why is this limited to future specifications? A priori, using existing BGP mechanism (AFI/SAFI, attributes, * >communities) one could exceed the size of 4096 octets. How does the BGP speaker supposed to behave in this ?>case? This should be described in this specification. Note that this is not a case of error handling, as every BGP
> speaker is behaving as specified.

I fail to see your logic.  At this point, all BGP data (AFI/SAFI and attributes) must fit within a 4096 message.   Therefore, a BGP peer A which connects on link 1 to BGP peer B with 4096 message and on link to a BGP peer C with extended messages only has to worry about reformatting the data.

In the future, the first group that defines attribute or an NLRI that does not fit within a 4096 BGP message and cannot be split, will need to define how that works in the specification.  There is no need to refine the current set of specifications.
[Bruno]
1) cf above, I believe that there are existing BGP Attributes which may not fit into a bytes 4096 BGP message
2)Even in the absence of such attribute, you have the issue of 2 smaller attributes whose cumulative size does not fit into a 4096 bytes message.

The problem is introduced by this specification, hence I believe this specification needs to specify how to handle this case.

[Sue]:  IHMO the message transmission and reception error handling issues are complete in the combination of RFC4271 and the extended message section 5.

What is not in this specification is the policy issues about what happens if you combine 2 smaller attributes.  For example, if you have a large community attribute of 400 communities (4800 bytes), you will likely run into problems when combining with some longer AS path lengths.  [4800--> 5096 bytes is a short jump]. People on the list and in-person argued this was policy and implementation specific details.
[Bruno2] It's not implementation specific. This is an issue created by the specification. Let's discuss it in public.

I agree with you that it is extremely useful to the operator community to have this type of policy details discussed to see if there is some desire to have common policies and error communities.   It is the type of thing that GROW and IDR should be deeply involved for large communities.   BESS, LSVR, SPRING and others should consider what policy should be adapted for their applications.

>----
>Depending on the above specification, a section describing the operational consequences in a network (such as the >Internet, BGP Enabled ServiceS/VPN networks) is probably needed. Possible consequences could be BGP NLRI >being removed in the middle of such network, or (extended) community (such as Route Targets) been removed. >Both having significant consequences on the availability provided by the network.

At this point, there are no attributes or NLRI groupings that do not fit within 4096.   If you define one (as SIDR did), then you will need to have it reviewed by IDR, BESS, SPRING, and operational groups (GROW, etc).
[Bruno] no, cf above.
[Sue]:  Thank you for reminding me of the potential of large communities to blow up BGP.  I have a new worry to add to my BGP concerns list!
[Bruno2] IMHO, the problem is more with Extended Message than with large community. And we can work to consider and address this problem.
--Bruno

Thanks,
--Bruno


---
>§4
>OLD: The Extended Message Capability only applies to all messages except for the >OPEN message.
>Probably
>NEW: The Extended Message Capability applies to all message types except for the >OPEN message (type 1).
>----
>§8

>"This extension to BGP does not change BGP's underlying security issues »

>Before evaluating this, I think this document should first specified how a BGP

>messages bigger than 4096 octets is handled when it needs to be sent to a

>received not supporting this extension.

Nits:
>OLD : to reduce compexity
>NEW : to reduce complexity

Thanks,
--Bruno

From: Idr [mailto:[e-mail]] On Behalf Of Susan Hares
Sent: Tuesday, January 29, 2019 12:33 PM
To: [e-mail]<mailto:[e-mail]>
Subject: [Idr] WG Last Call on Extened Message Support


This begins a 2 week WG LC on Extended Message Support for BGP (draft-ietf-idr-bgp-extended-messages-27).  You can access the draft at:

https://datatracker.ietf.org/doc/draft-ietf-idr-bgp-extended-messages/

The authors should indicate whether they know of any IPR.   Implementers are encouraged to update the  implementation data at:

https://trac.ietf.org/trac/idr/wiki/draft-ietf-idr-bgp-extended-implementations

The draft provides a means for expanding the BGP message to 65535 octets for all messages except OPEN messages.  BGP message space is running short for all of the potential attributes or additions proposed by BGP-LS features.

The WG chairs intend to forward this draft to the IESG with the current level of implementation.

As you comment on the draft, please consider if: a) the technology is mature, b) the additional space in a BGP message would be helpful for those deploying BGP-LS or SR, and c) if the specification is ready for publication.

Sue Hares (WG Chair, Shepherd)



_________________________________________________________________________________________________________________________



Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc

pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler

a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,

Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.



This message and its attachments may contain confidential or privileged information that may be protected by law;

they should not be distributed, used or copied without authorisation.

If you have received this email in error, please notify the sender and delete this message and its attachments.

As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.

Thank you.

_________________________________________________________________________________________________________________________



Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc

pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler

a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,

Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.



This message and its attachments may contain confidential or privileged information that may be protected by law;

they should not be distributed, used or copied without authorisation.

If you have received this email in error, please notify the sender and delete this message and its attachments.

As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.

Thank you.

_________________________________________________________________________________________________________________________



Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc

pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler

a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,

Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.



This message and its attachments may contain confidential or privileged information that may be protected by law;

they should not be distributed, used or copied without authorisation.

If you have received this email in error, please notify the sender and delete this message and its attachments.

As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.

Thank you.

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.