Re: [nvo3] 答复: Comments on NVO3 data plane requirements for OAM

[Tom Herbert <[e-mail]>]

On Fri, Nov 21, 2014 at 12:37 AM, Mach Chen <[e-mail]> wrote:
>
> Hi Deepak,
>
> > >In addition, I see the value of you proposed optional "measurement"
> > >field, it could be used to carry some correlation (e.g., block/period
> > >number) and timestamp information, then combine with the marking bit,
> > >it can greatly simplify the marking based solution.
> >
> > +++DK:
> > I think adding information regarding measurement field, block, period, etc. is not
> > required in data path as more information reduces the mtu, and this can easily be
> > added if required by TLV to OAM functionality with new subtype (as this is control
> > or configuration functionality).
>
> I am talking about two things here:
>
> 1) the fixed marking bits, I think they are necessary for passive PM;
>
> 2) the correlation information, timestamps, counters, they could be communicated either through in-band or out-or-band, each way has its pros and cons;
>
> > Also even passive oam loss measurement solution to calculating loss is not
> > accurate as packets can arrive late outside the measuring blocks. Even in that
>
> If it only depends on the marking bit and the measuring period is set to a very small interval, indeed, that will affect the accuracy. But from engineering point view, an operator and an implementation will not set (or support) to a very aggressive period.

I don't think this would be true in our data center. The very reason
we would enable a passive mechanism is for getting accurate
measurements of high granularity. For instance, if we want to use this
as real time feedback for congestion control we need fine grained
information. Also, as part of debugging a customers problems it may
come down to us being able to identify specific packets that are being
dropped or experiencing unusual latency, I don't see how marking with
a couple of bits is sufficient for that. We already have this deployed
in the pre-NV world (mostly provided by TCP), in an NV world there are
many cases we won't have visibility into the customer's protocol so
we'll need to find alternative methods (which likely results in
annotating packets).

> And if the packets can carry some correlation information (e.g., the block/period number), then the accuracy should be no problem.
>
> In theory, you are right, if the delay of the packets of block exceed a threshold (e.g., a block period), the packets may be mis-counted into another block.
>
> > case to get accurate measurement instead of ipfix method, better to use OAM to
> > exchange these marked packet counters on both ends and do loss measurement
> > between two consecutive loss measurement replies.
>
> I am fine with either way for communicating the counters and timestamps.
>
> >
> > For loss measurement, why we have to count traffic for marked packets only and
> > not maintain counters per flow?
>
> I'm not sure what's your question here.
>
> To calculate the packet loss, counters maintenance (no matter at where) is necessary, it depends on the specific implementation.
>
> Best regards,
> Mach
>
>
> > -----Original Message-----
> > From: Deepak Kumar (dekumar) [mailto:[e-mail]]
> > Sent: Friday, November 21, 2014 2:34 PM
> > To: Mach Chen; Tom Herbert
> > Cc: [e-mail]
> > Subject: Re: [nvo3] 答复: Comments on NVO3 data plane requirements for OAM
> >
> > Hi Mach,
> >
> > Please see inline +++DK:
> >
> >
> > On 11/20/14 5:02 PM, "Mach Chen" <[e-mail]> wrote:
> >
> > >Hi Tom,
> > >
> > >Please see my response inline...
> > >
> > >> -----Original Message-----
> > >> From: Tom Herbert [mailto:[e-mail]]
> > >> Sent: Friday, November 21, 2014 1:28 AM
> > >> To: Mach Chen
> > >> Cc: [e-mail]
> > >> Subject: Re: [nvo3] 答复: Comments on NVO3 data plane requirements for
> > >> OAM
> > >>
> > >> On Wed, Nov 19, 2014 at 5:54 PM, Mach Chen <[e-mail]>
> > wrote:
> > >> > Hi Tissa,
> > >> >
> > >> > Thanks for your response!
> > >> >
> > >> > Please see my response inline...
> > >> >
> > >> >> -----Original Message-----
> > >> >> From: nvo3 [mailto:[e-mail]] On Behalf Of Tissa
> > >> >> Senevirathne
> > >> >> (tsenevir)
> > >> >> Sent: Wednesday, November 19, 2014 8:45 PM
> > >> >> To: Haoweiguo; Tom Herbert
> > >> >> Cc: Greg Mirsky; Tapraj Singh; Deepak Kumar (dekumar);
> > >> >> [e-mail]
> > >> >> Subject: Re: [nvo3] 答复: Comments on NVO3 data plane requirements
> > >> >> for OAM
> > >> >>
> > >> >> Hi Weiguo, Mach et,al
> > >> >>
> > >> >> The discussion here is NVO3 data plane requirements for OAM. Like
> > >> >> I have said
> > >> >
> > >> > You are right, this discussion is about "NVO3 data plane
> > >> > requirements
> > >>for OAM",
> > >> but recently the focus is Performance Measurement (PM) requirement to
> > >>NVO3
> > >> that is also one of the OAM functions.
> > >> >
> > >> >> earlier,  we do not need to complicate the Data Plane. Can you
> > >> >> explain to me
> > >> >
> > >> > "Complicate/simple" is not the goal, the goal is to define a
> > >>reasonable solution
> > >> that can satisfy the requirement. That's why I agree with Greg that
> > >>we should  firstly make the agreement on the requirement.
> > >> >
> > >> Mach,
> > >>
> > >> The nvo3 OAM requirements draft
> > >> (draft-ashwood-nvo3-oam-requirements-01) seems to already contain a
> > >>fairly  comprehensive list of requirements. Particularly apropos to
> > >>this discussion are:
> > >>
> > >>    R13) NVO3 OAM frames MUST be forwarded along the same path (i.e.,
> > >>    links (including LAG members) and nodes) as the NVO3 data frames.
> > >>
> > >>    R16) NVO3 OAM should be extensible such that new functionality and
> > >>    information elements related to this functionality can be introduced
> > >>    in the future.
> > >>
> > >> I believe that an an active OAM message format like Tissa describes
> > >>would meet  these and most of the others in that draft.
> > >
> > >There are also the following requirements listed in the draft:
> > >
> > >   R7) NVO3 OAM MUST support measurement of per VNI frame loss between
> > >   two NV Edge devices that support the same VNI within a given NVO3
> > >   domain.
> > >
> > >   R8) NVO3 OAM MUST support measurement of per VNI two-way frame
> > delay
> > >   between two NV edge devices that support the same VNI within a given
> > >   NVO3 domain.
> > >
> > >   R9) NVO3 OAM MUST support measurement of per VNI one-way frame
> > delay
> > >   between two NV Edge devices that support the same VNI within a given
> > >   NVO3 domain.
> > >
> > >   R10) NVO3 OAM MUST support measurement of per VNI frame delay
> > >   variation between two NV Edge devices that support the same VNI
> > >   within a given NVO3 domain.
> > >
> > >
> > >>
> > >> If a passive mechanism is indeed required,
> > >
> > >I personally think it is indeed required, and we also received such
> > >requirements from the operators.
> > >
> > >> then we need to consider how to
> > >> meet the extensibility requirement. I don't believe that allocating
> > >>two bit flags in  the encapsulation header is at all an extensible
> > >>solution. The reserved header  bits are too a precious resource to be
> > >>allocated for such a narrow purpose and for
> > >
> > >Looking through the bits in some headers, we could find that every bit
> > >in a header has its own purpose. It's better that one bit could be
> > >defined for as many usages as possible, but there is always tradeoff.
> > >
> > >As for the two bits for passive PM that include loss, one/two way
> > >delay, delay variation and throughput, I may not think this is a narrow purpose.
> > >And if you want, you may use the marking bit for some policies control.
> > >
> > >> something not required for protocol operation. As previously
> > >>discussed in this  thread, using one bit to get one-way time delay
> > >>measurements is not even viable  in a lot deployments-- in this case
> > >>we probably need timestamps to get RTT.
> > >
> > >For the deployments in question, the challenge for one way time delay
> > >is time synchronization and its accuracy. The solution of using one-bit
> > >for one-way delay is really workable, there have been some prototypes
> > >and experiments show that.
> > >
> > >Since time synchronization is not needed for RTT, IMHO, measure RTT
> > >should be the easiest way to go.
> > >
> > >
> > >>
> > >> To support passive OAM support in GUE, I would probably propose to
> > >>add a  generic optional "measurement" field. This would provide some
> > >>number of bits  in the header that can be used for passive measurement
> > >>(possibly a few  different sizes say 32, 64, 128 bits). The field can
> > >>be structured to allow different  mechanisms (e.g. include timestamps
> > >>for RTT measurement). This also reduces  the constraints on the
> > >>measurement techniques, for instance the marking  technique might no
> > >>longer limited to use a single bit which should reduce the  complexity
> > >>needed to deal with OOO or packet loss.
> > >
> > >Even with the solution as above, seems there needs at least one bit(at
> > >the fix position of the header) that indicates there is an optional
> > >field exist. In the case we could have opportunity and "enough"
> > >reserved bits to allocate for the marking bits, I'd like to suggest
> > >allocating two bits for passive PM.
> > >
> > >In addition, I see the value of you proposed optional "measurement"
> > >field, it could be used to carry some correlation (e.g., block/period
> > >number) and timestamp information, then combine with the marking bit,
> > >it can greatly simplify the marking based solution.
> >
> > +++DK:
> > I think adding information regarding measurement field, block, period, etc. is not
> > required in data path as more information reduces the mtu, and this can easily be
> > added if required by TLV to OAM functionality with new subtype (as this is control
> > or configuration functionality).
> > Also even passive oam loss measurement solution to calculating loss is not
> > accurate as packets can arrive late outside the measuring blocks. Even in that
> > case to get accurate measurement instead of ipfix method, better to use OAM to
> > exchange these marked packet counters on both ends and do loss measurement
> > between two consecutive loss measurement replies.
> >
> > For loss measurement, why we have to count traffic for marked packets only and
> > not maintain counters per flow?
> >
> > Thanks,
> > Deepak
> > >
> > >Thanks,
> > >Mach
> > >
> > >>
> > >> Tom
> > >>
> > >> >
> > >> >> what difference it make to the data plane whether it is
> > >> >> active/passive or some other means of OAM.
> > >> >
> > >> > Active/passive is mainly regarding to PM which normally includes
> > >>Active and
> > >> Passive PM.
> > >> >
> > >> > Active PM measures the injected packets (e.g., OAM packets) to
> > >>evaluate the
> > >> performance of a path. Passive PM measures the performance of the
> > >>real/live  traffic of a path, it reflects the real performance of the
> > >>path. For more detail  about active/passive PM, you may refer to the
> > >>material of IPPM WG.
> > >> >
> > >> >>
> > >> >> All what it needs to know is that the packet is an OAM packet and
> > >> >> it is addressed to the local device,
> > >> >
> > >> > What you are talking are just part of the OAM functions (e.g., CC,
> > >>CV), for
> > >> passive PM, OAM packets may not be needed.
> > >> >
> > >> >
> > >> > Best regards,
> > >> > Mach
> > >> >
> > >> >>
> > >> >> -----Original Message-----
> > >> >> From: Haoweiguo [mailto:[e-mail]]
> > >> >> Sent: Tuesday, November 18, 2014 7:50 PM
> > >> >> To: Tissa Senevirathne (tsenevir); Tom Herbert
> > >> >> Cc: Greg Mirsky; Tapraj Singh; Deepak Kumar (dekumar);
> > >> >> [e-mail]
> > >> >> Subject: RE: [nvo3] 答复: Comments on NVO3 data plane requirements
> > >> >> for OAM
> > >> >>
> > >> >> Hi  Tissa,
> > >> >> Your solution is active OAM,  i think it is a basic and important
> > >> >> solution in whole OAM framework.The disccussed thread is about
> > >>passive
> > >> OAM.
> > >> >>  Both active and passive OAM have its pros/cons, both have its
> > >> >> usecases and scenarios.The regular method for passive OAM is to
> > >> >> add marking bits in packet header, in NVO3 case, the marking bits
> > >> >> had better be set in NVO3 header.But just as Greg said,currently
> > >> >> it's unfortunate that there is no accepted OAM requirements, gap
> > >> >> analysis, and etc in the WG. We hope this work could be progressed
> > >> >> more
> > >>quickly.
> > >> >> Thanks
> > >> >> weiguo
> > >> >> ________________________________________
> > >> >> From: Tissa Senevirathne (tsenevir) [[e-mail]]
> > >> >> Sent: Wednesday, November 19, 2014 8:25
> > >> >> To: Tom Herbert
> > >> >> Cc: Greg Mirsky; Tapraj Singh; Deepak Kumar (dekumar);
> > >> >> [e-mail]
> > >> >> Subject: Re: [nvo3] 答复: Comments on NVO3 data plane requirements
> > >> >> for OAM
> > >> >>
> > >> >> Hi Tom
> > >> >>
> > >> >> Your interpretation is correct. The entropy allows OAM packets to
> > >> >> follow the same path as the data packet.
> > >> >>
> > >> >> As I noted earlier on in the thread, OAM processing would not kick
> > >> >> in unless address matches the MEP/MIP. If address match MEP/MIP
> > >> >> and OAM bit is set, then OAM processing begins.
> > >> >>
> > >> >> -----Original Message-----
> > >> >> From: Tom Herbert [mailto:[e-mail]]
> > >> >> Sent: Tuesday, November 18, 2014 4:20 PM
> > >> >> To: Tissa Senevirathne (tsenevir)
> > >> >> Cc: Greg Mirsky; Tapraj Singh; [e-mail]; Deepak Kumar
> > >> >> (dekumar)
> > >> >> Subject: Re: [nvo3] 答复: Comments on NVO3 data plane requirements
> > >> >> for OAM
> > >> >>
> > >> >> On Tue, Nov 18, 2014 at 3:54 PM, Tissa Senevirathne (tsenevir)
> > >> >> <[e-mail]> wrote:
> > >> >> > Greg
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > I disagree with you on FM and PM cannot be achieved in ECMP
> > >> environment.
> > >> >> > Significant amount of work has gone in to this area during TRILL
> > >>OAM.
> > >> >> > Please check the use of Flow entropy functionality proposed in
> > >>NVO3 OAM.
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > https://tools.ietf.org/html/draft-tissa-nvo3-oam-fm-00
> > >> >> >
> > >> >> Tissa,
> > >> >>
> > >> >> If I am reading this correctly, the OAM message would be composed
> > >> >> of the encapsulation header, followed by 128 bytes of which
> > >> >> contains a pseudo header for switching, followed by a self defining OAM
> > message.
> > >> >> The OAM bit is only used at the receiver to distinguish data
> > >> >> messages for OAM messages for processing. Is this interpretation correct?
> > >> >>
> > >> >> Thanks,
> > >> >> Tom
> > >> >>
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > From: nvo3 [mailto:[e-mail]] On Behalf Of Greg
> > >> >> > Mirsky
> > >> >> > Sent: Tuesday, November 18, 2014 3:03 PM
> > >> >> > To: Tapraj Singh
> > >> >> > Cc: [e-mail]
> > >> >> > Subject: Re: [nvo3] 答复: Comments on NVO3 data plane requirements
> > >> >> > for OAM
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > Hi Tapraj,
> > >> >> >
> > >> >> > though I agree and support with idea of having OAM flag in NVO3
> > >> >> > header I have to point to:
> > >> >> >
> > >> >> > absence of WG agreed upon OAM Requirements; no gap analysis of
> > >> >> > tools for NVO3 OAM; OAM flag does not help passive performance
> > >> >> > measurement marking method (two bit-long field for marking in
> > >> >> > fixed
> > >> position).
> > >> >> >
> > >> >> > I agree that PW VCCV and GAL/G-ACh can be viewed as MPLS
> > >> >> > identification of OAM packet (though not necessarily OAM). But
> > >> >> > IP clearly doesn't have such identification for OAM and that, in
> > >> >> > part, why in-band requirement for IP OAM, both FM and Active PM,
> > >> >> > is not attainable
> > >> >> (ECMP environment).
> > >> >> >
> > >> >> > Regards,
> > >> >> >
> > >> >> > Greg
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > On Tue, Nov 18, 2014 at 1:31 PM, Tapraj Singh
> > >> >> > <[e-mail]>
> > >>wrote:
> > >> >> >
> > >> >> > Hi All,
> > >> >> >
> > >> >> >  I totally agree with the point made by Deepak and Tissa here.
> > >> >> > Our OAM should follow the data path for services as much as
> > >> >> > possible and all other protocol specific information should be
> > >> >> > in the OAM protocol specific TLVs.
> > >> >> >
> > >> >> > LAYER2 OAM
> > >> >> >
> > >> >> > In term of identify the OAM packet, first level of
> > >> >> > identification for
> > >> >> > L2 OAM Should be the MAC address and send level of hierarchy
> > >> >> > should be the ether type or OUI.
> > >> >> > No other OAM Specific field should be allowed in the packet header.
> > >> >> >
> > >> >> >  Please note that L3 OAM and MPLS also follow the same principle.
> > >> >> >
> > >> >> > Thanks
> > >> >> > Tapraj
> > >> >> >
> > >> >> >
> > >> >> > On 11/17/14 12:39 PM, "Deepak Kumar (dekumar)"
> > >> >> > <[e-mail]>
> > >> >> wrote:
> > >> >> >
> > >> >> >>I Agree with Tissa below. My Goal also was to point out that
> > >> >> >>instead of complicating the header, we can do OAM performance
> > >> >> >>within OAM channel itself and this is extensible and can be done
> > >> >> >>in hardware which is why mostly things are added in header.
> > >> >> >>
> > >> >> >>Also, Operators keep asking for new OAM tools (Fault detection,
> > >> >> >>verification, isolation, Interworking, alarm, putting service in
> > >> >> >>maintenance and perform test)  and Performance tools, eg:
> > >> >> >>(Delay/Jitter, Actual Loss Measurement, Synthetic Loss, loopback
> > >> >> >>signaling like TDM, Generate frames to verify qos etc.) and so
> > >> >> >>OAM Channel solution will be extensible.
> > >> >> >>
> > >> >> >>Thanks,
> > >> >> >>Deepak
> > >> >> >>
> > >> >> >>On 11/17/14 8:47 AM, "Tissa Senevirathne (tsenevir)"
> > >> >> >><[e-mail]>
> > >> >> >>wrote:
> > >> >> >>
> > >> >> >>>I think we are complicating OAM beyond what it is needed.
> > >> >> >>>
> > >> >> >>>As far as packet encapsulation is concern, all what is needed
> > >> >> >>>is single bit. This bit is needed to prevent OAM packets
> > >> >> >>>leaking out from the domain.
> > >> >> >>>
> > >> >> >>>Termination of OAM and processing of it happen based on the
> > >> >> >>>addressing in the packet.
> > >> >> >>>
> > >> >> >>>E.g. if Address matches and OAM bit is set then it is an OAM
> > >> >> >>>packet addressed to the local MEP/MP.
> > >> >> >>>
> > >> >> >>>Not other way around. Why? Because we want OAM to be as closely
> > >> >> >>>as possible follow the Data path.
> > >> >> >>>
> > >> >> >>>If we need to have performance and delay measurements, we
> > >> >> >>>SHOULD NOT mutate the packet header.
> > >> >> >>>
> > >> >> >>>Instead OAM specific extensions should be in the OAM shim.
> > >> >> >>>
> > >> >> >>>As an example. You could have packet fragment (which is
> > >> >> >>>sometimes called flow entropy) and at the end of that you can
> > >> >> >>>have all of the stuff you need in the world of OAM.
> > >> >> >>>
> > >> >> >>>Hope this clarify
> > >> >> >>>
> > >> >> >>>Thanks
> > >> >> >>>Tissa
> > >> >> >>>-----Original Message-----
> > >> >> >>>From: nvo3 [mailto:[e-mail]] On Behalf Of Tom
> > >> >> >>>Herbert
> > >> >> >>>Sent: Monday, November 17, 2014 8:02 AM
> > >> >> >>>To: Marc Binderberger
> > >> >> >>>Cc: Greg Mirsky; Mach Chen; Deepak Kumar (dekumar);
> > >> >> >>>[e-mail]; Haoweiguo; Larry Kreeger (kreeger); Vero Zheng;
> > >> >> >>>Jon Hudson
> > >> >> >>>Subject: Re: [nvo3] 答复: Comments on NVO3 data plane
> > >> >> >>>requirements for OAM
> > >> >> >>>
> > >> >> >>>On Mon, Nov 17, 2014 at 12:01 AM, Marc Binderberger
> > >> >> >>><[e-mail]>
> > >> >> >>>wrote:
> > >> >> >>>> Hello Deepak et al.,
> > >> >> >>>>
> > >> >> >>>> so this sounds like we need more than just a (2nd) bit for
> > >> >> >>>>delay measurement.
> > >> >> >>>> Seems we need an optional header extension or a TLV to carry
> > >> >> >>>>all the  information (timestamps, oam Subtype). Sounds
> > >> >> >>>>definitely more than a  32/64bit header could carry (*).
> > >> >> >>>>
> > >> >> >>>> The optional header extension, when done similar to GUE, has
> > >> >> >>>> a fixed position. For the TLV this would be an additional
> > >> >> >>>> requirement. This would allow for hardware-stamping.
> > >> >> >>>>
> > >> >> >>>The alternative is to do active delay measurement using
> > >>request/reply.
> > >> >> >>>We should be able to define the requirements so that an OAM
> > >> >> >>>message corresponding to a flow which would be routed in
> > >> >> >>>exactly the same way as a data message for the flow. Larry
> > >> >> >>>mentioned that we might even want to put a "fake" packet header
> > >> >> >>>as the first part of the encapsulated payload of an OAM message for
> > instance.
> > >> >> >>>
> > >> >> >>>> Now if we introduce such an OAM extension header it could as
> > >> >> >>>> well carry the "first" bit we discussed for packet loss
> > >>measurement (?).
> > >> >> >>>>
> > >> >> >>>>
> > >> >> >>>> Regards, Marc
> > >> >> >>>>
> > >> >> >>>> (*: at least all proposals so far have a base header that
> > >> >> >>>> fits into
> > >> >> >>>> 32/64 bit, plus IP and potential UDP)
> > >> >> >>>>
> > >> >> >>>>
> > >> >> >>>>
> > >> >> >>>>
> > >> >> >>>> On Sun, 16 Nov 2014 16:44:54 +0000, Deepak Kumar (dekumar)
> > >>wrote:
> > >> >> >>>>> Hi,
> > >> >> >>>>>
> > >> >> >>>>> Please see inline +++DK:
> > >> >> >>>>>
> > >> >> >>>>> On 11/14/14 11:09 AM, "Jon Hudson" <[e-mail]>
> > >>wrote:
> > >> >> >>>>>
> > >> >> >>>>>>
> > >> >> >>>>>> One comment in line....
> > >> >> >>>>>>
> > >> >> >>>>>>> On Nov 13, 2014, at 11:47 PM, Vero Zheng
> > >> >> >>>>>>><[e-mail]>
> > >> >> >>>>>>>wrote:
> > >> >> >>>>>>>
> > >> >> >>>>>>> Hi Tom,
> > >> >> >>>>>>>
> > >> >> >>>>>>> Please see in-line.
> > >> >> >>>>>>>
> > >> >> >>>>>>> BR, Vero
> > >> >> >>>>>>>
> > >> >> >>>>>>>> -----Original Message-----
> > >> >> >>>>>>>> From: nvo3 [mailto:[e-mail]] On Behalf Of
> > >> >> >>>>>>>> Tom Herbert
> > >> >> >>>>>>>> Sent: Friday, November 14, 2014 4:27 PM
> > >> >> >>>>>>>> To: Mach Chen
> > >> >> >>>>>>>> Cc: Greg Mirsky; Haoweiguo; Marc Binderberger; Larry
> > >> >> >>>>>>>> Kreeger; [e-mail]
> > >> >> >>>>>>>> Subject: Re: [nvo3] 答复: Comments on NVO3 data plane
> > >> >> >>>>>>>> requirements for OAM
> > >> >> >>>>>>>>
> > >> >> >>>>>>>> On Wed, Nov 12, 2014 at 5:13 PM, Mach Chen
> > >> >> >>>>>>>> <[e-mail]>
> > >> >> >>>>>>>> wrote:
> > >> >> >>>>>>>>> Hi Tom,
> > >> >> >>>>>>>>>
> > >> >> >>>>>>>>>> -----Original Message-----
> > >> >> >>>>>>>>>> From: Tom Herbert [mailto:[e-mail]]
> > >> >> >>>>>>>>>> Sent: Thursday, November 13, 2014 3:11 AM
> > >> >> >>>>>>>>>> To: Marc Binderberger
> > >> >> >>>>>>>>>> Cc: Mach Chen; Greg Mirsky; Haoweiguo; [e-mail];
> > >> >> >>>>>>>>>> Larry Kreeger
> > >> >> >>>>>>>>>> Subject: Re: [nvo3] 答复: Comments on NVO3 data plane
> > >> >> >>>>>>>>>> requirements for OAM
> > >> >> >>>>>>>>>>
> > >> >> >>>>>>>>>> On Wed, Nov 12, 2014 at 2:11 AM, Marc Binderberger
> > >> >> >>>>>>>>>> <[e-mail]>
> > >> >> >>>>>>>> wrote:
> > >> >> >>>>>>>>>>> Hello Mach,
> > >> >> >>>>>>>>>>>
> > >> >> >>>>>>>>>>> so for delay measurement you use the color flag to
> > >> >> >>>>>>>>>>> mark a single packet, which helps the receiver to pick
> > >> >> >>>>>>>>>>> the right packet?  And repeat this every time period T ?
> > >> >> >>>>>>>>>>>
> > >> >> >>>>>>>>>>>    ...000100000010000001000...
> > >> >> >>>>>>>>>> Is there there a draft or description of how this
> > >> >> >>>>>>>>>> algorithm would work? Seems like there would need to be
> > >> >> >>>>>>>>>> quite a bot of synchronization needed between end
> > >> >> >>>>>>>>>> points (synchronized clocks, provisions to correlate
> > >> >> >>>>>>>>>> measurements correctly with lost packets, replicated packets,
> > etc.).
> > >> >> >>>>>>>>>> Also, what is envisioned for range for the period?
> > >> >> >>>>>>>>>
> > >> >> >>>>>>>>> Here is a reference
> > >> >> >>>>>>>>
> > >> >> >>>>>>>> https://datatracker.ietf.org/doc/draft-chen-ippm-coloring
> > >> >> >>>>>>>> -ba
> > >> >> >>>>>>>> sed
> > >> >> >>>>>>>> -ip
> > >> >> >>>>>>>> fpm-fr
> > >> >> >>>>>>>> amew
> > >> >> >>>>>>>> ork/.
> > >> >> >>>>>>>>
> > >> >> >>>>>>>> Thanks for the pointer. Regarding the need for
> > >> >> >>>>>>>>synchronized clocks  to measure delay, I consulted our local
> > NTP expert.
> > >> >> >>>>>>>>The host clock  jitter we currently see in our network is
> > >> >> >>>>>>>>currently usually  greater than one-way packet delay (in
> > >> >> >>>>>>>>some cases much greater), so  in his words:
> > >> >> >>>>>>>> "measuring one-way packet delays using host clocks is a
> > >> >> >>>>>>>>lost cause".
> > >> >> >>>>>>>> Please take this as just one data point!
> > >> >> >>>>>>
> > >> >> >>>>>> <Jon> Thank you. As someone who has managed NTP more times
> > >> and
> > >> >> >>>>>>for  more years than I care to admit, this is a very good
> > >> >> >>>>>>datapoint to consider.
> > >> >> >>>>>> NTP helps many understand that time is relative.
> > >> >> >>>>>
> > >> >> >>>>> +++DK: As per our experience in carrier Ethernet we
> > >> >> >>>>> +++supported one way
> > >> >> >>>>> delay and never found NTP useful even for our lab networks
> > >> >> >>>>>(I am referring software based NTP NTPv3).
> > >> >> >>>>> As mentioned below IEEE 1588v2 will vary based on equipment
> > >> >> >>>>>and operator networks but in our testing we found it very
> > >> >> >>>>>precise if properly deployed.
> > >> >> >>>>> IEEE 1588v2 is very precise if phy based timestamping is used.
> > >> >> >>>>>Even  timestamping at NP level provided great results for one
> > >>way
> > >> delay.
> > >> >> >>>>>
> > >> >> >>>>> If we want to accurately measure two way delay we need 4
> > >> >> >>>>> timestamp total on receiver of frame (this is to avoid
> > >> >> >>>>> processing time that's taken for reply by software as
> > >> >> >>>>> hardware can put timestamp at lower layer without doing
> > >> >> >>>>> delay and jitter
> > >> calculation).
> > >> >> >>>>> For one way delay we will require 2 timestamp, so lower
> > >> >> >>>>> layer hardware can timestamp before packet is punted to software.
> > >> >> >>>>>
> > >> >> >>>>> As mentioned below I agree 8 byte IEEE 1588 timestamp is
> > >>required.
> > >> >> >>>>>
> > >> >> >>>>> We should also look for Synthetic OAM applicability for
> > >> >> >>>>>performance ('O'
> > >> >> >>>>> bit can be overloaded to do both Fault and performance if
> > >> >> >>>>>OAM is defined with different oam Subtype for Delay and Loss
> > >> >> >>>>>frames and it will not be too deep hardware inspection) as
> > >> >> >>>>>that give large flexibility (synthetic/real loss measurement,
> > >> >> >>>>>Availability/unavailability, on-demand and pro-active
> > >> >> >>>>>performance) and can be run on all flows of ECMP.
> > >> >> >>>>>
> > >> >> >>>>> Thanks,
> > >> >> >>>>> Deepak
> > >> >> >>>>>>
> > >> >> >>>>>>
> > >> >> >>>>>>>
> > >> >> >>>>>>> [Vero] Thanks for this. What about the current experience
> > >> >> >>>>>>> with
> > >> >> >>>>>>> 1588v2 then?
> > >> >> >>>>>>>>
> > >> >> >>>>>>>>> Yes, it does need some synchronization. As for the
> > >> >> >>>>>>>>> range, it depends on two
> > >> >> >>>>>>>> factors, one is the implementation limitation, the other
> > >> >> >>>>>>>> the requirement of the operators. In the above reference,
> > >> >> >>>>>>>> the suggested periods are 1s, 10s, 1min, 10min and 1h.
> > >> >> >>>>>>>> I think if we were implementing delay measurement in GUE,
> > >> >> >>>>>>>> I would advocate add a 64 bit optional field for
> > >> >> >>>>>>>> timestamp, probably containing source time stamp, and
> > >> >> >>>>>>>> echoed timestamp for a flow (usec resolution and similar
> > >> >> >>>>>>>> in design TCP timestamp option). This easily gives a
> > >> >> >>>>>>>> precise RTT, and if clocks are precisely synchronized
> > >> >> >>>>>>>> then one way latency
> > >>could be
> > >> calculated also.
> > >> >> >>>>>>> [Vero] If the source timestamp could be carried, it could
> > >> >> >>>>>>> also be used for packet loss calculation/correlation.
> > >> >> >>>>>>>
> > >> >> >>>>>>>> Thanks,
> > >> >> >>>>>>>> Tom
> > >> >> >>>>>>>>
> > >> >> >>>>>>>>> Best regards,
> > >> >> >>>>>>>>> Mach
> > >> >> >>>>>>>>>>
> > >> >> >>>>>>>>>> Thanks,
> > >> >> >>>>>>>>>> Tom
> > >> >> >>>>>>>>>>
> > >> >> >>>>>>>>>>>
> > >> >> >>>>>>>>>>> One question I still have is: why is the measurement
> > >> >> >>>>>>>>>>> done in the NVE
> > >> >> >>>>>>>> header?
> > >> >> >>>>>>>>>>> The outer header is IP/IPv6, so couldn't we use the
> > >> >> >>>>>>>>>>> coloring for the
> > >> >> >>>>>>>>>>> IP/IPv6 header, assuming this is defined?
> > >> >> >>>>>>>>>>>
> > >> >> >>>>>>>>>>>
> > >> >> >>>>>>>>>>> Thanks & Regards,
> > >> >> >>>>>>>>>>> Marc
> > >> >> >>>>>>>>>>>
> > >> >> >>>>>>>>>>>
> > >> >> >>>>>>>>>>>
> > >> >> >>>>>>>>>>>> On Wed, 12 Nov 2014 09:34:52 +0000, Mach Chen wrote:
> > >> >> >>>>>>>>>>>> Hi Tom,
> > >> >> >>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>> -----Original Message-----
> > >> >> >>>>>>>>>>>>> From: Tom Herbert [mailto:[e-mail]]
> > >> >> >>>>>>>>>>>>> Sent: Wednesday, November 12, 2014 5:06 PM
> > >> >> >>>>>>>>>>>>> To: Mach Chen
> > >> >> >>>>>>>>>>>>> Cc: Greg Mirsky; Haoweiguo; [e-mail]; Larry
> > >> >> >>>>>>>>>>>>> Kreeger
> > >> >> >>>>>>>>>>>>> (kreeger)
> > >> >> >>>>>>>>>>>>> Subject: Re: [nvo3] 答复: Comments on NVO3 data
> > plane
> > >> >> >>>>>>>>>>>>> requirements for OAM
> > >> >> >>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>> On Wed, Nov 12, 2014 at 12:55 AM, Mach Chen
> > >> >> >>>>>>>>>>>>> <[e-mail]>
> > >> >> >>>>>>>>>>>>> wrote:
> > >> >> >>>>>>>>>>>>>> Hi Greg and all,
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> Single bit is not sufficient if someone wants to
> > >> >> >>>>>>>>>>>>>>perform loss and delay measurement  simultaneously,
> > >> >> >>>>>>>>>>>>>>then two bits needed.
> > >> >> >>>>>>>>>>>>> Is that necessary? Can they share the same time
> > >> >> >>>>>>>>>>>>>quantum (as  well as other metrics maybe to be added
> > >> >> >>>>>>>>>>>>>later)? In all the  protocols mentioned, the reserved
> > >> >> >>>>>>>>>>>>>bits are a somewhat precious resource.
> > >> >> >>>>>>>>>>>>
> > >> >> >>>>>>>>>>>> Yes, it's necessary if there is ECMP.
> > >> >> >>>>>>>>>>>>
> > >> >> >>>>>>>>>>>> Given one bit is used for both loss and delay
> > >> >> >>>>>>>>>>>> measurement, for loss measurement, it periodically
> > >> >> >>>>>>>>>>>> set and clear the marking bit, a flow is divided into
> > >> >> >>>>>>>>>>>> consecutive blocks, and then the counting and
> > >>calculating are
> > >> based on each block.
> > >> >> >>>>>>>>>>>> This is fine for loss measurement.
> > >> >> >>>>>>>>>>>>
> > >> >> >>>>>>>>>>>> For delay measurement, it has to make sure the
> > >> >> >>>>>>>>>>>> timestamps (collected at sender and receiver) are for
> > >>the
> > >> same packet.
> > >> >> >>>>>>>>>>>> Presumably, the time when changing the marking bit is
> > >> >> >>>>>>>>>>>> right time to get
> > >> >> >>>>>>>> the timestamps.
> > >> >> >>>>>>>>>>>> Since there is ECMP, the first packet of a block at
> > >> >> >>>>>>>>>>>>the sender  may probably different from the first
> > >> >> >>>>>>>>>>>>packet at the receiver,  thus it will get the
> > >> >> >>>>>>>>>>>>mismatched timestamps to calculate the delay.
> > >> >> >>>>>>>>>>>>
> > >> >> >>>>>>>>>>>> Best regards,
> > >> >> >>>>>>>>>>>> Mach
> > >> >> >>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>> Tom
> > >> >> >>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> Best regards,
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> Mach
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> From: nvo3 [mailto:[e-mail]] On Behalf
> > >> >> >>>>>>>>>>>>>> Of Greg Mirsky
> > >> >> >>>>>>>>>>>>>> Sent: Wednesday, November 12, 2014 8:05 AM
> > >> >> >>>>>>>>>>>>>> To: Haoweiguo
> > >> >> >>>>>>>>>>>>>> Cc: [e-mail]; Larry Kreeger (kreeger)
> > >> >> >>>>>>>>>>>>>> Subject: Re: [nvo3] 答复: Comments on NVO3 data
> > plane
> > >> >> >>>>>>>> requirements
> > >> >> >>>>>>>>>>>>>> for OAM
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> Dear All,
> > >> >> >>>>>>>>>>>>>> agree with Weiguo, single bit flag in fixed
> > >> >> >>>>>>>>>>>>>> position would be sufficient and HW-friendly.
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> Regards,
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> Greg
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> On Tue, Nov 11, 2014 at 3:51 PM, Haoweiguo
> > >> >> >>>>>>>>>>>>>> <[e-mail]>
> > >> >> >>>>>>>>>>>>> wrote:
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> Hi Larry,
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> For marking purpose, i think one bit maybe OK,
> > >> >> >>>>>>>>>>>>>> fixed fields in
> > >> >> >>>>>>>>>>>>>> NVO3 header is precious. I would like it is set in
> > >> >> >>>>>>>>>>>>>> fixed field, rather than in option field. Because
> > >> >> >>>>>>>>>>>>>> chipset normally can't process optional field, it
> > >> >> >>>>>>>>>>>>>> is hard to realize in-band performance measurement
> > >> >> >>>>>>>>>>>>>> if using optional
> > >> >> >>>>>>>>>>>>> field for marking.
> > >> >> >>>>>>>>>>>>>> For other real time congestion control function,
> > >> >> >>>>>>>>>>>>>> maybe more bits are needed.
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> Thanks
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> weiguo
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> ________________________________
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> 发件人: Larry Kreeger (kreeger) [[e-mail]]
> > >> >> >>>>>>>>>>>>>> 发送时间: 2014年11月12日 4:33
> > >> >> >>>>>>>>>>>>>> 收件人: Haoweiguo; Greg Mirsky
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> 抄送: [e-mail]
> > >> >> >>>>>>>>>>>>>> 主题: Re: [nvo3] Comments on NVO3 data plane
> > >> >> requirements
> > >> >> >>>>>>>>>>>>>> for
> > >> >> >>>>>>>> OAM
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> Hi Weiguo,
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> What do you envision this marking looking like?  e.g.
> > >> >> >>>>>>>>>>>>>> is it just a single flag bit, or large field with a
> > >> >> >>>>>>>>>>>>>> counter or sequence number, or some kind of flow ID?
> > >> >> >>>>>>>>>>>>>> If not a single flag, how large do you see the
> > >> >> >>>>>>>>>>>>>> field
> > >> >> >>>>>>>>>>>>> being?
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> If it is more than a flag (and I assume it would
> > >> >> >>>>>>>>>>>>>> be), and is not mandatory for all implementations,
> > >> >> >>>>>>>>>>>>>> then it seems to fall into the category of optional
> > >>extensions.
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> Thanks, Larry
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> From: Haoweiguo <[e-mail]>
> > >> >> >>>>>>>>>>>>>> Date: Tuesday, November 11, 2014 10:18 AM
> > >> >> >>>>>>>>>>>>>> To: Greg Mirsky <[e-mail]>
> > >> >> >>>>>>>>>>>>>> Cc: "[e-mail]" <[e-mail]>
> > >> >> >>>>>>>>>>>>>> Subject: [nvo3] 答复: Comments on NVO3 data plane
> > >> >> >>>>>>>>>>>>>> requirements
> > >> >> >>>>>>>> for
> > >> >> >>>>>>>>>>>>>> OAM
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> Hi Greg,
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> I fully agree with you.
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> The real time OAM is passive performance
> > >> >> >>>>>>>>>>>>>> measurement methods. I would like
> > >> >> >>>>>>>>>>>>>> NVO3 data encapsulation has a field for marking and
> > >> >> >>>>>>>>>>>>>> not affect forwarding of packets, the marking field
> > >> >> >>>>>>>>>>>>>> is only used for performance measurement. The
> > >> >> >>>>>>>>>>>>>> NVO3 packet with this marking flag don't need to be
> > >> >> >>>>>>>>>>>>>> sent to control plane, it is different from
> > >> >> >>>>>>>>>>>>>> OAM(ping/Trace) packet processing.
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> Thanks
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> weiguo
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> ________________________________
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> 发件人: Greg Mirsky [[e-mail]]
> > >> >> >>>>>>>>>>>>>> 发送时间: 2014年11月12日 4:07
> > >> >> >>>>>>>>>>>>>> 收件人: Haoweiguo
> > >> >> >>>>>>>>>>>>>> 抄送: [e-mail]
> > >> >> >>>>>>>>>>>>>> 主题: Re: [nvo3] Comments on NVO3 data plane
> > >> >> requirements
> > >> >> >>>>>>>>>>>>>> for
> > >> >> >>>>>>>> OAM
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> Hi Weiguo,
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> marking groups of packets that belong to the
> > >> >> >>>>>>>>>>>>>> particular flow to facilitate measurement of some
> > >> >> >>>>>>>>>>>>>> performance metric, whether loss or delay/delay
> > >> >> >>>>>>>>>>>>>> variation, may be viewed as one of passive
> > >> >> >>>>>>>>>>>>>> performance
> > >> >> >>>>>>>>>>>>> measurement methods.
> > >> >> >>>>>>>>>>>>>> But such marking should not alter, at least not
> > >> >> >>>>>>>>>>>>>> significantly alter, treatment of data flow in the
> > >>network.
> > >> >> >>>>>>>>>>>>>> Because of that, I believe, OAM flag should not be
> > >> >> >>>>>>>>>>>>>> used for marking as that will force punting marked
> > >> >> >>>>>>>>>>>>>> packets from fast forwarding path to the control
> > >> >> >>>>>>>>>>>>>> plane. But it might be good to have a field in NVO3
> > >> >> >>>>>>>>>>>>>> header that may be used for marking and not affect
> > >> >> >>>>>>>>>>>>>> forwarding of
> > >> >> >>>>>>>> packets if altered.
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> Regards,
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> Greg
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> On Tue, Nov 11, 2014 at 12:34 AM, Haoweiguo
> > >> >> >>>>>>>>>>>>>> <[e-mail]>
> > >> >> >>>>>>>>>>>>> wrote:
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> Hi All,
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> I maybe not clearly said in today’s NVO3 meeting,
> > >> >> >>>>>>>>>>>>>>pls allow  me to reiterate the OAM data plane
> > >> >> >>>>>>>>>>>>>>requirements on the mail list.
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> Currently NVO3 data plane encapsulation only
> > >> >> >>>>>>>>>>>>>>includes one OAM flag, it is used for Ping/Trace
> > >> >> >>>>>>>>>>>>>>similar
> > >>applications.
> > >> >> >>>>>>>>>>>>>> This kind of OAM application is initiated by
> > >> >> >>>>>>>>>>>>>>operators for  network connectivity verification,
> > >> >> >>>>>>>>>>>>>>normally when network failure occurs.
> > >> >> >>>>>>>>>>>>>> There is another OAM requirements of real time OAM
> > >> >> >>>>>>>>>>>>>>or synthesizing OAM. It can be used for
> > >> >> >>>>>>>>>>>>> packet loss detection in real time.
> > >> >> >>>>>>>>>>>>>> When ingress NVE receives traffic from local TS, it
> > >> >> >>>>>>>>>>>>>>gets packet statistics, and mark(coloring) the OAM
> > >> >> >>>>>>>>>>>>>>flag relying on local policy when it performs
> > >> >> >>>>>>>>>>>>>> NVO3 encapsulation. When egress NVEs receives the
> > >> >> >>>>>>>>>>>>>>traffic,  it decapsulates
> > >> >> >>>>>>>>>>>>>> NVO3 encapsulation, and gets packet statistics with
> > >> >> >>>>>>>>>>>>>>the real  time OAM flag marking. By comparing the
> > >> >> >>>>>>>>>>>>>>packet number of  ingress NVE and the sum of all
> > >> >> >>>>>>>>>>>>>>egress NVEs, packet loss can be deduced.
> > >> >> >>>>>>>>>>>>>> This method can be applicable for both unicast and
> > >> >> >>>>>>>>>>>>>>multicast  traffic. Local policy on ingress NVE is
> > >> >> >>>>>>>>>>>>>>configured by  operators or automatically acquired
> > >> >> >>>>>>>>>>>>>>from centralized orchestration.
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> Thanks
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>> weiguo
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> _______________________________________________
> > >> >> >>>>>>>>>>>>>> nvo3 mailing list
> > >> >> >>>>>>>>>>>>>> [e-mail]
> > >> >> >>>>>>>>>>>>>> https://www.ietf.org/mailman/listinfo/nvo3
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> >> >>>>>>>>>>>>>>
> > >> _______________________________________________
> > >> >> >>>>>>>>>>>>>> nvo3 mailing list
> > >> >> >>>>>>>>>>>>>> [e-mail]
> > >> >> >>>>>>>>>>>>>> https://www.ietf.org/mailman/listinfo/nvo3
> > >> >> >>>>>>>>>>>>
> > _______________________________________________
> > >> >> >>>>>>>>>>>> nvo3 mailing list
> > >> >> >>>>>>>>>>>> [e-mail]
> > >> >> >>>>>>>>>>>> https://www.ietf.org/mailman/listinfo/nvo3
> > >> >> >>>>>>>>> _______________________________________________
> > >> >> >>>>>>>>> nvo3 mailing list
> > >> >> >>>>>>>>> [e-mail]
> > >> >> >>>>>>>>> https://www.ietf.org/mailman/listinfo/nvo3
> > >> >> >>>>>>>>
> > >> >> >>>>>>>> _______________________________________________
> > >> >> >>>>>>>> nvo3 mailing list
> > >> >> >>>>>>>> [e-mail]
> > >> >> >>>>>>>> https://www.ietf.org/mailman/listinfo/nvo3
> > >> >> >>>>>>> _______________________________________________
> > >> >> >>>>>>> nvo3 mailing list
> > >> >> >>>>>>> [e-mail]
> > >> >> >>>>>>> https://www.ietf.org/mailman/listinfo/nvo3
> > >> >> >>>>>>
> > >> >> >>>>>>
> > >> >> >>>>>
> > >> >> >>>>> _______________________________________________
> > >> >> >>>>> nvo3 mailing list
> > >> >> >>>>> [e-mail]
> > >> >> >>>>> https://www.ietf.org/mailman/listinfo/nvo3
> > >> >> >>>
> > >> >> >>>_______________________________________________
> > >> >> >>>nvo3 mailing list
> > >> >> >>>[e-mail]
> > >> >> >>>https://www.ietf.org/mailman/listinfo/nvo3
> > >> >> >>
> > >> >> >>_______________________________________________
> > >> >> >>nvo3 mailing list
> > >> >> >>[e-mail]
> > >> >> >>https://www.ietf.org/mailman/listinfo/nvo3
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > _______________________________________________
> > >> >> > nvo3 mailing list
> > >> >> > [e-mail]
> > >> >> > https://www.ietf.org/mailman/listinfo/nvo3
> > >> >> >
> > >> >> _______________________________________________
> > >> >> nvo3 mailing list
> > >> >> [e-mail]
> > >> >> https://www.ietf.org/mailman/listinfo/nvo3
> > >> > _______________________________________________
> > >> > nvo3 mailing list
> > >> > [e-mail]
> > >> > https://www.ietf.org/mailman/listinfo/nvo3
>