Re: [nvo3] 答复: Comments on NVO3 data plane requirements for OAM

[Mach Chen <[e-mail]>]

Hi Tom,

Please see my response inline...

> -----Original Message-----
> From: Tom Herbert [mailto:[e-mail]]
> Sent: Tuesday, November 25, 2014 3:13 AM
> To: Mach Chen
> Cc: Deepak Kumar (dekumar); [e-mail]
> Subject: Re: [nvo3] 答复: Comments on NVO3 data plane requirements for OAM
> 
> On Fri, Nov 21, 2014 at 12:37 AM, Mach Chen <[e-mail]> wrote:
> >
> > Hi Deepak,
> >
> > > >In addition, I see the value of you proposed optional "measurement"
> > > >field, it could be used to carry some correlation (e.g.,
> > > >block/period
> > > >number) and timestamp information, then combine with the marking
> > > >bit, it can greatly simplify the marking based solution.
> > >
> > > +++DK:
> > > I think adding information regarding measurement field, block,
> > > period, etc. is not required in data path as more information
> > > reduces the mtu, and this can easily be added if required by TLV to
> > > OAM functionality with new subtype (as this is control or configuration
> functionality).
> >
> > I am talking about two things here:
> >
> > 1) the fixed marking bits, I think they are necessary for passive PM;
> >
> > 2) the correlation information, timestamps, counters, they could be
> > communicated either through in-band or out-or-band, each way has its
> > pros and cons;
> >
> > > Also even passive oam loss measurement solution to calculating loss
> > > is not accurate as packets can arrive late outside the measuring
> > > blocks. Even in that
> >
> > If it only depends on the marking bit and the measuring period is set to a very
> small interval, indeed, that will affect the accuracy. But from engineering point
> view, an operator and an implementation will not set (or support) to a very
> aggressive period.
> 
> I don't think this would be true in our data center. The very reason we would
> enable a passive mechanism is for getting accurate measurements of high
> granularity. For instance, if we want to use this as real time feedback for
> congestion control we need fine grained information. 

OK, let's put the solution part aside for a while. Could you please share what's your passive PM requirement here? For example, the granularity, the period interval, etc. I think these are important inputs for choosing the appropriate solution. 

Thanks,
Mach

> Also, as part of debugging a
> customers problems it may come down to us being able to identify specific
> packets that are being dropped or experiencing unusual latency, I don't see how
> marking with a couple of bits is sufficient for that. We already have this deployed
> in the pre-NV world (mostly provided by TCP), in an NV world there are many
> cases we won't have visibility into the customer's protocol so we'll need to find
> alternative methods (which likely results in annotating packets).

> 
> > And if the packets can carry some correlation information (e.g., the
> block/period number), then the accuracy should be no problem.
> >
> > In theory, you are right, if the delay of the packets of block exceed a threshold
> (e.g., a block period), the packets may be mis-counted into another block.
> >
> > > case to get accurate measurement instead of ipfix method, better to
> > > use OAM to exchange these marked packet counters on both ends and do
> > > loss measurement between two consecutive loss measurement replies.
> >
> > I am fine with either way for communicating the counters and timestamps.
> >
> > >
> > > For loss measurement, why we have to count traffic for marked
> > > packets only and not maintain counters per flow?
> >
> > I'm not sure what's your question here.
> >
> > To calculate the packet loss, counters maintenance (no matter at where) is
> necessary, it depends on the specific implementation.
> >
> > Best regards,
> > Mach
> >
> >
> > > -----Original Message-----
> > > From: Deepak Kumar (dekumar) [mailto:[e-mail]]
> > > Sent: Friday, November 21, 2014 2:34 PM
> > > To: Mach Chen; Tom Herbert
> > > Cc: [e-mail]
> > > Subject: Re: [nvo3] 答复: Comments on NVO3 data plane requirements for
> > > OAM
> > >
> > > Hi Mach,
> > >
> > > Please see inline +++DK:
> > >
> > >
> > > On 11/20/14 5:02 PM, "Mach Chen" <[e-mail]> wrote:
> > >
> > > >Hi Tom,
> > > >
> > > >Please see my response inline...
> > > >
> > > >> -----Original Message-----
> > > >> From: Tom Herbert [mailto:[e-mail]]
> > > >> Sent: Friday, November 21, 2014 1:28 AM
> > > >> To: Mach Chen
> > > >> Cc: [e-mail]
> > > >> Subject: Re: [nvo3] 答复: Comments on NVO3 data plane requirements
> > > >> for OAM
> > > >>
> > > >> On Wed, Nov 19, 2014 at 5:54 PM, Mach Chen <[e-mail]>
> > > wrote:
> > > >> > Hi Tissa,
> > > >> >
> > > >> > Thanks for your response!
> > > >> >
> > > >> > Please see my response inline...
> > > >> >
> > > >> >> -----Original Message-----
> > > >> >> From: nvo3 [mailto:[e-mail]] On Behalf Of Tissa
> > > >> >> Senevirathne
> > > >> >> (tsenevir)
> > > >> >> Sent: Wednesday, November 19, 2014 8:45 PM
> > > >> >> To: Haoweiguo; Tom Herbert
> > > >> >> Cc: Greg Mirsky; Tapraj Singh; Deepak Kumar (dekumar);
> > > >> >> [e-mail]
> > > >> >> Subject: Re: [nvo3] 答复: Comments on NVO3 data plane
> > > >> >> requirements for OAM
> > > >> >>
> > > >> >> Hi Weiguo, Mach et,al
> > > >> >>
> > > >> >> The discussion here is NVO3 data plane requirements for OAM.
> > > >> >> Like I have said
> > > >> >
> > > >> > You are right, this discussion is about "NVO3 data plane
> > > >> > requirements
> > > >>for OAM",
> > > >> but recently the focus is Performance Measurement (PM)
> > > >>requirement to
> > > >>NVO3
> > > >> that is also one of the OAM functions.
> > > >> >
> > > >> >> earlier,  we do not need to complicate the Data Plane. Can you
> > > >> >> explain to me
> > > >> >
> > > >> > "Complicate/simple" is not the goal, the goal is to define a
> > > >>reasonable solution
> > > >> that can satisfy the requirement. That's why I agree with Greg
> > > >>that we should  firstly make the agreement on the requirement.
> > > >> >
> > > >> Mach,
> > > >>
> > > >> The nvo3 OAM requirements draft
> > > >> (draft-ashwood-nvo3-oam-requirements-01) seems to already contain
> > > >>a fairly  comprehensive list of requirements. Particularly apropos
> > > >>to this discussion are:
> > > >>
> > > >>    R13) NVO3 OAM frames MUST be forwarded along the same path
> (i.e.,
> > > >>    links (including LAG members) and nodes) as the NVO3 data frames.
> > > >>
> > > >>    R16) NVO3 OAM should be extensible such that new functionality and
> > > >>    information elements related to this functionality can be introduced
> > > >>    in the future.
> > > >>
> > > >> I believe that an an active OAM message format like Tissa
> > > >>describes would meet  these and most of the others in that draft.
> > > >
> > > >There are also the following requirements listed in the draft:
> > > >
> > > >   R7) NVO3 OAM MUST support measurement of per VNI frame loss
> between
> > > >   two NV Edge devices that support the same VNI within a given NVO3
> > > >   domain.
> > > >
> > > >   R8) NVO3 OAM MUST support measurement of per VNI two-way frame
> > > delay
> > > >   between two NV edge devices that support the same VNI within a given
> > > >   NVO3 domain.
> > > >
> > > >   R9) NVO3 OAM MUST support measurement of per VNI one-way frame
> > > delay
> > > >   between two NV Edge devices that support the same VNI within a given
> > > >   NVO3 domain.
> > > >
> > > >   R10) NVO3 OAM MUST support measurement of per VNI frame delay
> > > >   variation between two NV Edge devices that support the same VNI
> > > >   within a given NVO3 domain.
> > > >
> > > >
> > > >>
> > > >> If a passive mechanism is indeed required,
> > > >
> > > >I personally think it is indeed required, and we also received such
> > > >requirements from the operators.
> > > >
> > > >> then we need to consider how to
> > > >> meet the extensibility requirement. I don't believe that
> > > >>allocating two bit flags in  the encapsulation header is at all an
> > > >>extensible solution. The reserved header  bits are too a precious
> > > >>resource to be allocated for such a narrow purpose and for
> > > >
> > > >Looking through the bits in some headers, we could find that every
> > > >bit in a header has its own purpose. It's better that one bit could
> > > >be defined for as many usages as possible, but there is always tradeoff.
> > > >
> > > >As for the two bits for passive PM that include loss, one/two way
> > > >delay, delay variation and throughput, I may not think this is a narrow
> purpose.
> > > >And if you want, you may use the marking bit for some policies control.
> > > >
> > > >> something not required for protocol operation. As previously
> > > >>discussed in this  thread, using one bit to get one-way time delay
> > > >>measurements is not even viable  in a lot deployments-- in this
> > > >>case we probably need timestamps to get RTT.
> > > >
> > > >For the deployments in question, the challenge for one way time
> > > >delay is time synchronization and its accuracy. The solution of
> > > >using one-bit for one-way delay is really workable, there have been
> > > >some prototypes and experiments show that.
> > > >
> > > >Since time synchronization is not needed for RTT, IMHO, measure RTT
> > > >should be the easiest way to go.
> > > >
> > > >
> > > >>
> > > >> To support passive OAM support in GUE, I would probably propose
> > > >>to add a  generic optional "measurement" field. This would provide
> > > >>some number of bits  in the header that can be used for passive
> > > >>measurement (possibly a few  different sizes say 32, 64, 128
> > > >>bits). The field can be structured to allow different  mechanisms
> > > >>(e.g. include timestamps for RTT measurement). This also reduces
> > > >>the constraints on the measurement techniques, for instance the
> > > >>marking  technique might no longer limited to use a single bit
> > > >>which should reduce the  complexity needed to deal with OOO or packet
> loss.
> > > >
> > > >Even with the solution as above, seems there needs at least one
> > > >bit(at the fix position of the header) that indicates there is an
> > > >optional field exist. In the case we could have opportunity and "enough"
> > > >reserved bits to allocate for the marking bits, I'd like to suggest
> > > >allocating two bits for passive PM.
> > > >
> > > >In addition, I see the value of you proposed optional "measurement"
> > > >field, it could be used to carry some correlation (e.g.,
> > > >block/period
> > > >number) and timestamp information, then combine with the marking
> > > >bit, it can greatly simplify the marking based solution.
> > >
> > > +++DK:
> > > I think adding information regarding measurement field, block,
> > > period, etc. is not required in data path as more information
> > > reduces the mtu, and this can easily be added if required by TLV to
> > > OAM functionality with new subtype (as this is control or configuration
> functionality).
> > > Also even passive oam loss measurement solution to calculating loss
> > > is not accurate as packets can arrive late outside the measuring
> > > blocks. Even in that case to get accurate measurement instead of
> > > ipfix method, better to use OAM to exchange these marked packet
> > > counters on both ends and do loss measurement between two consecutive
> loss measurement replies.
> > >
> > > For loss measurement, why we have to count traffic for marked
> > > packets only and not maintain counters per flow?
> > >
> > > Thanks,
> > > Deepak
> > > >
> > > >Thanks,
> > > >Mach
> > > >
> > > >>
> > > >> Tom
> > > >>
> > > >> >
> > > >> >> what difference it make to the data plane whether it is
> > > >> >> active/passive or some other means of OAM.
> > > >> >
> > > >> > Active/passive is mainly regarding to PM which normally
> > > >> > includes
> > > >>Active and
> > > >> Passive PM.
> > > >> >
> > > >> > Active PM measures the injected packets (e.g., OAM packets) to
> > > >>evaluate the
> > > >> performance of a path. Passive PM measures the performance of the
> > > >>real/live  traffic of a path, it reflects the real performance of
> > > >>the path. For more detail  about active/passive PM, you may refer
> > > >>to the material of IPPM WG.
> > > >> >
> > > >> >>
> > > >> >> All what it needs to know is that the packet is an OAM packet
> > > >> >> and it is addressed to the local device,
> > > >> >
> > > >> > What you are talking are just part of the OAM functions (e.g.,
> > > >> > CC,
> > > >>CV), for
> > > >> passive PM, OAM packets may not be needed.
> > > >> >
> > > >> >
> > > >> > Best regards,
> > > >> > Mach
> > > >> >
> > > >> >>
> > > >> >> -----Original Message-----
> > > >> >> From: Haoweiguo [mailto:[e-mail]]
> > > >> >> Sent: Tuesday, November 18, 2014 7:50 PM
> > > >> >> To: Tissa Senevirathne (tsenevir); Tom Herbert
> > > >> >> Cc: Greg Mirsky; Tapraj Singh; Deepak Kumar (dekumar);
> > > >> >> [e-mail]
> > > >> >> Subject: RE: [nvo3] 答复: Comments on NVO3 data plane
> > > >> >> requirements for OAM
> > > >> >>
> > > >> >> Hi  Tissa,
> > > >> >> Your solution is active OAM,  i think it is a basic and
> > > >> >> important solution in whole OAM framework.The disccussed
> > > >> >> thread is about
> > > >>passive
> > > >> OAM.
> > > >> >>  Both active and passive OAM have its pros/cons, both have its
> > > >> >> usecases and scenarios.The regular method for passive OAM is
> > > >> >> to add marking bits in packet header, in NVO3 case, the
> > > >> >> marking bits had better be set in NVO3 header.But just as Greg
> > > >> >> said,currently it's unfortunate that there is no accepted OAM
> > > >> >> requirements, gap analysis, and etc in the WG. We hope this
> > > >> >> work could be progressed more
> > > >>quickly.
> > > >> >> Thanks
> > > >> >> weiguo
> > > >> >> ________________________________________
> > > >> >> From: Tissa Senevirathne (tsenevir) [[e-mail]]
> > > >> >> Sent: Wednesday, November 19, 2014 8:25
> > > >> >> To: Tom Herbert
> > > >> >> Cc: Greg Mirsky; Tapraj Singh; Deepak Kumar (dekumar);
> > > >> >> [e-mail]
> > > >> >> Subject: Re: [nvo3] 答复: Comments on NVO3 data plane
> > > >> >> requirements for OAM
> > > >> >>
> > > >> >> Hi Tom
> > > >> >>
> > > >> >> Your interpretation is correct. The entropy allows OAM packets
> > > >> >> to follow the same path as the data packet.
> > > >> >>
> > > >> >> As I noted earlier on in the thread, OAM processing would not
> > > >> >> kick in unless address matches the MEP/MIP. If address match
> > > >> >> MEP/MIP and OAM bit is set, then OAM processing begins.
> > > >> >>
> > > >> >> -----Original Message-----
> > > >> >> From: Tom Herbert [mailto:[e-mail]]
> > > >> >> Sent: Tuesday, November 18, 2014 4:20 PM
> > > >> >> To: Tissa Senevirathne (tsenevir)
> > > >> >> Cc: Greg Mirsky; Tapraj Singh; [e-mail]; Deepak Kumar
> > > >> >> (dekumar)
> > > >> >> Subject: Re: [nvo3] 答复: Comments on NVO3 data plane
> > > >> >> requirements for OAM
> > > >> >>
> > > >> >> On Tue, Nov 18, 2014 at 3:54 PM, Tissa Senevirathne (tsenevir)
> > > >> >> <[e-mail]> wrote:
> > > >> >> > Greg
> > > >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >> > I disagree with you on FM and PM cannot be achieved in ECMP
> > > >> environment.
> > > >> >> > Significant amount of work has gone in to this area during
> > > >> >> > TRILL
> > > >>OAM.
> > > >> >> > Please check the use of Flow entropy functionality proposed
> > > >> >> > in
> > > >>NVO3 OAM.
> > > >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >> > https://tools.ietf.org/html/draft-tissa-nvo3-oam-fm-00
> > > >> >> >
> > > >> >> Tissa,
> > > >> >>
> > > >> >> If I am reading this correctly, the OAM message would be
> > > >> >> composed of the encapsulation header, followed by 128 bytes of
> > > >> >> which contains a pseudo header for switching, followed by a
> > > >> >> self defining OAM
> > > message.
> > > >> >> The OAM bit is only used at the receiver to distinguish data
> > > >> >> messages for OAM messages for processing. Is this interpretation
> correct?
> > > >> >>
> > > >> >> Thanks,
> > > >> >> Tom
> > > >> >>
> > > >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >> > From: nvo3 [mailto:[e-mail]] On Behalf Of Greg
> > > >> >> > Mirsky
> > > >> >> > Sent: Tuesday, November 18, 2014 3:03 PM
> > > >> >> > To: Tapraj Singh
> > > >> >> > Cc: [e-mail]
> > > >> >> > Subject: Re: [nvo3] 答复: Comments on NVO3 data plane
> > > >> >> > requirements for OAM
> > > >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >> > Hi Tapraj,
> > > >> >> >
> > > >> >> > though I agree and support with idea of having OAM flag in
> > > >> >> > NVO3 header I have to point to:
> > > >> >> >
> > > >> >> > absence of WG agreed upon OAM Requirements; no gap analysis
> > > >> >> > of tools for NVO3 OAM; OAM flag does not help passive
> > > >> >> > performance measurement marking method (two bit-long field
> > > >> >> > for marking in fixed
> > > >> position).
> > > >> >> >
> > > >> >> > I agree that PW VCCV and GAL/G-ACh can be viewed as MPLS
> > > >> >> > identification of OAM packet (though not necessarily OAM).
> > > >> >> > But IP clearly doesn't have such identification for OAM and
> > > >> >> > that, in part, why in-band requirement for IP OAM, both FM
> > > >> >> > and Active PM, is not attainable
> > > >> >> (ECMP environment).
> > > >> >> >
> > > >> >> > Regards,
> > > >> >> >
> > > >> >> > Greg
> > > >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >> > On Tue, Nov 18, 2014 at 1:31 PM, Tapraj Singh
> > > >> >> > <[e-mail]>
> > > >>wrote:
> > > >> >> >
> > > >> >> > Hi All,
> > > >> >> >
> > > >> >> >  I totally agree with the point made by Deepak and Tissa here.
> > > >> >> > Our OAM should follow the data path for services as much as
> > > >> >> > possible and all other protocol specific information should
> > > >> >> > be in the OAM protocol specific TLVs.
> > > >> >> >
> > > >> >> > LAYER2 OAM
> > > >> >> >
> > > >> >> > In term of identify the OAM packet, first level of
> > > >> >> > identification for
> > > >> >> > L2 OAM Should be the MAC address and send level of hierarchy
> > > >> >> > should be the ether type or OUI.
> > > >> >> > No other OAM Specific field should be allowed in the packet header.
> > > >> >> >
> > > >> >> >  Please note that L3 OAM and MPLS also follow the same principle.
> > > >> >> >
> > > >> >> > Thanks
> > > >> >> > Tapraj
> > > >> >> >
> > > >> >> >
> > > >> >> > On 11/17/14 12:39 PM, "Deepak Kumar (dekumar)"
> > > >> >> > <[e-mail]>
> > > >> >> wrote:
> > > >> >> >
> > > >> >> >>I Agree with Tissa below. My Goal also was to point out that
> > > >> >> >>instead of complicating the header, we can do OAM
> > > >> >> >>performance within OAM channel itself and this is extensible
> > > >> >> >>and can be done in hardware which is why mostly things are added
> in header.
> > > >> >> >>
> > > >> >> >>Also, Operators keep asking for new OAM tools (Fault
> > > >> >> >>detection, verification, isolation, Interworking, alarm,
> > > >> >> >>putting service in maintenance and perform test)  and Performance
> tools, eg:
> > > >> >> >>(Delay/Jitter, Actual Loss Measurement, Synthetic Loss,
> > > >> >> >>loopback signaling like TDM, Generate frames to verify qos
> > > >> >> >>etc.) and so OAM Channel solution will be extensible.
> > > >> >> >>
> > > >> >> >>Thanks,
> > > >> >> >>Deepak
> > > >> >> >>
> > > >> >> >>On 11/17/14 8:47 AM, "Tissa Senevirathne (tsenevir)"
> > > >> >> >><[e-mail]>
> > > >> >> >>wrote:
> > > >> >> >>
> > > >> >> >>>I think we are complicating OAM beyond what it is needed.
> > > >> >> >>>
> > > >> >> >>>As far as packet encapsulation is concern, all what is
> > > >> >> >>>needed is single bit. This bit is needed to prevent OAM
> > > >> >> >>>packets leaking out from the domain.
> > > >> >> >>>
> > > >> >> >>>Termination of OAM and processing of it happen based on the
> > > >> >> >>>addressing in the packet.
> > > >> >> >>>
> > > >> >> >>>E.g. if Address matches and OAM bit is set then it is an
> > > >> >> >>>OAM packet addressed to the local MEP/MP.
> > > >> >> >>>
> > > >> >> >>>Not other way around. Why? Because we want OAM to be as
> > > >> >> >>>closely as possible follow the Data path.
> > > >> >> >>>
> > > >> >> >>>If we need to have performance and delay measurements, we
> > > >> >> >>>SHOULD NOT mutate the packet header.
> > > >> >> >>>
> > > >> >> >>>Instead OAM specific extensions should be in the OAM shim.
> > > >> >> >>>
> > > >> >> >>>As an example. You could have packet fragment (which is
> > > >> >> >>>sometimes called flow entropy) and at the end of that you
> > > >> >> >>>can have all of the stuff you need in the world of OAM.
> > > >> >> >>>
> > > >> >> >>>Hope this clarify
> > > >> >> >>>
> > > >> >> >>>Thanks
> > > >> >> >>>Tissa
> > > >> >> >>>-----Original Message-----
> > > >> >> >>>From: nvo3 [mailto:[e-mail]] On Behalf Of Tom
> > > >> >> >>>Herbert
> > > >> >> >>>Sent: Monday, November 17, 2014 8:02 AM
> > > >> >> >>>To: Marc Binderberger
> > > >> >> >>>Cc: Greg Mirsky; Mach Chen; Deepak Kumar (dekumar);
> > > >> >> >>>[e-mail]; Haoweiguo; Larry Kreeger (kreeger); Vero
> > > >> >> >>>Zheng; Jon Hudson
> > > >> >> >>>Subject: Re: [nvo3] 答复: Comments on NVO3 data plane
> > > >> >> >>>requirements for OAM
> > > >> >> >>>
> > > >> >> >>>On Mon, Nov 17, 2014 at 12:01 AM, Marc Binderberger
> > > >> >> >>><[e-mail]>
> > > >> >> >>>wrote:
> > > >> >> >>>> Hello Deepak et al.,
> > > >> >> >>>>
> > > >> >> >>>> so this sounds like we need more than just a (2nd) bit
> > > >> >> >>>>for delay measurement.
> > > >> >> >>>> Seems we need an optional header extension or a TLV to
> > > >> >> >>>>carry all the  information (timestamps, oam Subtype).
> > > >> >> >>>>Sounds definitely more than a  32/64bit header could carry (*).
> > > >> >> >>>>
> > > >> >> >>>> The optional header extension, when done similar to GUE,
> > > >> >> >>>> has a fixed position. For the TLV this would be an
> > > >> >> >>>> additional requirement. This would allow for hardware-stamping.
> > > >> >> >>>>
> > > >> >> >>>The alternative is to do active delay measurement using
> > > >>request/reply.
> > > >> >> >>>We should be able to define the requirements so that an OAM
> > > >> >> >>>message corresponding to a flow which would be routed in
> > > >> >> >>>exactly the same way as a data message for the flow. Larry
> > > >> >> >>>mentioned that we might even want to put a "fake" packet
> > > >> >> >>>header as the first part of the encapsulated payload of an
> > > >> >> >>>OAM message for
> > > instance.
> > > >> >> >>>
> > > >> >> >>>> Now if we introduce such an OAM extension header it could
> > > >> >> >>>> as well carry the "first" bit we discussed for packet
> > > >> >> >>>> loss
> > > >>measurement (?).
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>> Regards, Marc
> > > >> >> >>>>
> > > >> >> >>>> (*: at least all proposals so far have a base header that
> > > >> >> >>>> fits into
> > > >> >> >>>> 32/64 bit, plus IP and potential UDP)
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>>
> > > >> >> >>>> On Sun, 16 Nov 2014 16:44:54 +0000, Deepak Kumar
> > > >> >> >>>> (dekumar)
> > > >>wrote:
> > > >> >> >>>>> Hi,
> > > >> >> >>>>>
> > > >> >> >>>>> Please see inline +++DK:
> > > >> >> >>>>>
> > > >> >> >>>>> On 11/14/14 11:09 AM, "Jon Hudson"
> > > >> >> >>>>> <[e-mail]>
> > > >>wrote:
> > > >> >> >>>>>
> > > >> >> >>>>>>
> > > >> >> >>>>>> One comment in line....
> > > >> >> >>>>>>
> > > >> >> >>>>>>> On Nov 13, 2014, at 11:47 PM, Vero Zheng
> > > >> >> >>>>>>><[e-mail]>
> > > >> >> >>>>>>>wrote:
> > > >> >> >>>>>>>
> > > >> >> >>>>>>> Hi Tom,
> > > >> >> >>>>>>>
> > > >> >> >>>>>>> Please see in-line.
> > > >> >> >>>>>>>
> > > >> >> >>>>>>> BR, Vero
> > > >> >> >>>>>>>
> > > >> >> >>>>>>>> -----Original Message-----
> > > >> >> >>>>>>>> From: nvo3 [mailto:[e-mail]] On Behalf
> > > >> >> >>>>>>>> Of Tom Herbert
> > > >> >> >>>>>>>> Sent: Friday, November 14, 2014 4:27 PM
> > > >> >> >>>>>>>> To: Mach Chen
> > > >> >> >>>>>>>> Cc: Greg Mirsky; Haoweiguo; Marc Binderberger; Larry
> > > >> >> >>>>>>>> Kreeger; [e-mail]
> > > >> >> >>>>>>>> Subject: Re: [nvo3] 答复: Comments on NVO3 data plane
> > > >> >> >>>>>>>> requirements for OAM
> > > >> >> >>>>>>>>
> > > >> >> >>>>>>>> On Wed, Nov 12, 2014 at 5:13 PM, Mach Chen
> > > >> >> >>>>>>>> <[e-mail]>
> > > >> >> >>>>>>>> wrote:
> > > >> >> >>>>>>>>> Hi Tom,
> > > >> >> >>>>>>>>>
> > > >> >> >>>>>>>>>> -----Original Message-----
> > > >> >> >>>>>>>>>> From: Tom Herbert [mailto:[e-mail]]
> > > >> >> >>>>>>>>>> Sent: Thursday, November 13, 2014 3:11 AM
> > > >> >> >>>>>>>>>> To: Marc Binderberger
> > > >> >> >>>>>>>>>> Cc: Mach Chen; Greg Mirsky; Haoweiguo;
> > > >> >> >>>>>>>>>> [e-mail]; Larry Kreeger
> > > >> >> >>>>>>>>>> Subject: Re: [nvo3] 答复: Comments on NVO3 data plane
> > > >> >> >>>>>>>>>> requirements for OAM
> > > >> >> >>>>>>>>>>
> > > >> >> >>>>>>>>>> On Wed, Nov 12, 2014 at 2:11 AM, Marc Binderberger
> > > >> >> >>>>>>>>>> <[e-mail]>
> > > >> >> >>>>>>>> wrote:
> > > >> >> >>>>>>>>>>> Hello Mach,
> > > >> >> >>>>>>>>>>>
> > > >> >> >>>>>>>>>>> so for delay measurement you use the color flag to
> > > >> >> >>>>>>>>>>> mark a single packet, which helps the receiver to
> > > >> >> >>>>>>>>>>> pick the right packet?  And repeat this every time
> period T ?
> > > >> >> >>>>>>>>>>>
> > > >> >> >>>>>>>>>>>    ...000100000010000001000...
> > > >> >> >>>>>>>>>> Is there there a draft or description of how this
> > > >> >> >>>>>>>>>> algorithm would work? Seems like there would need
> > > >> >> >>>>>>>>>> to be quite a bot of synchronization needed between
> > > >> >> >>>>>>>>>> end points (synchronized clocks, provisions to
> > > >> >> >>>>>>>>>> correlate measurements correctly with lost packets,
> > > >> >> >>>>>>>>>> replicated packets,
> > > etc.).
> > > >> >> >>>>>>>>>> Also, what is envisioned for range for the period?
> > > >> >> >>>>>>>>>
> > > >> >> >>>>>>>>> Here is a reference
> > > >> >> >>>>>>>>
> > > >> >> >>>>>>>> https://datatracker.ietf.org/doc/draft-chen-ippm-colo
> > > >> >> >>>>>>>> ring
> > > >> >> >>>>>>>> -ba
> > > >> >> >>>>>>>> sed
> > > >> >> >>>>>>>> -ip
> > > >> >> >>>>>>>> fpm-fr
> > > >> >> >>>>>>>> amew
> > > >> >> >>>>>>>> ork/.
> > > >> >> >>>>>>>>
> > > >> >> >>>>>>>> Thanks for the pointer. Regarding the need for
> > > >> >> >>>>>>>>synchronized clocks  to measure delay, I consulted our
> > > >> >> >>>>>>>>local
> > > NTP expert.
> > > >> >> >>>>>>>>The host clock  jitter we currently see in our network
> > > >> >> >>>>>>>>is currently usually  greater than one-way packet
> > > >> >> >>>>>>>>delay (in some cases much greater), so  in his words:
> > > >> >> >>>>>>>> "measuring one-way packet delays using host clocks is
> > > >> >> >>>>>>>>a lost cause".
> > > >> >> >>>>>>>> Please take this as just one data point!
> > > >> >> >>>>>>
> > > >> >> >>>>>> <Jon> Thank you. As someone who has managed NTP more
> > > >> >> >>>>>> times
> > > >> and
> > > >> >> >>>>>>for  more years than I care to admit, this is a very
> > > >> >> >>>>>>good datapoint to consider.
> > > >> >> >>>>>> NTP helps many understand that time is relative.
> > > >> >> >>>>>
> > > >> >> >>>>> +++DK: As per our experience in carrier Ethernet we
> > > >> >> >>>>> +++supported one way
> > > >> >> >>>>> delay and never found NTP useful even for our lab
> > > >> >> >>>>>networks (I am referring software based NTP NTPv3).
> > > >> >> >>>>> As mentioned below IEEE 1588v2 will vary based on
> > > >> >> >>>>>equipment and operator networks but in our testing we
> > > >> >> >>>>>found it very precise if properly deployed.
> > > >> >> >>>>> IEEE 1588v2 is very precise if phy based timestamping is used.
> > > >> >> >>>>>Even  timestamping at NP level provided great results for
> > > >> >> >>>>>one
> > > >>way
> > > >> delay.
> > > >> >> >>>>>
> > > >> >> >>>>> If we want to accurately measure two way delay we need 4
> > > >> >> >>>>> timestamp total on receiver of frame (this is to avoid
> > > >> >> >>>>> processing time that's taken for reply by software as
> > > >> >> >>>>> hardware can put timestamp at lower layer without doing
> > > >> >> >>>>> delay and jitter
> > > >> calculation).
> > > >> >> >>>>> For one way delay we will require 2 timestamp, so lower
> > > >> >> >>>>> layer hardware can timestamp before packet is punted to
> software.
> > > >> >> >>>>>
> > > >> >> >>>>> As mentioned below I agree 8 byte IEEE 1588 timestamp is
> > > >>required.
> > > >> >> >>>>>
> > > >> >> >>>>> We should also look for Synthetic OAM applicability for
> > > >> >> >>>>>performance ('O'
> > > >> >> >>>>> bit can be overloaded to do both Fault and performance
> > > >> >> >>>>>if OAM is defined with different oam Subtype for Delay
> > > >> >> >>>>>and Loss frames and it will not be too deep hardware
> > > >> >> >>>>>inspection) as that give large flexibility
> > > >> >> >>>>>(synthetic/real loss measurement,
> > > >> >> >>>>>Availability/unavailability, on-demand and pro-active
> > > >> >> >>>>>performance) and can be run on all flows of ECMP.
> > > >> >> >>>>>
> > > >> >> >>>>> Thanks,
> > > >> >> >>>>> Deepak
> > > >> >> >>>>>>
> > > >> >> >>>>>>
> > > >> >> >>>>>>>
> > > >> >> >>>>>>> [Vero] Thanks for this. What about the current
> > > >> >> >>>>>>> experience with
> > > >> >> >>>>>>> 1588v2 then?
> > > >> >> >>>>>>>>
> > > >> >> >>>>>>>>> Yes, it does need some synchronization. As for the
> > > >> >> >>>>>>>>> range, it depends on two
> > > >> >> >>>>>>>> factors, one is the implementation limitation, the
> > > >> >> >>>>>>>> other the requirement of the operators. In the above
> > > >> >> >>>>>>>> reference, the suggested periods are 1s, 10s, 1min, 10min
> and 1h.
> > > >> >> >>>>>>>> I think if we were implementing delay measurement in
> > > >> >> >>>>>>>> GUE, I would advocate add a 64 bit optional field for
> > > >> >> >>>>>>>> timestamp, probably containing source time stamp, and
> > > >> >> >>>>>>>> echoed timestamp for a flow (usec resolution and
> > > >> >> >>>>>>>> similar in design TCP timestamp option). This easily
> > > >> >> >>>>>>>> gives a precise RTT, and if clocks are precisely
> > > >> >> >>>>>>>> synchronized then one way latency
> > > >>could be
> > > >> calculated also.
> > > >> >> >>>>>>> [Vero] If the source timestamp could be carried, it
> > > >> >> >>>>>>> could also be used for packet loss calculation/correlation.
> > > >> >> >>>>>>>
> > > >> >> >>>>>>>> Thanks,
> > > >> >> >>>>>>>> Tom
> > > >> >> >>>>>>>>
> > > >> >> >>>>>>>>> Best regards,
> > > >> >> >>>>>>>>> Mach
> > > >> >> >>>>>>>>>>
> > > >> >> >>>>>>>>>> Thanks,
> > > >> >> >>>>>>>>>> Tom
> > > >> >> >>>>>>>>>>
> > > >> >> >>>>>>>>>>>
> > > >> >> >>>>>>>>>>> One question I still have is: why is the
> > > >> >> >>>>>>>>>>> measurement done in the NVE
> > > >> >> >>>>>>>> header?
> > > >> >> >>>>>>>>>>> The outer header is IP/IPv6, so couldn't we use
> > > >> >> >>>>>>>>>>> the coloring for the
> > > >> >> >>>>>>>>>>> IP/IPv6 header, assuming this is defined?
> > > >> >> >>>>>>>>>>>
> > > >> >> >>>>>>>>>>>
> > > >> >> >>>>>>>>>>> Thanks & Regards,
> > > >> >> >>>>>>>>>>> Marc
> > > >> >> >>>>>>>>>>>
> > > >> >> >>>>>>>>>>>
> > > >> >> >>>>>>>>>>>
> > > >> >> >>>>>>>>>>>> On Wed, 12 Nov 2014 09:34:52 +0000, Mach Chen
> wrote:
> > > >> >> >>>>>>>>>>>> Hi Tom,
> > > >> >> >>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>> -----Original Message-----
> > > >> >> >>>>>>>>>>>>> From: Tom Herbert [mailto:[e-mail]]
> > > >> >> >>>>>>>>>>>>> Sent: Wednesday, November 12, 2014 5:06 PM
> > > >> >> >>>>>>>>>>>>> To: Mach Chen
> > > >> >> >>>>>>>>>>>>> Cc: Greg Mirsky; Haoweiguo; [e-mail]; Larry
> > > >> >> >>>>>>>>>>>>> Kreeger
> > > >> >> >>>>>>>>>>>>> (kreeger)
> > > >> >> >>>>>>>>>>>>> Subject: Re: [nvo3] 答复: Comments on NVO3 data
> > > plane
> > > >> >> >>>>>>>>>>>>> requirements for OAM
> > > >> >> >>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>> On Wed, Nov 12, 2014 at 12:55 AM, Mach Chen
> > > >> >> >>>>>>>>>>>>> <[e-mail]>
> > > >> >> >>>>>>>>>>>>> wrote:
> > > >> >> >>>>>>>>>>>>>> Hi Greg and all,
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> Single bit is not sufficient if someone wants
> > > >> >> >>>>>>>>>>>>>>to perform loss and delay measurement
> > > >> >> >>>>>>>>>>>>>>simultaneously, then two bits needed.
> > > >> >> >>>>>>>>>>>>> Is that necessary? Can they share the same time
> > > >> >> >>>>>>>>>>>>>quantum (as  well as other metrics maybe to be
> > > >> >> >>>>>>>>>>>>>added later)? In all the  protocols mentioned,
> > > >> >> >>>>>>>>>>>>>the reserved bits are a somewhat precious resource.
> > > >> >> >>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>> Yes, it's necessary if there is ECMP.
> > > >> >> >>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>> Given one bit is used for both loss and delay
> > > >> >> >>>>>>>>>>>> measurement, for loss measurement, it
> > > >> >> >>>>>>>>>>>> periodically set and clear the marking bit, a
> > > >> >> >>>>>>>>>>>> flow is divided into consecutive blocks, and then
> > > >> >> >>>>>>>>>>>> the counting and
> > > >>calculating are
> > > >> based on each block.
> > > >> >> >>>>>>>>>>>> This is fine for loss measurement.
> > > >> >> >>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>> For delay measurement, it has to make sure the
> > > >> >> >>>>>>>>>>>> timestamps (collected at sender and receiver) are
> > > >> >> >>>>>>>>>>>> for
> > > >>the
> > > >> same packet.
> > > >> >> >>>>>>>>>>>> Presumably, the time when changing the marking
> > > >> >> >>>>>>>>>>>> bit is right time to get
> > > >> >> >>>>>>>> the timestamps.
> > > >> >> >>>>>>>>>>>> Since there is ECMP, the first packet of a block
> > > >> >> >>>>>>>>>>>>at the sender  may probably different from the
> > > >> >> >>>>>>>>>>>>first packet at the receiver,  thus it will get
> > > >> >> >>>>>>>>>>>>the mismatched timestamps to calculate the delay.
> > > >> >> >>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>> Best regards,
> > > >> >> >>>>>>>>>>>> Mach
> > > >> >> >>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>> Tom
> > > >> >> >>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> Best regards,
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> Mach
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> From: nvo3 [mailto:[e-mail]] On
> > > >> >> >>>>>>>>>>>>>> Behalf Of Greg Mirsky
> > > >> >> >>>>>>>>>>>>>> Sent: Wednesday, November 12, 2014 8:05 AM
> > > >> >> >>>>>>>>>>>>>> To: Haoweiguo
> > > >> >> >>>>>>>>>>>>>> Cc: [e-mail]; Larry Kreeger (kreeger)
> > > >> >> >>>>>>>>>>>>>> Subject: Re: [nvo3] 答复: Comments on NVO3 data
> > > plane
> > > >> >> >>>>>>>> requirements
> > > >> >> >>>>>>>>>>>>>> for OAM
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> Dear All,
> > > >> >> >>>>>>>>>>>>>> agree with Weiguo, single bit flag in fixed
> > > >> >> >>>>>>>>>>>>>> position would be sufficient and HW-friendly.
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> Regards,
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> Greg
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> On Tue, Nov 11, 2014 at 3:51 PM, Haoweiguo
> > > >> >> >>>>>>>>>>>>>> <[e-mail]>
> > > >> >> >>>>>>>>>>>>> wrote:
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> Hi Larry,
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> For marking purpose, i think one bit maybe OK,
> > > >> >> >>>>>>>>>>>>>> fixed fields in
> > > >> >> >>>>>>>>>>>>>> NVO3 header is precious. I would like it is set
> > > >> >> >>>>>>>>>>>>>> in fixed field, rather than in option field.
> > > >> >> >>>>>>>>>>>>>> Because chipset normally can't process optional
> > > >> >> >>>>>>>>>>>>>> field, it is hard to realize in-band
> > > >> >> >>>>>>>>>>>>>> performance measurement if using optional
> > > >> >> >>>>>>>>>>>>> field for marking.
> > > >> >> >>>>>>>>>>>>>> For other real time congestion control
> > > >> >> >>>>>>>>>>>>>> function, maybe more bits are needed.
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> Thanks
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> weiguo
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> ________________________________
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> 发件人: Larry Kreeger (kreeger)
> > > >> >> >>>>>>>>>>>>>> [[e-mail]]
> > > >> >> >>>>>>>>>>>>>> 发送时间: 2014年11月12日 4:33
> > > >> >> >>>>>>>>>>>>>> 收件人: Haoweiguo; Greg Mirsky
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> 抄送: [e-mail]
> > > >> >> >>>>>>>>>>>>>> 主题: Re: [nvo3] Comments on NVO3 data plane
> > > >> >> requirements
> > > >> >> >>>>>>>>>>>>>> for
> > > >> >> >>>>>>>> OAM
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> Hi Weiguo,
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> What do you envision this marking looking like?
> e.g.
> > > >> >> >>>>>>>>>>>>>> is it just a single flag bit, or large field
> > > >> >> >>>>>>>>>>>>>> with a counter or sequence number, or some kind of
> flow ID?
> > > >> >> >>>>>>>>>>>>>> If not a single flag, how large do you see the
> > > >> >> >>>>>>>>>>>>>> field
> > > >> >> >>>>>>>>>>>>> being?
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> If it is more than a flag (and I assume it
> > > >> >> >>>>>>>>>>>>>> would be), and is not mandatory for all
> > > >> >> >>>>>>>>>>>>>> implementations, then it seems to fall into the
> > > >> >> >>>>>>>>>>>>>> category of optional
> > > >>extensions.
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> Thanks, Larry
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> From: Haoweiguo <[e-mail]>
> > > >> >> >>>>>>>>>>>>>> Date: Tuesday, November 11, 2014 10:18 AM
> > > >> >> >>>>>>>>>>>>>> To: Greg Mirsky <[e-mail]>
> > > >> >> >>>>>>>>>>>>>> Cc: "[e-mail]" <[e-mail]>
> > > >> >> >>>>>>>>>>>>>> Subject: [nvo3] 答复: Comments on NVO3 data
> plane
> > > >> >> >>>>>>>>>>>>>> requirements
> > > >> >> >>>>>>>> for
> > > >> >> >>>>>>>>>>>>>> OAM
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> Hi Greg,
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> I fully agree with you.
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> The real time OAM is passive performance
> > > >> >> >>>>>>>>>>>>>> measurement methods. I would like
> > > >> >> >>>>>>>>>>>>>> NVO3 data encapsulation has a field for marking
> > > >> >> >>>>>>>>>>>>>> and not affect forwarding of packets, the
> > > >> >> >>>>>>>>>>>>>> marking field is only used for performance
> > > >> >> >>>>>>>>>>>>>> measurement. The
> > > >> >> >>>>>>>>>>>>>> NVO3 packet with this marking flag don't need
> > > >> >> >>>>>>>>>>>>>> to be sent to control plane, it is different
> > > >> >> >>>>>>>>>>>>>> from
> > > >> >> >>>>>>>>>>>>>> OAM(ping/Trace) packet processing.
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> Thanks
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> weiguo
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> ________________________________
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> 发件人: Greg Mirsky [[e-mail]]
> > > >> >> >>>>>>>>>>>>>> 发送时间: 2014年11月12日 4:07
> > > >> >> >>>>>>>>>>>>>> 收件人: Haoweiguo
> > > >> >> >>>>>>>>>>>>>> 抄送: [e-mail]
> > > >> >> >>>>>>>>>>>>>> 主题: Re: [nvo3] Comments on NVO3 data plane
> > > >> >> requirements
> > > >> >> >>>>>>>>>>>>>> for
> > > >> >> >>>>>>>> OAM
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> Hi Weiguo,
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> marking groups of packets that belong to the
> > > >> >> >>>>>>>>>>>>>> particular flow to facilitate measurement of
> > > >> >> >>>>>>>>>>>>>> some performance metric, whether loss or
> > > >> >> >>>>>>>>>>>>>> delay/delay variation, may be viewed as one of
> > > >> >> >>>>>>>>>>>>>> passive performance
> > > >> >> >>>>>>>>>>>>> measurement methods.
> > > >> >> >>>>>>>>>>>>>> But such marking should not alter, at least not
> > > >> >> >>>>>>>>>>>>>> significantly alter, treatment of data flow in
> > > >> >> >>>>>>>>>>>>>> the
> > > >>network.
> > > >> >> >>>>>>>>>>>>>> Because of that, I believe, OAM flag should not
> > > >> >> >>>>>>>>>>>>>> be used for marking as that will force punting
> > > >> >> >>>>>>>>>>>>>> marked packets from fast forwarding path to the
> > > >> >> >>>>>>>>>>>>>> control plane. But it might be good to have a
> > > >> >> >>>>>>>>>>>>>> field in NVO3 header that may be used for
> > > >> >> >>>>>>>>>>>>>> marking and not affect forwarding of
> > > >> >> >>>>>>>> packets if altered.
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> Regards,
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> Greg
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> On Tue, Nov 11, 2014 at 12:34 AM, Haoweiguo
> > > >> >> >>>>>>>>>>>>>> <[e-mail]>
> > > >> >> >>>>>>>>>>>>> wrote:
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> Hi All,
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> I maybe not clearly said in today’s NVO3
> > > >> >> >>>>>>>>>>>>>>meeting, pls allow  me to reiterate the OAM data
> > > >> >> >>>>>>>>>>>>>>plane requirements on the mail list.
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> Currently NVO3 data plane encapsulation only
> > > >> >> >>>>>>>>>>>>>>includes one OAM flag, it is used for Ping/Trace
> > > >> >> >>>>>>>>>>>>>>similar
> > > >>applications.
> > > >> >> >>>>>>>>>>>>>> This kind of OAM application is initiated by
> > > >> >> >>>>>>>>>>>>>>operators for  network connectivity
> > > >> >> >>>>>>>>>>>>>>verification, normally when network failure occurs.
> > > >> >> >>>>>>>>>>>>>> There is another OAM requirements of real time
> > > >> >> >>>>>>>>>>>>>>OAM or synthesizing OAM. It can be used for
> > > >> >> >>>>>>>>>>>>> packet loss detection in real time.
> > > >> >> >>>>>>>>>>>>>> When ingress NVE receives traffic from local
> > > >> >> >>>>>>>>>>>>>>TS, it gets packet statistics, and
> > > >> >> >>>>>>>>>>>>>>mark(coloring) the OAM flag relying on local
> > > >> >> >>>>>>>>>>>>>>policy when it performs
> > > >> >> >>>>>>>>>>>>>> NVO3 encapsulation. When egress NVEs receives
> > > >> >> >>>>>>>>>>>>>>the traffic,  it decapsulates
> > > >> >> >>>>>>>>>>>>>> NVO3 encapsulation, and gets packet statistics
> > > >> >> >>>>>>>>>>>>>>with the real  time OAM flag marking. By
> > > >> >> >>>>>>>>>>>>>>comparing the packet number of  ingress NVE and
> > > >> >> >>>>>>>>>>>>>>the sum of all egress NVEs, packet loss can be
> deduced.
> > > >> >> >>>>>>>>>>>>>> This method can be applicable for both unicast
> > > >> >> >>>>>>>>>>>>>>and multicast  traffic. Local policy on ingress
> > > >> >> >>>>>>>>>>>>>>NVE is configured by  operators or automatically
> > > >> >> >>>>>>>>>>>>>>acquired from centralized orchestration.
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> Thanks
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>> weiguo
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> _______________________________________________
> > > >> >> >>>>>>>>>>>>>> nvo3 mailing list [e-mail]
> > > >> >> >>>>>>>>>>>>>> https://www.ietf.org/mailman/listinfo/nvo3
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> >> >>>>>>>>>>>>>>
> > > >> _______________________________________________
> > > >> >> >>>>>>>>>>>>>> nvo3 mailing list
> > > >> >> >>>>>>>>>>>>>> [e-mail]
> > > >> >> >>>>>>>>>>>>>> https://www.ietf.org/mailman/listinfo/nvo3
> > > >> >> >>>>>>>>>>>>
> > > _______________________________________________
> > > >> >> >>>>>>>>>>>> nvo3 mailing list
> > > >> >> >>>>>>>>>>>> [e-mail]
> > > >> >> >>>>>>>>>>>> https://www.ietf.org/mailman/listinfo/nvo3
> > > >> >> >>>>>>>>>
> _______________________________________________
> > > >> >> >>>>>>>>> nvo3 mailing list
> > > >> >> >>>>>>>>> [e-mail]
> > > >> >> >>>>>>>>> https://www.ietf.org/mailman/listinfo/nvo3
> > > >> >> >>>>>>>>
> > > >> >> >>>>>>>>
> _______________________________________________
> > > >> >> >>>>>>>> nvo3 mailing list
> > > >> >> >>>>>>>> [e-mail]
> > > >> >> >>>>>>>> https://www.ietf.org/mailman/listinfo/nvo3
> > > >> >> >>>>>>> _______________________________________________
> > > >> >> >>>>>>> nvo3 mailing list
> > > >> >> >>>>>>> [e-mail]
> > > >> >> >>>>>>> https://www.ietf.org/mailman/listinfo/nvo3
> > > >> >> >>>>>>
> > > >> >> >>>>>>
> > > >> >> >>>>>
> > > >> >> >>>>> _______________________________________________
> > > >> >> >>>>> nvo3 mailing list
> > > >> >> >>>>> [e-mail]
> > > >> >> >>>>> https://www.ietf.org/mailman/listinfo/nvo3
> > > >> >> >>>
> > > >> >> >>>_______________________________________________
> > > >> >> >>>nvo3 mailing list
> > > >> >> >>>[e-mail]
> > > >> >> >>>https://www.ietf.org/mailman/listinfo/nvo3
> > > >> >> >>
> > > >> >> >>_______________________________________________
> > > >> >> >>nvo3 mailing list
> > > >> >> >>[e-mail]
> > > >> >> >>https://www.ietf.org/mailman/listinfo/nvo3
> > > >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >> > _______________________________________________
> > > >> >> > nvo3 mailing list
> > > >> >> > [e-mail]
> > > >> >> > https://www.ietf.org/mailman/listinfo/nvo3
> > > >> >> >
> > > >> >> _______________________________________________
> > > >> >> nvo3 mailing list
> > > >> >> [e-mail]
> > > >> >> https://www.ietf.org/mailman/listinfo/nvo3
> > > >> > _______________________________________________
> > > >> > nvo3 mailing list
> > > >> > [e-mail]
> > > >> > https://www.ietf.org/mailman/listinfo/nvo3
> >