Re: [mpls] [PWE3] Pseudowire Sequence Number in draft-ietf-mpls-forwarding: a minor comment

[Alexander Vainshtein <[e-mail]>]

Re-sending due to an unspecified issue with my mail server... 
________________________________________
From: Alexander Vainshtein
Sent: Saturday, December 14, 2013 8:27 AM
To: [e-mail]
Cc: [e-mail]; [e-mail]; [e-mail]; [e-mail]; [e-mail]; [e-mail]; Yaakov Stein ([e-mail]); pwe3 ([e-mail])
Subject: RE: [PWE3] Pseudowire Sequence Number in draft-ietf-mpls-forwarding: a minor comment

Curtis,
Lots of thanks for a prompt and most encouraging response.

I believe that the last proposed text fully addreses both my original comment and the misunderstanding that has been encountered at some stage in this email thread.

Regards,
     Sassha
________________________________________
From: Curtis Villamizar <[e-mail]>
Sent: Friday, December 13, 2013 9:17 PM
To: Alexander Vainshtein
Cc: [e-mail]; [e-mail]; [e-mail]; [e-mail]; [e-mail]; [e-mail]; [e-mail]; Yaakov Stein ([e-mail]); pwe3 ([e-mail])
Subject: Re: [PWE3] Pseudowire Sequence Number in draft-ietf-mpls-forwarding: a minor comment

Sasha,

Thanks for pointing out the lack of clarity regarding the context of
the statement we were discussing.  These are the first three paragraphs
(originally one) of "2.1.8.1.  Pseudowire Sequence Number" that we've
been discussing

 Original (draft-ietf-mpls-forwarding-03, short but inaccurate)

   Pseudowire (PW) sequence number support is most important for PW
   payload types with a high expectation of in-order delivery.
   Resequencing support, rather than dropping at egress on out of order
   arrival, is most important for PW payload types with a high
   expectation of lossless delivery.  For example, TDM payloads require
   sequence number support and require resequencing support.  The same
   is true of ATM CBR service.  ATM VBR or ABR may have somewhat relaxed
   requirements, but generally require ATM Early Packet Discard (EPD) or
   ATM Partial Packet Discard (PPD) [ATM-EPD-and-PPD].  Though sequence
   number support and resequencing support are beneficial to PW packet
   oriented payloads such as FR and Ethernet, they are highly desirable
   but not as strongly required.

 First change

   Pseudowire (PW) sequence number support is most important for PW
   payload types with a high expectation of lossless and/or in-order
   delivery.  Identifying the position of any lost packets is important
   for PW services which are attempting to reconstruct a bit stream
   which maintains bit timing, such as time division multiplexing (TDM)
   services.  TDM and other PW services which require strict ordering
   also require that misordered packets be either dropped or reordered.

   PW services which are not timing critical bit streams in nature are
   cell oriented or frame oriented.  Though resequencing support is
   beneficial to PW cell and frame oriented payloads such as ATM, FR and
   Ethernet, they are highly desirable but not required.

 Suggested new para (Sasha) - part of second para rewritten

   Pseudowire (PW) sequence number support is most important for PW
   payload types with a high expectation of lossless and/or in-order
   delivery.  Identifying lost PW packets and exact amount of lost
   payload is critical for PW services which maintain bit timing, such
   as Time Division Multiplexing (TDM) services since these services
   MUST compensate lost payload on a bit-for-bit basis.

   With these services PW packets that have been received out of order
   also MUST also be identified and may be either re-ordered or
   dropped.  Reordering requires, in addition to sequence numbering, a
   "de-jitter buffer" in the egress PE, and ability to reorder is
   limited by the depth of this buffer. The down side of maintaining a
   de-jitter buffer is added end-to-end service delay.

   PW services which are not timing critical bit streams in nature are
   cell oriented or frame oriented.  Though resequencing support is
   beneficial to PW cell and frame oriented payloads such as ATM, FR and
   Ethernet, they are highly desirable but not required.

 Suggested change to above (Curtis) - minor plus add last sentence

   Pseudowire (PW) sequence number support is most important for PW
   payload types with a high expectation of lossless and/or in-order
   delivery.  Identifying lost PW packets and exact amount of lost
   payload is critical for PW services which maintain bit timing, such
   as Time Division Multiplexing (TDM) services since these services
   MUST compensate lost payload on a bit-for-bit basis.

   With these services PW packets that have been received out of order
   also MUST also be identified and may be either re-ordered or
   dropped.  Reordering requires, in addition to sequence numbering, a
   "jitter buffer" in the egress PE, and ability to reorder is limited
   by the depth of this buffer. The down side of maintaining a large
   jitter buffer is added end-to-end service delay.  A small jitter
   buffer is always necessary and the jitter buffer is needed
   regardless of whether reordering is done.

   PW services which are not timing critical bit streams in nature are
   cell oriented or frame oriented.  Though resequencing support is
   beneficial to PW cell and frame oriented payloads such as ATM, FR
   and Ethernet, they are highly desirable but not required.

 Reorder paragraphs and edit for clarity of context

   Pseudowire (PW) sequence number support is most important for PW
   payload types with a high expectation of lossless and/or in-order
   delivery.  Identifying lost PW packets and exact amount of lost
   payload is critical for PW services which maintain bit timing, such
   as Time Division Multiplexing (TDM) services since these services
   MUST compensate lost payload on a bit-for-bit basis.

   With PW services which maintain bit timing, packets that have been
   received out of order also MUST be identified and may be either
   re-ordered or dropped.  Reordering requires, in addition to
   sequence numbering, a "reorder buffer" in the egress PE, and ability
   to reorder is limited by the depth of this buffer. The down side of
   maintaining a large reorder buffer is added end-to-end service
   delay.

   For PW services which maintain bit timing or any other service
   where jitter must be bounded, a jitter buffer is always necessary.
   The jitter buffer is needed regardless of whether reordering is
   done.  In order to be effective, a reorder buffer must often be
   larger than a jitter buffer needs to be creating a tradeoff between
   reducing loss and minimizing delay.

   PW services which are not timing critical bit streams in nature are
   cell oriented or frame oriented.  Though resequencing support may
   be beneficial to PW cell and frame oriented payloads such as ATM,
   FR and Ethernet, this support is desirable but not required.
   Requirements to hamdle out of order packets at all vary among
   services and deployments.  For example for Ethernet PW, occasional
   (very rare) reordering is usually acceptable.  If the Ethernet PW
   is carrying MPLS-TP, then this reordering may be acceptable.

   Reducing jitter is best done by an end-system, given that the
   tradeoff of loss vs delay varies among services.  For example with
   interactive real time services low delay is preferred, while with
   non-interactive (one way) real time services low loss is preferred.
   The same end-site may be receiving both types of traffic.
   Regardless of this, bounded jitter is sometimes a requiremnet for
   specific deployments.

In the prior iteration a paragraph began with "With these services"
following the prior paragraph discussing "PW services which maintain
bit timing, such as Time Division Multiplexing (TDM) services" which
did not clearly carry over the context.

At this point, each paragraph clearly indicates its context, either
bit timing, not bit timing, or all PW.  It also makes a distinction
between a reorder buffer and a jitter buffer.

This is a more thorough treatment of the tradeoffs of using PW
sequence in supporting de-jitter and reordering in PW services than
what was originally in the document (but given that it wasn't correct
some change was needed).

Curtis


ps - Needed to top post a summary since your repeated top posting was
obscuring the context of the (longish) conversation thread.  You've
probably seen something like this before:

  You're welcome.
  I get it.  Thanks.
  Because it is the default in most pee-cee mailers.
  Why do people do it so often then?
  Because it loses context and reverses the order of the discussion.
  Why?
  Yes.
  Is top posting a bad practice?


In message <[e-mail]>
Alexander Vainshtein writes:

Curtis,
Lots of hanks for a long and detailed explanation.

It appears that your original comment about the need of jitter buffer
refered to TDM (or TDM-like) PWs (bit-stream PWs in the Architecture
RFC). If this was indeed the case, there is nothing to argue about,
because I agree completely that jitter buffer is absolutely necessary
for these.

My impression from reading the original text has been that it referred
to ALL kinds of PWs (bit-stream, celland packet) alike. Apparently
this has been my mis-interptetation; if this is indeed so, then maybe
some clarification (placing the jiter buffer discussion in the proper
context) would hopefully help.

Regards,
     Sasha
________________________________________
From: Curtis Villamizar <[e-mail]>
Sent: Thursday, December 12, 2013 6:53 AM
To: Alexander Vainshtein
Cc: [e-mail]; [e-mail]; [e-mail]; [e-mail]; [e-mail]; [e-mail]; [e-mail]; Yaakov Stein ([e-mail]); pwe3 ([e-mail])
Subject: Re: [PWE3] Pseudowire Sequence Number in draft-ietf-mpls-forwarding: a minor comment

In message <[e-mail]>
Alexander Vainshtein writes:

> Curtis,
> Lots of thanks for sending out the new text.
>
> I still have a couple of issues with this version of the text: one
> technical and one editorial.
>
> The technical issue is about the fragment "A small jitter buffer is
> always necessary".
>
> I am not sure this is correct, since I am aware of several PW
> implementations that pass the payload of a received PW packet directly
> for transmission via the corresdponding attachment circuit. Of course
> this payload would be stored in some Tx queue associated with the
> attachment circuit and controlled by the appropriate egress scheduler,
> but IMO this does not constitue a PW jitter bufer, since the PW
> interworking function does not exersize any control over these
> queues.

Well if its a technical argument you want, then here goes ...

Even in pure TDM systems some jitter exist due to the longer time it
takes if multiple passes of the FEC algorithm are needed.  In that
case there is a very small jitter buffer that can be set to the
maximum needed for a worst case FEC, or set lower to decrease delay
with the potential to make a greater number of errors uncorrectable.

For IP, even if IP EF service is used and there is a very small amount
of EF traffic, some jitter exists.  For most systems there is some
jitter crossing the fabric and internal jitter elsewhere.  The egress
buffer in most chips is very small but needed for that jitter.  For
TDM, the egress buffer can never run dry or there will be a timing
slip.

Consider what happens when packets arrive late in a system with only a
non-zero egress buffer.  For example, assume 1% of packets arrive
about one packet time late (one other EF packet in queue somewhere on
the path), 0.1% arrive two packets late.  Note that this would be an
extremely light EF load.  Generally most packets arrive at least zero
to one minus epsilon packet times late due to a lower priority packet
partially transmited when the EF packet is moved to the front of the
queue.  If there is no egress buffer at all or considerably less than
a packet time, lots of data would be dropped.  If the egress buffer
could hold just a few packets data drop would be very low.

If a packet arrives late and there is no standing queue the egress
runs dry, there is a timing slip.  When the packet does arrive a
little late the packet is sent, albiet late.  Every packet to follow
is now late by that amount of time (if the egress buffer has not
overflowed on a later packet).

At that point there is an standing queue in the egress buffer which
acts as a jitter buffer.  Most packets arrive early.  A few arrive a
little later but as long as the queue doesn't run dry again, the
jitter is compensated for.

If the egress buffer is large, then it will tend to grow to the worst
case packet delay and keep a large standing queue.  This is with no
reorder and no drop.  If the egress buffer is small (or configured to
be small) then when a packet arrives very late followed by a set of
packets in close succession, the buffer runs dry until that first late
packet arrives.  This packet is transmitted but very late.  Subsequent
packets arrive and some of them have to be dropped due to the limited
queue capacity.  This in effect puts a cap on the jitter buffer.

With only an egress buffer there are multiple strategies for dealing
with dropped packets.

  1.  Just ignore it and send the next one.
  2.  Put one packet time worth of all zeros in the queue.

Now consider what happens with a dropped packet and a standing queue.
The first strategy creates a hidden timing slip.  The second strategy
avoids the timing slip.

So my argument is that the egress buffer is in effect the jitter
buffer.  When jitter occurs a standing buffer forms that reflects the
lesser of the limit of the buffer size or the worst case jitter.
There is always a jitter buffer.  There always needs to be a jitter
buffer of at least one large packet time, otherwise high loss of data
will occur.  There should be a jitter buffer that can be made larger
so that jitter of multiple packet times doesn't turn in to loss.

Therefore I assert that a jitter buffer is always present and is
always needed because jitter is always non-zero.  Therefore this
statement is true:

   A small jitter buffer is always necessary and the jitter buffer is
   needed regardless of whether reordering is done.

> In particular, these queues could not be used for re-ordering
> misordered PW packets.

That does not mean that they can't be used to compensate for jitter
which is by definition what a jitter buffer is for.

When reordering is done, the egress buffer is made small and a buffer
preceding that is used which has some ability to reorder packets.  In
hardware a circular queue makes sense with packets place according to
their sequence number.  If you prescribe to the "just insert zeros"
idea when a packet is dropped, then after moving a packet to the
egress queue, zero the buffer or mark it as unoccupied and more the
circular buffer a bit.  If the packet that is later supposed to occupy
that slot never arrives, then it is already zerod (or marked as
missing in which case a packet of all zeros is put on the egress
queue).

So ... reording within a small jitter buffer is not hard to do.

Reordering can be supported with a small jitter buffer but adding
reordering to a small buffer may be ineffective.  If reordering is due
to a path change, then the buffer of a small number of packet times
may yield no improvement.  To be effective in that situation, the
buffer may need to be larger.  OTOH if moving EF traffic from one
queue with no standing EF queue to another with no standing EF queue
and the path change is just moving to another wave on the WDM,
reordering with a small jitter buffer may help.

> IMHO and FWIW this understanding matches RFC 3985 which mentions (in
> Section 5.2.1.1 "Frame Ordering") that reodering as the strategy for
> handling mis-ordered packets introduces additional delay (i.e.,
> requires a ddicated jitter buffer), while the alternative strategy of
> discarding mis-ordered packets does not introduce such additional
> delay.

You quoted the framework and not a TDM document.  Nowhere in that text
is there mention of a jitter buffer because that text is not specific
to TDM.

For packet traffic you have no idea if a prior packet is dropped or
will arrive out of order and so when a drop occurs packets have to be
held for a non-zero amount of time rather than transmitted as soon as
they arrive.  This applies to all cell or packet AC, such as FR, ATM,
Ethernet, etc.

You might want to look at section 5.2.2.1.  Clock Recovery which does
apply to TDM.  It points to RFC3550 (RTP) which assumes a jitter
buffer is always needed.

> The editorial issue is about the entire sentence that contains the
> above-mentioned fragment, i.e., "A small jitter buffer is always
> necessary and the jitter buffer is needed regardless of whether
> reordering is done" - to me this looks like an unnecessary repetition
> if if it were techically correct.

The second part of the sentence was to emphasis that the need for a
jitter buffer is not contingent on implementation or reordering.
Perhaps it would be more clear if the two points were separated.  How
about this

 OLD new sentence

   A small jitter buffer is always necessary and the jitter buffer
   is needed regardless of whether reordering is done.

 NEW new sentence

   A small jitter buffer is always necessary.  A jitter buffer is
   needed regardless of whether reordering is done.

> Hence I would like to suggest discrading the problematic sentence as
> the simplest way to resolve both issues.

I think the sentence provides an important clarification.  I think the
need to have this discussion proves that the clarification is needed.

If we can agree on keeping this, I can submit the updated draft.

> Hopefully these notes will e helpful.
>
> Regards,
>      Sasha

Thanks again for the review and good comments.

Regards,

Curtis


> ________________________________________
> From: Curtis Villamizar <[e-mail]>
> Sent: Tuesday, December 10, 2013 5:47 PM
> To: Alexander Vainshtein
> Cc: [e-mail]; [e-mail]; [e-mail]; [e-mail]; [e-mail]; [e-mail]; [e-mail]; Yaakov Stein ([e-mail]); pwe3 ([e-mail])
> Subject: Re: [PWE3] Pseudowire Sequence Number in draft-ietf-mpls-forwarding: a minor comment
>
> In message <[e-mail]>
> Alexander Vainshtein writes:
> >
> > Curtis,
> >
> > Lots of thanks for a prompt and very detailed response.
> > May I suggest an alternative version of the following text fragment?
> >
> > <Curtis>
> > >   Identifying the position of any lost packets is important
> > >    for PW services which are attempting to reconstruct a bit stream
> > >    which maintains bit timing, such as time division multiplexing (TDM)
> > >    services.  TDM and other PW services which require strict ordering
> > >    also require that misordered packets be either dropped or reordered.
>
> Invalid XML - you didn't close this element.  :-)
>
> >  <Sasha>
> >
> >    Identifying lost PW packets and exact amount of lost payload is
> >    critical for PW services which maintain bit timing, such as Time
> >    Division Multiplexing (TDM) services since these services MUST
> >    compensate lost payload on a bit-for-bit basis.
> >
> >    With these services PW packets that have been received out of order
> >    also MUST also be identified and may be either re-ordered or
> >    dropped.  Reordering requires, in addition to sequence numbering, a
> >    "de-jitter buffer" in the egress PE, and ability to reorder is
> >    limited by the depth of this buffer. The down side of maintaining a
> >    de-jitter buffer is added end-to-end service delay.
> >
> >   </Sasha>
>
> Accepted with a small change.
>
> After the words "down side"
> s/de-jitter buffer/large jitter buffer/
> Elsewhere
> s/de-jitter buffer/jitter buffer/g
> Then add the sentence.
>
>    A small jitter buffer is always necessary and the jitter buffer
>    is needed regardless of whether reordering is done.
>
> This yields a slightly changed second paragraph.
>
>    With these services PW packets that have been received out of order
>    also MUST also be identified and may be either re-ordered or
>    dropped.  Reordering requires, in addition to sequence numbering, a
>    "jitter buffer" in the egress PE, and ability to reorder is limited
>    by the depth of this buffer. The down side of maintaining a large
>    jitter buffer is added end-to-end service delay.  A small jitter
>    buffer is always necessary and the jitter buffer is needed
>    regardless of whether reordering is done.
>
> Is this wording OK with you?
>
> Curtis
>
> > Hopefully this will be useful.
> >
> >
> > Regards,
> >      Sasha
> >
> > > -----Original Message-----
> > > From: Curtis Villamizar [mailto:[e-mail]]
> > > Sent: Monday, November 25, 2013 8:30 PM
> > > To: Alexander Vainshtein
> > > Cc: [e-mail]; [e-mail]; [e-mail];
> > > [e-mail]; [e-mail]; [e-mail]; Yaakov Stein
> > > ([e-mail]); pwe3 ([e-mail])
> > > Subject: Re: [PWE3] Pseudowire Sequence Number in draft-ietf-mpls-
> > > forwarding: a minor comment
> > >
> > >
> > > In message
> > > <[e-mail].c
> > > om>
> > > Alexander Vainshtein writes:
> > > >
> > > > Hi all,
> > > >
> > > > I would like to comment on the text in Section 2.1.8.1 "Pseudowire
> > > > Sequence Number" in draft-ietf-mpls-forwarding-03 - MPLS Forwarding
> > > > Compliance and Performance Requirements
> > > > <http://tools.ietf.org/html/draft-ietf-mpls-forwarding-03>.
> > > >
> > > > This section states, in short, that the main drive for using the
> > > > sequence number in the PW Control Word is handling of packet
> > > > reordering events. TDM PWs are presented as a major example, with CBR
> > > > ATM services as another example.
> > > >
> > > > In fact, this statement is not accurate.
> > >
> > > Yes.  You are correct in pointing this out as inaccurate by omitting
> > > the other important function of identifying drops for the purpose of
> > > recunstructing TDM bit streams.
> > >
> > > Andy brought up resequencing being a strong provider request.
> > > Reordering is far more common than loss particularly for high priority
> > > services on provider networks and without resequencing reorder results
> > > in PW loss in what could otherwise be lossless service.
> > >
> > > So we should get the base requirements right but still reflect this,
> > > but strictly as advice with no normative wording.  We had used the
> > > phrase "is beneficial" and will retain that.
> > >
> > > Andy brought up the topic.  The incorrect wording in the existing
> > > draft is my fault.  This text went in fairly early with a lot of other
> > > changes and it appears that no one had since given it a careful enough
> > > read and review until you came along.
> > >
> > > > The main drive for mandating the use of sequence number in TM PWs is
> > > > the need to detect and count lost packets because the egress PE MUST
> > > > compensate the lost payload bit for bit.
> > > >
> > > > Ability to compensate reordering of PW packets at egress is a side
> > > > effect of (a) sequence number usage and (b) usage of the de-jitter
> > > > buffer in the egress PW. It is not mandatory and in any case is
> > > > limited by the depth of the de-jitter buffer: re-ordered packets that
> > > > cannot be accommodated within this buffer are treated as lost.
> > > >
> > > > Additional details can be found, e.g., in section 6.2.2 of RFC
> > > > 4553<http://tools.ietf.org/html/rfc4553>. Ability to re-order
> > > > mis-ordered PW packets is defined there as OPTIONAL, while replacement
> > > > of the payload of lost PW packets is defined as MANDATORY.
> > >
> > > The change below at the top of the section more accurately reflects
> > > requirements in the RFCs but still states that resequencing can be
> > > beneficial.  The comments about EPD and PPD are dropped and therefore
> > > also the informative reference.
> > >
> > >  Context:
> > >
> > >  2.1.8.1.  Pseudowire Sequence Number
> > >
> > >  OLD
> > >
> > >    Pseudowire (PW) sequence number support is most important for PW
> > >    payload types with a high expectation of in-order delivery.
> > >    Resequencing support, rather than dropping at egress on out of order
> > >    arrival, is most important for PW payload types with a high
> > >    expectation of lossless delivery.  For example, TDM payloads require
> > >    sequence number support and require resequencing support.  The same
> > >    is true of ATM CBR service.  ATM VBR or ABR may have somewhat relaxed
> > >    requirements, but generally require ATM Early Packet Discard (EPD) or
> > >    ATM Partial Packet Discard (PPD) [ATM-EPD-and-PPD].  Though sequence
> > >    number support and resequencing support are beneficial to PW packet
> > >    oriented payloads such as FR and Ethernet, they are highly desirable
> > >    but not as strongly required.
> > >
> > > NEW
> > >
> > >    Pseudowire (PW) sequence number support is most important for PW
> > >    payload types with a high expectation of lossless and/or in-order
> > >    delivery.  Identifying the position of any lost packets is important
> > >    for PW services which are attempting to reconstruct a bit stream
> > >    which maintains bit timing, such as time division multiplexing (TDM)
> > >    services.  TDM and other PW services which require strict ordering
> > >    also require that misordered packets be either dropped or reordered.
> > >
> > >    PW services which are not timing critical bit streams in nature are
> > >    cell oriented or frame oriented.  Though resequencing support is
> > >    beneficial to PW cell and frame oriented payloads such as ATM, FR and
> > >    Ethernet, they are highly desirable but not required.
> > >
> > > NEW (end of subsection after list of possible reording causes)
> > >
> > >    In provider networks which use multipath techniques and which may
> > >    occassionally rebalance traffic or which may change PW paths
> > >    occasionally for other reasons, reordering may be far more common
> > >    than loss.  Where reordering is more common than loss, resequencing
> > >    packets is beneficial, rather than dropping packets at egress when
> > >    out of order arrival occus.  Resequencing is most important for PW
> > >    payload types with a high expectation of lossless delivery since in
> > >    such cases out of order delivery within the network results in PW
> > >    loss.
> > >
> > > The final paragraph sums up the motivation for highlighting
> > > resequencing as beneficial and desirable.  It does so without any
> > > normative wording.
> > >
> > > > Hopefully these notes will be useful.
> > > >
> > > > Regards,
> > > >      Sasha
> > >
> > > These notes are very useful.  Thank you.
> > >
> > > Please let us know if you (and the WG) are OK with the rewording
> > > proposed above.
> > >
> > > Curtis