Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt

[Tom Herbert <[e-mail]>]

On Tue, May 20, 2014 at 8:07 PM, Zhou, Han <[e-mail]> wrote:

>  Hi Tom,
>
>
>
> > There has been a lot of work recently to get software variants (GSO and
> GRO) working in Linux with tunnels.
>
> Are you referring UDP GRO for VXLAN? I am aware of this implementation in
> kernel 3.14.
>

GRO and GSO for UDP tunnels, VXLAN is one use case.


> The performance gain should be similar, but still requires careful
> configuration of VM MTU to avoid IP fragmentation on physical MTU, rather
> than decoupling from underlay. I think each of these solutions has its own
> advantages.
>
>
In the case of TCP it's really the path MTU that is relevant. We really
want the MSS to be the largest value which avoids fragmentation in the path.

For UDP it is less straightforward. If the VM MTU is greater than the
physical MTU or even the path MTU then fragmentation is possible. Most UDP
applications really want to avoid fragmentation (much worse than doing TCP
segmentation), so they need to ascertain what size to send packets-- either
they choose a fixed minimum value, use the MTU, or some out of band
negotiation. With the occurrence of transport protocols running over UDP
(such as QUIC), this becomes important. In this case, it's really hard to
meaningfully decouple physical MTU (path MTU) from those of the overlay.

>
>
> > I think this would be applicable to about all tunnel protocols. Then
> you would also want to do reassembly at each hop? Sounds expensive. Once
> you segment, I think you'd only only want to reassemble at the end host.
>
> There is no segmentation/reassembly needed at each hop, but instead
> “offload” hop by hop, just save the metadata to the next hop headers. And
> finally segmentation should be avoided if the destination is a VM on a
> hypervisor.
>
> This scenario is like in Example 2 (section 5.2), just replace the
> VXLAN-SOE between the 2 gateways by any other Tunnel protocol that can
> support similar offloading feature, such as STT.
>
>
>
This where your proposal confuses me, we already have a lot of offloading
support of protocols without having to change protocols to support
offloading. I don't know if the concept of offloading is generic enough to
need intrinsic protocol support, or if it is at what protocol level this
should be (for instance I could conceive of this being an option in TCP to
allow mega sized segments).

A example where use case SOE is usable but GSO/GRO wouldn't be might be
enlightening!

Thanks,
Tom

 Best regards,
>
> Han
>
>
>
> *From:* Tom Herbert [mailto:[e-mail]]
> *Sent:* Wednesday, May 21, 2014 10:42 AM
>
> *To:* Zhou, Han
> *Cc:* [e-mail]; [e-mail];
> [e-mail];
> [e-mail]; Erik Nordmark
> *Subject:* Re: FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt
>
>
>
>
>
>
>
> On Tue, May 20, 2014 at 5:28 PM, Zhou, Han <[e-mail]> wrote:
>
> Hi Tom,
>
>
>
> Thanks for your comments.
>
> Yes TSO/LRO with VXLAN support should provide similar or even better
> performance gains, but the mechanism proposed by this draft decouples
> overlay and underlay, and it is hardware independent.
>
> Secondly, hardware offloading usually support TCP only (TSO). The
> mechanism here can help on large UDP packet performance, also verified by
> the prototype.
>
>  BTW, how many types of off-the-shelf NIC support VXLAN offloading? Any
> performance data?
>
>
>
>  HW is not a requirement for this offloading. There has been a lot of
> work recently to get software variants (GSO and GRO) working in Linux with
> tunnels. These do show 2-3x performance improvements. HW support would be
> mostly an incremental improvement to that, or becomes interesting for OS
> bypass like in SR-IOV. GSO/GRO can be presented to the guest as TSO and LRO
> so that it's possible to plumb use of large packets from the guest all the
> way to the host driver. It should also be possible to plumb two VMs on the
> same host to communicate without segmentation, i.e. output from TSO on one
> VM becomes input for another.
>
>
>
> The advantage that I see in your draft is that it allows an intermediate
> device to perform segmentation/reassembly instead of fragmentation.
>
>
>
>  Likewise, setting large MTU on overlay interfaces achieves similar
> result, but still the overlay/underlay decoupling issue. It is usually
> advised that overlay MTU is slightly smaller than underlay to avoid
> inefficient fragmentation after adding the outer header, but to achieve
> really high performance between VMs, large MTU is preferred.
>
>
>
> Depends on the performance dimension to be optimized. Larger MTUs could
> increase latency of high priority small packets for instance (HOL
> blocking), or UDP based application might try to use MTU to decide how
> large it should send it's packets to avoid fragmentation.
>
>
>
>  And considering overlay <-> physical connection, path MTU discovery is
> not always work.
>
>   This kind of configuration complexity and pain-point can be resolved
> simply by decoupling overlay and underlay MTU, as suggested by this draft.
> Here is an example of configuration confusion:
>
> http://openvswitch.org/pipermail/discuss/2014-May/013898.html
>
>
>
> Ideally, all NV tunnel protocols should support similar metadata, thus
> overlay segmentation can be offloaded hop-by-hop.
>
>
>
> I think this would be applicable to about all tunnel protocols. Then you
> would also want to do reassembly at each hop? Sounds expensive. Once you
> segment, I think you'd only only want to reassemble at the end host.
>
>
>
> One important thing to keep in mind, and a hard lesson in real deployment
> :-), segmentation offload is opportunistic and very dependent on the
> conditions of the traffic. It's value can be fleeting in real workloads.
> For instance imagine a host with a lot of active high throughout
> connections (like a video serving) which hits a hiccup causing all
> congestions windows to shrink. If the system is not provisioned for this
> event it can be very hard to recover (a lot more CPU is required to achieve
> same throughput). This differentiates a larger MTU from relying on
> segmentation, the former offers more predictable CPU savings.
>
>
>
> Tom
>
>
>
>
>
> Best regards,
>
> Han
>
>
>
> *From:* Tom Herbert [mailto:[e-mail]]
> *Sent:* Wednesday, May 21, 2014 2:44 AM
> *To:* Zhou, Han
> *Cc:* [e-mail]; [e-mail];
> [e-mail];
> [e-mail]; Erik Nordmark
> *Subject:* Re: FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt
>
>
>
> Hi Zou, a couple of questions inline.
>
>
>
> On Mon, May 19, 2014 at 8:01 PM, Zhou, Han <[e-mail]> wrote:
>
> Hi,
>
> We have updated the VXLAN-SOE draft according to earlier comments. Now it
> is fully compatible with VXLAN-GPE. And some examples are added for better
> understanding
>
>
>
>  A prototype is also implemented here (patch based on Open vSwitch):
>
> https://github.com/hzhou8/openvswitch/commit/9a7deb8b432ce83a9c09d7d4ff85fa050f7dd2be
>
> netperf TCP_STREAM test result between a pairs of VMs on hosts with 10G
> interfaces:
>
> Before the change: 2.62 Gbits/sec
> After the change: 6.68 Gbits/sec
> Speedup is ~250%.
>
>  Can you provide some more details on this benefit? It seems like plain
> TSO/LRO that understands encapsulation should provide similar benefits when
> going between hosts.
>
>
>
>
>
> The patch attracted some interests in OVS community, but since this RFC
> draft is in very early stage so it is regarded as inappropriate by Jesse to
> apply the change to OVS tree.
> The discuss mail-thread:
> http://openvswitch.org/pipermail/discuss/2014-May/013981.html
> http://openvswitch.org/pipermail/discuss/2014-May/013898.html
>
> So we would like to request a review here by NVO3/TOFOO groups and VXLAN
> authors: is this VXLAN extension is worth formally put into VXLAN as a
> standard, so that more people can benefit from it?
>
>  Could you get the same effect by setting larger MTUs on the overlay
> network interface and relying in path MTU discovery when going over
> physical network?
>
>
>
>  Best regards,
> Han
>
> -----Original Message-----
> From: I-D-Announce [mailto:[e-mail]] On Behalf Of
> [e-mail]
> Sent: Friday, May 02, 2014 8:09 PM
> To: [e-mail]
> Subject: I-D Action: draft-zhou-li-vxlan-soe-01.txt
>
>
> A New Internet-Draft is available from the on-line Internet-Drafts
> directories.
>
>
>         Title           : Segmentation Offloading Extension for VXLAN
>         Authors         : Han Zhou
>                           Chengyuan Li
>         Filename        : draft-zhou-li-vxlan-soe-01.txt
>         Pages           : 13
>         Date            : 2014-05-02
>
> Abstract:
>    Segmentation offloading is nowadays common in network stack
>    implementation and well supported by para-virtualized network device
>    drivers for virtual machine (VM)s. This draft describes an extension
>    to Virtual eXtensible Local Area Network (VXLAN) so that segmentation
>    can be decoupled from physical/underlay networks and offloaded
>    further to the remote end-point thus improving data-plane performance
>    for VMs running on top of overlay networks.
>
>
> The IETF datatracker status page for this draft is:
> https://datatracker.ietf.org/doc/draft-zhou-li-vxlan-soe/
>
> There's also a htmlized version available at:
> http://tools.ietf.org/html/draft-zhou-li-vxlan-soe-01
>
> A diff from the previous version is available at:
> http://www.ietf.org/rfcdiff?url2=draft-zhou-li-vxlan-soe-01
>
>
> Please note that it may take a couple of minutes from the time of
> submission
> until the htmlized version and diff are available at tools.ietf.org.
>
> Internet-Drafts are also available by anonymous FTP at:
> ftp://ftp.ietf.org/internet-drafts/
>
> _______________________________________________
> I-D-Announce mailing list
> [e-mail]
> https://www.ietf.org/mailman/listinfo/i-d-announce
> Internet-Draft<https://www.ietf.org/mailman/listinfo/i-d-announceInternet-Draft>directories:
> http://www.ietf.org/shadow.html
> or ftp://ftp.ietf.org/ietf/1shadow-sites.txt
>
>
>
>
>