Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt

["Zhou, Han" <[e-mail]>]

Hi Joe,

This is an interesting topic.

> TCP offloading is fine when the OS hands off user data, and the offload
> engine creates the entire segment.
Existing TSO/GSO mechanisms deliver full (large) TCP segment to "offload engine", which then create smaller segments according to physical MTU, and recalculates checksums.
This is the case even without overlay considered. So I suppose the problem you pointed out is not related to my change, but a general limitation for TSO/GSO, right?

For my understanding the TCP implementation should decide whether to use offloading or not according to the feature/options required by a TCP connection. If the option required (such as MD5) is not supported by offloading, the TCP stack should do the segmentation by itself instead of utilizing offloading.

In fact, the proposal in this draft should be able to alleviate the limitation for TCP connections between VMs behind same gateways, because in this case there is no real TCP segmentation performed by "offload engine". 

Let me know if you have more concerns, or maybe an example of how an option is broken by TSO/GSO, then we can check what's the current solution in kernel.

Best regards,
Han

> -----Original Message-----
> From: Joe Touch [mailto:[e-mail]]
> Sent: Wednesday, May 21, 2014 9:14 AM
> To: Zhou, Han; [e-mail]; [e-mail];
> [e-mail];
> [e-mail]
> Cc: Erik Nordmark; Tom Herbert
> Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt
> 
> Hi, Han,
> 
> This helps, but doesn't quite address my concern.
> 
> TCP offloading is fine when the OS hands off user data, and the offload
> engine creates the entire segment.
> 
> The situation you're describing seems to be a hybrid, where the guest OS
> makes a TCP segment, and then something lower (in the VM system) parses
> that segment to create one or more segments on the wire.
> 
> If that happens, it will be incompatible with a number of existing TCP
> options, and will also cause side-effects with ACK clocking, timeouts,
> and more than a few other TCP features.
> 
> Although I appreciate the goal of efficiency and speed, there is a
> severe compatibility issue that doesn't seem to be addressed.
> 
> We can discuss this off-list if useful, FWIW.
> 
> Joe
> 
> On 5/20/2014 6:01 PM, Zhou, Han wrote:
> > Hi Joe,
> >
> > Thanks for your comment.
> >
> > Yes you are right that "length" fields in packet headers can be regarded as hard
> limit of MTU, but here in the draft we were referring interface MTU. We will
> make it more precise in next versions.
> >
> > For your question, it seems there are misunderstandings. It is always Guest OS
> (VM) handles the TCP, but segmentation is offloaded to host. Let me explain the
> code change:
> > - TX side:
> > -- before the change:
> >    TCP segmentation offloaded by TSO of Guest OS virt-driver from guest to
> host, MSS carried by GSO metadata in skbuff.
> >    VXLAN layer add encapsulation, and overlay TCP segmentation is carried
> right before sending to host IP layer, which is the idea of GSO - segment at the
> latest point.
> >    Host do IP fragmentation only if overlay segment + outer header exceed
> physical interface MTU. (e.g. when both guest MTU and host MTU are configured
> to 1500)
> > -- after the change:
> >    TCP segmentation offloaded by TSO of Guest OS virt-driver from guest to
> host, MSS carried by GSO metadata in skbuff.
> >    VXLAN layer removes GSO metadata and store it in VXLAN-SOE header. So
> overlay TCP segmentation is skipped.
> >    Host do IP fragmentation according to physical interface MTU.
> >
> > -RX side:
> > -- before the change:
> >    Host do IP reassembly if necessary. (e.g. when both guest MTU and host
> MTU are configured to 1500)
> >    Overlay TCP segments are decapsulated and delivered all the way to guest,
> and guest OS do TCP handling. (high cost here)
> > -- after the change:
> >    Host do IP reassembly, and large packets decapsulated and delivered to
> guest, and guest OS do TCP handling. (cost reduced here because of reduced
> number of packets)
> >
> > I hope this clarifies.
> >
> > Best regards,
> > Han
> >
> >> -----Original Message-----
> >> From: Joe Touch [mailto:[e-mail]]
> >> Sent: Wednesday, May 21, 2014 1:29 AM
> >> To: Zhou, Han; [e-mail]; [e-mail];
> >> [e-mail];
> >> [e-mail]
> >> Cc: Erik Nordmark; Tom Herbert
> >> Subject: Re: [Tofoo] FW: I-D Action: draft-zhou-li-vxlan-soe-01.txt
> >>
> >> Hi, all,
> >>
> >> I had a comment and a question:
> >>
> >> Comment - (from the doc) overlays do have a hard MTU limit; it is the
> >> limit of the encapsulation mechanism. E.g., without additional layers,
> >> for UDP in IPv4 this would be a at most 65507 bytes (i.e., IPv4 max -
> >> (min IP header + UDP header)). Using additional adaptation layers, this
> >> could be larger (e.g., see SEAL).
> >>
> >> Question - the code appears to have the VXLAN layer do the
> >> fragmentation, with the OS layer implementing the rest of TCP. There are
> >> a lot of interactions, notably:
> >>
> >> 	- any mechanism outside of the TCP source and TCP destination
> >> 	that interprets the TCP header will result in a decrease in
> >> 	functionality
> >> 		i.e., the TCP connection will support only the
> >> 		intersection of options and features supported
> >> 		by the source, dest, *and* VXLAN layers
> >>
> >> 		(rather than being limited only by the
> >> 		source-dest pair)
> >>
> >> 	- if passed a full TCP segment, this mechanism will be
> >> 	incompatible with TCP security (e.g., TCP MD5, TCP-AO, and
> >> 	the results of the TCPCRYPT WG.
> >>
> >> I'm not quite sure from your doc whether you're re-segmenting TCP
> >> segments, or merely collecting them for aggregate transit (e.g., as is
> >> done in burst-mode Ethernet).
> >>
> >> Can you please clarify?
> >>
> >> Joe
> >>
> >>
> >> On 5/19/2014 8:01 PM, Zhou, Han wrote:
> >>> Hi,
> >>>
> >>> We have updated the VXLAN-SOE draft according to earlier comments. Now
> it
> >> is fully compatible with VXLAN-GPE. And some examples are added for better
> >> understanding.
> >>>
> >>> A prototype is also implemented here (patch based on Open vSwitch):
> >>>
> >>
> https://github.com/hzhou8/openvswitch/commit/9a7deb8b432ce83a9c09d7d
> >> 4ff85fa050f7dd2be
> >>>
> >>> netperf TCP_STREAM test result between a pairs of VMs on hosts with 10G
> >> interfaces:
> >>>
> >>> Before the change: 2.62 Gbits/sec
> >>> After the change: 6.68 Gbits/sec
> >>> Speedup is ~250%.
> >>>
> >>> The patch attracted some interests in OVS community, but since this RFC
> draft
> >> is in very early stage so it is regarded as inappropriate by Jesse to apply the
> >> change to OVS tree.
> >>> The discuss mail-thread:
> >>> http://openvswitch.org/pipermail/discuss/2014-May/013981.html
> >>> http://openvswitch.org/pipermail/discuss/2014-May/013898.html
> >>>
> >>> So we would like to request a review here by NVO3/TOFOO groups and
> VXLAN
> >> authors: is this VXLAN extension is worth formally put into VXLAN as a
> standard,
> >> so that more people can benefit from it?
> >>>
> >>> Best regards,
> >>> Han
> >>>
> >>> -----Original Message-----
> >>> From: I-D-Announce [mailto:[e-mail]] On Behalf Of
> >> [e-mail]
> >>> Sent: Friday, May 02, 2014 8:09 PM
> >>> To: [e-mail]
> >>> Subject: I-D Action: draft-zhou-li-vxlan-soe-01.txt
> >>>
> >>>
> >>> A New Internet-Draft is available from the on-line Internet-Drafts
> directories.
> >>>
> >>>
> >>>           Title           : Segmentation Offloading Extension for
> VXLAN
> >>>           Authors         : Han Zhou
> >>>                             Chengyuan Li
> >>> 	Filename        : draft-zhou-li-vxlan-soe-01.txt
> >>> 	Pages           : 13
> >>> 	Date            : 2014-05-02
> >>>
> >>> Abstract:
> >>>      Segmentation offloading is nowadays common in network stack
> >>>      implementation and well supported by para-virtualized network
> device
> >>>      drivers for virtual machine (VM)s. This draft describes an extension
> >>>      to Virtual eXtensible Local Area Network (VXLAN) so that
> segmentation
> >>>      can be decoupled from physical/underlay networks and offloaded
> >>>      further to the remote end-point thus improving data-plane
> performance
> >>>      for VMs running on top of overlay networks.
> >>>
> >>>
> >>> The IETF datatracker status page for this draft is:
> >>> https://datatracker.ietf.org/doc/draft-zhou-li-vxlan-soe/
> >>>
> >>> There's also a htmlized version available at:
> >>> http://tools.ietf.org/html/draft-zhou-li-vxlan-soe-01
> >>>
> >>> A diff from the previous version is available at:
> >>> http://www.ietf.org/rfcdiff?url2=draft-zhou-li-vxlan-soe-01
> >>>
> >>>
> >>> Please note that it may take a couple of minutes from the time of submission
> >>> until the htmlized version and diff are available at tools.ietf.org.
> >>>
> >>> Internet-Drafts are also available by anonymous FTP at:
> >>> ftp://ftp.ietf.org/internet-drafts/
> >>>
> >>> _______________________________________________
> >>> I-D-Announce mailing list
> >>> [e-mail]
> >>> https://www.ietf.org/mailman/listinfo/i-d-announce
> >>> Internet-Draft directories: http://www.ietf.org/shadow.html
> >>> or ftp://ftp.ietf.org/ietf/1shadow-sites.txt
> >>>
> >>> _______________________________________________
> >>> Tofoo mailing list
> >>> [e-mail]
> >>> https://www.ietf.org/mailman/listinfo/tofoo
> >>>