RE: [Ipoverib] Please read - proposed WG termination

[Michael Krause <[e-mail]>]

At 11:19 AM 9/1/2005, H.K. Jerry Chu wrote:
>[co-chair hat off]
>
>...
><snip>
>
> >These performance problems are primarily implementation-specific and have
> >little to do with IB technology itself.  In addition, nearly all IB
> >solutions use a 2KB not the smallest MTU to transfer data - no different
> >than Ethernet.
>
>Ethernet is adopting jumboframe to get more firing power. Where is IB's
>equivalent of jumboframe?

Jumboframe causes QoS issues for multiple flows.  The whole reason for the 
variable MTU was to allow people to configure the different virtual lanes / 
paths to meet QoS concerns.  Examination of the hardware-based SAR provided 
by IB illustrates that there is no definitive performance difference 
between using a 2KB per and say a larger 9KB MTU.  BTW, examination of many 
IP workloads shows that majority of the units of work exchange fit quite 
nicely in a 2KB payload.  Those that are larger, are easily SAR without 
impacting either side of the communication, i.e. requiring a connection to 
be established.

Keep in mind that Ethernet is also doing other activities to solve various 
issues - multi-path support to compensate for the spanning tree 
limitations, conceptually similar virtual lanes are being investigated, 
improved security, link-level congestion management, etc.  All of these 
changes combined with de facto use of large send / receive-side scaling, 
etc. will likely make the issue of 9KB rather moot except for bulk data 
transfers in the WAN (even then the QoS issues of multiple flows across a 
common link will likely limit the use of 9KB).

In the end, IB HCA implementations can adopt the same techniques as 
Ethernet and deliver the same performance as a connected model without the 
overhead / complexity and ecosystem cost of a new specification.  IB 
already has enough problems getting the basics into the world 
today.  Adding in yet another spec isn't going to solve these problems or 
make life easier - too many things within the ecosystem are impacted.


> >As I and others have raised over the years, the enablement
> >of IP over IB to perform well is a local HCA issue not a standards
> >issue.  Addition of checksum off-load support to the HCA is rather trivial
> >and does not require standardization (this is what is done for Ethernet
> >today and is non-standard).  Addition of large send off-load support is a
> >local HCA issue not a standards issue and effectively provides the same
> >benefit as connected mode.
>
>Yes LSO (or TSO as some call it) is relatively easy. But LRO (large receive
>offload) is a heck more difficult. IB connected transports already have all
>silicons to do it. Why not just use it?

Large receive off-load isn't that hard - really an implementation problem 
from most of the designs I've examined.  I helped implement something 
equivalent to TSO and LRO over 10 years ago and it wasn't that bad or 
expensive.  As for why not just use it?  Simple, cost to specify, 
implement, get adopted within the industry is already high for IB.  Adding 
more to the stack of things when there is questionable benefit compared to 
simply re-using the existing Ethernet infrastructure seems like a poor 
choice of resource expenditure.  Of course, most people only care about 
making Linux happen these days so I guess feel free to knock yourselves out 
if that is all that is required.


> >The use of multiple QP to spread work across
> >CPU for both send / receive ala the multi-queue support I've worked with
> >various Ethernet IHV to get in place is again a local HCA issue (does not
> >have to be visible as part of the layer 2 address resolution).  One can
> >construct a very nice performing IP over IB solution but there hasn't been
> >much public progress to implement these de facto capabilities found in
> >Ethernet solutions on IB.  Getting these into a HCA implementation is a
> >heck of a lot easier and faster to do than to develop a standard and
> >getting all of the OS changes made (the HCA implementation issues can all
> >be done underneath the IP stack just like with Ethernet so no real OS 
> impacts).
>
>I don't understand the large MTU issue to the OS (requiring continguous 
>physical
>addresses). Aren't all decent hardware capable of scatter/gather these days?

Nothing above indicates any issue regarding physical addresses.  IB 
hardware as does Ethernet comprehends V-to-P mapping and makes no 
requirements on it being physically contiguous beyond the base physical 
page size.


>What's more hairy to the OS stack is the per-destination MTU and different
>MTU for multicast than for unicast inherited in IPoIB CM.

Agreed.  This is one aspect of the additional complexity that is simply 
easier to avoid.   The cost / benefit of connected mode is quite 
questionable compared to leveraging the existing Ethernet implementation 
concepts and associated infrastructure.

Mike


>Jerry
>
> >
> >
> >>For commercial clusters, if IB is used for storage, then you save a network
> >>by having fast IP performance and can use the IB network for both. Why use
> >>IB and another network for the commercial cluster, when the other network
> >>supports similar bandwidth for storage and IP.
> >
> >There will always be Ethernet in any cluster so the fabric is there.  The
> >question is whether it is just for low-bandwidth / management services or
> >for applications.  For storage, need to separate the discussion into
> >whether it is block or file.  For block, IB gateways to Fibre Channel, etc.
> >can and are being used today quite nicely.  Performance is reasonable and
> >the ecosystem costs, target availability, customer "pain", etc. are much
> >lower than attempting to move to native IB storage.  The same applies to
> >file based where IB gateways to Ethernet which then attaches to file
> >servers works quite nicely.  In fact, the original vision of IB was that of
> >an I/O fabric to create modular server solutions.  The addition of IPC came
> >later in the process when it was found to be relatively low cost to
> >define.  So, IB is successful in the HPC world and slowly entering some
> >commercial solutions.  To state that its future relies on getting an IP
> >over IB RC solution is perhaps blowing it a bit out of proportion.   The
> >easier path for all is to simply use the techniques I and others have
> >advocated for years now and solve the problems within the HCA
> >implementation.  Much lower costs and will result in delivering a good
> >performance solution.
> >
> >BTW, RNIC / Ethernet solutions implement these techniques today.  With the
> >arrival of 10 GbE and the lower prices of RNIC and 10 GbE switch ports,
> >lower latency switches (competitive enough with IB for commercial and many
> >HPC clusters), etc. the success of IB must lie elsewhere and not on an IETF
> >spec.   This was noted at the recent IEEE Hot Interconnects conference as
> >well so isn't just my opinion.
> >
> >Mike
> >
> >>Implementing IPoIB-CM makes IB viable in the HPC cluster and some
> >>commercial clusters. Otherwise I don't think it competes economically with
> >>other network technologies.
> >>
> >>Regards.
> >>
> >>Bernie King-Smith
> >>IBM Corporation
> >>Server Group
> >>Cluster System Performance
> >>[e-mail]    [phone]
> >>Tie. 293-8483 or wombat2 on NOTES
> >>
> >>"We are not responsible for the world we are born into, only for the world
> >>we leave when we die.
> >>So we have to accept what has gone before us and work to change the only
> >>thing we can,
> >>-- The Future." William Shatner
> >>
> >>
> >>
> >>              Dror Goldenberg
> >>              <gdror@mellanox.c
> >>              o.il>                                                      To
> >>              Sent by:                  [e-mail],
> >>              ipoverib-bounces@         "H.K. Jerry Chu"
> >>              ietf.org                  <[e-mail]>
> >>                                                                         cc
> >>                                        [e-mail],
> >>              08/30/2005 09:32          [e-mail],
> >>              AM                        [e-mail]
> >>                                                                    Subject
> >>                                        RE: [Ipoverib] Please read -
> >>                                        proposed WG termination
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> > From: Vivek Kashyap [mailto:[e-mail]]
> >> > Sent: Tuesday, August 30, 2005 8:39 AM
> >> >
> >> > On Mon, 29 Aug 2005, H.K. Jerry Chu wrote:
> >> >
> >>
> >>
> >><snip>
> >>
> >>
> >> > > 1. IPoIB connected mode draft-ietf-ipoib-connected-mode-00.txt
> >> > > updated recently
> >> >
> >> > Well, in recent days there has been a discussion going on
> >> > based on Dror's input. I also made some updates after some
> >> > discussion on OpenIB (not on
> >> > IETF though).  This draft itself became a working group draft
> >> > this february
> >> > after some lively discussion just before that.  It appears to
> >> > me that we
> >> > should be possible to finalise this draft soon enough.
> >> >
> >> > 20th sept. might be long enough to know one way or the other...
> >> >
> >> > vivek
> >> >
> >>
> >>
> >>We would like to see IPoIB-CM being finalized in IETF. We see
> >>great value in having a standard for connected mode which effectively
> >>increases the MTU. We are willing to contribute to the standardization
> >>effort. We're also looking at the implementation of IPoIB-CM in Linux.
> >>
> >>
> >>-Dror _______________________________________________
> >>IPoverIB mailing list
> >>[e-mail]
> >>https://www1.ietf.org/mailman/listinfo/ipoverib
> >>
> >>
> >>
> >>
> >>
> >>_______________________________________________
> >>IPoverIB mailing list
> >>[e-mail]
> >>https://www1.ietf.org/mailman/listinfo/ipoverib
>
>
>_______________________________________________
>IPoverIB mailing list
>[e-mail]
>https://www1.ietf.org/mailman/listinfo/ipoverib

_______________________________________________
IPoverIB mailing list
[e-mail]
https://www1.ietf.org/mailman/listinfo/ipoverib