Comments on: RDMA on Converged Ethernet http://digitalvampire.org/blog/index.php/2009/03/25/rdma-on-converged-ethernet/ Linux hacker, recovering mathematician, former athlete Fri, 11 Mar 2011 20:17:40 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: James http://digitalvampire.org/blog/index.php/2009/03/25/rdma-on-converged-ethernet/comment-page-1/#comment-1017 James Thu, 06 May 2010 11:36:09 +0000 http://digitalvampire.org/blog/?p=76#comment-1017 Apparently you can use SoftRoCE by systemfabricworks to use standard ethernet NICs to try out RoCE without any specific hardware as such. Obviously the performance will not be nearly as good as it would with some of the Mellanox cards or whatever, but it could be a good starting point! Apparently you can use SoftRoCE by systemfabricworks to use standard ethernet NICs to try out RoCE without any specific hardware as such. Obviously the performance will not be nearly as good as it would with some of the Mellanox cards or whatever, but it could be a good starting point!

]]> By: James http://digitalvampire.org/blog/index.php/2009/03/25/rdma-on-converged-ethernet/comment-page-1/#comment-979 James Wed, 05 May 2010 00:03:53 +0000 http://digitalvampire.org/blog/?p=76#comment-979 @roland, I loved your article and pretty much agree with it entirely. I do believe that RoCE will only pick up if you can use fairly standard ethernet NICs, is this the case? Roland, do you know if it's so? Some datacenters, and some circumstances don't make TCP the ideal protocol, and although iWARP does make a huge difference, if you want to avoid TCP, RoCE does seem like a fairly decent alternative.... the fact that infiniband uses a credit/token based protocol sometimes does yield it's benefits with regard to flow control, although it's true that it's implementation of reliability may be more complex than TCP altogether. Love your blog Roland!! :-) @roland, I loved your article and pretty much agree with it entirely. I do believe that RoCE will only pick up if you can use fairly standard ethernet NICs, is this the case? Roland, do you know if it’s so? Some datacenters, and some circumstances don’t make TCP the ideal protocol, and although iWARP does make a huge difference, if you want to avoid TCP, RoCE does seem like a fairly decent alternative…. the fact that infiniband uses a credit/token based protocol sometimes does yield it’s benefits with regard to flow control, although it’s true that it’s implementation of reliability may be more complex than TCP altogether. Love your blog Roland!! :-)

]]> By: Roland’s Blog » Blog Archive » Rocky roads http://digitalvampire.org/blog/index.php/2009/03/25/rdma-on-converged-ethernet/comment-page-1/#comment-926 Roland’s Blog » Blog Archive » Rocky roads Mon, 19 Apr 2010 23:16:52 +0000 http://digitalvampire.org/blog/?p=76#comment-926 [...] announced the “RDMA over Converged Ethernet (RoCE)” specification today.  I’ve already discussed my thoughts on the underlying technology (although I have a bit more to say), so for now I just [...] [...] announced the “RDMA over Converged Ethernet (RoCE)” specification today.  I’ve already discussed my thoughts on the underlying technology (although I have a bit more to say), so for now I just [...]

]]> By: roland http://digitalvampire.org/blog/index.php/2009/03/25/rdma-on-converged-ethernet/comment-page-1/#comment-880 roland Mon, 28 Sep 2009 17:06:34 +0000 http://digitalvampire.org/blog/?p=76#comment-880 @Pete: A) not sure why you wouldn't look at it as IBoE -- taking IB transport protocol and changing the L2 encapsulation to ethernet sure looks like exactly what IBoE means. You might be able to make a proprietary protocol quickly but are you going to be able to do enough research/simulation to handle congestion stably, etc? And you have to figure out how to tie into ethernet multicast management (most 10 gig ethernet switches are going to do IGMP snooping), do address discovery to know how to talk to other endpoints, etc. B) Yes, TCP introduces some overhead on the NIC, but you need some reliability protocol (and if anything, the IB RC protocol is more complex in some ways than TCP, with multiple kinds of ACK/NAKs etc). And TCP offload is a solved problem at this point anyway -- cf Chelsio, NetEffect/Intel, etc. @Pete: A) not sure why you wouldn’t look at it as IBoE — taking IB transport protocol and changing the L2 encapsulation to ethernet sure looks like exactly what IBoE means. You might be able to make a proprietary protocol quickly but are you going to be able to do enough research/simulation to handle congestion stably, etc? And you have to figure out how to tie into ethernet multicast management (most 10 gig ethernet switches are going to do IGMP snooping), do address discovery to know how to talk to other endpoints, etc.

B) Yes, TCP introduces some overhead on the NIC, but you need some reliability protocol (and if anything, the IB RC protocol is more complex in some ways than TCP, with multiple kinds of ACK/NAKs etc). And TCP offload is a solved problem at this point anyway — cf Chelsio, NetEffect/Intel, etc.

]]> By: Pete http://digitalvampire.org/blog/index.php/2009/03/25/rdma-on-converged-ethernet/comment-page-1/#comment-879 Pete Fri, 25 Sep 2009 22:44:50 +0000 http://digitalvampire.org/blog/?p=76#comment-879 Doing some investigation and I have to agree with Andy. A) I do not look at it as IBoE. I think I could make a proprietary RDMA protocol between two intelligent NICs pretty quick. And B) the effort and overhead of running the TCP/IP on the Nic I would like to avoid. This is the research that brought me here. Looking for a non-proprietary standard non-TCP method. These writeups may have motivated me to think more about proprietary, though still researching. Doing some investigation and I have to agree with Andy. A) I do not look at it as IBoE. I think I could make a proprietary
RDMA protocol between two intelligent NICs pretty quick. And B) the effort and overhead of running the TCP/IP on the Nic I would like to avoid. This is the research that brought me here. Looking for a non-proprietary standard non-TCP method.
These writeups may have motivated me to think more about proprietary, though still researching.

]]>