As I mentioned on Twitter (by the way, are you following @rolanddreier?), I’ll be speaking at the Linux Foundation Collaboration Summit in San Francisco on April 7. My general mandate is to give an introduction to RDMA and InfiniBand on Linux, and to talk about recent developments and what might be coming next in the area. However, I’d like to make my talk a little less boring than my usual talks, so I’d be curious to hear about specific topics you’d like me to cover. And if you’re at the summit, stop by and say hello.
Archive for the ‘infiniband’ Category
Since I changed jobs, I left behind a lot of my test systems, but I now have a couple of test systems set up. Here is the rather crazy set of non-chipset devices I now have in one box:
$ lspci -nn|grep -v 8086: 03:00.0 InfiniBand [0c06]: Mellanox Technologies MT25208 [InfiniHost III Ex] [15b3:6282] (rev 20) 04:00.0 Ethernet controller : Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] [15b3:6750] (rev b0) 05:00.0 InfiniBand [0c06]: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] [15b3:673c] (rev b0) 84:00.0 Ethernet controller : NetEffect NE020 10Gb Accelerated Ethernet Adapter (iWARP RNIC) [1678:0100] (rev 05) 85:00.0 Ethernet controller : Chelsio Communications Inc T310 10GbE Single Port Adapter [1425:0030] 86:00.0 InfiniBand [0c06]: Mellanox Technologies MT25204 [InfiniHost III Lx HCA] [15b3:6274] (rev 20)
(I do have a couple of open slots if you have some RDMA cards that I’m missing to complete my collection :))
Fences (2016) HD
|Producer||:||Scott Rudin, Denzel Washington.|
|Release||:||December 16, 2016|
|Country||:||United States of America.|
|Production Company||:||Paramount Pictures, Scott Rudin Productions, Bron Creative, MACRO.|
Movie ‘Fences’ was released in December 16, 2016 in genre Drama. Denzel Washington was directed this movie and starring by Denzel Washington. This movie tell story about In 1950s Pittsburgh, a frustrated African-American father struggles with the constraints of poverty, racism, and his own inner demons as he tries to raise a family.
Do not miss to Watch movie Fences (2016) Online for free with your family. only 2 step you can Watch or download this movie with high quality video. Come and join us! because very much movie can you watch free streaming.
download full film Fences 2016
Fences 2016 Episodes Online
Fences 2016 Full Episodes Online
Fences 2016 English Full Episodes Free Download
Fences 2016 Online Free Megashare
live streaming movie Fences online
Fences 2016 HD Full Episodes Online
watch full film Fences 2016
watch full Fences 2016 film online
Fences 2016 English Full Episodes Download
live streaming film Fences 2016 online
download film Fences
Watch Fences 2016 Online Free megashare
Fences 2016 English Episodes
Fences 2016 HD English Full Episodes Download
Watch Fences 2016 Online Free Viooz
watch full Fences 2016 movie
streaming film Fences 2016
Watch Fences 2016 Online Free
Watch Fences 2016 Online Putlocker
Fences 2016 For Free Online
Fences 2016 For Free online
Watch Fences 2016 Online Viooz
Fences 2016 Full Episodes Watch Online
trailer movie Fences
Fences 2016 Full Episode
download movie Fences now
Fences 2016 English Episodes Free Watch Online
watch full movie Fences 2016
Watch Fences 2016 Online Free Putlocker
Watch Fences 2016 Online Free putlocker
download film Fences 2016 now
watch movie Fences 2016 now
Fences 2016 Watch Online
Watch Fences 2016 Online Megashare
Fences 2016 English Full Episodes Online Free Download
Fences 2016 Episodes Watch Online
Fences 2016 English Full Episodes Watch Online
Fences 2016 English Episode
watch full movie Fences online
film Fences 2016
I want to mention two things about IBoE. (I’m using the term InfiniBand-over-Ethernet, or IBoE for short, for what the IBTA calls RoCE for reasons already discussed)
First, we merged IBoE support on mlx4 devices into the upstream kernel in 2.6.37-rc1, so IBoE will be in upstream kernel for the 2.6.37 release — one fewer reason to use OFED. (And by the way, we used the term IBoE in the kernel) The requisite libibverbs and libmlx4 patches are not merged yet, but I hope to get to that soon and release new versions of the userspace libraries with IBoE support.
Second, a while ago I promised to detail some of my specific critiques of the IBoE spec (more formally, “Annex A16: RDMA over Converged Ethernet (RoCE)” to the “InfiniBand Architecture Specification Volume 1 Release 1.2.1”; if you want to follow along at home, you can download a copy from the IBTA). So here are two places where I think it’s really obvious that the spec is a half-assed rush job, to the detriment of trying to create interoperable implementations. (Fortunately everyone will just copy what the Linux stack does if they don’t actually just reuse the code, but still it would have been nice if the people writing the standards had thought things through instead of letting us just make something up and hope it there are no corner cases that will bite us later)
- The annex has this to say about address resolution in A16.5.1, “ADDRESS ASSIGNMENT AND RESOLUTION”:
The means for resolving a GID to a local port address (i.e. SMAC or DMAC) are outside the scope of this annex. It is assumed that standard Ethernet mechanisms, such as ARP or Neighbor Discovery are used to maintain an appropriate address cache for RoCE ports.
It’s easy to say that something is “outside the scope” but, uh, who else is going to specify how to turn an IB GID into an Ethernet address, if not the spec about how to run IB over Ethernet packets? And how could ARP conceivably be used, given that GIDs are 128-bit IPv6 addresses? If we’re supposed to use neighbor discovery, a little more guidance about how to coordinate the IPv6 stack and the IB stack might be helpful. In the current Linux code, we finesse all this by assuming that (unicast) GIDs are always local-scope IPv6 addresses with the Ethernet address encoded in them, so converting a GID to a MAC is trivial (cf
- This leads to the second glaring omission from the spec: nowhere are we told how to send multicast packets. The spec explicitly says that multicast should work in IBoE, but nowhere does it say how to map a multicast GID to the Ethernet address to use when sending to that MGID. In Linux we just used the standard mapping from multicast IPv6 addresses to multicast Ethernet addresses, but this is a completely arbitrary choice not supported by the spec at all.
You may hear people defending these omissions from the IBoE spec by saying that these things should be specified elsewhere or are out of scope for the IBTA. This is nonsense: who else is going to specify these things? In my opinion, what happened is simply that (for non-technical reasons) some members of the IBTA wanted to get a spec out very quickly, and this led to a process that was too short to produce a complete spec.
I saw that the InfiniBand Trade Association announced the “RDMA over Converged Ethernet (RoCE)” specification today. I’ve already discussed my thoughts on the underlying technology (although I have a bit more to say), so for now I just want to say that I really, truly hate the name they chose. There are at least two things that suck about the name:
- Calling the technology “RDMA over” instead of “InfiniBand over” is overly vague and intentionally deceptive. We already have “RDMA over Ethernet” — except we’ve been calling it iWARP. Choosing “RoCE” is somewhat like talking about “Storage over Ethernet” instead of “Fibre Channel over Ethernet.” Sure, FCoE is storage over ethernet, but so is iSCSI. As for the intentionally deceptive part: I’ve been told that “InfiniBand” was left out of the name because the InfiniBand Trade Association felt that InfiniBand is viewed negatively in some of the markets they’re going after. What does that say about your marketing when you are running away from your own main trademark?
- The term “Converged Ethernet” is also pretty meaningless. The actual technology has nothing to do with “converged” ethernet (whatever that is, exactly); the annex that was just release simply describes how to stick InfiniBand packets inside a MAC header and Ethernet FCS, so simply “Ethernet” would be more accurate. At least the “CE” part is an improvement over the previous try, “Converged Enhanced Ethernet” or “CEE”; not only does the technology have nothing to do with CEE either, “CEE” was an IBM-specific marketing term for what eventually became Data Center Bridging or “DCB.” (At Cisco we used to use the term “Data Center Ethernet” or “DCE”)
So both the “R” and the “CE” of “RoCE” aren’t very good choices. It would be a lot clearer and more intellectually honest if we could just call InfiniBand over Ethernet by its proper name: IBoE. And explaining the technology would be a bit simpler too, since the analogy with FCoE becomes a lot more explicit.
davejâ€™s recent post reminded me of a story from The Game. The Game, BTW, is a very amusing book about learning to pick up women â€“ if you havenâ€™t read it, I recommend it (even if youâ€™re not single).
Anyway, thereâ€™s a scene in the book where Style, our hero who happens to have a shaved head, is at a club, and he has a phenomenally easy time meeting women. Everything is working for him, and heâ€™s feeling great about himself, until just as he leaves, the hostess says, â€œArenâ€™t you Moby?â€
But I donâ€™t think anyone is going to mix up davej and Moby.
I recently read Andy Grover’s post about converged fabrics, and since I particupated in the OpenFabrics panel in Sonoma that he alluded to, I thought it might be worth sharing my (somewhat different) thoughts.
The question that Andy is dealing with is how to run RDMA on “Converged Ethernet.” I’ve already explained what RDMA is, so I won’t go into that here, but it’s probably worth talking about Ethernet, since I think the latest developments are not that familiar to many people. The IEEE has been developing a few standards they collectively refer to as “Data Center Bridging” (DCB) and that are also sometimes referred to as “Converged Enhanced Ethernet” (CEE). This refers to high speed Ethernet (currently 10 Gb/sec, with a clear path to 40 Gb/sec and 100 Gb/sec), plus new features. The main new features are:
- Priority-Based Flow Control (802.1Qbb), sometimes called “per-priority pause”
- Enhanced Transmission Selection (802.1Qaz)
- Congestion Notification (802.1Qau)
The first two features let an Ethernet link be split into multiple “virtual links” that operate pretty independently — bandwidth can be reserved for a given virtual link so that it can’t be starved, and by having per-virtual-link flow control, we can make sure certain traffic classes don’t overrun their buffers and avoid dropping packets. Then congestion notification means that we can tell senders to slow down to avoid congestion spreading caused by that flow control.
The main use case that DCB was developed for was Fibre Channel over Ethernet (FCoE). FC requires a very reliable network — it simply doesn’t work if packets are dropped because of congestion — and so DCB provides the ability to segregate FCoE traffic onto a “no drop” virtual link. However, I think Andy misjudges the real motivation for FCoE; the TCP/IP overhead of iSCSI was not really an issue (and indeed there are many people running iSCSI with very high performance on 10 Gb/sec Ethernet).
The real motivation for FCoE is to give a way for users to continue using all the FC storage they already have, while not requiring every server that wants to talk to the storage to have both a NIC and an FC HBA. With a gateway that’s easy to build an scale, legacy FC storage can be connected to an FCoE fabric, and now servers with a “converged network adapter” that functions as both an Ethernet NIC and an FCoE HBA can talk to network and storage over one (Ethernet) wire.
Now, of course for servers that want to do RDMA, it makes sense that they want a triple-threat converged adapter that does Ethernet NIC, FCoE HBA, and RDMA. The way that people are running RDMA over Ethernet today is via iWARP, which runs an RDMA protocol layered on top of TCP. The idea that Andy and several other people in Sonoma are pushing is to do something analogous to FCoE instead, that is, take the InfiniBand transport layer and stick it into Ethernet somehow. I see a number of problems with this idea.
First, one of the big reasons given for wanting to use InfiniBand on Ethernet instead of iWARP is that it’s the fastest path forward. The argument is, “we just scribble down a spec, and everyone can ship it easily.” That ignores the fact that iWARP adapters are already shipping from multiple vendors (although, to be fair, none with support for the proposed IEEE DCB standards yet; but DCB support should soon be ubiquitous in all 10 gigE NICs, iWARP and non-iWARP alike). And the idea that an IBoE spec is going to be quick or easy to write flies in the face of the experience with FCoE; FCoE sounded dead simple in theory (just stick an Ethernet header on FC frames, what more could there be?) it turns out that the standards work has taken at least 3 years, and a final spec is still not done. I believe that IBoE would be more complicated to specify, and fewer resources are available for the job, so a realistic view is that a true standard is very far away.
Andy points at a TOE page to say why running TCP on an iWARP NIC sucks. But when I look at that page, pretty much all the issues are still there with running the IB transport on a NIC. Just to take the first few on that page (without quibbling about the fact that many of the issues are just wrong even about TCP offload):
- Security updates: yup, still there for IB
- Point-in-time solution: yup, same for IB
- Different network behavior: a hundred times worse if you’re running IB instead of TCP
- Performance: yup
- Hardware-specific limits: yup
And so on…
Certainly, given infinite resources, one could design an RDMA protocol that was cleaner than iWARP and took advantage of all the spiffy DCB features. But worse is better and iWARP mostly works well right now; fixing the worst warts of iWARP has a much better chance of success than trying to shoehorn IB onto Ethernet and ending up with a whole bunch of unforseen problems to solve.
I’ve been trying to get a udev rule added to Ubuntu so that /dev/infiniband/rdma_cm is owned by group “rdma” instead of by root, so that unprivileged user applications can be given permission to use it by adding the user to the group rdma. This matches the practice in the Debian udev rules and is a simple way to allow unprivileged use of RDMA while still giving the administrator some control over who exactly uses it.
I created a patch to the Ubuntu librdmacm package containing the appropriate rule and opened a Launchpad bug report requesting that it be applied. After two months of waiting, I got a response that basically said, “no, we don’t want to do that.” After another month of asking, I finally found out what solution Ubuntu would rather have:
Access to system devices is provided through the HAL or DeviceKit interface. Permission to access is managed through the PolicyKit layer, where the D-Bus system bus service providing the device access negotiates privilege with the application requesting it.
Because of course, rather than having an application simply open a special device node, mediated by standard Unix permissions, we’d rather have to run a daemon (bonus points for using DBus activation, I guess) and have applications ask that daemon to open the node for them. More work to implement, harder to administer, less reliable for users — everyone wins!
At long last, after several requests, I’ve posted the slides, notes, and client and server examples from the tutorial I gave at LinuxConf.eu 2007 in Cambridge back in September. Hyper-observant readers will notice that the client program I posted does not match the listing in the notes I handed out; this is because I fixed a race condition in how completions are collected.
I’m not sure how useful all this is without me talking about it, but I guess every little bit helps. And of course, if you have questions about RDMA or InfiniBand programming, come on over to the mailing list and fire away.
With yesterday’s release of kernel 2.6.23, I thought it might be a good time to look back at what significant changes are in 2.6.23, and what we have queued up for 2.6.24..
So first I looked at the kernel git log from the v2.6.22 tag to the v2.6.23 tag, and I was surprised to find that nothing really stood out. We merged something like 158 patches that touched 123 files, but I couldn’t really find any headline-worthy new features in there. There were just tons of fixes and cleanups all over, although mostly in the low-level hardware drivers. For some reason, 2.6.23 was a pretty calm development cycle for InfiniBand and RDMA, which means that at least that part of 2.6.23 should be rock solid.
2.6.24 promises to be a somewhat more exciting release for us. In my for-2.6.24 branch, in addition to the usual pile of fixes and cleanups, I have a couple of interesting changes queued up to merge as soon as Linus starts pulling things in:
- Sean Hefty’s quality-of-service support. These changes allow administrators to configure the network to give different QoS parameters to different types of traffic (eg IPoIB, SRP, and so on).
- A patch from me (based on Sean Hefty’s work) to handle multiple P_Keys for userspace management applications. This is one of the last pieces to make the InfiniBand stack support IB partitions fully.
Also, bonding support for IP-over-InfiniBand looks set to go in through Jeff Garzik’s tree. This is something that I’ve been wanting to see for years now; the patches allow the standard bonding module to enslave IPoIB interfaces, which means that multiple IB ports can finally be used for IPoIB high-availability failover. Moni Shoua and others did a lot of work and stuck with this for a long time, and the final set of patches turned out to be very clean and nice, so I’m really pleased to see this get merged.