I’ve said “so long” to Typo and migrated my blog to WordPress. WordPress just seems to have more momentum than Typo, even though PHP seems kind of 90s to me.
Anyway, I’ve set up a redirect so the previous RSS feed should continue to work, but you’ll probably want to update your URL if for some strange reason you’ve actually subscribed to my blog’s feed.
I’m back from a little more than two weeks of relaxing vacation. I didn’t check my work email once, so if you sent me an email @cisco.com, please understand if it takes me a few days to reply:
$ from|wc -l 5645 $ ls -lh /var/mail/rdreier -rw------- 1 rdreier floppy 61M 2007-01-04 07:38 /var/mail/rdreier
I just converted the main repositories for two libraries that I maintain, libibverbs and libmthca, from subversion to git. And even though I work with git a lot when working on the kernel, it’s still a shock to see how much easier git makes everything.
A few amusing examples:
$ du -sh libibverbs.* 980K libibverbs.git 1.3M libibverbs.svn
Yes, the git checkout with full development history takes up less disk space than a svn working copy with just the tip of the trunk (accessing history goes over the network for svn). And svn needing the network to do stuff has implications beyond just “work on my laptop on a plane”:
$ cd libibverbs.svn $ time svn log > /dev/null real 0m0.820s user 0m0.032s sys 0m0.004s $ cd ../libibverbs.git $ time git log > /dev/null real 0m0.005s user 0m0.004s sys 0m0.000s
Yes, git is more than 100 times faster for showing the log!
And these performance differences make a real productivity difference. With git, I’m much more likely to look through the history, examine past changes, and so on, which means that I waste less time figuring out things that I used to know.
I recently acquired a Lenovo ThinkPad X60s laptop (which is a really sweet machine if you want a small laptop). I installed the latest Ubuntu Edgy Eft development version, and I ran into one gotcha that I’m going to document here in case it bites you too.
Since the X60s has no CD-ROM drive, I started the Ubuntu installer via network boot, which worked very smoothly. The installer worked great, asking minimal questions and handling everything smoothly, including resizing the existing NTFS Windows partition and adding a Windows option to the grub menu.
However, when I booted into my new Ubuntu system, I was mystified by the fact that there was no wireless interface. lspci confirmed that my laptop did, as documented, have a Intel IPW3945 wireless device, and lsmod showed the ipw3945 module was loaded. A look at the kernel log showed that ipw3945 found its device and seemed to be happy, but ifconfig -a stubbornly showed only the wired eth0 interface.
After some head scratching and web searching, I noticed that there was no ipw3945d binary blob running in userspace. After doing some more research, I discovered that ipw3945d is contained in the restricted modules package — linux-restricted-modules-generic in my case. After installing that package and reloading the ipw3945 module, eth1 showed up and everything worked great.
So if you are missing an interface for your ipw3945, make sure you have ipw3945d running.
Here’s a short summary of what I plan to merge for 2.6.19. I sent this out via email to all the relevant lists, but I figured it can’t hurt to blog it too. Some of this is already in infiniband.git (git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git), while some still needs to be merged up. Highlights:
Ishai Rabinovitz:
IB/srp: Add port/device attributes
James Lentini:
IB/mthca: Include the header we really want
Michael S. Tsirkin:
IB/mthca: Don't use privileged UAR for kernel access
IB/ipoib: Fix flush/start xmit race (from code review)
Roland Dreier:
IB/uverbs: Use idr_read_cq() where appropriate
IB/uverbs: Fix lockdep warning when QP is created with 2 CQs
I read quite a bit of SF, but I’m not really a “fan” in the sense that I’ve never been to a Worldcon or cast a vote for the Hugos. However, I just belatedly looked at the 2006 Hugo nominees, and I was really surprised to see John Scalzi’s Old Man’s War is one of the finalists for best novel.
I happened to read this book when it appeared on the new books shelf of my library, and I found an amusing enough bit of literary junk food. But the thought “Hey! This is the best novel of the year!” never crossed my mind.
I’m not offended by the militarism of the book, as some people were — heck, I was fine with it when Seaton destroyed the whole Chloran galaxy in Skylark DuQuesne. I’m just appalled by the mediocrity of Scalzi’s book, now that it’s a Hugo nominee. The writing and characterization are adequate at best, and let’s just say that the plot has been done before. And the science is not good — this is SF at the Star Trek level. The aliens are basically people in makeup, and the technology is just souped up versions of present day stuff — apparently there have been no revolutions after the cell phone in this universe.
Wars with alien species on distant planets seem a lot like 20th century wars, which is pretty far fetched given that even war in the 21st century is not much like 20th century wars. Just as an example, let’s say you had advanced nanotech and you wanted to use it militarily. What would you do? Create a smart mist that turns enemy forces into gray goo? Nah, in Scalzi’s world they just make rifles that manufacture bullets on the fly — and they’re not even nanotech bullets that do something cool like subvert the enemy forces they hit, they’re just plain old bullets.
I mean really: quality SF should make futuristic stuff seem futuristic. I really got a kick out of the line “Simply grasping how such weapons were in some way disadvantageous to something loosely analagous to an enemy would have required such a comprehensive remapping of the human mind that it would be pointless calling it human anymore” in Alastair Reynolds’s Absolution Gap. Scalzi’s future just seems like the 1990s with starships and aliens added. (BTW, if you want to read a good space opera, with a nicely twisty plot, interesting ideas, and even decent characters, I recommend starting with Reynolds’s Revelation Space)
Maybe the explanation is that this is a weak year and Scalzi’s book really is the fifth best book. But I don’t buy it. John C. Wright’s very strong Orphans of Chaos was eligible. Even Karl Schroeder’s somewhat unsatisfying Lady of Mazes seems much more like a Hugo nominee (although it shouldn’t win).
I guess I’m left wondering what the Hugo voters saw in Scalzi’s book that I don’t.
gcc supports an extension that allows structures or structure members to be marked with __attribute__((packed)), which tells gcc to leave out all padding between members. Sometimes you need this, when you want to make sure of the layout of a structure. For example, you might have something like
struct my_struct {
uint8_t field1;
uint16_t field2;
} __attribute__((packed));
Without the packed attribute, the struct will have padding between field1 and field2, and that/s no good if this struct is something that has to match hardware or be sent on a wire.
However, it’s actively harmful to add the attribute to a structure that’s already going to be laid out with no padding. Sometimes, when a structure needs to be laid out without padding (because of hardware or wire protocol), people are tempted to add the attribute to a struct like the following “just to let the compiler know”
struct my_struct {
uint32_t field1;
uint32_t field2;
};
But adding __attribute__((packed)) goes further than just telling gcc that it shouldn’t add padding — it also tells gcc that it can’t make any assumptions about the alignment of accesses to structure members. And this leads to disastrously bad code on some architectures.
To see this, consider the simple code
struct foo { int a; };
struct bar { int b; } __attribute__((packed));
int c(struct foo *x) { return x->a; }
int d(struct bar *x) { return x->b; }
On architectures like x86, x86-64 and powerpc, both functions generate the same code. But take a look at what happens on ia64:
0000000000000000 <c>:
0: 13 40 00 40 10 10 [MBB] ld4 r8=[r32]
6: 00 00 00 00 10 80 nop.b 0x0
c: 08 00 84 00 br.ret.sptk.many b0;;
0000000000000010 <d>:
10: 09 70 00 40 00 21 [MMI] mov r14=r32
16: f0 10 80 00 42 00 adds r15=2,r32
1c: 34 00 01 84 adds r32=3,r32;;
20: 19 80 04 1c 00 14 [MMB] ld1 r16=[r14],1
26: f0 00 3c 00 20 00 ld1 r15=[r15]
2c: 00 00 00 20 nop.b 0x0;;
30: 09 70 00 1c 00 10 [MMI] ld1 r14=[r14]
36: 80 00 80 00 20 e0 ld1 r8=[r32]
3c: f1 78 bd 53 shl r15=r15,16;;
40: 01 00 00 00 01 00 [MII] nop.m 0x0
46: e0 70 dc ee 29 00 shl r14=r14,8
4c: 81 38 9d 53 shl r8=r8,24;;
50: 0b 70 40 1c 0e 20 [MMI] or r14=r16,r14;;
56: f0 70 3c 1c 40 00 or r15=r14,r15
5c: 00 00 04 00 nop.i 0x0;;
60: 11 00 00 00 01 00 [MIB] nop.m 0x0
66: 80 78 20 1c 40 80 or r8=r15,r8
6c: 08 00 84 00 br.ret.sptk.many b0;;
gcc gets scared about unaligned accesses and generates six times as much code (96 bytes vs. 16 bytes)! sparc64 goes similarly crazy, bloating from 12 bytes to 52 bytes:
0000000000000000 <c>:
0: 81 c3 e0 08 retl
4: d0 42 00 00 ldsw [ %o0 ], %o0
8: 30 68 00 06 b,a %xcc, 20 <d>
0000000000000020 <d>:
20: c6 0a 00 00 ldub [ %o0 ], %g3
24: c2 0a 20 01 ldub [ %o0 + 1 ], %g1
28: c4 0a 20 02 ldub [ %o0 + 2 ], %g2
2c: 87 28 f0 18 sllx %g3, 0x18, %g3
30: d0 0a 20 03 ldub [ %o0 + 3 ], %o0
34: 83 28 70 10 sllx %g1, 0x10, %g1
38: 82 10 40 03 or %g1, %g3, %g1
3c: 85 28 b0 08 sllx %g2, 8, %g2
40: 84 10 80 01 or %g2, %g1, %g2
44: 90 12 00 02 or %o0, %g2, %o0
48: 81 c3 e0 08 retl
4c: 91 3a 20 00 sra %o0, 0, %o0
50: 30 68 00 04 b,a %xcc, 60 <d+0x40>
So the executive summary is: don’t add __attribute__((packed)) to your code unless you know you need it.
After a smooth trip and suprisingly faster-than-scheduled trip, I’m safely in Ottawa, ready for the Kernel Summit and OLS. If you see me, say hello (of course I would expect you to do that even if you don’t read my blog).
By the way, the network in my hotel seems to have a broken DNS server that doesn’t respond to AAAA requests. This makes Firefox take forever to do anything, because it tries an IPv6 lookup that has to time out before it finds a site’s regular IPv4 address. However, I discovered the network.dns.disableIPv6 configuration option – setting this to true cured my browsing problems.
A good way to kick this blog off is probably to explain what this RDMA stuff that I work on really is.
RDMA stands for Remote Direct Memory Access, but the term “RDMA” is usually used to refer to networking technologies that have a software interface with three features:
InfiniBand host channel adapters (HCAs) are an example of network adapters that offer such an interface, but RDMA over IP (iWARP) adapters are starting to appear as well.
Anyway, let’s take a look at what these three features really mean.
Remote DMA
Remote DMA is pretty much what it sounds like: DMA on a remote system. The adapter on system 1 can send a message to the adapter on system 2 that causes the adapter on system 2 to DMA data to or from system 2’s memory. The messages come in two main types:
These messages are “one-sided” in the sense that they will be processed by the adapter that receive them without involving the CPU on the system that receives the messages.
Letting a remote system DMA into your memory sounds pretty scary, but RDMA adapters give fine-grained control over what remote systems are allowed to do. Going into the details now will make this entry way too long, so for now just trust me that things like protection domains and memory keys allow you to control connection-by-connection and byte-by-byte with separate read and write permissions.
To see why RDMA is useful, you can think of RDMA operations as “direct placement” operations: data comes along with information about where it’s supposed to go. For example, there is a spec for NFS/RDMA, and it’s pretty easy to see why RDMA is nice for NFS. The NFS/RDMA server can service requests in whatever order it wants and return responses via RDMA as they become available; by using direct placement, the responses can go right into the buffers where the client wants them, without requiring the NFS client to do any copying of data.
(There are actually some more complicated RDMA operations that are supported on InfiniBand, namely atomic fetch & add, and atomic compare & swap, but those aren’t quite as common so you can ignore them for now)
Asynchronous work queues
Software talks to RDMA adapters via an aynchronous interface. This doesn’t really have all that much to do with remote DMA, but when we talk about RDMA adapters, we expect this type of interface (which is called a “verbs” interface for some obscure historical reason).
Basically, to use an RDMA adapter, you create objects called queue pairs (or QPs), which as the name suggests are a pair of work queues: a send queue and a receive queue, and completion queues (or CQs). When you want to do something, you tell the adapter to post an operation to one of your work queues. The operation executes asynchronously, and when it’s done, the adapter adds work completion information onto the end of your CQ. When you’re ready, you can go retrieve completion information from the CQ to see which requests have completed.
Operating asynchronously like this makes it easier to overlap computation and communication.
(Incidentally, as the existence of receive queues might make you think, RDMA adapters support plain old “two-sided’ send/receive operations, in addition to one-sided RDMA operations. You can post a receive request to your local receive queue, and then next send message that comes in will be received into the buffer you provided. RDMA operations and send operations can be mixed on the same send queue, too)
Kernel bypass
The last feature which is common to RDMA adapters also has nothing to do with remote DMA per se. But RDMA adapters allow userspace processes to do fast-path operations (posting work requests and retreiving work completions) directly with the hardware without involving the kernel at all. This is nice because these adapters are typically used for high-performance, latency-sensitive applications, and saving the system call overhead is a big win when you’re counting nanoseconds.
OK, that’s it for today’s edition of “RDMA 101.†Now we have some background to talk about some of the more interesting features of RDMA networking.
I couldn’t begin to guess what Gillette will put in the razor they come out with after their latest Gillette Fusion Power, since its hard to imagine how you top a razor with five blades and a “patented on-board micro-chip” that “optimizes the performance of the razor.”
But I bet the next Gillette razor, whatever it is, will run Linux.