Do you feel lucky, punk?
Sun just introduced their Constellation supercomputer at ISC Dresden. They’ve managed to get a lot of hype out of this, including mentions in places like the New York Times. But the most interesting part to me is the 3,456-port “Magnum” InfiniBand switch. I haven’t seen many details about it and I couldn’t find anything about it on Sun’s Constellation web site.
However I’ve managed to piece together some info about the switch from the new stories as well as the pictures in this blog entry. Physically, this thing is huge–it looks like it’s about half a rack high and two racks wide. The number 3,456 gives a big clue as to the internal architecture: 3456 = 288 * 12. Current InfiniBand switch chips have 24 ports, and the biggest non-blocking switch one can build with two levels (spine and leaf) is 24 * 12 = 288 ports: 24 leaf switches each of which have 12 ports to the outside and 12 ports to the spines (one port to each of the 12 spine switches).
Then, using 12 288-port switches as spines, one can take 288 24-port leaf switches that each have 12 ports to the outside and end up with 288 * 12 = 3456 ports, just like Sun’s Magnum switch. From the pictures of the chassis, it looks like Magnum has the spine switches on cards on one side of the midplane and the leaf switches on the other side, using the cute trick of having one set of cards be vertical and one set horizontal to get all-to-all connections between spines and leaves without having too-long midplane traces.
All of this sounds quite reasonable until you start to consider putting all of this in one box. Each 288 port switch (which is on one card in this design!) has 36 switch chips on it. At about 30 Watts per switch chip, each of this cards is over 1 kilowatt, and there are 12 of these in a system. In fact, with 720 switch chips in the box, the total system is well over 20 kW!
It also seems that the switch is using proprietary high-density connectors that bring three IB ports out of each connector, which reduces the number of external connectors on the switch down to a mere 1152.
One other thing I noticed is that the Sun press stuff is billing the Constellation as running Solaris, while the actual TACC page about the Ranger system says the cluster will be running Linux. I’m inclined to believe TACC, since running Solaris for an InfiniBand cluster seems a little silly, given how far behind Solaris’s InfiniBand support is when compared to Linux, whose InfiniBand stack is lovingly maintained by yours truly.
[…] this section of Roland Dreier’s blog discussing Sun’s new Constellation system quite hum…: One other thing I noticed is that the Sun press stuff is billing the Constellation as running […]
Roland, Sun, when you talk to them, is I believe, quite straightforward that TACC is Linux, not OpenSolaris. Bill.
Roland, Sun, if one asks them is, I believe, quite clear that TACC is Linux, not OpenSolaris. Bill.
Bill: I’m sure that most of the people at Sun have their story straight, but if you look at http://www.sun.com/servers/hpc/SunConstellationPreview.jsp then you can see what their marketing is saying:
which I think you have to agree is far from clear/straightforward about TACC’s system running Linux. (I couldn’t find the word Linux once on that page)
Hello! Yes, the magnum pulls alot of power. But when you consider the overall needed power for the entire system (3 Megawatts!), it’s not all that much. The real kicker is the scream of the hypersonic fans in the back, each turning at 23,000 rpms! It literally hurts your ears to stand near them. Also, if you have long hair, it would literally be blown back like a supermodel photo shoot.
And, yes, we will be running Linux on Ranger. Solaris, while a decent OS, is not yet ready for primetime in the hpc market.
Thanks for the link!
-SuperGeek.
A press release from Sun without Solaris means the system is running
Linux 🙂
About your description of the switch, maybe their 3456 port switch is more something like 144 * 24.
Looking at the boards at https://photos.sun.com/ISC2007 (32/33 and 33/33), there is a 24 chips board which makes a 144-ports switches with 144 links up ; and one 8 chips board.
So, I’d see something like 24 144-ports switches (with 144 links up) topped by 144 24-ports switches (grouped by 8 , so 18 boards total). So, 24 boards vertical, 18 boards horizontal, and you’ve got a nice switch.