Speed may kill, but it also thrills. At Washington Apple Pi Labs, we’re always seeking the best, the brightest and the fastest, usually while munching on pizza. Sometimes our seeking has seduced us into strange speculations, such as our “Wall of Macs” idea, where we pondered taking 60-odd Power Mac 6100s stacked in the Pi office and turning them into a RAIC. Though a RAIC (Redundant Array of Inexpensive Computers) sounded like a cool idea, a few quick calculations revealed that even 60 Power Mac 6100s would be less powerful than a single Power Mac G4 running Mac OS X.
But let your mind roam free for a moment. What could you do if you had, say, 1,100 Power Mac G5 computers, each of them with twin two-gigahertz PPC970 processors? That would give you 2,200 total processors to work with, all running Mac OS X. Think of the power! Think of the pizza bill…
Dr. Srinidhi Varadarajan, an assistant professor at Virginia Tech (Virginia Polytechnic Institute and State University), didn’t think of Macs at all. He’d never touched one prior to 2003. Varadarajan heads a team that was trying to put Virginia Tech on the map as a world-class computer science and engineering center, despite the fact that the state of Virginia, facing massive deficits, brutally slashed the budget for higher education. Varadarajan wanted to build a world-class supercomputer, on the cheap.
And he did. As this article goes to press, the final results aren’t in; they will be announced in mid-November 2003 at SC2003, the annual supercomputing conference. But at the moment, “Big Mac,” as the Virginia Tech’s supercomputer is informally called, should break into the top 10 ranks of all the supercomputers in the world, possibly placing as high as 3rd or 4th on the list. It was built using 1,100 dual-processor Power Mac G5s, at a cost of $5.2 million. It will certainly be the cheapest supercomputer on the list, and the first one built with pizzas.
The road to “Big Mac” was not a direct one, but it wasn’t a particularly long road, either. The Virginia Tech team started working on a supercomputer proposal in March 2003, and started raising money in April and May. At that time, they also started looking at the options. The team didn’t even consider Cray, the company that virtually invented supercomputing, because their machines were too costly (and slow). Instead, they looked to Dell Computers and Intel, which offered cluster-style systems based on Itantium II processors; Advanced Micro Devices, which offered cluster-style systems based on their Opteron processors; Sun Microsystems, which offered cluster-style systems based on their UltraSPARC processors; and even IBM. IBM was, apparently, willing to offer something based on their new PPC970 processors – the same processors found in the Power Mac G5 -- but not until 2004.
Even the cheapest proposals came in at between $9 and 12 million dollars, and all were “too slow,” according to Varadarajan. The team was pondering the options when, on June 23, 2003, Steve Jobs announced the Power Mac G5. This was a complete surprise to the Virginia Tech group (and most of the computing world). On June 26, after looking over the specs, Virginia Tech called Apple and said they were interested in placing a “fairly large order.” On June 27, Varadarajan flew to California, to meet with Apple. Much to Apple’s surprise, Varadarajan was not a long-time Mac fanatic; he’d never owned one. He just wanted to build a supercomputer. In a hurry. Within a day, Apple agreed, and Virginia Tech made a “fairly large order” through Apple’s online store.
Starting September 5, Power Mac G5s started arriving in Blacksburg, Virginia. The “fairly large order” amounted to 22 tons of Power Macs, each shipped in the standard black boxes used for every other Power Mac G5. Virginia Tech did get an educational discount – the same as every other educational institution – but aside from that, the units were standard, off-the-shelf consumer products, the same Power Mac G5s you can buy at MacUgrades or CompUSA or Micro Center. Apple apparently did give Virginia Tech some engineering advice, and bumped their place to the front of the shipping queue. But aside from that, the “Big Mac” is the first supercomputer made from consumer computers, and purchased through consumer channels.
Dr. Varadarajan, by the way, now has two Macs: a 17-inch PowerBook and a Power Mac G5.
This isn’t the way supercomputers are normally designed, constructed, and built. For comparison, consider the Earth Simulator.
A joint project of three Japanese government agencies, the National Space Development Agency, Atomic Energy Research Institute and Marine Science and Technology Center, Earth Simulator is a massive custom-built facility in Yokohama, Japan. It uses 640 processor nodes, each with eight processors, custom-designed by NEC. Each of the 5,120 processors has a peak performance of 8 Gflops (8 billion floating point operations per second) and as of June 2003 had received the top benchmark rating in the world of 35.86 Tflops (35.86 trillion floating point operations per second). Designed to, among other things, do research on the international Global Change Prediction Plan, the Earth Simulator is hugely expensive, with an estimated cost of $250-300 million.
Virginia Tech’s “Big Mac,” on the other hand, uses dual-processor Power Mac G5s. Each PPC970 processor has two double-precision floating-point processors, so there are four floating-point processors per computer. Each floating-point processor can perform two floating-point operations per cycle, so a dual G5 running at 2 GHz can provide a peak performance of 16 Gflops of double-precision performance (2 floating point processors times two CPUs times 2 flops per cycle times two billion cycles per second equals 16 billion floating point operations per second). Given perfect efficiency, this translates to 17.6 Tflops (17.6 trillion floating point operations per second), but cluster computing has inherent bottlenecks to perfect efficiency. As of this writing, Varadarajan had managed to coax “Big Mac” to 9.55 Tflops, lagging behind only the Earth Simulator and ASCI Q, a massive supercomputer composed of Alpha clusters, based at Los Alamos National Laboratory.
Building “Big Mac” was also unusual. After refurbishing a building and beefing up the power and cooling (1,100 Power Mac G5s generate a lot of heat, and use a lot of electricity), Virginia Tech turned to volunteer labor. Free sodas and pizza were used to fuel student volunteers who worked setting up the Power Mac G5s. Varadarajan is quoted as saying the volunteers went through 600 to 700 pizzas. Virginia Tech President, Dr. Charles W. Steger, offered free football tickets to technicians who put in long hours on the project. If they get desperate, Virginia Tech can probably recover a few more bucks selling mice and keyboards, since they have over a thousand spares.
For Virginia Tech, “Big Mac” may be just the thing they need to enhance their reputation. Right now, they have a nationally-ranked football team, but “real tech”schools – California Institute of Technology, Massachusetts Institute of Technology, Carnagie-Mellon University, Rensselaer Polytechnic Institute – have nationally-ranked computer science and engineering departments. So how do you get nationally ranked, in a hurry?
The key may be fast food: Big Macs, sodas and pizza. Washington Apple Pi Labs understands, and approves.
Virginia Polytechnic’s Terrascale home page:
http://www.computing.vt.edu/research_computing/terascale/
O’Reilley’s MacDevCenter article:
http://www.macdevcenter.com/pub/a/mac/2003/
10/29/osxcon_g5cluster.html
Earth Simulator
http://www.es.jamstec.go.jp/
Top 500 list:
http://www.top500.org/