Thom Dunning posed Monday in front of one of the dozens of cabinets that house the new Blue Waters supercomputer, a photographer's lens an uncomfortable couple of feet from his face.

"We've been far more uncomfortable," the director of the National Center for Supercomputing Applications joked, looking at Bill Kramer, who led the project to build what is among the world's fastest computers at the University of Illinois.They joke now, with the $300 million machine running, but in late in 2011, some feared that the Blue Waters project was circling the drain.

A big, $72 million building sat mostly empty waiting to house Blue Waters as IBM, the original builder, pulled out after essentially concluding it couldn't build and maintain the machine and make money doing it. A new builder had to be found, and fast, to stay within thetimeline preferred by the primary financier, the National Science Foundation.

"We knew many of those people -- we knew that when they made this announcement that jobs were at risk, their role in national science computing was at risk," said Barry Bolding, vice president of corporate marketing at Cray Inc. The company, known for building supercomputers, took over after IBM backed out.This week, with about 30 separate groups now using Blue Waters, Dunning, Kramer and all of those people Bolding talked about can finally breathe easy.

Gov. Pat Quinn is expected on Thursday to join Dunning and others from the NCSA, along with people from Cray and the National Science Foundation to proclaim a job well doneBlue Waters operates much like it was designed to do in 2007.

The plan was to build what would be, at least for a short time, the fastest supercomputer in the world. But more importantly, Blue Waters would also be able to sustain a speed of at least one petaflop -- a thousand trillion operations a second, a long-sought speed that makes massive computational projects possible.

The NSF awarded the contract to the university and IBM, and agreed to provide most of the money, just over $200 million. The university contributed $100 million.

When IBM pulled out, about $160 million had already been spent. And the NSF's deadline to have the computer in the building by the fall of 2012 was still very much in place.

"It was very, very dicey," Kramer said.Pulling the plug was real possibility, according to Allen Blatecky, director of NSF's Division of Advanced Cyberinfrastructure.Cray and the university were essentially told, "Convince us" it should continue, Blatecky said. "And they did."
The biggest worry, Bolding said, was whether the Seattle-based company could source the parts required for a computer that fits inside about 300 person-sized cabinets.

"I could name probably 10 things we had to juggle because they were late," Bolding said. "We ran into those over and over and over again, and each time we had to fix it and move on. ... We took huge risks that individual pieces might not be delivered on time."

It worked, and at a profit, Bolding says. The company doesn't break out profit information for any one project, he said, but its annual report for last year showed Cray's revenue -- $421 million -- and profits were up over 2011. The Blue Waters contract paid the company $188 million.

"It was a major contributor," Bolding said.There is almost no end to the list of numbers you can use to explain what Blue Waters can do.
At a maximum speed of 11.6 petaflops, it is probably the third-fastest supercomputer in the world behind a Cray-built computer known as Titan at Oak Ridge National Laboratory in Tennessee and an IBM machine, Sequoia, at the Lawrence Livermore National Laboratory in California, according to a website that tracks supercomputers.

There are also those 30-plus cabinets -- they sit on a tiled floor that's raised six feet to accommodate the cooling system that such as massive computer requires. Kramer lifts one of those heavy, concrete-reinforced tiles to reveal a ladder, climbing down and -- shouting over the continuous, loud hum from above -- shows off hundreds of yards of pipe carrying a constant flow cold water -- thousands of gallons of it a minute.

There's evidence down here of how far along IBM was when it pulled out -- Dunning points out a sort of long basket installed to hold cable's for IBM's computer. Cray doesn't use them, stringing its cables over the computer cabinets instead.

Blue Waters also has a 300-petabyte storage system. According to Dunning, "The amount of storage we have and the bandwith of storage we have in the Blue Waters system, as far as we know, is the largest, most intense in the world."

Thomas Jordan is director of the Southern California Earthquake Center at the University of Southern California and one of the early users of Blue Waters, a sort of beta tester.

He's working with others on a project to simulate the range of possible outcomes of earthquakes along hundreds of fault lines in Southern California.

"Our ambition is to eventually do this for all of California," he said. Blue Waters has, so far, provided a mostly smooth ride aside from a few glitches, he said.

Other projects are studying everything from how stars explode to how viruses affect cells and how contagious diseases spread.
They all amount to a great reward -- and a huge relief -- for Kramer and Dunning, particularly Dunning, who will retire later this year.

"It's very nice to have a big computer room that actually has a computer in it," said the understated Dunning. For a time, "it looked very lonely."