Blue Gene Baby

Illinois' new supercomputer

Mapping genes, treating Parkinson's disease, exploring the universe and combatting global warming are just a few of the problems that Illinois' new supercomputer may solve.

With the signing of the Comprehensive Nuclear-Test-Ban Treaty of 1996, the government needed to find another way to examine the weapons without so much as a spark. "That treaty forced the requirement to be able to do something to test the state [of the nuclear arsenal]," says Herb Schultz, marketing manager for supercomputing at IBM. "That's a very hard problem."

Like many of the hardest problems in the world, it was assumed that a supercomputer might help find a solution. Lawrence Livermore National Laboratory in California joined heads with IBM to spawn Blue Gene, a line of supercomputers they hoped would unravel the mysteries of protein-folding and give new direction to other scientific dead-ends. The $100 million, five-year research initiative sought to build a massively parallel computer, the world's most powerful. Today, Lawrence Livermore dedicates much research to the safety of the nuclear arsenal, "simulating how the nuclear stockpiles are faring over the years without having to do any testing," according to Schultz.

The term "supercomputer" has been around for decades, but its definition evolves as quickly as computer chips. Jack Dongarra, of the Innovative Computing Laboratory at the University of Tennessee, started an authoritative list of the most powerful computers in 1993 (www.top500.org). He describes a supercomputer as "a computer that's the most powerful system that we have today. Today's supercomputer will no longer be a supercomputer in one or two years. It changes with the same pace that our computer chips change, and our chips change by doubling their speed every 18 months. That's the same effort that is driving supercomputing. Just as an analogy, the most powerful system that we had back in 1995 is the equivalent of my laptop today." Dongarra started his list, which is updated twice a year, when a friend in Germany suggested it as a good way to track the supercomputing industry and perhaps project its future.

"These supercomputers are really quite powerful devices, and they have the ability to help solve problems that we have not really been able to tackle until now," Dongarra says. "For many years, science has been constructed out of theory and experiment, and today, computing is playing sort of a third pillar. These supercomputers provide us with almost a crystal ball in how we can see the future. They really touch almost every area--aerospace, automobile design, biology, defense, electronics, energy, environment, finance, the gaming industry, medicine and so on."

The world of supercomputing became supercharged in June with news of IBM's Blue Gene/P, an evolution of the original Blue Gene/L. The Blue Gene/L at Lawrence Livermore, which is a cluster of 65,536 computers and has a theoretical peak performance of 360 teraflops, has maintained the number one spot on Dongarra's list since its debut in 2004. ("Flops" stands for "floating point operations per second." A teraflop is a trillion operations per second.)

There are nearly 30 computers currently on the Top 500 list that use Blue Gene/L architecture, but IBM says the Blue Gene/P will nearly triple the performance of its predecessor and will be the first supercomputer to break the petaflop barrier, which is one quadrillion operations per second. The company says that the Blue Gene/P is actually capable of reaching three petaflops, but that design would hinge on who has the budget to build such a computer. According the IBM, Blue Gene/P "is 100,000 times more powerful than a home PC and can process more operations in one second than the combined power of a stack of laptop computers nearly 1.5 miles high."

Supercomputers are composed of racks, and the first racks of Blue Gene/P, which are each roughly six feet tall, will be shipped this fall to Argonne National Laboratory, which was awarded the computing facility out of a competition held by the Department of Energy (DOE). The first installment will be a 111-teraflop system, which has approximately 32,000 processors, and it will become operational for the national community in the spring.

"There are plans already to have even a larger system here, so we're deploying the machines in phases and building up to what will be one of the largest computer centers in the world," says Ray Bair, director of the Argonne National Laboratory Leadership Computing Facility. Accessing via remote, scientists and engineers across the nation will use the supercomputer around the clock, at their designated time, and the time is free. "If you wanted to compare it to your PC at home, this machine can do a problem in a day that, if your PC at home was able to do, would literally take you a lifetime." There will be 20 large projects and a number of smaller projects, and the DOE currently has a call out for scientific proposals.

"Having the fastest computer means the people at Argonne will have this tremendous capability in solving whatever problems they're looking to solve," Dongarra says. "It will help the Argonne scientists and the other people who are there to use it to investigate phenomenon we couldn't explore before. You and I have computers, but they're nowhere near the capability of these supercomputers. Think of it like if you had a telescope or binoculars--it's nowhere near the match for the Hubble telescope that's up there in space. These devices are very expensive, and they shouldn't be used by just anyone. They have to be used by the people who have the most challenging problems. So at Argonne, they're planning on having the scientists who have the most challenging problems engage in the use of that equipment."

The most challenging of problems fall into a variety of areas. The name "Blue Gene" came from the notion that the supercomputer would model protein-folding, thought to hold answers to treating many diseases. The supercomputer has been successful, though, because "it isn't just a one-trick pony," according to Schultz. Today's supercomputers are used to study solve global warming, nanotechnology, automotive and airplane testing and energy efficiency.

"I think that people have this notion that supercomputing means just running things fast," says Schultz. Since Blue Gene's inception, IBM has sold 158 Blue Gene racks to 34 customer sites around the world. The customers include a group in Switzerland that's modeling the human brain and how it functions and an organization in the Netherlands that's analyzing signals from space. "I think if they were to see all the kinds of research that scientists and industry are doing with supercomputing in general, people would be rather astonished. As an example, in addition to the things that might come to mind--like molecular modeling or studying the weather, climate change, that kind of thing--there's a lot of work that goes on designing automobiles, airplanes, understanding how fluids flow through pipes, through a reactor, or how engines can be more efficient, how to get more oil out of the ground, logistical problems, trying to understand how you manage traffic. Understanding interactions between things makes for a prime candidate for supercomputers."

Like all else in the world of technology, as supercomputers continue to get faster and more powerful, they're also getting smaller. Argonne's first model of Blue Gene/P will be the size of eight refrigerators, along with a few other cabinets. While today's supercomputers can be delivered in a truck, "15, 20 years ago, it was not unusual for 20 semi-trucks to show up at the customer [site] because we had that much stuff," says Schultz. IBM also deploys someone to assemble the computer. "It's not as big of an undertaking as it was in the old days." Another breakthrough feature of the Blue Gene/P is that it's seven to 10 times more energy-efficient than its predecessor because it uses smaller, low-power chips. Supercomputers use an abundance of power in both heating and cooling. Argonne's 111-teraflop system uses 300 kilowatts, much less than other supercomputers.

"One of the things we're most enthusiastic about Blue Gene is the way it's designed, we're able to efficiently apply more computing power, more processors, to a problem than we can on many other machines," says Bair of Argonne. "It scales very well. The communication between the processors on the IBM Blue Gene/P is about 1,000 times the speed of the internet connection to your home. It uses about a seventh of the power of other computers that are currently available. That's quite green. Certainly that's becoming an increasing perk for places that run large supercomputers because power isn't getting any cheaper."

Year after year, supercomputers cost roughly the same, approximately $1 million, but the price per operation continues to fall. "How much do 100,000 laptops cost?" says Schultz. "Basically, if you look at the price per operation, we're down to less than 10 cents per million operations per second. Just maybe 10 years ago, that number would have been $25 to $50. The machine's always a million-dollar thing, but every year you can do a hundred times more stuff for that power. Maybe it seems like a lot of money to shell out for that kind of product, but the amount of things it can do..."

The world of supercomputers, once exclusively open to the most technical of sciences, has opened its doors to industry. The idea of supercomputing "has caught fire in the industrial sector," according to Schultz. "The reason people are interested in supercomputing is because it helps them innovate." A supercomputer is today's latest tool for a successful business and economy--for those industries that can support the costs. More than half the supercomputers on Dongarra's Top 500 list are in the commercial sector.

"Supercomputers can help drive business competitiveness, and they do that by reducing design costs through virtual prototyping, so you don't actually have to build something," Dongarra says. "You can build it on a computer through a model, and that helps by reducing the time it takes to bring certain things to market. It can help from the standpoint of the economy. Just as an example, the weather. The U.S. economy is around $13 trillion, and about 40 percent of it is adversely affected by weather and the climate. If we had better ways of predicting what's going to happen, we might be in a position to avoid that effect. When a hurricane strikes a coast, what we do is evacuate people from the coastline. Evacuation costs about $1 million per mile to move people out of that area. Today, when a hurricane's about to strike, we over-warn by about 3 times. That warning costs about $200 million in terms of loss of economy. Understanding where things are going to happen, we're in a much better position to adjust and carry on with that."

Along with those by IBM, there are supercomputers made by Cray, Fujitsu and Sun Microsystems, among many others. Although IBM's Blue Gene has remained at the top in recent years, there's ever-heightening competition as to who can come up with the most powerful supercomputer. Sun Microsystems thought they were edging out until the announcement of the Blue Gene/P.

"[Supercomputing] is tremendously competitive," Dongarra says. "It's constantly increasingly competitive on a number of levels. There's competition between government labs. So for instance, I work at Oak Ridge National Lab, and we compete with Argonne to have the fastest machine. Today, we have the faster machine, but they're going to have a faster machine pretty soon. Competition extends far beyond that. It extends to industry, which uses computing to help them get a competitive advantage over their competition, and we're seeing more and more of that. People realize that these computers can in fact provide them with a commercial edge over the people they compete with. In some sense, the only way to face that is to have a bigger, faster, more powerful computer. Industry can really no longer compete on traditional cost and quality terms, and the ability to create new value and innovate will really determine the competitive advantage."

Igor Tsigelny researches the molecular basis of Parkinson's disease. His project is one of nine on Argonne's Blue Gene/L. Most of the nine will likely want to upgrade and continue their research on the Blue Gene/P, according to Bair. Tsigelny, who accesses the Blue Gene from University of California, San Diego, is in his third year of the project, which deals with a reasonably new hypothesis about Parkinson's disease.

"We try to elucidate the molecular mechanism of Parkinson's disease," Tsigelny says. "You can't study things that are changing all the time. Here the supercomputers come to help. This simulation is mostly theoretical. We take snapshots from the long, dynamic simulation. This work can only happen in the era of supercomputers. We can't even imagine what will happen to supercomputers. It changes mankind. Absolutely, I can imagine using a three-petaflop computer. The next step will be supercomputers in houses."

"In 12 years, the machine that the Argonne people are about to have will be the machines that you and I use in our daily life," Dongarra says. "I have a hard time understanding what I'm going to be doing with that computational power, but by that same token, 12 years ago I didn't have a concept of what we'd be doing with our computers that we're doing today--by that I mean things like Google and being able to use the internet in a way that provides a seamless mechanism for acquiring information and buying and selling things.

"I can project where the fastest computers will be in 10 years," Dongarra continues. "Because of that power, we can solve more challenging problems--the problems will have more detail and will be larger. For instance, if I'm looking at the climate, I can get better predications of where the climate will be going in terms of small changes in the number of years I look at. I can understand what the weather will be like in a smaller area. What I have a harder time understanding is what I'll do with all the power that will be placed in my laptop. The power in my laptop of tomorrow is the supercomputer of today."