Sure the popular open source platform Hadoop can crunch Big Data. But we’re talking Really Big Data. At Stanford University in Northern California, researchers just tapped into the world’s largest supercomputer and ran an application that crunched information across more than one million processor cores.
Joseph Nichols and his team are the first to run live code on the Lawrence Livermore National Laboratories’ Sequoia IBM Bluegene/Q supercomputer, a machine that spans over 1.5 million cores in total. The team used just over one million of those cores to simulate the amount of noise produced by an experimental jet engine, apparently setting a supercomputer record in the process.
Nichols and crew had never run the code on a machine with over 200,000 cores before, and they spent the past few weeks working closely with the Lawrence Livermore researchers to optimize the software for Sequoia. “I had no idea if it was going to work or not,” Nichols says.
The experiment shows that despite the rise of open source distributing computing tools such as Hadoop — which uses dirt-cheap, commodity hardware — old school supercomputers still provide much larger data crunching platforms. The largest Hadoop cluster likely spans around 8,800 cores.
Supercomputers work by breaking down very large problems into smaller problems and distributing them across many machines and many processor cores. Typically, adding more cores makes the calculations faster, but it also adds complexity. At a certain point, calculations can actually become slower due to bottlenecks introduced by the communications between processors.
But Sequoia’s processors are organized and networked in a new way — using a “5D Torus” interconnect. Each processor is directly connected to ten other processors, and can connect, with lower latency, to processors further away. But some of those processors also have an 11th connection, which taps into a central input/output channel for the entire system. These special processors collect signals from the processors and write the results to disk. This allowed most of the necessary communications to occur between the processors without a need to hit the disk.
The team hopes the results will help create quieter jet engines. Under the direction of Professors Parviz Moin and Sanjiva Lele, the Stanford team has been working with the NASA Glenn Research Center in Ohio and the NAVAIR branch of the U.S. Navy to predict how loud an experimental engine will be without having to actually construct a prototype. That’s harder than it sounds. Nichols explains that the acoustic energy of an engine is less than one percent of its total energy. Calculations have to be extremely precise in order to accurately model the noise an engine will generate.
But thanks to the Sequoia, Nichols thinks their research could go beyond just modeling into prescriptive design — in other words, figuring out what the optimum design would be.
There are many other possibilities. Nichols says that the code they’re working with — originally developed by former Stanford senior research associate Frank Ham — enables other researchers at Stanford to simulate the full flow of an entire aircraft wing and to model hypersonic scramjets, propulsion systems for flight at several times the speed of sound.
“It gave pause to a lot of people,” Nichols says. “We were like: ‘Whoa we can actually do that.’”
Researchers Set Record With Million-Core Calculation
This article
Researchers Set Record With Million-Core Calculation
can be opened in url
http://spraynewstar.blogspot.com/2013/01/researchers-set-record-with-million.html
Researchers Set Record With Million-Core Calculation
