On-Demand Supercomputing for Emergencies

By Nancy Wilkins-Diehr and Ilya Mirman

K-means clustering algorithm, used for data mining, developed in MATLAB and accelerated with Star-P.

When a natural calamity strikes without warning, detailed information is needed immediately by emergency responders as well as the public. Seismic researchers, for example, may be able to quickly disseminate statistics such as the epicenter, size, and scope of an earthquake. But additional critical analysis and modeling can take hours, days, even weeks to accomplish.

  That’s why some of this country’s national supercomputing centers are making their high-performance computing (HPC) resources publicly available to support event-driven science, for those situations when it’s most needed. Urgent applications that will make use of on-demand supercomputing range from making movies of Southern California earthquakes to systems that will help give near real-time warnings of a tornado’s or a hurricane’s predicted path, or foretell the most likely direction of a toxic plume released by an industrial accident or a terrorist incident.

  As computer simulations have increasingly become a part of everyday life, we naturally look at how we might marshal very large-scale resources to tackle some of life’s most urgent problems. A recent urgent-computing workshop featured presentations on using HPC resources for modeling pandemics, tornadoes, hurricanes, and wildfires.

  Other applications included modeling for medical emergencies, simulations for the United Nations’ International Security Assistance Force, and NASA’s event-driven modeling of the space shuttle. The domains are quite varied, but the needs are the same — access to supercomputers right away.

SPICE-based circuit simulation, parallelized with Star-P from Interactive Supercomputing.

Priority for Emergencies
While powerful parallel computers have been publicly available at supercomputing centers for some time, what’s new is the nation’s researchers’  ability to “talk” to supercomputers interactively and in real time. Supercomputers are typically accessed via a queuing system whereby researchers can wait days for access to a machine. For urgent applications, such a wait renders a time-critical simulation useless.

  Because of increasing interest in the different use models mentioned above, the San DiegoSupercomputer Center has dedicated a Dell cluster with 64 Intel dual-socket, dual-core compute nodes for a total of 256 processors for urgent computing calculations to any U.S.-based researcher. The 2.33GHz, four-way nodes have 8GB of memory. When not in use for urgent computing, the system can be used for calculations that can be interrupted at any time.

  A number of key technical developments have emerged to make on-demand computing very effective in a production environment.

  The types of calculations that will be valuable in an emergency scenario can be quite complex. Many researchers conduct their work with the assistance of familiar desktop applications such as MATLAB from The MathWorks. As their computing needs increase, they tend to limit the types of problems that they study rather than make what is typically a very large investment in parallel programming. But new technology has eliminated this barrier, enabling ordinary scientists to tap parallel computers with the same ease and interactivity of their favorite desktop tools to quickly crank out computationally challenging answers in emergency situations.

MATLAB-based photonic circuit simulation, parallelized with Star-P.

Use of Star-P
The San DiegoSupercomputer Center,  in collaboration with Interactive Supercomputing, has installed Star-P, which provides parallel application development capabilities for familiar desktop tools like MATLAB. Star-P is a parallel application development platform that enables users to code algorithms and models on their desktops using familiar mathematical software packages such as MATLAB, Python, and R — and run them instantly and interactively on large-scale multicore clusters like On-Demand. It eliminates the need to reprogram applications in C, C++, Fortran, or MPI in order to run on HPC servers, which can take months to complete for large,  complex problems.

  Urgent computing demands are by nature unpredictable. Expensive supercomputing facilities must be available to designated researchers when needed, but can be put to work on other calculations when not in use for urgent computing.

  SPRUCE (special priority and urgent computing environment) —  developed at Argonne National Laboratory and the University of Chicago through the National Science Foundation’s TeraGrid project — provides a cohesive infrastructure to provide urgent computing cycles. SPRUCE provides token-based access to supercomputers, which allows authorized researchers to run their calculations immediately.

MATLAB-based computational photonicsmodel, parallelized with Star-P.

The response policies vary by site, ranging from next-to-run to preemption using checkpoint-restart capabilities and are advertised to the users. It also has the capability to provide high-priority bandwidth over national research networks so scientists can quickly move large datasets to supercomputers.

  In addition, the SPRUCE team has studied issues such as “warm standby,” where testing is done to ensure that a code that might be run only infrequently is ready to go full-tilt in an urgent situation.

MRI image-processing application created in MATLAB, and extended to a parallel environment with Star-P using a handful of data tagsand commands.

Healthcare Benefits As Well
On-demand computing models are also playing an increasingly critical role in solving some of the nation’s urgent healthcare problems. For example, ultrasound scans have helped save tens of thousands of lives by catching early-stage breast cancers. But the imaging technology is far from flawless, often leading to false alarms or worse, undetected tumors.

  Conventional beamforming algorithms have been used in ultrasound scanners for nearly a half-century to help diagnose breast cancer and other life-threatening conditions. The problem is that they typically result in degraded images that are blurry or cluttered. The culprit: off-axis signals or the sound wave reflections coming from undesired locations within the organ or tissue.

  Biomedical engineers at the University of Virginia (U.Va.) School ofEngineering and Applied Science (SEAS) set out to develop a new algorithmic approach to solve the problem. The biomedical engineering research team created an advanced beamforming algorithm — called the Time-domain Optimized Near-field Estimator (TONE) — which significantly improves contrast and resolution in ultrasound images. The TONE research project was funded by a grant from the U.S. Army Congressionally Directed Medical Research Program (CDMRP) in Breast Cancer.

Finite element analysis simulation of a human heart created in MATLAB, and extended to a parallel environment with Star-P using ahandful of data tags and commands.

The TONE algorithm reduces undesired off-axis signals,  resulting in much higher definition images, but at the price of a much greater computational load. The algorithm developed on desktop computers overwhelmed the computer’s processing ability. The team solved this problem by automatically parallelizing their algorithms with Star-P to run them remotely on an on-demand, memory-rich Linux cluster based in Waltham, Mass.

  Until it gained access to on-demand HPC resources, the researchers were unable to generate images with such a fine sampling pitch due to the huge amount of memory and computational resources required to execute the algorithm. Typical resolution for ultrasound imaging systems is in the 200-300 micron range. With the new on-demand computing model, it can tap into powerful supercomputing clusters to generate ultra high-resolution images of 67 microns in near real time, without having to become parallel programming experts.

  Real-time Responsiveness
Making HPC resources instantly available to hosts of new researchers, while allowing them to work in their familiar development environments, is enabling urgent science problems to be solved quickly when and where they are needed. The notoriously complex programming requirements of parallel systems are a thing of past, yielding huge gains in research productivity while providing real-time responsiveness to event-driven science.

Argonne National Laboratory
Argonne,IL
anl.gov

Dell
Round Rock, TX
dell.com

Interactive Supercomputing
Waltham,MA
interactivesupercomputing.com

MATLAB
The MathWorks
Natick, MA
mathworks.com

NASA
Washington,DC
nasa.gov

National Science Foundation
North Arlington,VA
nsf.gov

San Diego
SupercomputerCenter
San Diego,CA
sdsc.edu

U.S. Army CDMRP
in Breast Cancer
Fort Detrick, MD
cdmrp.army.mil/bcrp

Universityof Chicago
Chicago,IL
uchicago.edu

Universityof Virginia
Schoolof Engineering and Applied Science (SEAS)
Charlottesville,VA
seas.virginia.edu

United Nations’ International Security Assistance Force
Brussels,Belgium
nato.int/ISAF

Nancy Wilkins-Diehr is part of the San Diego SupercomputingCenter’s scientific computing division. Ilya Mirman is vice president of marketing at Interactive Supercomputing. Please send your comments to [email protected].