When Serial Apps Aren’t Enough

By Loren Dean

The International Linear Collider project uses The MathWorks’distributed computing tools to run more than 100 simulations in parallel on a highthroughput computer cluster such as this one.

New and expanded capabilities that enable engineers and scientists to interactively develop distributed or parallel applications and adapt existing applications to run in distributed environments are calling for more power. They include commercial off-the shelf (COTS) computer clusters that provide affordable high-performance distributed environments; operating systems that add features to simplify cluster management, with many schedulers now supporting multiple cluster configurations; and the availability of software libraries and tools for distributed and parallel computing.

Where Technical Computing is Going

Technical computing applications can generally be classified as serial applications, distributed applications, or parallel applications. Serial applications, in which one instance of an application runs on one computer, describe the vast majority of software that people use on their personal computers.

Distributed applications leverage multiple computational engines that can run on one or many computers and contain independent tasks that do not interact with each other. These include applications like Monte Carlo simulations, which typically execute the same algorithm over and over with different input parameters. And parallel applications also leverage multiple computational engines, which can run on one or many computers, but contain interdependent tasks that exchange data when the application executes.

Instead of using technical computing solely for algorithm development, there is a trend to quickly convert ideas into production-ready algorithms that can be included in commercial products. Researchers who previously focused on problem analysis are moving into data analysis in order to design applications, applying tools to gain theoretical insight from their data. Application development and delivery, which involves the creation of enabling tools and methods that are deployed and used throughout the organization, is also becoming standard. All of these trends contribute to an increase in the size and complexity of technical computing applications. As a result, researchers find that applications often either exceed the memory capacity of their computers or exhibit extremely long run times. Such limitations are driving serial applications to become distributed or parallel applications. 

Low-Cost Hardware

Due to improvements over the past decade, standard commercial off-the-shelf (COTS) computing platforms nearly approach the performance of supercomputers. The same computing power in the million-dollar high-performance computing (HPC) platforms of 10 years ago is now available for a few thousand dollars in COTS computer clusters.

While the lower end of the HPC market is growing significantly, demand for larger enterprise clusters has declined. This rapid adoption of clusters for interactive personal use is significant because it enables interactive prototyping. In the past, prototyping had to be done in a batch manner that involved submitting and retrieving work.

The somewhat recent shift toward multiple cores on one processor, or even multiple processors, is driving a rapid change in the requirement that a single machine support distributed and parallel applications. Because today’s complex problems demand that an application scale beyond a single processor, engineers and scientists must address whether applications should use multiple threads on one machine; multiple processes, which may each contain multiple threads, on one machine; or a cluster of many machines.

Developing distributed and parallel applications using MATLAB and Distributed Computing Toolbox.

At the same time, many clusters are being linked across organizations in large multi-enterprise solution grids, such as Teragrid in the U.S. or Enabling Grids for E-science (EGEE) in Europe. These grid solutions represent another rapidly growing trend, especially in the academic community.

Schedulers and Operating Systems

A variety of additional resource management solutions are coming to the aid of engineers and scientists who want to take advantage of distributed computing. Operating system companies are working to simplify cluster management and reduce the reliance on IT resources. For example, Microsoft Cluster Computing Service (CCS) targets departmental and workgroup-level users who typically rely on dedicated IT support groups. Dozens of commercial and many freeware schedulers now support multiple cluster configurations. Schedulers offer advanced scheduling capabilities, batch workflow support, utilization and performance increases, and improvements in scalability, reliability, and security. Platform Computing, which develops and commercializes the LSF scheduler, now serves 1,700 Fortune 2000 companies.

Distributed and Parallel Tools

Taking advantage of clusters has historically required an advanced programming degree, which many engineers and scientists lack. Engineers and scientists developing HPC applications have had to either develop applications using Message Passing Interface (MPI) in low-level languages, such as C and Fortran, or program in higher-level languages and then recode their applications to use MPI in C or Fortran. These approaches were costly, time consuming, and error prone. Even traditional HPC users are looking for better programming tools.

The MathWorks has responded by providing tools that make it possible to interactively develop distributed and parallel applications with MATLAB and to distribute Simulink models for execution on a cluster, or on a multicore or multiprocessor computer. Distributed Computing Toolbox enables users to prototype distributed and parallel applications on their desktop computers. MATLAB Distributed Computing Engine lets users scale their application to a cluster without changing applications. Both tools support the interactive use of computational resources and the traditional batch use of the resources. They also take advantage of industry-standard interfaces, such as MPICH2 and ScaLAPACK. Additionally, MATLAB now supports multithreading of some of its core algorithms so that serial applications can also benefit from hardware changes.

With Distributed Computing Toolbox, distributed applications can be segmented into different independent tasks that can be distributed to cluster nodes. In the simplest case, where the problems can be divided into tasks consisting of the same function with the same number of input and output variables, a single function call parallelizes the problem. In more complex cases, only several lines of code are required.

For programming parallel applications, Distributed Computing Toolbox provides support for parallel for loops and global array semantics via distributed arrays. Distributed arrays store segments of an array on participating labs and appear as regular arrays on all labs. Distributed arrays enable users to develop parallel applications without having to manage the low-level details of message passing.

Interactive Programming and Reuse of Existing Applications

This new generation of programming tools has made it possible for the first time to interactively program both new and existing applications for distributed and parallel computers. Often, with little or no change to existing code, users can interactively prototype their applications on a cluster of machines. This capability has historically been reserved for serial applications, but it is now available for distributed and parallel applications. For example, in a parallel application, programmers may simply use the transpose function on an array, D, that is distributed across the processors in the cluster. The transposed matrix, E, is also distributed across the processors:

>> E = D’

This is the same code that users will write to transpose a matrix for a serial application and, in both cases, it can be interactively typed and executed.

A Real-World Example

The International Linear Collider (ILC) project, which currently runs at the University of London, provides an example of how distributed computing can benefit large-scale computations. The ILC consists of two linear accelerators, each 20 kilometers long, that accelerate beams of electrons and positrons toward each other to produce collision energies of up to 1,000 giga-electron volts.

At less than 5 nanometers thick, the ILC particle beams can become misaligned by such tiny disturbances as small seismic events, the tidal pull of the moon, and ground motion caused by trains and motor vehicles. To ensure that the particle beams collide head-on, researchers are developing a real-time beam alignment control system. This development effort relies upon accurate simulations requiring a comprehensive model of the entire ILC. Each simulation tracks millions of individual particles through the accelerator and incorporates the effects of ground motion.

Running a single simulation on a high-end PC used to take up to three days of processing time. The ILC now uses MathWorks distributed computing tools to run more than 100 simulations in parallel on a computer cluster running a Maui scheduler with a portable batch queue system. This approach can run a hundred simulations in the time previously required to run one, and it will reduce the total simulation time for the project by hundreds of days.

An Upward Trend

Several trends have converged to enable engineers and scientists to develop applications that can access many computing resources simultaneously. As a result, they can more quickly solve larger and more computationally intensive problems. Using MathWorks distributed computing tools, engineers and scientists can now much more easily develop distributed and parallel applications that can scale from one machine to many.

Loren Dean, senior engineering manager for The MathWorks’ MATLAB development team, is responsible for its distributed computing products as well as the Test & Measurement application area and the license management and installation group. Dean has B.S. and M.S. degrees in aeronautical engineering from Purdue University and an M.B.A. from Northeastern University. Send comments about this article to DE-Editorsmailto:[email protected].

Distributed Computing Toolbox
The MathWorks
Natick, MA
mathworks.com