U.S. Supercomputer Users Call for Better Performance Tests

SAN JOSE, CA – AUGUST 4, 2000 – Representatives of leading U.S. supercomputer user organizations meeting here called for the creation of better performance tests for this most-powerful class of computers.


At a meeting of the new International Data Corporation (IDC) High-Performance Computing User Forum, 53 attendees from government, industry, university and vendor organizations heard from IDC analyst Earl Joseph II that oft-cited theoretical "peak" performance figures have little meaning. Joseph also said the popular Linpack performance test that is the basis for the semi-annual `World's Top500 Supercomputer Sites' list "correlates almost perfectly with peak performance, so IDC has decided to launch a new benchmark called the IDC2000.

Erich Strohmaier, who publishes the `Top500' list with University of Tennessee colleague Jack Dongarra and Hans Meuer of the University of Mannheim (Germany), said, "Linpack was the best practical choice in 1993, when the list was first published. Linpack is a pure CPU benchmark and was not designed to measure other system attributes that may contribute significantly to real-world performance, including internal memory speed, disks and external networks."

Strohmaier said the emergence of multiple supercomputer architectures has made it necessary to supplement Linpack with additional tests. He outlined requirements for creating an improved suite of benchmark tests. "A new benchmark suite must be simple enough to be usable and maintainable, complex enough to reflect the influence of all system attributes of interest, and scalable to utilize resources in a variety of system sizes."

During the meeting, representatives from Ford and Dupont, the Department of Energy and Department of Defense presented their benchmarking successes and future requirements. The User Forum asked its newly formed Performance Advisory Group (PAG) to pursue the development of the new benchmark. PAG membership includes many HPC users, IDC, Strohmaier and DOE/NERSC benchmarking experts Horst Simon and David Bailey, who were responsible for the well-known NAS Parallel Benchmarks test suite.

Debra Goldfarb, IDC Group VP, said that the User Forum's mission is to assess the strengths and weaknesses of all high-performance computing architectures, including high-bandwidth, single-system supercomputers; clusters and `superclusters'; SMPs; `grid computers'; and `Internet computers' such as SETI@home. "The IDC concept is `strategic fit,' a taxonomy of where you'd run various types of applications," she said. "We look at applications profiles–where people are using a system and the kinds of efficiencies they're getting."

The HPC User Forum (www.idc.com/hpc) is a new initiative developed by IDC and leading HPC users to promote the interests of high-end HPC (High Performance Computing) users in industry, government and academia. The HPC User Forum is an extension of IDC's long-standing commitment to the global HPC community.

IDC is the world's leading provider of information technology (IT) industry analysis, market data and insight, and strategic and tactical guidance to builders, providers and users of information technology. IDC provides global research with local content through more than 600 analysts in 43 countries worldwide. IDC is a division of IDG, the world's leading IT media, research and exposition company.