High Performance Parallel I/O: 1st Edition (Hardback) book cover

High Performance Parallel I/O

1st Edition

Edited by Prabhat, Quincey Koziol

Chapman and Hall/CRC

434 pages | 14 Color Illus. | 131 B/W Illus.

Purchasing Options:$ = USD
Hardback: 9781466582347
pub: 2014-10-23
SAVE ~$22.00
eBook (VitalSource) : 9780429103117
pub: 2014-10-23
from $28.98

FREE Standard Shipping!


Gain Critical Insight into the Parallel I/O Ecosystem

Parallel I/O is an integral component of modern high performance computing (HPC), especially in storing and processing very large datasets to facilitate scientific discovery. Revealing the state of the art in this field, High Performance Parallel I/O draws on insights from leading practitioners, researchers, software architects, developers, and scientists who shed light on the parallel I/O ecosystem.

The first part of the book explains how large-scale HPC facilities scope, configure, and operate systems, with an emphasis on choices of I/O hardware, middleware, and applications. The book then traverses up the I/O software stack. The second part covers the file system layer and the third part discusses middleware (such as MPIIO and PLFS) and user-facing libraries (such as Parallel-NetCDF, HDF5, ADIOS, and GLEAN).

Delving into real-world scientific applications that use the parallel I/O infrastructure, the fourth part presents case studies from particle-in-cell, stochastic, finite volume, and direct numerical simulations. The fifth part gives an overview of various profiling and benchmarking tools used by practitioners. The final part of the book addresses the implications of current trends in HPC on parallel I/O in the exascale world.


"While I/O has become a major performance bottleneck in HPC, no single reference allows practitioners to understand the complex interactions between applications, libraries, and file systems. This book fills this major gap; written by the best experts in the field, it is likely to become a necessary reference to anybody who wants to understand I/O on HPC platforms."

—Marc Snir, University of Illinois at Urbana-Champaign

"It is…remarkable to have a broad overview of past, present, and future technologies, with a focus on real problems and clear solutions…"

Computing Reviews

Table of Contents

Parallel I/O in Practice

Parallel I/O at HPC Facilities Galen Shipman

National Energy Research Scientific Computing Center Jason Hick

National Center for Supercomputing Applications William Kramer, Michelle Butler, Gregory Bauer, Kalyana Chadalavada, and Celso Mendes

Argonne Leadership Computing Facility William E. Allcock and Kevin Harms

Livermore Computing Center Richard Hedges and Blaise Barney

Los Alamos National Laboratory Gary Grider

Texas Advanced Computing Center Karl W. Schulz

File Systems

Lustre Eric Barton and Andreas Dilger

GPFS Dean Hildebrand and Frank Schmuck

OrangeFS Walt Ligon and Boyd Wilson

OneFS Nick Kirsch

I/O Libraries

I/O Libraries: Past, Present and Future Mike Folk

MPI-IO Wei-keng Liao and Rajeev Thakur

PLFS: Software-Defined Storage for HPC John Bent

Parallel-NetCDF Rob Latham

HDF5 Quincey Koziol, Russ Rew, Mark Howison, Prabhat, and Marc Poinot

ADIOS Norbert Podhorszki, Scott Klasky, Qing Liu, Yuan Tian, Manish Parashar, Karsten Schwan, Matthew Wolf, and Sriram Lakshminarasimhan

GLEAN Venkatram Vishwanath, Huy Bui, Mark Hereld, and Michael E. Papka

I/O Case Studies

Parallel I/O for a Trillion-Particle Plasma Physics Simulation Surendra Byna, Prabhat, Homa Karimabadi, and William Daughton

Stochastic Simulation Data Management Dimitris Servis

Silo: A General-Purpose API and Scientific Database Mark Miller

Scaling Up Parallel I/O in S3D to 100-K Cores with ADIOS Scott Klasky, Gary Liu, Hasan Abbasi, Norbert Podhorszki, Jackie Chen, and Hemanth Kolla

In-Transit Processing: Data Analysis Using Burst Buffers Christopher Mitchell, David Bonnie, and Jonathan Woodring

I/O Profiling Tools

Overview of I/O Benchmarking Katie Antypas and Yushu Yao

TAU Sameer Shende and Allen D. Malony

Integrated Performance Monitoring David Skinner

Darshan Philip Carns

Iota Mark Howison, Prabhat, and Surendra Byna

Future Trends

Parallel Computing Trends for the Coming Decade John Shalf

Storage Models: Past, Present, and Future Dries Kimpe and Robert Ross

Resilience Gary Grider and Nathan DeBardeleben

Multi/Many Core Ramon Nou, Toni Cortes, Stelios Mavridis, Yannis Sfakianakis, and Angelos Bilas

Storage Networks and Interconnects Parks Fields and Benjamin McClelland

Power Consumption Matthew L. Curry, H. Lee Ward, and David Martinez, Jill Gemmill, Jay Harris, Gary Grider, and Anna Maria Bailey


About the Editors

Prabhat leads the NERSC Data and Analytics Services Group at Lawrence Berkeley National Laboratory. His main research interests include Big Data analytics, scientific data management, parallel I/O, HPC, and scientific visualization. He is also interested in atmospheric science and climate change.

Quincey Koziol is the director of core software development and HPC at The HDF Group, where he leads the HDF5 software project. His research interests include HPC, scientific data storage, and software engineering and management.

About the Series

Chapman & Hall/CRC Computational Science

Learn more…

Subject Categories

BISAC Subject Codes/Headings:
COMPUTERS / Operating Systems / General
COMPUTERS / Computer Engineering