Installing parallel I/O libraries¶

High-resolution simulations (e.g. modeling the entire Greenland ice sheet using the 900 m resolution and a grid of 1700 by 3000 points) can produce huge amounts of data and I/O can become a bottleneck.

PISM supports several parallel I/O approaches that take advantage of parallel NetCDF, PnetCDF, and NCAR ParallelIO. The administrators of your HPC system should be able to help you install these libraries, but it may be easier to install them in the “home” directory instead.

This section describes the steps needed to build

NetCDF with parallel I/O based on HDF5 (needed to use PISM’s option -o_format netcdf4_parallel),
PNetCDF (needed to use PISM’s option -o_format pnetfdf),
ParallelIO (needed to use options -o_format pio_netcdf4p, -o_format pio_netcdf4c, -o_format pio_pnetcdf, -o_format pio_netcdf).

Scripts below install libraries in ~/local/library_name, using ~/local/build/library_name to build them.

Section Building PISM with libraries in non-standard locations explains how build PISM with these libraries.

Section PISM’s I/O performance explains how to use them in PISM.

Installing HDF5-based parallel NetCDF¶

Installing HDF5¶

# Install HDF5 1.12.0 with parallel I/O in ~/local/hdf5,
# using ~/local/build/hdf5 as the build directory.

version=1.12.0
prefix=$HOME/local/hdf5
build_dir=~/local/build/hdf5
hdf5_site=https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.12
url=${hdf5_site}/hdf5-${version}/src/hdf5-${version}.tar.gz

mkdir -p ${build_dir}
pushd ${build_dir}

wget -nc ${url}
tar xzf hdf5-${version}.tar.gz

pushd hdf5-${version}

CC=mpicc CFLAGS=-w ./configure \
  --disable-static \
  --enable-parallel \
  --prefix=${prefix} 2>&1 | tee hdf5_configure.log

make all 2>&1 | tee hdf5_compile.log
make install 2>&1 | tee hdf5_install.log

popd
popd

To compile parallel HDF5 one should use the MPI compiler wrapper mpicc and run configure with the option --enable-parallel. The flag -w is not important: it hides numerous compiler warnings emitted when building HDF5.

Installing NetCDF¶

# Install parallel NetCDF using parallel HDF5 in ~/local/hdf5 and
# ~/local/build/netcdf as a build directory.

hdf5=~/local/hdf5

version=4.7.4
prefix=$HOME/local/netcdf
build_dir=~/local/build/netcdf
url=https://github.com/Unidata/netcdf-c/archive/refs/tags/v${version}.tar.gz

mkdir -p ${build_dir}
pushd ${build_dir}

wget -nc ${url}
tar zxf v${version}.tar.gz

pushd netcdf-c-${version}

export CFLAGS="-g -O0"
unset CFLAGS

CC=mpicc CPPFLAGS=-I${hdf5}/include LDFLAGS=-L${hdf5}/lib ./configure \
        --enable-netcdf4 \
        --disable-dap \
        --prefix=${prefix} 2>&1 | tee netcdf_configure.log

make all 2>&1 | tee netcdf_compile.log
make install 2>&1 | tee netcdf_install.log

popd
popd

Here we use the same compiler wrapper and set CPPFLAGS and LDFLAGS to select the parallel HDF5 library installed earlier. The option --enable-netcdf4 is required for parallel I/O; --disable-dap is not required (it disables a NetCDF feature not used by PISM).

Installing PnetCDF¶

# Install PnetCDF 1.12.1 in ~/local/pnetcdf,
# using ~/local/build/pnetcdf as a build directory.

version=1.12.1
prefix=$HOME/local/pnetcdf
build_dir=~/local/build/pnetcdf/
url=https://parallel-netcdf.github.io/Release/pnetcdf-${version}.tar.gz

mkdir -p ${build_dir}
pushd ${build_dir}

wget -nc ${url}
tar xzf pnetcdf-${version}.tar.gz

pushd pnetcdf-${version}

./configure \
      --prefix=${prefix} \
      --enable-shared \
      --disable-static \
      --disable-cxx \
      --disable-fortran

make all
make install

popd
popd

Here we disable PnetCDF’s C++ and Fortran APIs and build the shared library.

Installing NCAR ParallelIO¶

# Install NCAR ParallelIO in ~/local/parallelio
# using parallel NetCDF and PnetCDF installed in ~/local/netcdf
# and ~/local/pnetcdf. Uses ~/local/build/parallelio
# as a build directory.

netcdf_prefix=~/local/netcdf
pnetcdf_prefix=~/local/pnetcdf

url=https://github.com/NCAR/ParallelIO.git
build=~/local/build/parallelio
prefix=$HOME/local/parallelio

rm -rf ${build}
mkdir -p ${build}/build ${build}/sources

git clone ${url} ${build}/sources

pushd ${build}/sources
git checkout -b 2_5_7 pio2_5_7
popd

pushd ${build}/build

CC=mpicc cmake \
  -DCMAKE_C_FLAGS="-fPIC" \
  -DCMAKE_INSTALL_PREFIX=${prefix} \
  -DNetCDF_PATH=${netcdf_prefix} \
  -DPnetCDF_PATH=${pnetcdf_prefix} \
  -DPIO_ENABLE_FORTRAN=0 \
  -DPIO_ENABLE_TIMING=0 \
  ${build}/sources

make install

popd

Here we use CMake’s variable CMAKE_FIND_ROOT_PATH to tell CMake to use libraries in ~/local/netcdf and ~/local/pnetcdf, to install in ~/local/parallelio, and to disable ParallelIO features not used by PISM.