Spack

Spack & Cloud-HPC Package Management

Packages on Fluid-Slurm-GCP clusters are managed by Spack. Our solutions come with a preconfigured sitewide packages.yaml and compilers.yaml in addition to GCC/10.2, OpenMPI/4.0.3, and Singularity/3.7. All of these packages are managed via Spack.

Spack is an open-source HPC package manager that simplifies building applications from source code. It has built in dependency resolution and makes it easy to manage multiple build stacks with differing build specs, compilers, and MPI flavors.

Using Installed Packages

Packages on Fluid-Slurm-GCP clusters are installed by Spack and are exposed to users through lmod. Our solutions come with a preconfigured sitewide packages.yaml, compilers.yaml, and modules.yaml in addition to GCC/10.2, OpenMPI/4.0.2, and Singularity/3.7. All of these packages are managed via Spack. Additionally, fluid-slurm-gcp comes with an installation of the oneAPI™ HPC Toolkit from Intel, which can be brought into your path using environment modules.

Listing Packages

To show what packages are currently available on the cluster, you can use the module list command.

module list

Loading and Unloading Packages

To load packages that have been previously installed by spack, you can use the module load command.

For example, to load OpenMPI/4.0.2 compiled with the GCC/10.2.0 compiler, you can use the command

module load gcc/10.2.0 openmpi/4.0.2

If you want to remove a package from your path, you can use module unload command.

For example, to unload OpenMPI/4.0.2, you can use the command

module unload openmpi@4.0.2

Install additional packages with Spack

Spack supports the installation of a large number of packages. All packages are compiled from source, giving you the ability to control compilation flags and the types of features to include with each packages build.

Use the checklist below to help guide package installation with spack.

  • First check that the desired package can be built with spack using the spack list command. Alternatively, you can visit the spack packages list page to see if the desired package is supported.

  • Use the spack info command to familiarize yourself with the packages dependencies, available versions, and available build specs.

  • Preview what packages will be installed by using spack spec. Keep in mind that you can specify the package version and the compiler to build with. Additionally, you can concretize the package variants to customize the installation. For example, some packages like hdf5 can be built with or without mpi support. The example below will provide a spec for hdf5@1.10.7 using the provided gcc@10.2 and openmpi@4.0.2 packages. This eliminates the need for spack to install additional dependencies.

spack spec hdf5@1.10.7 % gcc@10.2 +cxx+fortran+mpi+threadsafe^openmpi@4.0.2

Input spec

--------------------------------

hdf5

Concretized

--------------------------------

hdf5@1.10.7%gcc@10.2+cxx~debug+fortran~hl~java+mpi+pic+shared~szip+threadsafe api=none arch=linux-debian10-haswell

  • Install the package using the spack install command.
    Since packages are installed from source, some packages may take a while to build. As an example, to install hdf5@1.10.7 using the provided gcc@10.2 and openmpi@4.0.2 packages,

spack install hdf5@1.10.7 % gcc@10.2 +cxx+fortran+mpi+threadsafe^openmpi@4.0.2

  • This system has been configured to expose Spack installed packages through lmod. Because of this, you can load them into your environment using module load commands.

module load gcc/10.2 hdf5/1.10.7

Adding Commercial Compilers

Compilers from commercial vendors like Intel and PGI are not open-source and cannot be installed using Spack. However, if you wish to use commercially licensed compilers, we recommend that you install the compilers underneath the /apps directory on the cluster and add an environment module file under /apps/modulefiles.

When installing a module file, we recommend that you make a subdirectory under /apps/modulefiles named after the compiler vendor and name the module file after the version. For example, if I wanted to install PGI compilers version 20.4, the modulefile would be found at /apps/modulefiles/pgi/20.4.

Once your commercial compiler and module file are installed, use the module load command to bring the compiler into your path. Then, use the spack compiler find command with the scope set to "site" to add the compiler to spacks compilers.yaml file. For example

module load pgi/20.4

spack compiler find --scope site

Read more about working with Vendor-specific compiler configuration.

Build your own code

Best Practices

If you'd like to build and install your own code on the cluster, you can easily install packages in your home directory. Alternatively, if you'd like to install your application for use by multiple users, you can follow these recommended best practices

  • Use spack to install your desired compilers and dependencies with spack install

  • When building your application, use the spack load command to bring the compilers and dependencies into your path

  • Install your application under the /apps directory (root privileges are required for this).

  • Add a module file under the /apps/modulefiles directory so that users can easily bring your application binaries into their path.

Create a Spack Package

We recommend that you consider writing a spack package for your application. This will greatly reduce the barrier to entry for other application users on Fluid-Slurm-GCP and will simplify the installation process on other systems using Spack. If your application's dependencies are already managed by spack, creating a spack package for your application is easy to do.

Create a Package for your Cluster

  • Create a tagged release of your application and push the tag to your Github, Gitlab, or Bitbucket repository.

  • In your browser, navigate to your release page and find the .zip or .tar.gz pacakage that contains the tagged release of your application. For example, the package feq-parse has one of its releases hosted on Github at https://github.com/FluidNumerics/feq-parse/releases/tag/v1.0.2.

  • Use spack create to create a package template for your application. This command will auto-detect your application's build system (ideally, autoconf or cmake) and open a text editor for you to fill out some details for your application. You may need to run this command as root on the cluster, since spack is installed under /apps. For example

spack create https://github.com/FluidNumerics/feq-parse/archive/v1.0.2.tar.gz

  • Modify the spack package file for your application. See the Spack Packaging Guide for more details.

  • Test your application install by using spack install

At this point, users on your cluster will be able to install and load your application via Spack.

Contribute your package to the official Spack repository

If you would like to list your package on the official Spack repository, there are a few steps you can take to make the process go as smooth as possible.

  • Run spack flake8 to verify that your package file passes Spack's formatting requirements. Resolve any issues.

  • Run git pull to update spack to the most recent version of the develop branch

  • Navigate to the /apps/spack directory on your cluster and run git pull to update spack to the most recent version of the develop branch.

  • Run git add and git commit to commit your local changes.

  • Create a fork of the official spack repository on Github.

  • Add another remote to your local spack repository on the cluster that points to your fork

git remote add upstream https://github.com/GITHUB-ACCOUNT/spack

  • Push your changes to this new remote.

git push upstream develop


Recommended Concretizations

OpenMPI

openmpi@4.0.5%gcc@10.2.0~atomics+cuda+cxx+cxx_exceptions+gpfs~java+legacylaunchers+lustre+memchecker+pmi+singularity~sqlite3+static~thread_multiple+vt+wrapper-rpath fabrics=auto schedulers=slurm