By default the controller hosts the /home, /apps, /etc/munge, and /opt directories for all other instances in your cluster. When packages are installed via apt or yum on the login or controller, they are not necessarily available on other instances within your cluster.
Installation in NFS mounted directories
When possible, it is recommended that packages are installed under /apps or /opt. You can add modulefiles that users can load to update their PATH, LD_LIBRARY_PATH, and other environment variables when they need to use the installed packages.
For Python packages, we highly recommend users adopt the use of virtual environments.
Custom Compute Images
Another option is to create custom compute node images that are built on top of the fluid-slurm-gcp compute images. Since v2.4.0 of the fluid-slurm-gcp deployment, you are able to specify images for each machine in your compute partitions. This allows you to set up Slurm partitions that are meant for running specific applications.
Fluid Numerics has provided an Open-Source and Publicly available repository on Github to help you get started with creating custom compute node images for your organization. This repository provides a template for creating custom compute node images for your applications using Packer and Google Cloud Build.
Reach out to firstname.lastname@example.org for assistance in using this repository.