Developing Spack Packages¶

Info

This guide uses the conterized official software version 2.9. A similar approach can be used with the CVMFS release.

Instructions are based on the Spack developement workflow.

Install and Configure Spack¶

For more details, see the Spack installation guide. The Spack version used by our release is listed under /opt/spack/var/mucoll-spack/.latest-commit.

# Set up working directory
export SPACK_INSTALL_DIR=${HOME}/mcdevel
cd ${SPACK_INSTALL_DIR}
# Download Spack repository
git clone -c feature.manyFiles=true https://github.com/spack/spack.git
# IF needed, switch to the exact version used in release v2.9
git -C spack checkout f5946c4621035dd466953c8d2664ff5f82f38138 
# IF running Docker image on macOS via Rosetta
export SPACK_ROOT=${PWD}/spack
# Load Spack into the environment
source spack/share/spack/setup-env.sh
spack compiler find --scope site

For any subsequent sessions you only need to load Spack into the environment:

cd ${SPACK_INSTALL_DIR}
source spack/share/spack/setup-env.sh

Configure Relases 2.9 as Upstream¶

For more details, see chaining Spack installations. This will reuse existing packages when appropriate from the release, so you don't waste time compiling everything.

cat << EOF >> ${SPACK_ROOT}/etc/spack/upstreams.yaml
upstreams:
  release_2.9:
    install_tree: /opt/spack/opt/spack
EOF

If you are using the CVMFS distribution, use /cvmfs/muoncollider.cern.ch/release/2.9 for install_tree instead.

Setup Key4HEP and MuColl Repositories¶

Add Key4HEP and MuonCollider repositories to Spack:

cd ${SPACK_INSTALL_DIR}

git clone https://github.com/key4hep/key4hep-spack.git
git -C key4hep-spack checkout dc0b2bc4aaad8f3ef5f3c7f62ff5b0277a554eee

git clone https://github.com/MuonColliderSoft/mucoll-spack.git
git -C mucoll-spack checkout v2.9

spack repo add --scope site ./key4hep-spack
spack repo add --scope site ./mucoll-spack

Modify an Installed Package¶

It is possible to modify code of one or more installed packages using the development workflow, which lets Spack build a package from your local source folder and use this custom version instead of the one already installed in the release.

NOTE: This should primarily be used for isolated code changes that do not affect the overall build process and list of dependencies. For more substantial changes it's preferred to set up a new release, potentially using the old one as an upstream.

We'll use LCIO package as an example, which many other packages in the stack depend on.

Load the Spack environment with the default release and find the exact version of the package already installed:

# Activate the default environment
spack env activate /opt/spack/var/spack/environments/sim/

# Get the LCIO package version
spack find lcio # -> lcio@2.22

To leave the original release untouched create a new Spack environment in ./dev-lcio folder, using the .yaml configuration of the original sim environment as a starting point:

spack env create -d ./dev-lcio $SPACK_ENV/spack.yaml
spack env activate ./dev-lcio

Get your custom source code and mark it for development in Spack:

# Create the custom source-code directory
mkdir -p dev/LCIO

# Download the source code e.g. from GitHub repository
git clone https://github.com/MuonColliderSoft/LCIO.git --branch v02-22-MC dev/LCIO

# Mark the package for development reusing the same spec as already installed
spack develop -p $PWD/dev/LCIO lcio@2.22

If instead you just want to get the original code and modify it later, it can be done with a single command:

spack develop lcio@2.22

This will download the source code into $SPACK_ENV/lcio/ and mark the package for development.

NOTE: It is important to use the same package version in the spack develop command as the one installed in the release, ensuring that the dependency tree won't change. Otherwise version conflicts may arise during concretization.

Reconcretize the stack:

spack concretize -f --reuse --deprecated

Compile the code with Spack:

# Build the modified package
spack install lcio

# Build the dependencies
spack install

Only this last step needs to be repeated for recompiling the code after it has been modified.

Finally, load the newly installed version of the release into your environment:

spack load mucoll-stack

NOTE: To use this custom installation in any new shell you should not use the precompiled commands like setup_mucoll or source /opt/setup_mucoll.sh available in the container.
Instead you should let Spack handle it using the following commands:
# Activate Spack
source ${SPACK_INSTALL_DIR}/spack/share/spack/setup-env.sh 
# OR if using CVMFS distribution
source /cvmfs/muoncollider.cern.ch/release/2.9/setup.sh

# Activate the environment
spack env activate ./dev-lcio

# Load the environment 
spack load mucoll-stack