Last modified: April 18, 2020
This article is written in: 🇺🇸
Environment Modules is a powerful and flexible tool that enables dynamic modification of a user's environment via modulefiles. Each modulefile contains the information necessary to configure the shell for a specific application or version, allowing users to seamlessly switch between different software environments. This utility is essential in complex computing environments, such as high-performance computing (HPC) clusters, where managing multiple software packages and versions can be challenging.
Environment Modules simplifies this process by handling environment variables like PATH
, LD_LIBRARY_PATH
, and others, ensuring that users can load and unload applications as needed without conflicts or compatibility issues. It promotes efficient use of system resources and enhances productivity by providing a consistent and user-friendly interface for environment management.
Main idea:
The installation process for Environment Modules varies depending on your Linux distribution. Below are the instructions for different systems:
Update the package list and install Environment Modules:
sudo apt update
sudo apt install environment-modules
Use yum
or dnf
to install Environment Modules:
# For CentOS/RHEL
sudo yum install environment-modules
# For Fedora
sudo dnf install environment-modules
Install using zypper
:
sudo zypper install environment-modules
If the Environment Modules package is not available through your operating system’s package manager, or if you require a specific version, you can install it directly from the source. This approach provides greater flexibility, particularly if you want to install a newer or custom version.
I. Download the Latest Version:
First, download the source code from the official Environment Modules repository on SourceForge. The following example uses wget
to fetch version 4.7.1, but be sure to replace the URL with the latest version if needed:
wget https://sourceforge.net/projects/modules/files/Modules/modules-4.7.1/modules-4.7.1.tar.gz
tar -xzf modules-4.7.1.tar.gz
cd modules-4.7.1
wget
is a command-line tool used to download files from the internet.tar -xzf
command extracts the downloaded .tar.gz
archive, unzipping it into a directory named modules-4.7.1
.cd modules-4.7.1
changes to this directory, where the source code is located, preparing for the installation steps.II. Configure and Install:
After extracting the source, you’ll need to configure the build settings and then compile and install the software:
./configure --prefix=/usr/local/modules
make
sudo make install
./configure --prefix=/usr/local/modules
initiates the configuration process and specifies the installation prefix. By setting the --prefix
option, you control where the software will be installed; in this case, /usr/local/modules
. You can customize this path as needed.make
compiles the source code, creating the binaries that will be installed. This step may take a few minutes depending on your system.sudo make install
installs the compiled software to the specified prefix directory. This command requires sudo
because it installs files in system directories that need root privileges.III. Configure Shell Initialization:
To make the module
command available in your shell, you need to add a line to your shell initialization file. This line loads the Environment Modules system each time you open a new shell session:
echo "source /usr/local/modules/init/bash" >> ~/.bashrc
source
line to your ~/.bashrc
file, ensuring the modules system is initialized whenever you start a new shell.bash
in the path if you use a different shell, such as zsh
or tcsh
. For example, if you use zsh
, the line would be: source /usr/local/modules/init/zsh
.To apply these changes immediately, either log out and back in or reload your shell configuration:
source ~/.bashrc
This command applies the new configuration without requiring a logout, so you can start using Environment Modules right away. Now you should be able to run the module
command and manage software modules as needed.
Modulefiles are scripts written in the Tcl language that define how to modify the environment for a specific application. They are typically stored in directories specified by the MODULEPATH
environment variable, such as /usr/share/modules/modulefiles
or /etc/modulefiles
.
Here's an example of what a directory tree structure might look like for a collection of modulefiles organized for various applications and versions. Each application has its own subdirectory, and within that, modulefiles for specific versions are stored. Additionally, a .version
file or symbolic link might be present to specify the default version.
/etc/modulefiles
├── python
│ ├── 2.7
│ ├── 3.6
│ ├── 3.8
│ └── .version
├── gcc
│ ├── 8.4.0
│ ├── 9.3.0
│ └── 10.2.0
├── my_app
│ ├── 1.0
│ ├── 2.1
│ ├── 2.2
│ └── default -> 2.1
├── cuda
│ ├── 10.1
│ ├── 11.0
│ └── .version
├── openmpi
│ ├── 3.1.4
│ ├── 4.0.5
│ └── 4.1.0
└── cmake
├── 3.17
├── 3.18
└── .version
This structure provides organization and makes it easy to manage multiple versions of different software using Environment Modules. Each application has its own directory under /etc/modulefiles
, and within those directories, version-specific modulefiles are stored.
To organize your modulefiles, it’s a good practice to create a directory structure that reflects the software you’re managing. This makes it easier to maintain and locate modulefiles as your collection grows.
I. Create a Directory for Your Application:
Use the following command to create a new directory for your application within the standard modulefile path:
sudo mkdir -p /usr/share/modules/modulefiles/my_app
-p
flag tells mkdir
to create the parent directories if they do not already exist, which helps avoid errors if any intermediate directories are missing.my_app
with the name of your application, making the modulefile directory structure more organized.II. Set Appropriate Permissions:
To ensure that users can access the modulefiles, set the correct permissions on the new directory:
sudo chmod 755 /usr/share/modules/modulefiles/my_app
755
permission setting allows the owner to read, write, and execute files in the directory, while others can read and execute files. This is usually suitable for shared module directories, as it allows others to access the files but not modify them.A modulefile is a simple script that configures the environment for a particular software version. By setting paths and environment variables, a modulefile helps users load the necessary settings for your application without modifying their shell configuration files.
I. Create the Modulefile:
To create a modulefile, use a text editor to open a new file with the version number you want to manage:
sudo nano /usr/share/modules/modulefiles/my_app/1.0
1.0
inside the my_app
directory, representing version 1.0 of your application.1.0
with any version number, following this naming convention for each version of the software you want to manage with modules.II. Edit the Modulefile:
Here’s an example modulefile for my_app
version 1.0. The file contains commands that configure the environment whenever the module is loaded.
#%Module1.0#####################################################################
##
## my_app modulefile
##
proc ModulesHelp { } {
puts stderr "This module loads my_app version 1.0"
}
module-whatis "Loads my_app version 1.0"
# Set the installation prefix
set root /opt/my_app/1.0
# Modify environment variables
prepend-path PATH $root/bin
prepend-path LD_LIBRARY_PATH $root/lib
prepend-path MANPATH $root/share/man
# Set custom environment variables
setenv MY_APP_HOME $root
# Handle conflicts with other versions
conflict my_app
module help my_app/1.0
command is run. This is useful for providing users with a brief description of what the module does.module whatis my_app/1.0
. It helps users quickly identify what this modulefile does.set
command defines a variable (root
) that points to the installation directory of your application. This makes the modulefile easier to update, as you can adjust the path in one place if needed.$root/bin
to PATH
, so the shell can locate the application's executables; $root/lib
to LD_LIBRARY_PATH
, enabling the linker to find the application’s libraries; and $root/share/man
to MANPATH
, allowing the man
command to access the application’s manual pages.MY_APP_HOME
, which points to the root directory of the application. Users can reference this variable directly, which can be helpful for scripts or commands related to the application.my_app
at the same time. If a user tries to load a second version without first unloading the current one, the module system will display an error message. This ensures that only one version of the application is loaded, avoiding potential conflicts.III. Save and Exit:
After editing the modulefile, save and close the editor:
Ctrl + O
to save your changes.Ctrl + X
to exit nano
.The MODULEPATH
environment variable tells the module
command where to look for available modulefiles. By adjusting MODULEPATH
, you can customize where the module system searches for additional modules, allowing you to access modulefiles that may not be in the default path.
I. Add to MODULEPATH Temporarily:
To add a directory to MODULEPATH
for your current session only, use the module use
command:
module use /usr/share/modules/modulefiles
This command tells the module system to include /usr/share/modules/modulefiles
when searching for available modulefiles. However, it only applies to the current terminal session. Once you log out or close the terminal, the addition to MODULEPATH
is reset.
II. Add to MODULEPATH Permanently:
To make the change permanent, add the module use
line to your shell initialization file, such as ~/.bashrc
(for Bash users). This ensures the path is always added when you open a new terminal:
echo "module use /usr/share/modules/modulefiles" >> ~/.bashrc
source ~/.bashrc
module
CommandThe module
command provides several subcommands to interact with environment modules. These subcommands allow you to load, unload, list, display, switch, and manage module configurations.
I. Load a Module:
To load a module and make its associated software available for use:
module load my_app
This command loads the specified module and updates environment variables such as PATH
, LD_LIBRARY_PATH
, or other variables defined by the modulefile. If multiple versions of a module exist and no default is set, you may need to specify the desired version:
module load my_app/1.0
II. Load Multiple Modules:
To load multiple modules at once, separate each module name with a space:
module load gcc/9.3.0 openmpi/4.0.5 my_app/1.0
This example loads the GCC compiler, the Open MPI library, and my_app
. Modules are loaded in the order specified, which can be important if later modules depend on those loaded earlier.
I. Unload a Module:
To remove a module from your environment and reset any associated environment variables:
module unload my_app
This command removes the specified module's changes from the environment, effectively "unloading" it.
II. Unload Multiple Modules:
You can also unload multiple modules at once by listing them with spaces:
module unload gcc openmpi my_app
This is useful for clearing several modules without removing all loaded modules.
I. List Loaded Modules:
To view a list of all currently loaded modules:
module list
This command displays the names and versions of each module currently loaded into your environment.
II. List Available Modules:
To see a list of all modules available for loading, use:
module avail
If you want to filter the results, you can specify part of a module name or a pattern:
module avail my_app
This command will show only those modules that match my_app
.
I. Show Module Details:
To view detailed information about a specific module, including the environment variables it modifies:
module show my_app/1.0
This command displays the modulefile’s contents and how it affects your environment, helping you understand what it changes when loaded.
II. Get Help for a Module:
Some modules include a help message to provide additional usage information. To access this help:
module help my_app/1.0
If you want to replace one version of a module with another, you can use module switch
:
module switch my_app/1.0 my_app/2.0
This command unloads the first module and immediately loads the second, effectively switching between versions.
The module purge
command removes all loaded modules from your environment:
module purge
This is useful when you need a clean environment but should be used with caution, as it will unload every loaded module, including those you may still need.
I. Save Current Module Set:
To save your current set of loaded modules, use:
module save my_default_modules
This creates a named set of modules that you can reload later, making it easy to return to a specific configuration.
II. Restore Module Set:
To reload a saved module set, use:
module restore my_default_modules
This command reloads all the modules saved under the specified name, allowing you to quickly set up your environment.
III. List Saved Module Sets:
To view a list of all saved module sets:
module savelist
This displays the names of all saved sets, helping you remember what configurations you’ve saved for later use.
Environment Modules is especially useful when you need to switch between different versions of the same software. For example, you may need to switch between Python 2.7 and Python 3.8 for different projects. The steps below illustrate how you can use Environment Modules to handle this task.
First, you need a directory where your Python module files will reside. This is typically located in /etc/modulefiles/
, but you can adjust the path based on your system’s configuration. Create a directory for Python modulefiles:
sudo mkdir /etc/modulefiles/python
This command creates a directory /etc/modulefiles/python
, where you'll store the configuration files for different Python versions.
Next, create a modulefile for Python 2.7. A modulefile is essentially a script that modifies environment variables like PATH
and LD_LIBRARY_PATH
to make the specific Python version available in your environment.
sudo touch /etc/modulefiles/python/2.7
sudo nano /etc/modulefiles/python/2.7
The touch
command creates an empty file named 2.7
, and nano
opens it for editing.
In the opened nano
editor, add the necessary lines that define how the environment should be configured when the Python 2.7 module is loaded. Be sure to modify the prepend-path
lines based on the actual location of Python 2.7 on your system.
#%Module 1.0
##
## Module for the Python 2.7 programming language.
##
module-whatis "Module for Python 2.7"
prepend-path PATH /usr/local/python2.7/bin
prepend-path LD_LIBRARY_PATH /usr/local/python2.7/lib
conflict python
module-whatis
provides a short description of the module.prepend-path PATH
ensures that the directory containing the Python 2.7 binary is placed at the front of your PATH
environment variable.prepend-path LD_LIBRARY_PATH
adds the directory containing Python 2.7 libraries to LD_LIBRARY_PATH
.conflict python
ensures that only one Python version is loaded at a time, preventing conflicts.Save the file and exit nano
by pressing CTRL + X
, then Y
, and Enter
.
Similarly, you need to create a modulefile for Python 3.8:
sudo touch /etc/modulefiles/python/3.8
sudo nano /etc/modulefiles/python/3.8
Then, add the following lines to configure Python 3.8. Again, adjust the prepend-path
lines based on your system’s Python 3.8 location.
#%Module 1.0
##
## Module for the Python 3.8 programming language.
##
module-whatis "Module for Python 3.8"
prepend-path PATH /usr/local/python3.8/bin
prepend-path LD_LIBRARY_PATH /usr/local/python3.8/lib
conflict python
This modulefile works similarly to the one for Python 2.7, but it's configured for Python 3.8. The conflict python
directive ensures that Python 2.7 and Python 3.8 cannot be loaded at the same time.
Once the modulefiles are created and configured, you can easily switch between the two Python versions using the module
command.
To load Python 2.7, run:
module load python/2.7
This will modify your shell environment, updating PATH
and LD_LIBRARY_PATH
so that Python 2.7 becomes the default.
If you later need to switch to Python 3.8, first unload Python 2.7, then load Python 3.8:
module unload python/2.7
module load python/3.8
Unloading Python 2.7 ensures that no conflicting versions are loaded simultaneously.
After switching between Python versions, you can verify which version is active by using the following command:
python --version
This will display the version of Python currently set as default in your environment.
I. To ensure reproducibility and avoid unexpected behavior, always specify the version when loading modules.
module load my_app/1.0
II. While module purge
can help reset your environment, it may also unload critical modules. Check which modules are loaded before purging.
III. Include module load commands in scripts or job submission files to ensure the correct environment is set up for your tasks.
IV. Be aware of module conflicts, especially when working with compilers and MPI libraries. The conflict
command in modulefiles helps prevent incompatible modules from being loaded together.
Environment Modules support Tcl scripting, which allows for advanced features like conditional logic and dynamic behavior. This makes it possible to create more flexible and powerful modulefiles that adapt to the environment in which they are loaded.
You can use Tcl's scripting features to implement conditional logic within a modulefile. For example, you might want to load specific dependencies only if another module is already loaded.
# Check if a specific module is loaded
if { [ is-loaded gcc/9.3.0 ] } {
# Load dependencies
module load dependency_module
} else {
puts stderr "Error: gcc/9.3.0 must be loaded before my_app."
exit 1
}
is-loaded gcc/9.3.0
: This checks if the gcc/9.3.0
module is already loaded in the environment.module load dependency_module
: Loads a required dependency if the condition is true.puts stderr
: Prints an error message to the standard error if the condition is false.exit 1
: Exits with an error code, signaling a failure due to the unmet condition.This ensures that the required dependencies are in place before loading your application module.
Administrators can set a default module version so that users automatically load the default version of a module if they don’t specify a specific one. This can be done using either a .version
file or a symbolic link.
I. Using a .version
File:
You can create a .version
file in the application's modulefile directory to define the default version of the module.
#%Module1.0
set ModulesVersion "1.0"
The .version
file ensures consistency by defaulting to a stable version of the software.
II. Using a Symbolic Link:
Another method to set the default module version is by using symbolic links. This method involves creating a symbolic link named default
pointing to the desired version.
ln -s 1.0 /usr/share/modules/modulefiles/my_app/default
ln -s 1.0
: Creates a symbolic link named default
that points to version 1.0
./usr/share/modules/modulefiles/my_app/default
: Specifies the path where the symbolic link will reside, which is the modulefile directory for my_app
.This approach allows for easy switching of default versions by updating the symbolic link to point to a new version.
Even with well-written modulefiles, users may encounter issues when loading or using modules. Here are common issues and their solutions.
I. Module Not Found:
If the module you are trying to load is not found, it could be due to the modulefile directory not being included in the MODULEPATH
. To verify the directories in MODULEPATH
, run:
echo $MODULEPATH
If the desired directory is not listed, you can add it using the following command:
module use /path/to/modulefiles
Ensure that the correct directory is permanently added if you need it frequently by adding the command to your shell's initialization script (e.g., .bashrc
or .bash_profile
).
II. Environment Variables Not Set:
If certain environment variables aren’t being set as expected when you load a module, check the modulefile for errors in syntax or incorrect paths. To inspect what a module does when loaded, use the following command:
module show <module_name>
III. Conflicts with Other Modules:
Sometimes, multiple loaded modules can conflict with each other, causing unexpected behavior. If this happens, the easiest solution is to unload all currently loaded modules and start fresh:
module purge
After purging, you can selectively load the required modules for your task.
IV. Module Command Not Found:
If you receive an error like module: command not found
, it indicates that the Environment Modules package is either not installed or not set up properly in your shell environment. To resolve this:
.bashrc
, .bash_profile
) sources the appropriate environment setup script. For example, for bash, you would typically add:
source /usr/share/modules/init/bash
This ensures that the module
command is available in your shell sessions.
apt install environment-modules
for Debian-based systems or yum install environment-modules
for CentOS). Explain the purpose of Environment Modules and how they simplify the management of software versions on shared systems./etc/modulefiles
for a new application. For example, set up a directory for Python by running sudo mkdir /etc/modulefiles/python
. Discuss why /etc/modulefiles
is commonly used for storing modulefiles and how it enables centralized module management.sudo touch /etc/modulefiles/python/3.8
to create a modulefile. Briefly describe the structure of modulefile directories and the purpose of organizing applications by version.prepend-path
directive to add the bin
and lib
directories to the PATH
and LD_LIBRARY_PATH
environment variables, respectively. Include the conflict
directive to prevent the loading of incompatible versions simultaneously. Discuss how prepend-path
and conflict
ensure smooth version management.module load
command, and confirm the module is active by checking the output of module list
and verifying the application’s version (e.g., python --version
). Explain how the module load
command modifies the user environment to include the application paths defined in the modulefile.module unload
command. Verify that the module has been removed by running module list
and checking that the application’s paths are no longer in your environment. Discuss why it’s essential to unload modules, especially when switching between different versions.module avail
command to list all available modules on your system, including the one you created. Explain how the module avail
command helps users discover available software and manage multiple applications more effectively.module purge
command, and confirm that no modules remain active by checking the output of module list
. Discuss the utility of module purge
for resetting the environment and ensuring a clean state before loading other modules.
Here are two additional challenges to complete the list, focusing on advanced modulefile customization and persistent module environments:module-whatis
to provide a summary and module help
to give users more details when they load the module. Discuss how adding metadata improves module discoverability and provides helpful information for users..bashrc
or .bash_profile
) to automatically load a specific module or set of modules on login. Test this configuration by logging out and back in, then verifying that the module(s) load automatically. Explain the benefits of persistent module environments for frequent users of specific software.