Hi to all,
i have access to one HPC cluster that use RL8 as OS for the nodes.
Since I want to create the environments module for python in this HPC, some one have best praticte to suggest the way to do this in simplest way?
Is is enough to compile the requirements version of python or more setup is needed?
Thanks
Antonio
Environment modules are TCL or Lua files that the helper tool module (from package environment-modules or EPEL package Lmod) uses to modify your current shell session’s environment.
See Environment Modules (software) - Wikipedia
The modifications are usually prepending (or appending) directories to PATH (and similar lists),
setting environment variables, and creating shell aliases. Variables that application needs and non-standard directories that shell would not search from by default.
Adding a module for python makes no sense, because python is usually installed from RPM packages, into default location.
RHEL 7 had Software Collections that had RPM-packaged content in non-standard location and helper tool scl was used to “enable” such alternative version. One could convert the modifications into “environment module” and then used the module to enable a software collection for use.
Which environment module implementation is in your cluster?
What do you get with module av now?
in the cluster there are the first that you mentioned, and in the details:/modules-5.3.1.tar.gz
On the nodes of the cluster there are not python3.10 but only python3.6 that is installed with RockyLinux8.
Since we need to use python3.10 i want to add is as module like the consultant add miniconda as module.
I have to compile and install python on the location path that is defined in modulefiles ?
And also need to compile python statically as suggested: BuildStatically - Python Wiki
The simplest way to create your module is to install miniconda and use conda or mamba to create a python environment suited to your needs. (And maybe also pip if you have a requirements.txt file.) You can create an environment module to load any conda environment.
Here’s an example lua module written for Lmod:
local appRoot = "/opt/miniconda"
local condaShellProfile = appRoot .. "/etc/profile.d/conda.sh"
local mambaShellProfile = appRoot .. "/etc/profile.d/mamba.sh"
setenv("CONDA_SHELL_PROFILE", condaShellProfile)
setenv("MAMBA_SHELL_PROFILE", mambaShellProfile)
prepend_path("PATH", appRoot .. "/bin")
prepend_path("CPATH", appRoot .. "/include")
prepend_path("LD_LIBRARY_PATH", appRoot .. "/lib")
prepend_path("MANPATH", appRoot .. "/man")
prepend_path("MANPATH", appRoot .. "/share/man")
source_sh("bash", condaShellProfile)
source_sh("bash", mambaShellProfile)
This is what I use for loading the basic miniconda environment. For a separate environment created with the --prefix option (e.g. conda create --prefix /some/path ...) all that should be needed to adapt it for the new environment is to change the appRoot variable to match the path defined with the --prefix option.
I need to create the module that use python for all the users.
On the hpc cluster there is already the conda module.
So If I understand I can load module conda, create the python environments and after i create the modulefiles for that new environments on what i installed the python app.
I had el7 cluster and NFS server. Each node in the cluster did NFS-mount (with autofs) shares from server. One of the mountpoints was /site/app7. The non-RPM applications were installed under that directory. There was also directory /site/app7/modulefiles.
Every node had a config file:
$ cat /etc/profile.d/my-modules.sh
source /etc/profile.d/modules.sh
module use /site/app7/modulefiles
All users had thus their module command look from /site/app7/modulefiles.
For each application, I had directory within and the TCL-format files under, one per version. For example:
$cat /site/app7/modulefiles/vmd/1.9.3
#%Module 1.0
#
module-whatis "VMD 1.9.3"
append-path PATH /site/app7/vmd/1.9.3/bin
The user could thus do module add vmd or module add vmd/1.9.3 and the /site/app7/vmd/1.9.3/bin would append to PATH.
The python has same name as the system’s executable, so you would prepend, rather than append to PATH for your version to be found first.
Thank’s a lot.
I follow yuor suggestion and the module work fine
We use environment modules for Python because no-one with half a clue wants the default python3.6. We have 5 or 6 versions of Python, as well as around 10 versions of R all as environment modules. All our apps (~200 scientific apps) are run via environment modules and there are multiple versions of all apps - it’s a great system when you get used to it.
Indeed.
Spack appears one method to install (multiple versions of scientific) applications and it generates the modules too, but some were less excited about it. https://spack.io/
I hadn’t heard of spack but it looks interesting.
Our process is to build apps into singularity containers with wrapper scripts then load the module to put the wrappers in the $PATH. Often the containers contain conda/mamba envs so it’s all very easy to build.
This is great for reproducible science as I can compare the output of one version of a package with the updated version to see if it’s consistent. It also means if I want to re-run some analysis from 3 years ago I can use the exact versions of all the packages.
How do you built the python module?
Just install from sources?
Just set teh prefix when you build Python,
wget https://www.python.org/ftp/python/3.8.10/Python-3.8.10.tgz
tar xvzf Python-3.8.10.tgz
cd Python-3.8.10
./configure --prefix=/opt/bioinf/python/3.8.10 --enable-optimizations --with-ensurepip=install
make -j 4
make altinstall
Then the modulefile looks like this:
#%Module 1.0
#
# Python-core module for use with 'environment-modules' package:
# based on https://github.com/shawfdong/modulefiles/blob/master/python/rh-python36
#
set root /opt/bioinf/python/3.8.10
prepend-path LD_LIBRARY_PATH $root/lib64
prepend-path MANPATH $root/share/man
prepend-path PATH $root/bin
prepend-path PKG_CONFIG_PATH $root/lib64/pkgconfig
prepend-path XDG_DATA_DIRS $root/share
Then users just load the module for the version they require:
# default system Python
muthur$ which python3
/usr/bin/python3
muthur$ python3 --version
Python 3.6.8
muthur$
# load a module
muthur$ module load python/3.8.10
muthur$ python3.8 --version
Python 3.8.10
muthur$
# clear the loaded module and use a different Python
muthur$ module purge
muthur$ module load python/3.9.5
muthur$ python3.9 --version
Python 3.9.5
muthur$
Modules with R is a slightly different case as R compiled with a 3.x kernel won’t run under a 4.x kernel, and as we have a mixed environment while we’re still migrating to Rocky 8, I came up with this solution in the modulefile:
#%Module 1.0
#
# R-core module for use with 'environment-modules' package:
#
set KERNEL [lindex $tcl_platform(osVersion) 0]
switch -glob $KERNEL {
3* {
prepend-path PATH /opt/bioinf/R/R-4.3.0/bin
prepend-path MANPATH /opt/bioinf/R/R-4.3.0/share/man
prepend-path LD_LIBRARY_PATH /opt/bioinf/R/R-4.3.0/lib64/R/lib
}
4* {
prepend-path PATH /opt/admin/rocky8_apps/R/R-4.3.0/bin
prepend-path MANPATH /opt/admin/rocky8_apps/R/R-4.3.0/share/man
prepend-path LD_LIBRARY_PATH /opt/admin/rocky8_apps/R/R-4.3.0/lib64/R/lib
}
default {
puts "not sure what this is with kernel $KERNEL"
}
}
Hope this helps 
I finally find the mistake that i did.
I try to compile and install python as static library and it is a nightmare instead all i need it to use the prepend-path.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.