The ‘bash’ sources ‘/etc/profile’, which includes all the /etc/profile.d/*.sh
Adding a file there adds it “for everyone”.
Note how the /etc/profile defines pathmunge function for those files to use. Convenient, although only for the PATH.
I did recently see the installation of some CUDA version that it did actually use the alternatives to put symlinks to some commands on the path.
Note also that /usr/local/cuda is a symlink; each CUDA version is in distinct /usr/local/cuda-*
The different versions can be simultaneously installed, because they are in distinct paths. Only the /usr/local/cuda and alternatives symlinks would be “to one version”.
If you have one binary compiled with 11.2 and another with 12.1, are you sure that you want to – and can – run both with the 12.1 libs?
A SLURM job does inherit the submitter’s environment, but the job can adjust its environment too.
We have used environment modules, now the Lmod Lua-based version from EPEL.
Yes, the job script has to have one extra line, like:
module load cuda/11.2
to set the environment for the CUDA-application, but it is also explicitly recorded within the job script, which CUDA was/is used and the user has control.