We provide prebuilt
conda environments containing GPU-supported builds of
tensorflow (both with
horovod support for multi-node calculations),
jax, and many other commonly-used Python modules.
Users can activate this environment by first loading the
conda module, and then activating the base environment.
Explicitly (either from an interactive job, or inside a job script):
$ module load conda $ conda activate base (base) $ which python3 /soft/datascience/conda/2022-07-19/mconda3/bin/python3
module load conda; conda activate. This can be performed on a compute node, as well as a login node.
As of writing, the latest
conda module on Polaris is built on Miniconda3 version 4.13.0 and contains Python 3.8.13. Future modules may contain entirely different major versions of Python, PyTorch, TensorFlow, etc.; however, the existing modules will be maintained as-is as long as feasible.
While the shared Anaconda environment encapsulated in the module contains many of the most commonly used Python libraries for our users, you may still encounter a scenario in which you need to extend the functionality of the environment (i.e. install additional packages)
There are two different approaches that are currently recommended.
Virtual environments via
Creating your own (empty) virtual Python environment in a directory that is writable to you is simple:
python3 -m venv /path/to/new/virtual/environment
You activate the new environment whenever you want to start using it via running the activate script in that folder:
In many cases, you do not want an empty virtual environment, but instead want to start from the
conda base environment's installed packages, only adding and/or changing a few modules.
To extend the base Anaconda environment with
my_env in the current directory) and inherit the base enviroment packages, one can use the
module load conda; conda activate python -m venv --system-site-packages my_env source my_env/bin/activate # Install additional packages here...
--system-site-packagesflag state for this virtual environment by editing
my_env/pyvenv.cfgand changing the value of the line
include-system-site-packages = false.
To install a different version of a package that is already installed in the base environment, you can use:
pip install --ignore-installed ... # or -I
pipcommand should shadow those installed in the base environment.
Cloning the base Anaconda environment
If you need more flexibility, you can clone the conda environment into a custom path, which would then allow for root-like installations via
conda install <module> or
pip install <module>. Unlike the
venv approach, using a cloned Anaconda environment requires you to copy the entirety of the base environment, which can use significant storage space.
This can be performed by:
$ module load conda $ conda activate base (base) $ conda create --clone base --prefix /path/to/envs/base-clone (base) $ conda activate /path/to/envs/base-clone (base-clone) $ which python3 /path/to/base-clone/bin/python3
In the above commands,
path/to/envs/base-clone should be replaced by a
suitably chosen path.