Conda
Another example of conda environment setup.
Login on the submit node
Login on the cluster submission node, check the page How to Access for more information:
$ ssh -l <username> hpc7.ncg.ingrid.pt
[username@hpc7 ~]$ _
Prepare a conda virtual environment
The default python version for CentOS 7.x is 2.7.5 which is not suitable for many applications. So, we will create a python virtual environment:
[username@hpc7 ~]$ conda create -n myconda python=3.6
[username@hpc7 ~]$ conda activate myconda
On the first command, where we create the conda virtual environment, you can include a list of applications to include on your environmnet, for example:
[username@hpc7 ~]$ conda create -n myconda python=3.6 ipython-notebook numpy=1.6
Manage the conda virtual environment
It is possible to include additional packages to your conda environment, for example:
[username@hpc7 ~]$ conda activate myconda
[username@hpc7 ~]$ conda install numpy
You can update your software bandle on the conda virtual environment with the command:
[username@hpc7 ~]$ conda update [scipy ...]
or remove a specific application:
[username@hpc7 ~]$ conda uninstall tensorflow-gpu
Check the conda help for more information:
[username@hpc7 ~]$ conda help
[username@hpc7 ~]$ conda install --help
Manage the conda packages list with pip
It is possible to complemment the conda virtual environment packages list with pip. For example:
[username@hpc7 ~]$ conda activate myconda
[username@hpc7 ~]$ pip install --upgrade pip
[username@hpc7 ~]$ pip install --upgrade setuptools
[username@hpc7 ~]$ pip install tensorflow-gpu
[username@hpc7 ~]$ pip install keras
Manage packages versions
If the applications available on conda virtual environment do not match your version requirements you may need to use packages from pip repostory; check the availability of conda search and pip search command line interfaces.
As an example we have the tensorflow-gpu package, when used with keras, the conda repository downgrades *tensorflow-gpu to version 1.15, but you most likely will prefer version 2.0. The pip repository has the right combination of tensorflow-gpu and keras packages.
We advise the user to install a package from only one repository in order to guarantee perfect behaviour.
Load conda environment on a batch job
Create a submit script:
[username@hpc7 ~]$ cat submit.sh
#!/bin/bash
# Set parallell environment "mp", and excute in 4 cores on same node
#$ -pe mp 4
# Queue selection
#$ -q hpc
# check python version
python --version
# Load conda environment
conda activate myconda
# recheck python version
python --version
# your job payload
#....
Submit:
[username@hpc7 ~]$ qsub submit.sh
Your job 2037792 ("submit.sh") has been submitted
After completion:
[username@hpc7 ~]$ ls -l
-rwxr-----+ 1 username hpc 668 Jan 7 12:19 submit.sh
-rw-r-----+ 1 username hpc 44 Jan 7 12:18 submit.sh.e2037792
-rw-r-----+ 1 username hpc 0 Jan 7 12:18 submit.sh.o2037792
[username@hpc7 ~]$ cat submit.sh.e2037792
Python 2.7.5
Python 3.6.9 :: Anaconda, Inc.