Jupyter Hub

Help Desk

Email: support@alcf.anl.gov

Theta and ThetaGPU
Updated: 11/08/2019

Using Jupyter Hub

Jupyter provides an interactive computing environment for different languages. The most popular use case is Python; Project Jupyter is a spin-off of the interactive Python (iPython) project.

With Jupyter Hub, you can run Python and R notebooks, or open a terminal directly on your browser.

  • You can log in to Theta or Cooley instances of Jupyter Hub at https://jupyter.alcf.anl.gov using your Theta/Cooley credentials.
  • Theta Jupyter Hub instances runs on an external server (jupyter02.mcp.alcf.anl.gov), but not on Theta login, mom, or compute nodes.
  • Theta Jupyer Hub has access to your home folder (/home/$USER) and projects folder (/projects), but does not have access to /opt/cray, /opt/intel, etc.—that is, you cannot use any Theta modules or any Cray libraries.
  • Cooley Jupyter Hub instances have access to the user's home folder (/home/$USER), the Mira projects folder (/projects), and the Theta project folder (/lus/theta-fs0/projects).
  • You can submit jobs to run on Theta/Cooley with !qsub myjob.sh.
  • You can customize your environment and add new kernels. To do so, you could run shell commands on a notebook cell by prepending them with ! (i.e., !pwd). However, it would be much easier to use the Jupyter terminal. You can open a terminal on the Jupyter Hub home page under Files/new menu. Using the terminal, you can create a new conda environment and a new Jupyter kernel with the commands below.

For more information on how to manage conda environments, refer to this page.

From terminal:

conda create -n jhub_myenv
source activate jhub_myenv
conda install jupyter nb_conda ipykernel
conda install any_module_you_need
source deactivate

 

From notebook:

!conda create -y -n jhub_myenv
!source activate jhub_myenv; conda install -y jupyter nb_conda ipykernel
!source activate jhub_myenv; conda install -y any_module_you_need
Copy to Clipboard

This will complete the installation of a new environment. To create a Jupyter kernel, additional work is necessary:

From terminal:

source activate jhub_myenv
python -m ipykernel install --user --name jhub_myenv
source deactivate

 

From notebook:

!source activate;python -m ipykernel install --user --name jhub_myenv
Copy to Clipboard

After completing these steps, you will see jhub_myenv kernel when you click new on the Jupyter Hub home page or when you use Kernel menu on a Jupyter notebook.

  • Use a name identifier (such as jhub_ prefix used above) on your Jupyter Hub conda environment to make sure you don’t mix them with conda environments you create on Cooley/Theta nodes.

Accessing Project Folders:

Jupyterhub limits to a user home directory the user's ability to browse the file system. To access a given project directory, the user must create a symbolic link in their home directory to the project directory:

If a user is connected to the Cooley instances and wants to access Mira project ABC and Theta project EFG, the user should execute the following commands to create the links:

From terminal:

ln -s /project/ABC ABC_project
ln -s /lus/theta-fs0/projects/EFG EFG_project

 

From notebook:

!ln -s /project/ABC ABC_project
!ln -s /lus/theta-fs0/projects/EFG EFG_project
Copy to Clipboard

Hardware 

  • CPU : Intel(R) Xeon(R) CPU E5-2683
  • RAM : 512GB
  • GPU: None 

Training Assets: user support + software + data analytics and visualization + Machine Learning