You are here

Create custom kernel for Jupyter Hub

This is how you can setup your JLab jupyterhub notebook to run inside a python virtual environment. This is useful for installing python packages via pip from your own account without needing admin privliges (which you don't have). The following will need to be run from a terminal on a CUE computer. I recommend doing this via a terminal launched via jupyterhub since you can use a python version available there that should have all necessary system libraries installed.

; ; Options to allow pip install to work from behind ; the hallgw firewall ; [install] trusted-host = pypi.org files.pythonhosted.org [search] trusted-host = pypi.org files.pythonhosted.org
/opt/conda/bin/python3.8 -m venv venv
source venv/bin/activate.csh
pip install --upgrade pip
pip install tensorflow keras pandas matplotlib
pip install ipykernel python -m ipykernel install --user --name=venv
  • The JLab firewall will prevent pip from downloading packages using default options. The easiest thing to do is setup a pip configuration file in the home directory of your CUE account that sets the necessary options so you don't have to type them with every pip command. Create a file named "~/.config/pip/pip.conf" and write the following into it:
  • NOTE: To create a venv that works with jupyterhub you must use the /opt/conda/bin/python3.7 version and NOT the one installed in /apps. This directory is not available on the ifarms so you must execute the commands below in a terminal launched in jupyterhub itself. However, if you wish to run the same code on the ifarm machines (including sciml machines) and jupyterhub, then you can obtain a mirrored version of your venv on jupyterhub by following the steps shown at the bottom of this tutorial in Optional section.
  • In order to use GPUs with your custom kernel, you need to follow steps shown in the Advance section below.
  • Create a python virtual environment in the directory of your choosing with:
  • Source the new environment so it will be used for all subsequent pip commands:
  • Upgrade the pip package itself to see that everything is working:
  • Install python packages you want:
  • Install the ipykernel into the venv and the venv into your list of kernels:
  • At this point a new option for "venv" should be there in your launcher for both "Notebook" and "Console".
  • The virtual environment is now part of a "kernel" and you can switch existing notebooks to using it via a menu in the upper right-hand corner of the notebook. (you may need to close and reopen the notebook to see the new kernel in the menu)
  • You can list available kernels via command line using "jupyter kernelspec list"
  • You can remove an old kernels using "jupyter kernelspec uninstall old-kernel"
  • Note: We have not tested these steps with different systems and we do not provide support for custom kernels.
Optional: Follow these steps install all the python libraries from another venv in one shot. This way you can mirror your venv from ifarm to jupyterhub.
pip freeze > ~/requirements.txt
pip install -r ~/requirements.txt
Advance: To use GPUs with your custom kernel, go to ~/.local/share/jupyter/kernels/venv. Inside this directory there's a file named kernel.json that must look like as follows, please add env variable to this as highlighted below. NOTE: currently, jupyterhub only supports cuda 10 (compatible with tf 2.3.x).
{ "argv":[ "< path to the python source >", "-m", "ipykernel_launcher", "-f", "{Connection_file}", ], "display_name": "venv", "language": "python",
"env": {"LD_LIBRARY_PATH":"/apps/cuda/10.1/lib64:/usr/local/lib:/.singularity.d/libs"}

}

  • Activate your venv on ifarm and run the following command to save requirements.txt to your home directory.
  • Now in jupyterhub terminal, activate your custom kernel virtual environment and run following command to install all the same set of libraries with exact version numbers in the kernel.