Create custom kernel for Jupyter Hub

This is how you can setup your JLab jupyterhub notebook to run inside a python virtual environment. This is useful for installing python packages via pip from your own account without needing admin privliges (which you don't have). The following will need to be run from a terminal on a CUE computer. I recommend doing this via a terminal launched via jupyterhub since you can use a python version available there that should have all necessary system libraries installed.
  • The JLab firewall will prevent pip from downloading packages using default options. The easiest thing to do is setup a pip configuration file in the home directory of your CUE account that sets the necessary options so you don't have to type them with every pip command. Create a file named "~/.config/pip/pip.conf" and write the following into it:
  • ;
    ; Options to allow pip install to work from behind
    ; the hallgw firewall
    ;
    [install]
    trusted-host =
    pypi.org
    files.pythonhosted.org
    [search]
    trusted-host =
    pypi.org
    files.pythonhosted.org
  • NOTE: To create a venv that works with jupyterhub you must use the /opt/conda/bin/python3.7 version and NOT the one installed in /apps. This directory is not available on the ifarms so you must execute the commands below in a terminal launched in jupyterhub itself. However, if you wish to run the venv on the ifarm machines (including sciml machines), then you should use the /apps version and run these commands from an ifarm machine. If you use /apps version there's one extra step you need to follow shown at the bottom of this tutorial in advance section.
  • Create a python virtual environment in the directory of your choosing with:
    /opt/conda/bin/python3.7 -m venv venv
    or to use apps/ version do this:
    module load python/3.7.3
    python3.7 -m venv venv
  • Source the new environment so it will be used for all subsequent pip commands:
  • source venv/bin/activate.csh
    or
    source venv/bin/activate.sh
  • Upgrade the pip package itself to see that everything is working:
  • pip install --upgrade pip
  • Install python packages you want:
  • pip install tensorflow keras pandas matplotlib
  • Install the ipykernel into the venv and the venv into your list of kernels:
  • pip install ipykernel
    python -m ipykernel install --user --name=venv
  • Advance: Follow this step only if you are using the /aaps version of the python.
    On ifarm go to ~/.local/share/jupyter/kernels/venv . Inside this directory there's a file named kernel.json that must look like as follows, please add env variable to this as highlighted below.
  • {
    "argv":[
    "< path to the python source >",
    "-m",
    "ipykernel_launcher",
    "-f",
    "{Connection_file}",
    ], "display_name": "venv",
    "language": "python",
    "env": {"LD_LIBRARY_PATH":"/apps/python/3.7.3/lib:/apps/cuda/10.1/lib64"}
    }
  • At this point a new option for "venv" should be there in your launcher for both "Notebook" and "Console".
  • The virtual environment is now part of a "kernel" and you can switch existing notebooks to using it via a menu in the upper right-hand corner of the notebook. (you may need to close and reopen the notebook to see the new kernel in the menu)
  • You can list available kernels via command line using "jupyter kernelspec list"
  • You can remove an old kernels using "jupyter kernelspec uninstall old-kernel"
  • Note: We have not tested these steps with different systems and we do not provide support for custom kernels.