Reason for not using Conda: For Conda environments installed in a home directory, both the Python interpreter and all of it’s modules reside on the shared parallel file system. The operations to look up the needed files are metadata heavy and can strain the parallel file system, resulting in a slower startup performance.
Although you can install Jupyter via Conda, we highly recommend to install using regular Python techniques. The following is known to work:
[alice@dev1 ~]$ python3 -m pip install --user notebook
Jupyter Notebook should only be run on the Wynton HPC development nodes. However, you cannot connect from outside Wynton HPC directly to a development node, but rather either need to use SSH port forwarding to establish the connection with a local web browser, else use X2Go to redirect the desktop from the development server to your local desktop and launch a remote web browser there.
Running Jupyter Notebook on Wynton HPC involves three steps. On an development node,
find an available TCP port
launch Jupyter Notebook on select port, and
set up a TCP port tunnel from your local machine to the Wynton development node where Jupyter runs
Jupyter Notebook makes itself available via the web browser. In order for multiple users to run Jupyter at the same time, each Jupyter instance must be served on a unique port in [1024,65535]. This port has to be free, that is, it must not already be used by any other processes on the same machine. One way to find such a port is to simply pick a random port and hope no one else is already using the port. An alternative approach, is to use the port4me tool, which will find a free port, e.g.
[alice@dev1 ~]$ module load CBI port4me
[alice@dev1 ~]$ port4me --tool=jupyter
47467
Make a note of the port number you get - you will need it next.
Comment: The port4me tool is designed to find the same free port each time you use it. It is only when that port happens to be already occupied that you will get another port, but most of the time, you will be using the same one over time.
Next, we launch Jupyter Notebook on the same development node:
[alice@dev1]$ jupyter notebook --no-browser --port 47467
[I 2024-03-20 14:48:45.693 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2024-03-20 14:48:45.698 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2024-03-20 14:48:45.703 ServerApp] jupyterlab | extension was successfully linked.
[I 2024-03-20 14:48:45.708 ServerApp] notebook | extension was successfully linked.
[I 2024-03-20 14:48:46.577 ServerApp] notebook_shim | extension was successfully linked.
[I 2024-03-20 14:48:46.666 ServerApp] notebook_shim | extension was successfully loaded.
[I 2024-03-20 14:48:46.668 ServerApp] jupyter_lsp | extension was successfully loaded.
[I 2024-03-20 14:48:46.669 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[I 2024-03-20 14:48:46.675 LabApp] JupyterLab extension loaded from /wynton/home/boblab/alice/.local/lib/python3.11/site-packages/jupyterlab
[I 2024-03-20 14:48:46.675 LabApp] JupyterLab application directory is /wynton/home/boblab/alice/.local/share/jupyter/lab
[I 2024-03-20 14:48:46.677 LabApp] Extension Manager is 'pypi'.
[I 2024-03-20 14:48:46.707 ServerApp] jupyterlab | extension was successfully loaded.
[I 2024-03-20 14:48:46.711 ServerApp] notebook | extension was successfully loaded.
[I 2024-03-20 14:48:46.712 ServerApp] Serving notebooks from local directory: /wynton/home/boblab/alice
[I 2024-03-20 14:48:46.712 ServerApp] Jupyter Server 2.13.0 is running at:
[I 2024-03-20 14:48:46.712 ServerApp] http://localhost:47467/tree?token=8e37f8d62fca6a1c9b2da429f27df5ebcec706a808c3a8f2
[I 2024-03-20 14:48:46.712 ServerApp] http://127.0.0.1:47467/tree?token=8e37f8d62fca6a1c9b2da429f27df5ebcec706a808c3a8f2
[I 2024-03-20 14:48:46.712 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 2024-03-20 14:48:46.725 ServerApp]
To access the server, open this file in a browser:
file:///wynton/home/boblab/alice/.local/share/jupyter/runtime/jpserver-2853162-open.html
Or copy and paste one of these URLs:
http://localhost:47467/tree?token=8e37f8d62fca6a1c9b2da429f27df5ebcec706a808c3a8f2
http://127.0.0.1:47467/tree?token=8e37f8d62fca6a1c9b2da429f27df5ebcec706a808c3a8f2
However, these instructions do not work out of the box, because they are based on the assumption that you run Jupyter on your local machine. If you try to open one of these links, your browser produces a “site-not-found” error. To solve this, we need to complete the next step.
Above, Jupyter makes itself available on the local machine where it
runs, which in our case is development node dev1.
In order for us to access this from the web browser running on our
local machine, we need to tunnel the TCP port from on your local
machine to the port on the development node. This can be achieved
using SSH port forward. To do this, open a terminal on your local
machine, and run:
{local}$ ssh -J alice@log1.wynton.ucsf.edu -L 47467:localhost:47467 alice@dev1
...
[alice@dev1 ~]$
Importantly, in your case, you will need to replace both instance of
47467 with the port number that you used in Step 2.
Now your personal Jupyter Notebook instance running on
dev1 is accessible directly from the web
browser running on your local computer. To do this, open one of the
HTTP links outputted by Jupyter in Step 2, e.g.
An alternative method to run a Jupyter Notebook on one of the development nodes and interact with on your local computer is to use the X2Go software to connect to one of the development nodes, then launch a Jupyter Notebook and a web browser on the development node. The web browser will appear on your remote computer in the X2Go interface while the Jupyter Notebook will be running on Wynton HPC.
See the ‘Graphical User Interfaces (GUI)’ for how to setup and use X2Go via one of the Wynton HPC development nodes.
If you use Python via your own Conda installation, instead of the
Python version provided by Wynton HPC, you can launch a
Jupyter Notebook using the jupyter notebook command from the
terminal connected by X2Go, which will also launch the web browser.