JupyterHub

SHADE-AI Gym: JupyterHub Quickstart

A dedicated JupyterHub is provided as part of the DARPA SHADE-AI Gym. Unlike batch computing workflows on Frontera, LS6, and Longhorn, JupyterHub offers on-demand compute through interactive Jupyter notebooks. The SHADE JupyterHub is running on a Kubernetes cluster. After logging into the hub, a container is launched that hosts the Jupyter notebook. Currently, each running instance is provisioned with 12 CPUs and 28 GB of RAM. If you require additional CPUs and/or RAM please connect with the TACC team.

Accessing the SHADE-AI JupyterHub

In your browser, enter the following URL:

https://shade.jupyter.tacc.cloud

and login with your TACC username and password. If this is the first time you are logging in, it may take extra time to launch the instance.

Storage Volumes

Invidual users' $WORK directory and the DARPA-SHADE directory on Corral are directly mounted and are accessible from the Jupyter notebook. Notebooks and dataset should be stored on one of these file systems. Note: because the Jupyter notebook is running out of a container, any notebook files created in the root directory only persist for the life of the running instance.

Python Packages

A number of python packages are installed and available by default, most popular of which include pandas, tensorflow, torch, scikit-learn, among others. Select the dropdown below to see the list of default packages. Because the Jupyter notebook is running out of a container, any packages installed via the common pip install command will not persist and will only be available for the lifetime of the running instance. The most flexible option to customize the Jupyter environment will be to create an a docker image that descends from the base image we are using. Through the Dockerfile, you will be able to add additional packages. You can provide the TACC team this image, which we will make available through the SHADE JupyterHub.

Default python packages

Package Version
absl-py 0.11.0
agavepy 0.9.3
aiohttp 3.6.2
alembic 1.4.3
algorithmia 1.2.1
algorithmia-api-client 1.1.0
appnope 0.1.0
argon2-cffi 20.1.0
astunparse 1.6.3
async-generator 1.1
async-timeout 3.0.1
atomicwrites 1.4.0
attrs 19.3.0
backcall 0.1.0
backports.functools-lru-cache 1.6.1
backports.ssl-match-hostname 3.7.0.1
beautifulsoup4 4.9.3
bleach 3.1.0
blinker 1.4
blis 0.4.1
bokeh 1.4.0
Bottleneck 1.3.2
brotlipy 0.7.0
cachetools 4.1.1
catalogue 1.0.0
certifi 2020.11.8
certipy 0.1.3
cffi 1.14.3
chardet 3.0.4
Click 7
click-plugins 1.1.1
cligj 0.5.0
cloudpickle 1.6.0
conda 4.9.0
conda-package-handling 1.7.2
configparser 5.0.1
cryptography 2.9.2
cycler 0.10.0
cymem 2.0.3
Cython 0.29.14
cytoolz 0.11.0
dash 1.9.1
dash-core-components 1.8.1
dash-daq 0.2.1
dash-html-components 1.0.2
dash-renderer 1.2.4
dash-table 4.6.1
dash-table-experiments 0.6.0
dashserve 0.1.7
dask 2.30.0
dataclasses 0.6
DateTime 4.3
decorator 4.4.1
defusedxml 0.6.0
dill 0.3.3
distributed 2.30.1
Django 2.2.7
django-plotly-dash 1.0.2
dpd-components 0.1.0
en-core-web-lg 2.2.5
entrypoints 0.3
enum34 1.1.6
fastcache 1.1.0
Fiona 1.8.13.post1
Flask 1.1.1
Flask-Compress 1.4.0
freetype-py 2.1.0.post1
fsspec 0.8.4
future 0.18.2
gast 0.3.3
geopandas 0.7.0
gmpy2 2.1.0b1
google-auth 1.23.0
google-auth-oauthlib 0.4.2
google-pasta 0.2.0
grpcio 1.34.0
h5py 2.10.0
HeapDict 1.0.1
idna 2.8
imagecodecs 2020.5.30
imageio 2.9.0
import-ipynb 0.1.3
importlib-metadata 0.23
iniconfig 1.1.1
ipykernel 5.1.3
ipympl 0.5.8
ipython 7.9.0
ipython-genutils 0.2.0
ipywidgets 7.5.1
isodate 0.6.0
itsdangerous 1.1.0
jedi 0.15.1
Jinja2 2.10.3
joblib 0.14.0
json5 0.8.5
jsonschema 3.1.1
jupyter 1.0.0
jupyter-client 5.3.4
jupyter-console 6.0.0
jupyter-core 4.6.1
jupyter-plotly-dash 0.3.1
jupyter-server-proxy 1.3.0
jupyter-telemetry 0.0.5
jupyterhub 1.2.1
jupyterlab 1.2.2
jupyterlab-dash 0.1.0a3
jupyterlab-pygments 0.1.2
jupyterlab-server 1.0.6
Keras 2.4.3
Keras-Preprocessing 1.1.2
kiwisolver 1.1.0
lazy-object-proxy 1.5.2
llvmlite 0.34.0
locket 0.2.0
Mako 1.1.3
Markdown 3.1.1
MarkupSafe 1.1.1
matplotlib 3.1.2
mistune 0.8.4
mock 4.0.2
more-itertools 7.2.0
mpmath 1.1.0
msgpack 1.0.0
multidict 4.5.2
munch 2.5.0
murmurhash 1.0.2
mydcc 0.1.22
nbclient 0.5.1
nbconvert 5.6.1
nbformat 4.4.0
nbgitpuller 0.9.0
nbserverproxy 0.8.8
nest-asyncio 1.4.2
networkx 2.4
nltk 3.4.5
notebook 6.0.2
numba 0.51.2
numexpr 2.7.1
numpy 1.17.4
oauthlib 3.0.1
olefile 0.46
openapi-core 0.13.4
openapi-schema-validator 0.1.1
openapi-spec-validator 0.2.9
opt-einsum 3.3.0
packaging 19.2
pamela 1.0.0
pandas 0.25.3
pandocfilters 1.4.2
panel 0.6.4
param 1.9.2
parse 1.18.0
parso 0.5.1
partd 1.1.0
patsy 0.5.1
pexpect 4.7.0
pickleshare 0.7.5
Pillow 6.2.1
pip 20.2.4
plac 1.1.3
plotly 4.5.4
pluggy 0.13.1
preshed 3.0.2
prometheus-client 0.7.1
prompt-toolkit 2.0.10
protobuf 3.13.0
psutil 5.7.3
ptyprocess 0.6.0
py 1.9.0
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycosat 0.6.3
pycparser 2.2
pyct 0.4.6
pycurl 7.43.0.6
Pygments 2.4.2
PyJWT 1.7.1
pyOpenSSL 19.1.0
pyparsing 2.4.5
pyproj 2.6.0
pyrsistent 0.15.5
pyshp 2.1.0
PySocks 1.7.1
pytest 6.1.2
python-dateutil 2.8.1
python-editor 1.0.4
python-json-logger 2.0.1
pytz 2019.3
pyviz-comms 0.7.2
PyWavelets 1.1.1
PyYAML 5.3.1
pyzmq 18.1.0
qtconsole 4.5.5
reportlab 3.5.32
requests 2.23.0
requests-oauthlib 1.3.0
requests-toolbelt 0.9.1
retrying 1.3.3
rsa 4.6
ruamel-yaml-conda 0.15.80
ruamel.yaml 0.16.12
ruamel.yaml.clib 0.2.2
scikit-image 0.17.2
scikit-learn 0.21.3
scipy 1.3.2
seaborn 0.11.0
Send2Trash 1.5.0
setuptools 21.2.2
shap 0.32.1
Shapely 1.7.0
simpervisor 0.3
six 1.13.0
sortedcontainers 2.2.2
soupsieve 2.0.1
spacy 2.2.3
SQLAlchemy 1.3.20
sqlparse 0.3.0
srsly 1.0.1
statsmodels 0.12.1
strict-rfc3339 0.7
sympy 1.6.2
tables 3.6.1
tapipy 0.3.0
tblib 1.6.0
tensorboard 2.4.0
tensorboard-plugin-wit 1.7.0
tensorflow 2.3.1
tensorflow-estimator 2.3.0
termcolor 1.1.0
terminado 0.8.3
testpath 0.4.4
thinc 7.3.1
threadpoolctl 2.1.0
tifffile 2020.10.1
toml 0.10.2
toolz 0.11.1
torch 1.7.0
tornado 6.0.3
tqdm 4.39.0
traitlets 4.3.3
typing-extensions 3.7.4.3
urllib3 1.25.8
vincent 0.4.4
wasabi 0.6.0
wcwidth 0.1.7
webencodings 0.5.1
websocket-client 0.53.0
Werkzeug 0.16.0
wheel 0.35.1
widgetsnbextension 3.5.1
wrapt 1.12.1
xgboost 0.9
xlrd 1.2.0
yarl 1.3.0
zict 2.0.0
zipp 0.6.0
zope.interface 4.7.1