How to Select and Configure Your Container on the Data Science/Machine Learning Platform (DSMLP)


Launch Script Overview


Please review "Launching Containers From the Command Line" on how to access and launch containers.

Each launch script on dsmlp-login.ucsd.edu (or ieng6) in ../public/bin specifies the default Docker image to use, the required number of CPU cores, GPU cards, and GB RAM assigned to its containers.  Instructors and TAs may directly modify the coursewide scripts located in ../public/bin. Otherwise, users may copy an existing launch script into their home directory, then modify that private copy: 

  $ cp -p `which launch-pytorch.sh` $HOME/my-launch-pytorch.sh
$ nano $HOME/my-launch-pytorch.sh  
$ $HOME/my-launch-pytorch.sh

An example of such a launch configuration, which launches the DSMLP machine learning notebook server, and calls the run_jupyter.sh entrypoint script defined in that dockerfile (see "An Overview of DSMLP Containers"), is as follows:

K8S_DOCKER_IMAGE="ucsdets/scipy-ml-notebook:2021.1-stable"
K8S_ENTRYPOINT="./run_jupyter.sh"

K8S_NUM_GPU=1  # max of 1 (contact ETS to raise limit)
K8S_NUM_CPU=4  # max of 8 ("")
K8S_GB_MEM=32  # max of 64 ("")

# Controls whether an interactive Bash shell is started
SPAWN_INTERACTIVE_SHELL=YES

# Sets up proxy URL for Jupyter notebook inside
PROXY_ENABLED=YES
PROXY_PORT=8888

Launch Script Command Line Options


Defaults set within launch scripts' environment variables may be overridden using the following command-line options:

Option

Description

Example

-c N

Adjust # CPU cores

-c 8

-g N

Adjust # GPU cards

-g 2

-m N

Adjust # GB RAM

-m 64

-i IMG

Docker image name

-i nvidia/cuda:latest 
(see below)

-e ENTRY

Docker image ENTRYPOINT/CMD. 
Review the Dockerfile for the name of the launch script. 

-e /setup.sh

-n N

Request specific cluster node (1-10)

-n 7

-v

Request specific GPU (gtx1080ti,k5200,titan)

-v k5200

-b

Request background pod

(see below)

Example:

[cs190f @ieng6-201]:~:56$  launch-py3torch-gpu.sh -m 64 -v k5200

Adjusting CPU/GPU/RAM limits


The maximum limits (8 CPU, 64GB, 1 GPU) apply to all of your running containers:  

Increases to GPU allocations require consent of TA, instructor or advisor.  

Background Execution / Long-Running Jobs


To support longer training runs, we permit background execution of student containers, up to 12 hours execution time, via the "-b" command line option.  

Use the ‘kubesh <pod-name>’ command to connect or reconnect to a background container, and ‘kubectl delete pod <pod-name>’ to terminate.

Please be considerate and terminate idle containers:  while containers share system RAM and CPU resources under the standard Linux/Unix model, the cluster’s 80 GPU cards are assigned to users on an exclusive basis. When attached to a container they become unusable by others even if completely idle.

Creating and Specifying a Custom Docker Image


If you need to create a custom container from one of the standard containers, please see Instructions on Building a Custom Image.  Typically these are used for entire courses using the DSMLP platform, or if individual users need to install operating system-level packages (e.g. 'apt-get install', 'yum').  To create and use a custom image: 

  1. Select any standard image (ucsdets/datahub-base-notebook, ucsdets/datascience-notebook, etc.) to base your image from
  2. Follow this guide to create, build and host a docker image.  As of June 2021 we recommend github actions. 
  3. SSH to either dsmlp-login.ucsd.edu (or ieng6.ucsd.edu if available to your course).
  4. Run the command launch.sh -i {MY_DOCKER_ACCOUNT/REPO:TAG} to launch your container.  More information: Launching Containers from the Command Line

It may take some time to run your container as it has to download onto the DSMLP servers. Run the command:  kubectl describe pod -n {USERNAME} to see the state of your container

Adding Custom Python Packages to Your Container


To customize your environment within your existing container, please see: How To: Customize your environment in DSMLP/Datahub (including jupyter notebooks)


For the original version of this guide, click here.

For more information about datahub.ucsd.edu, check out the FAQ.

Your instructor or TA will be your best resource for course-specific questions.

If you still have questions or need additional assistance, please email dsmlp@ucsd.edu or visit support.ucsd.edu.