UC San Diego's Data Science/Machine Learning Platform (DSMLP) provides undergraduate and graduate students with access to research-class CPU/GPU resources for coursework, formal independent study, and student projects.
Built and operated by IT Services (ITS), with additional financial contributions from Cognitive Science and Jacobs School of Engineering, DSMLP leverages Qualcomm Institute's current research into cost-effective machine-learning cyberinfrastructure using Kubernetes and Docker container technologies.
For course-specific questions, such as a problem with a homework assignment, please contact your Teaching Assistant or Instructor.
To report problems with DSMLP/datahub.ucsd.edu, or to request assistance, please contact the ITS Service Desk by emailing email@example.com to create a problem ticket. Include the following information:
If necessary, you may also reach the ITS Service Desk by phone/walk-in.
If you have modified your environment using pip, see "How Do I Fix My Local Environment" below.
There are also several self-help documents in the Specialized Instructional Computing Knowledge Base.
Log out of all Google accounts or open an incognito window. When prompted, enter your full UCSD email, "firstname.lastname@example.org", as your credentials. Currently only '@ucsd.edu' addresses are accepted in the Data Science and Machine Learning Platform, not departmental or divisional addresses such as '@eng.ucsd.edu' or '@physics.ucsd.edu'
Access is restricted to authorized users. Students may obtain access to DSMLP through the following methods:
For more information on eligibility, visit the DSMLP page on blink.ucsd.edu.
Note that we perform required weekly patching on datahub/DSMLP every Wednesday from 6-8 AM Pacific Time. This may result in reduced capacity, and depending on the nature of the update, the service as a whole may be inaccessible.
After logging in, make sure you are selecting the correct container for your course on the "Select your (Course) Environment" page. The course name will be identified on the container. Some courses have multiple containers available; consult with your TA/instructor for guidance.
After launching your container, you will see the jupyter notebook server environment. To open a new jupyter notebook, select "New" (top right of page), then "Python 3".
When your work is complete, please shut down your container via "Control Panel" (top right), then click the Stop my Server button.
If you recently registered in a course, please allow 1-2 business days for resources to be listed. Otherwise, please file a IT Services support ticket, include reasons for access, what course you are registered in, and whether you are auditing the course. If auditing a course, please get the instructor's permission prior to submitting a request.
If you are no longer enrolled in any courses, please see "How long does access last after the end of term?" below on how to retrieve your files when you can no longer access your course environment.
Please reset your profile:
Check if you have received a "disk quota exceeded" email. If so, please see the disk space quota section below.
Note that some errors are detected and fixed automatically within a few minutes by our system.
Server capacity is limited, and during peak times (e.g. 10th week/finals) occasional delays are to be expected. GPU resources are the most constrained, so you may have success re-launching your job as CPU-only. If you experience capacity errors for prolonged periods, or at unexpected times of the quarter, please file a Service Desk ticket. ETS staff will respond within one business day.
DSMLP is primarily an instructional resource. Users enrolled in a course have higher priority than independent study and research users. When capacity is exhausted, or course users require resources, you may receive an eviction message notifying you to save your work and exit within ten minutes. Temporary hardware or systems issues can also reduce capacity and result in evictions.
During busy periods, we encourage you to visit the cluster status (requires login) page to check resource availability.
Access is retained for one additional term, e.g. a fall course is available until the end of winter. Please submit an independent study request (https://go.ucsd.edu/2wc5gH0) to request an extension.
To retrieve your files, please scp them from dsmlp-login.ucsd.edu.
scp -r <username>@dsmlp-login.ucsd.edu:/directory/to/send /local/where/to/put
Each user has a predefined amount GPU/CPU/RAM based on their enrolled course. To see the available RAM, look at the upper right corner of your notebook server (after opening a notebook). Please be considerate and terminate any unused background jobs, since GPU cards are assigned to containers on an exclusive basis, and when attached to a container are unusable by others even if idle.
If available for your course, we encourage you to use non-GPU (CPU-only) containers until your code is fully tested and a simple training run is successful. (PyTorch, Tensorflow, and Caffe toolkits can easily switch between CPU and GPU.)
Each student receives 10GB of storage for current-quarter work. If you receive an email asking to reduce your file system usage, open a terminal in your environment to remove files, or ssh to dsmlp-login.ucsd.edu to do the same. (Use the command "
du -h -d 1" to see the file space usage of your account).
Large input datasets should reside in a single location rather than being downloaded to each student's home directory. Students who require more storage should ask their instructor (or TA) to submit a request via the instructors' course Service Desk ticket. For Independent Study students (not enrolled in a datahub course) who require more storage, please submit a Service Desk ticket.
Note: when you delete a file, it goes into the trash folder at .local/share/Trash. These files can accumulate, and to manually delete them, open a terminal and cd to that directory and delete the files using "rm". Files should get automatically deleted out of Trash after 7 days.
There are several large datasets available for use with DSMLP/Datahub. To access these datasets from datahub.ucsd.edu, launch your environment, select “New->Terminal” at the top right of the notebook server interface, and enter:
cd /datasets. To access these datasets from the command line on dsmlp-login, ssh to dsmlp-login.ucsd.edu and use the previous command to access the datasets directory.
Groups of users can be created in DSMLP/datahub for sharing datasets, code, etc. Your TA or instructor can set this up in Canvas using the instructions here: https://support.ucsd.edu/services?id=kb_article_view&sysparm_article=KB0030588. Groups will have 100 GB of storage by default; please email email@example.com to request additional space.
Use the control panel (button top right of the Jupyter server) to stop the notebook, there may be a delay in updating interface, please let the process complete. Logging Out Does NOT Stop the Server (but it may stop due to inactivity).
If you have a job that needs to run for an extended period of time, we recommend running from the command line. See "How do I launch a container from the command line" below. If you are running from within datahub.ucsd.edu, you must keep your browser window open. More information: https://zero-to-jupyterhub.readthedocs.io/en/stable/jupyterhub/customizing/user-management.html
If your course container has additional software versions available, these can be accessed via the "New" button at the top right of your notebook server interface in datahub.ucsd.edu. Additional kernels containing those software versions will appear in the dropdown menu below the standard python notebook option.
See the current installed version of cuda by referring to the "cuda install cudatoolkit" line in our scipy-ml-notebook container here: https://github.com/ucsd-ets/scipy-ml-notebook/blob/master/Dockerfile
This version may be behind the newest cuda release(s). This is because upgrading cuda requires coordinating an update of its drivers and various toolkits. Therefore we typically perform the update process during the summer academic terms.
For example, in Summer 2021, current cuda version = 10.1, updated to cuda 11+ for Fall 2021 term.
Each GPU has a limited amount of physical memory available to it, independent of the amount of RAM available to your pod. This message is seen when you exhaust that memory, and indicates that you need to work with your TA to modify how much GPU memory your work requires.
The GPUs on different compute nodes in the DataHub cluster have different amounts of memory available. Most nodes have 11GB, but others may have more or less. To view the capacities of the GPU attached to your current pod, run the command "nvidia-smi". (Note that you will not be able to see the list of processes running on the GPU from your pod.)
The amount of free memory can also be determined programmatically by using the cudaMemGetInfo() function in the CUDA Runtime API.
Subtle choices in PyTorch library usage can result in memory leaks that exhaust GPU memory. For suggestions on how to manage CUDA memory in PyTorch, see the section My model reports "cuda runtime error(2): out of memory" on the PyTorch FAQ.
1. Open New -> Terminal
2. rm ~/.local/share/jupyter/nbsignatures.db
3. Close terminal window
4. Control Panel -> Stop
5. Restart notebook
Information on using R and RStudio with DSMLP/Datahub can be found here.
After starting your server, replace "/tree" in the URL with "/lab", e.g., https://datahub.ucsd.edu/user/<username>/lab.
Extension students may receive separate accounts for the purpose of accessing Datahub. To look up your account information and reset any additional account passwords, visit https://sdacs.ucsd.edu/~icc/index.php and enter your AX account in the username field and your UID in the Student ID field (e.g. "axNNNN", "cs120sp20aa", etc.)
This occurs if a read-only cell/an autograded cell has been copied. Do not copy and paste these cells in your notebook. Similar errors, such as "Failed validating 'required' in notebook" might occur if you edit or delete cells provided by the instructor. To resolve this problem, you (or your student) will need to: