UC San Diego's Data Science/Machine Learning Platform (DSMLP) provides undergraduate and graduate students with access to research-class CPU/GPU resources for coursework, formal independent study, and student projects.
Built and operated by IT Services (ITS), with additional financial contributions from Cognitive Science and Jacobs School of Engineering, DSMLP leverages Qualcomm Institute's current research into cost-effective machine-learning cyberinfrastructure using Kubernetes and Docker container technologies.
Please note, DSMLP is not intended to store personally identifiable information (PII) or other sensitive data, see Policies below.
For course-specific questions, such as a problem with a homework assignment, please contact your Teaching Assistant or Instructor.
To report problems with DSMLP/datahub.ucsd.edu, or to request assistance, please contact the ITS Service Desk by emailing email@example.com to create a problem ticket. Include the following information:
If necessary, you may also reach the ITS Service Desk by phone/walk-in.
If you have modified your environment using pip, see "How Do I Fix My Local Environment" below.
There are also several self-help documents in the Specialized Instructional Computing Knowledge Base.
Log out of all Google accounts or open an incognito window. When prompted, enter your full UCSD email, "firstname.lastname@example.org", as your credentials. Currently only '@ucsd.edu' addresses are accepted in the Data Science and Machine Learning Platform, not departmental or divisional addresses such as '@eng.ucsd.edu' or '@physics.ucsd.edu'
Access is restricted to authorized users. Students may obtain access to DSMLP through the following methods:
For more information on eligibility, visit the DSMLP page on blink.ucsd.edu.
Note that we perform required weekly patching on datahub/DSMLP every Wednesday from 6-8 AM Pacific Time. This may result in reduced capacity, and depending on the nature of the update, the service as a whole may be inaccessible.
After logging in, make sure you are selecting the correct container for your course on the "Select your (Course) Environment" page. The course name will be identified on the container. Some courses have multiple containers available; consult with your TA/instructor for guidance.
After launching your container, you will see the jupyter notebook server environment. To open a new jupyter notebook, select "New" (top right of page), then "Python 3".
When your work is complete, please shut down your container via "Control Panel" (top right), then click the Stop my Server button.
If you recently registered in a course, please allow 1-2 business days for resources to be listed. Otherwise, please file a IT Services support ticket, include reasons for access, what course you are registered in, and whether you are auditing the course. If auditing a course, please get the instructor's permission prior to submitting a request.
If you are no longer enrolled in any courses, please see "How long does access last after the end of term?" below on how to retrieve your files when you can no longer access your course environment.
Please reset your profile:
Check if you have received a "disk quota exceeded" email. If so, please see the disk space quota section below.
Note that some errors are detected and fixed automatically within a few minutes by our system.
Each user has a course home directory for each course and may have a team directory. Users also have a personal home directory that isn't associated with any course.
The simplest way to access your files is to log into datahub.ucsd.edu and launch the course notebook. Here you will see all of your files. You can upload and download files from the Jupyter UI.
When you login to dsmlp-login you will be in your personal home directory. To change to the course workspace use "/opt/launch-sh/bin/workspace".
|Files||Path in Jupyter notebook UI||Container Location||Workspace Location|
|Course home directory||/||/home/USERNAME||WORKSPACE/home/USERNAME|
|Course team directory||/teams||/home/USERNAME/teams/TEAM||WORKSPACE/teams/TEAM|
|Course public directory||/public||/home/USERNAME/public||WORKSPACE/public|
|Personal directory||/private||/home/USERNAME /private||N/A|
Course, team, and public files are specific to the course notebook that was launched. For example, if you are enrolled in two courses each course has it's own set of files specific to that course. You will lose access to these files one quarter after the course ends and they will eventually be deleted. If you want to save these files it's recommended to download them to your own computer. You may also move them to your personal directory.
Personal files in /private are accessible in any course and persist between quarters. Be careful not to access files in your /private directory in an assignment, because the grader doesn't have access to them.
If you're uploading large files (> 64MB) or a lot of files, then uploading them with the Jupyter UI may not be ideal.
One option is to ZIP the files, upload the ZIP, and then extract the ZIP. Or you could put the files into git and do a git pull.
Another option is to use scp to transfer the files. To do this ssh to dsmlp-login. This is your personal directory. You will notice that this path name is different than what you see inside a course notebook.
You can use scp to copy the files here from your own pc to datahub, or backup files from datahub. Type "pwd" to show the path to your home directory.
scp /local/path email@example.com:~/remote/path
scp firstname.lastname@example.org:~/remote/path /local/path
If you need to copy files to your course home directory run "workspace --list" to show the directory they are located in. It should be something like /dsmlp/fs0x-workspaces/COURSE/home/USERNAME. The directory may be invisible until you cd to it. Team files are located under the same path, but with /teams instead of home, e.g. /dsmlp/fs0x-workspaces/COURSE/teams/TEAM. Use this path in scp instead of your personal home.
Server capacity is limited, and during peak times (e.g. 10th week/finals) occasional delays are to be expected. GPU resources are the most constrained, so you may have success re-launching your job as CPU-only. If you experience capacity errors for prolonged periods, or at unexpected times of the quarter, please file a Service Desk ticket. ETS staff will respond within one business day.
DSMLP is primarily an instructional resource. Users enrolled in a course have higher priority than independent study and research users. When capacity is exhausted, or course users require resources, you may receive an eviction message notifying you to save your work and exit within ten minutes. Temporary hardware or systems issues can also reduce capacity and result in evictions.
During busy periods, we encourage you to visit the cluster status page to check resource availability (requires login).
Access is retained for one additional term, e.g. a fall course is available until the end of winter. Please submit an independent study request (https://go.ucsd.edu/2wc5gH0) to request an extension.
To retrieve your files, please scp them from dsmlp-login.ucsd.edu.
scp -r <username>@dsmlp-login.ucsd.edu:/directory/to/send /local/where/to/put
Each user has a predefined amount GPU/CPU/RAM based on their enrolled course. To see the available RAM, look at the upper right corner of your notebook server (after opening a notebook). Please be considerate and terminate any unused background jobs, since GPU cards are assigned to containers on an exclusive basis, and when attached to a container are unusable by others even if idle.
If available for your course, we encourage you to use non-GPU (CPU-only) containers until your code is fully tested and a simple training run is successful. (PyTorch, Tensorflow, and Caffe toolkits can easily switch between CPU and GPU.)
GPU types are listed on the cluster status page.
Each student receives 10GB of storage for current-quarter work. If you receive an email asking to reduce your file system usage, open a terminal in your environment to remove files, or ssh to dsmlp-login.ucsd.edu to do the same. (Use the command "
du -h -d 1" to see the file space usage of your account). If the quota is exceeded in a course run "workspace" to change to your workspace in the course.
To check your quota login to DataHub and select Services -> disk-quota-service.
Large input datasets should reside in a single location rather than being downloaded to each student's home directory. Students who require more storage should ask their instructor (or TA) to submit a request via the instructors' course Service Desk ticket. For Independent Study students (not enrolled in a datahub course) who require more storage, please submit a Service Desk ticket.
Note: when you delete a file, it goes into the trash folder at .local/share/Trash. These files can accumulate, and to manually delete them, open a terminal and cd to that directory and delete the files using "rm". Files should get automatically deleted out of Trash after 7 days.
Your course may provide datasets in the public directory.
There are are also several large datasets available for use with DSMLP/Datahub. To access these datasets from datahub.ucsd.edu, launch your environment, select “New->Terminal” at the top right of the notebook server interface, and enter:
cd /datasets. To access these datasets from the command line on dsmlp-login, ssh to dsmlp-login.ucsd.edu and use the previous command to access the datasets directory.
Groups of users can be created in DSMLP/datahub for sharing datasets, code, etc. Your TA or instructor can set this up in Canvas using the instructions here: https://support.ucsd.edu/services?id=kb_article_view&sysparm_article=KB0030588. Groups will have 100 GB of storage by default; please email email@example.com to request additional space.
Use the control panel (button top right of the Jupyter server) to stop the notebook, there may be a delay in updating interface, please let the process complete. Logging Out Does NOT Stop the Server (but it may stop due to inactivity).
If you have a job that needs to run for an extended period of time, we recommend running from the command line. See "How do I launch a container from the command line" below. If you are running from within datahub.ucsd.edu, you must keep your browser window open. More information: https://zero-to-jupyterhub.readthedocs.io/en/stable/jupyterhub/customizing/user-management.html
If your course container has additional software versions available, these can be accessed via the "New" button at the top right of your notebook server interface in datahub.ucsd.edu. Additional kernels containing those software versions will appear in the dropdown menu below the standard python notebook option.
See the current installed version of cuda by referring to the "cuda install cudatoolkit" line in our scipy-ml-notebook container here: https://github.com/ucsd-ets/scipy-ml-notebook/blob/master/Dockerfile
This version may be behind the newest cuda release(s). This is because upgrading cuda requires coordinating an update of its drivers and various toolkits. Therefore we typically perform the update process during the summer academic terms.
For example, in Summer 2021, current cuda version = 10.1, updated to cuda 11+ for Fall 2021 term.
Each GPU has a limited amount of physical memory available to it, independent of the amount of RAM available to your pod. This message is seen when you exhaust that memory, and indicates that you need to work with your TA to modify how much GPU memory your work requires.
The GPUs on different compute nodes in the DataHub cluster have different amounts of memory available. Most nodes have 11GB, but others may have more or less. To view the capacities of the GPU attached to your current pod, run the command "nvidia-smi". (Note that you will not be able to see the list of processes running on the GPU from your pod.)
The amount of free memory can also be determined programmatically by using the cudaMemGetInfo() function in the CUDA Runtime API.
Subtle choices in PyTorch library usage can result in memory leaks that exhaust GPU memory. For suggestions on how to manage CUDA memory in PyTorch, see the section My model reports "cuda runtime error(2): out of memory" on the PyTorch FAQ.
1. Open New -> Terminal
2. rm ~/.local/share/jupyter/nbsignatures.db
3. Close terminal window
4. Control Panel -> Stop
5. Restart notebook
Information on using R and RStudio with DSMLP/Datahub can be found here.
After starting your server, replace "/tree" in the URL with "/lab", e.g., https://datahub.ucsd.edu/user/<username>/lab.
Extension students may receive separate accounts for the purpose of accessing Datahub. To look up your account information and reset any additional account passwords, visit https://sdacs.ucsd.edu/~icc/index.php and enter your AX account in the username field and your UID in the Student ID field (e.g. "axNNNN", "cs120sp20aa", etc.)
This occurs if a read-only cell/an autograded cell has been copied. Do not copy and paste these cells in your notebook. Similar errors, such as "Failed validating 'required' in notebook" might occur if you edit or delete cells provided by the instructor. To resolve this problem, you (or your student) will need to:
UC Data Protection Levels are defined at: https://security.ucop.edu/files/documents/uc-protection-level-classification-guide.pdf
DSMLP is not suitable for storage or processing of Category P3/P4 data, which includes:
Please visit the link above for a full list of P3/P4 data.
If your project may involve P3/P4 data, please contact ETS or Research IT Services for a consultation.