datahub.ucsd.edu and Data Science/Machine Learning Platform FAQ


Overview


UC San Diego's Data Science/Machine Learning Platform (DSMLP) provides undergraduate and graduate students with access to research-class CPU/GPU resources for coursework, formal independent study, and student projects.

Built and operated by IT Services (ITS), with additional financial contributions from Cognitive Science and Jacobs School of Engineering, DSMLP leverages Qualcomm Institute's current research into cost-effective machine-learning cyberinfrastructure using Kubernetes and Docker container technologies.

Please note, DSMLP is not intended to store personally identifiable information (PII) or other sensitive data, see Policies below.

General


How to get help, report a problem, request a feature, or provide feedback?

For course-specific questions, such as a problem with a homework assignment, please contact your Teaching Assistant or Instructor.

To report problems with DSMLP/datahub.ucsd.edu, or to request assistance, please contact the ITS Service Desk by emailing datahub@ucsd.edu to create a problem ticket. Include the following information:

  1. Your course code, if any (e.g., COGS 108)
  2. Whether you are using https://datahub.ucsd.edu, or logging into dsmlp-login.ucsd.edu
  3. The container/environment that you are launching (e.g, "scipy-ml-notebook"; "cogs108-notebook")
  4. Any scripts you are running on dsmlp-login, e.g., launch.sh
  5. Screenshot(s) of the problem

If necessary, you may also reach the ITS Service Desk by phone/walk-in.  

If you have modified your environment using pip, see "How Do I Fix My Local Environment" below.

There are also several self-help documents in the Specialized Instructional Computing Knowledge Base.

 

Accessing Datahub


Why can't I log in?

Log out of all Google accounts or open an incognito window. When prompted, enter your full UCSD email, "username@ucsd.edu", as your credentials. Currently only '@ucsd.edu' addresses are accepted in the Data Science and Machine Learning Platform, not departmental or divisional addresses such as '@eng.ucsd.edu' or '@physics.ucsd.edu'

Access is restricted to authorized users. Students may obtain access to DSMLP through the following methods:

UC San Diego Extension students enrolled in a DSMLP/datahub course should fill out the Concurrent Enrollment Account form and ask their instructor to add them to the course via Canvas.

For more information on eligibility, visit the DSMLP page on blink.ucsd.edu.

Note that we perform required weekly patching on datahub/DSMLP every Wednesday from 6-8 AM Pacific Time.  This may result in reduced capacity, and depending on the nature of the update, the service as a whole may be inaccessible.    

How do I get to my course environment?

After logging in, make sure you are selecting the correct container for your course on the "Select your (Course) Environment" page. The course name will be identified on the container. Some courses have multiple containers available; consult with your TA/instructor for guidance.

After launching your container, you will see the jupyter notebook server environment.  To open a new jupyter notebook, select "New" (top right of page), then "Python 3".

When your work is complete, please shut down your container via "Control Panel" (top right), then click the Stop my Server button.

Why isn't my course environment listed after I log in?

If you recently registered in a course, please allow 1-2 business days for resources to be listed.  Otherwise, please file a IT Services support ticket, include reasons for access, what course you are registered in, and whether you are auditing the course. If auditing a course, please get the instructor's permission prior to submitting a request.

If you are no longer enrolled in any courses, please see "How long does access last after the end of term?" below on how to retrieve your files when you can no longer access your course environment.

How can I fix "spawn failed when starting server" errors?  

Please reset your profile:

  1. Navigate to datahub.ucsd.edu
  2. Click on the "services" dropdown
  3. Select "manual-resetter", and click on the reset button. This will stop any servers you are running, log you out, and reset your profile, while preserving all work/files.

More information: How To: Customize your environment in DSMLP/Datahub (including jupyter notebooks)

What are some other reasons for my server (course container) not starting?  

Check if you have received a "disk quota exceeded" email.  If so, please see the disk space quota section below.

Note that some errors are detected and fixed automatically within a few minutes by our system. 

Accessing Files

Each user has a course home directory for each course and may have a team directory. Users also have a personal home directory that isn't associated with any course.

The simplest way to access your files is to log into datahub.ucsd.edu and launch the course notebook. Here you will see all of your files. You can upload and download files from the Jupyter UI.

When you login to dsmlp-login you will be in your personal home directory. To change to the course workspace use "/opt/launch-sh/bin/workspace".

FilesPath in Jupyter notebook UIContainer LocationWorkspace Location
Course home directory//home/USERNAMEWORKSPACE/home/USERNAME
Course team directory/teams/home/USERNAME/teams/TEAMWORKSPACE/teams/TEAM
Course public directory/public/home/USERNAME/publicWORKSPACE/public
Personal directory/private/home/USERNAME /privateN/A

Course, team, and public files are specific to the course notebook that was launched. For example, if you are enrolled in two courses each course has it's own set of files specific to that course. You will lose access to these files one quarter after the course ends and they will eventually be deleted. If you want to save these files it's recommended to download them to your own computer. You may also move them to your personal directory.

Personal files in /private are accessible in any course and persist between quarters. Be careful not to access files in your /private directory in an assignment, because the grader doesn't have access to them.

Uploading Large/Bulk Files

If you're uploading large files (> 64MB) or a lot of files, then uploading them with the Jupyter UI may not be ideal.

One option is to ZIP the files, upload the ZIP, and then extract the ZIP. Or you could put the files into git and do a git pull.

Another option is to use scp to transfer the files. To do this ssh to dsmlp-login. This is your personal directory. You will notice that this path name is different than what you see inside a course notebook.

You can use scp to copy the files here from your own pc to datahub, or backup files from datahub. Type "pwd" to show the path to your home directory.

Uploading files

scp /local/path username@dsmlp-login.ucsd.edu:~/remote/path

Downloading files

scp username@dsmlp-login.ucsd.edu:~/remote/path /local/path

If you need to copy files to your course home directory run "workspace --list" to show the directory they are located in. It should be something like /dsmlp/fs0x-workspaces/COURSE/home/USERNAME. The directory may be invisible until you cd to it. Team files are located under the same path, but with /teams instead of home, e.g. /dsmlp/fs0x-workspaces/COURSE/teams/TEAM. Use this path in scp instead of your personal home.

Why am I receiving an out of capacity error, and/or being evicted?

Server capacity is limited, and during peak times (e.g. 10th week/finals) occasional delays are to be expected. GPU resources are the most constrained, so you may have success re-launching your job as CPU-only. If you experience capacity errors for prolonged periods, or at unexpected times of the quarter, please file a Service Desk ticket. ETS staff will respond within one business day.

DSMLP is primarily an instructional resource. Users enrolled in a course have higher priority than independent study and research users. When capacity is exhausted, or course users require resources, you may receive an eviction message notifying you to save your work and exit within ten minutes. Temporary hardware or systems issues can also reduce capacity and result in evictions.

During busy periods, we encourage you to visit the cluster status (requires login) page to check resource availability. 

How long does access last after the end of term?

Access is retained for one additional term, e.g. a fall course is available until the end of winter. Please submit an independent study request (https://go.ucsd.edu/2wc5gH0) to request an extension.

To retrieve your files, please scp them from dsmlp-login.ucsd.edu.

scp -r <username>@dsmlp-login.ucsd.edu:/directory/to/send /local/where/to/put

 

General Questions/Errors/Problems


What are the default resource limits?

Each user has a predefined amount GPU/CPU/RAM based on their enrolled course. To see the available RAM, look at the upper right corner of your notebook server (after opening a notebook). Please be considerate and terminate any unused background jobs, since GPU cards are assigned to containers on an exclusive basis, and when attached to a container are unusable by others even if idle.

If available for your course, we encourage you to use non-GPU (CPU-only) containers until your code is fully tested and a simple training run is successful. (PyTorch, Tensorflow, and Caffe toolkits can easily switch between CPU and GPU.)

What is the default disk space quota?

Each student receives 10GB of storage for current-quarter work. If you receive an email asking to reduce your file system usage, open a terminal in your environment to remove files, or ssh to dsmlp-login.ucsd.edu to do the same. (Use the command "du -h -d 1" to see the file space usage of your account). If the quota is exceeded in a course run "workspace" to change to your workspace in the course.

To check your quota login to DataHub and select Services -> disk-quota-service.

Large input datasets should reside in a single location rather than being downloaded to each student's home directory. Students who require more storage should ask their instructor (or TA) to submit a request via the instructors' course Service Desk ticket. For Independent Study students (not enrolled in a datahub course) who require more storage, please submit a Service Desk ticket.

Note: when you delete a file, it goes into the trash folder at .local/share/Trash.  These files can accumulate, and to manually delete them, open a terminal and cd to that directory and delete the files using "rm".  Files should get automatically deleted out of Trash after 7 days.

What datasets are available?

Your course may provide datasets in the public directory.

There are are also several large datasets available for use with DSMLP/Datahub.  To access these datasets from datahub.ucsd.edu, launch your environment, select “New->Terminal” at the top right of the notebook server interface, and enter: cd /datasets. To access these datasets from the command line on dsmlp-login, ssh to dsmlp-login.ucsd.edu and use the previous command to access the datasets directory. 

How do I upload a large file or files?

See: How to: File/Data Transfer - Data Science/Machine Learning Platform (DSMLP)

Are groups available?

Groups of users can be created in DSMLP/datahub for sharing datasets, code, etc.  Your TA or instructor can set this up in Canvas using the instructions here: https://support.ucsd.edu/services?id=kb_article_view&sysparm_article=KB0030588. Groups will have 100 GB of storage by default; please email datahub@ucsd.edu to request additional space.  

How do I kill / stop a notebook?

Use the control panel (button top right of the Jupyter server) to stop the notebook, there may be a delay in updating interface, please let the process complete. Logging Out Does NOT Stop the Server (but it may stop due to inactivity).

How can I run a long-running job?

If you have a job that needs to run for an extended period of time, we recommend running from the command line.  See "How do I launch a container from the command line" below.  If you are running from within datahub.ucsd.edu, you must keep your browser window open.  More information: https://zero-to-jupyterhub.readthedocs.io/en/stable/jupyterhub/customizing/user-management.html   

How do I customize, fix or reset my environment?

Please see How To: Customize your environment in DSMLP/Datahub (including jupyter notebooks)

What additional kernels/software are available in my datahub course container? 

If your course container has additional software versions available, these can be accessed via the "New" button at the top right of your notebook server interface in datahub.ucsd.edu.  Additional kernels containing those software versions will appear in the dropdown menu below the standard python notebook option. 

How do I launch a container from the command line (outside datahub.ucsd.edu)?

See "Launching Containers from the Command Line - DSMLP" and "How to Select and Configure Your Container - DSMLP"

What version of cuda is installed in the machine learning notebook container?

See the current installed version of cuda by referring to the "cuda install cudatoolkit" line in our scipy-ml-notebook container here: https://github.com/ucsd-ets/scipy-ml-notebook/blob/master/Dockerfile

This version may be behind the newest cuda release(s).  This is because upgrading cuda requires coordinating an update of its drivers and various toolkits.  Therefore we typically perform the update process during the summer academic terms. 

For example, in Summer 2021, current cuda version = 10.1, updated to cuda 11+ for Fall 2021 term.

Why am I receiving the error "RuntimeError: CUDA out of memory"?

Each GPU has a limited amount of physical memory available to it, independent of the amount of RAM available to your pod. This message is seen when you exhaust that memory, and indicates that you need to work with your TA to modify how much GPU memory your work requires.

The GPUs on different compute nodes in the DataHub cluster have different amounts of memory available. Most nodes have 11GB, but others may have more or less. To view the capacities of the GPU attached to your current pod, run the command "nvidia-smi". (Note that you will not be able to see the list of processes running on the GPU from your pod.)

The amount of free memory can also be determined programmatically by using the cudaMemGetInfo() function in the CUDA Runtime API.

Subtle choices in PyTorch library usage can result in memory leaks that exhaust GPU memory. For suggestions on how to manage CUDA memory in PyTorch, see the section My model reports "cuda runtime error(2): out of memory" on the PyTorch FAQ.

What if I get a database locked error message?

1. Open New -> Terminal
2. rm ~/.local/share/jupyter/nbsignatures.db
3. Close terminal window
4. Control Panel -> Stop
5. Restart notebook

How do I launch my first RStudio session?

Information on using R and RStudio with DSMLP/Datahub can be found here.

Where is JupyterLab? 

After starting your server, replace "/tree" in the URL with "/lab", e.g., https://datahub.ucsd.edu/user/<username>/lab.

Where can I look up my UCSD Extension student account?

Extension students may receive separate accounts for the purpose of accessing Datahub.  To look up your account information and reset any additional account passwords, visit https://sdacs.ucsd.edu/~icc/index.php and enter your AX account in the username field and your UID in the Student ID field (e.g. "axNNNN", "cs120sp20aa", etc.)

I'm getting a "failed to validate" or "source of the following cell has changed" error/I can't autograde a particular submission due to "corrupt metadata"

This occurs if a read-only cell/an autograded cell has been copied. Do not copy and paste these cells in your notebook. Similar errors, such as "Failed validating 'required' in notebook" might occur if you edit or delete cells provided by the instructor. To resolve this problem, you (or your student) will need to:

  1. Back up your assignment: Rename the existing notebook by adding '-corrupted' to the end of the notebook filename, and download a copy to your computer in case you need to start over.
  2. Re-fetch your notebook from the 'Assignments' tab. You will now see both the fresh copy and the corrupted copy in your Files.
  3. Open both copies of the notebook, and copy your answers from the corrupted notebook to the fresh copy
  4. Re-submit your assignment

 

Policies


Sensitive Data

UC Data Protection Levels are defined at: https://security.ucop.edu/files/documents/uc-protection-level-classification-guide.pdf 

DSMLP is not suitable for storage or processing of Category P3/P4 data, which includes:

Please visit the link above for a full list of P3/P4 data.

If your project may involve P3/P4 data, please contact ETS or Research IT Services for a consultation.

If you still have questions or need additional assistance, please email datahub@ucsd.edu to create a support ticket, or visit support.ucsd.edu.