Core Service Functionality

The Data Science and Machine Learning Platform (DSMLP) provides a standard set of features for instructional use, including web and command-line access to jupyter notebook servers, GPU access, student home directories and storage, and large dataset access. Learn more about DSMLP.

datahub.ucsd.edu is the web-based DSMLP platform.

For more information about the default and custom containers available, see "An overview of Standard Datahub/DSMLP Containers maintained by Educational Technology Services."

Service Level Objectives and Roles/Responsibilities

IT Services’ Responsibilities

• Provide reliable system operations and core service functionality/restoration
• Enroll students (including waitlisted students) in courses
• Aim to resolve individual user issues within 1-2 business days
• Incidents affecting multiple courses, many users, or at critical points in the quarter (e.g. exams) will have a target response time of 30 minutes from first contact to Service Desk, and a target resolution time of 8 hours
• Assist instructors/TAs with incident resolution and minor configuration change requests
• Maintain our online FAQ and additional Knowledge Base documentation

Instructor/TA Responsibilities

• Adding TAs and Observers to courses (see How-To below).  Formerly-waitlisted students who have been removed can be re-added as an Observer.
• Provide first-tier support for students with Jupyter notebook usage questions
• Basic knowledge of jupyter notebooks and nbgrader/formgrader
  
  • Please do not ask students to copy read-only cells in their notebooks, since this causes a validation error: https://github.com/jupyter/nbgrader/issues/1083
• Request course resources via an Instructional Technology Request (CINFO)
  
  • On page 2 of the form (Web & Cloud Resources), select "Datahub" for the web-based version of the platform, "Data Science / Machine Learning GPU Cluster" for the command-line version, or both.
  • In the comments, indicate:
    
    • Which containers are needed for the course (default or custom software packages required)
    • Important course dates (e.g., course setup required by; midterm; final project due dates)
    • Typical assignment due dates (e.g., every Friday at midnight)
• Provide timely communication of incidents/requests via the course's support ticket, or by:
  
  • Web: https://support.ucsd.edu/its
  • Email: dsmlp@ucsd.edu
  • Phone: (858) 246-4357
  • Walk-in: AP&M 1313, First floor; Weekdays 8 a.m. – 4:30 p.m

Service Timeline

• 2-4 weeks prior to term start:
  
  • Instructor submits an Instructional Technology Request (CINFO): https://cinfo.ucsd.edu/.  See above for details.
  • If necessary, instructor requests meetings with ITS staff to discuss any non-standard requirements, and/or to assist TAs with nbgrader/formgrader setup
    
  • Early access granted for instructors/TAs to dsmlp-login.ucsd.edu and/or https://datahub.ucsd.edu.  Student-test accounts available by request.
  • Each course receives a shared nbgrader TA grader account for access to the formgrader UI in https://datahub.ucsd.edu, used for assignment creation/distrubution/collection and grading. 
• Week 1:
  
  • Students enrolled in course are granted ssh access to dsmlp-login.ucsd.edu and/or https://datahub.ucsd.edu
  • IT Service Hub ticket for course is open for instructor and TA communication with IT Services, including new software package requests
• Week 1-10:
  
  • nbgrader/formgrader student roster automatically updated based on course enrollment
• 11 days after the last day of class:
  
  • IT Service Hub ticket closed
• 45 days after the last day of class:
  
  • Shared nbgrader TA and TA access to course environment is removed
• 90 days after the last day of class:
  
  • Instructor and student access to course environment is removed.  Instructors are still able to access platform with a generic environment.

How-To's for Instructors and TAs

Add TAs or Observers to Course

1. Log into Canvas https://canvas.ucsd.edu/
2. Click Courses on left nav bar
3. Select course by clicking link
4. Click Add TAs on right nav bar
5. Click Access Level dropdown
6. Select Teaching Assistant
7. Enter user's email address to add
8. Click Add Teaching Assistant or Add Observer
   
   1. Note: if observer is a UC San Diego Extension student, they will need to fill out the Concurrent Enrollment Account form first

Add Groups to Course

Note: if you need to add a student to more than one group, please request this via your course ticket in support.ucsd.edu, or email awsed@ucsd.edu

1. Log into Canvas https://canvas.ucsd.edu and navigate to the course
2. Click People on the left nav bar
3. Click +Group Set to create a group set, e.g. Group Projects
   
   
   
   
4. Click the newly create group set
5. Click +Group to create a group, e.g. Project 1
   
   
   
   
6. Drag students into the group
   
   
   
   
7. Repeat for each student and group

Accessing Datasets

There are several large datasets available for use with DSMLP/Datahub. To navigate to the directory containing the datasets, use the following command in a terminal: cd /datasets.

Request a Shared Course Directory

To request a shared course directory for your course, please include this in the comments of your Instructional Technology Request (CINFO; see below) or follow up in the support.ucsd.edu ticket that is created for your course when the CINFO is submitted.

Set up the "nb2canvas" Tool (LTI) in your Canvas course to upload datahub assignments/submissions

More information: How To: Configure and Use nb2canvas

For Winter 2022 courses, please email datahub@ucsd.edu and we will set this up for you.  We plan to allow instructors to directly install this tool in their Canvas course for Spring 2022 term.

Naming assignments

"nbgrader" requires unique assignment names for any assignment across all instructor assignments.

Removing old "nbgrader" courses from "Assignments" dropdown

1. Open a new Terminal
   
   1. New -> Terminal
2. Navigate to the "nbgrader_cache" directory
   
   1. Enter command: "cd ~/.local/share/jupyter/nbgrader_cache"
3. List all files in the directory
   
   1. Enter command: "ls -al"
4. Delete the old directories

Service Resource Limitations

Resource exhaustion (insufficient GPUs/CPUs) can occur during periods of heavy usage. If eviction of a user’s server is required, they will be notified 10 minutes before termination and instructed to save their data.

Scheduled Weekly Patching and Potential Downtime

Datahub/DSMLP Weekly Maintenance Window: Tuesdays, 6AM-8AM (Pacific)

The worldwide increase in ransomware and similar cybersecurity attacks has prompted UC San Diego to adopt strict policies regarding updating/patching of campus servers.

To minimize interruption to your students' work, we've designated Tuesdays from 6AM-8AM as a weekly maintenance period.

During this time, DataHub/DSMLP may operate at reduced capacity, and depending on the nature of the update, the cluster as a whole may be inaccessible.

Scheduling policies will try to run jobs on nodes that won’t be impacted, but the risk remains that user jobs, i.e. Jupyter notebooks, may be terminated. It is important to periodically save state during long running jobs ("checkpointing") so they can be resumed when service is restored.

We recognizing the impact of any disruption during 10th week and Finals week, and will attempt to defer patching during these critical instructional periods unless the nature of the vulnerability requires immediate action.

Finally, as campus policy requires that "Critical" severity threads be addressed within 24 hours of discovery, we may be forced to patch outside of the Tuesday 6AM-8AM window. In such cases we will provide advance warning to instructors and TAs to the extent possible.

Troubleshooting

• Student's Notebook Failed to Validate/Cannot autograde assignment/Metadata Corrupted
  
  • This occurs if a read-only cell/an autograded cell has been copied. To resolve this problem, your student will need to:
    
    • Back up their assignment: Rename the existing notebook by adding '-corrupted' to the end of the notebook filename, and download a copy to their computer in case they need to start over.
    • Re-fetch their notebook from the 'Assignments' tab. They will now see both the fresh copy and the corrupted copy in their Files.
    • Open both copies of the notebook, and copy their answers from the corrupted notebook to the fresh copy.
    • Re-submit their assignment.
    • You re-collect the assignment in Formgrader.
• Nbgrader export failed
  
  • This can happen if an assignment is created with a space in the title, or if an assignment is deleted through the filesystem rather than the "nbgrader db assignment remove" command.
    
    • Delete the assignment and rerun the command.
• Not Enough GPUs
  
  • Your students may need to wait some time (on average 5-10 minutes) before a GPU is available for scheduling. If so, make sure they are not running their launch scripts with the "-f" flag, as that will kill the containers after the given command is executed, instead of launching the container's shell.