Datahub/DSMLP Grading Tool - nbgrader


Overview


nbgrader is a tool for Jupyter notebooks that allows instructors to create, distribute, and grade assignments through the Jupyter notebook interface (Formgrader). Instructors can easily create notebook-based assignments for both coding exercises and written responses. nbgrader also provides a streamlined interface for quickly grading completed assignments.

The following documentation is an overview of the basic concepts of Formgrader/nbgrader on Datahub/DSMLP to assist graders in creating assignments for their courses.

Using nbgrader


Accessing nbgrader via its Formgrader user interface

Note: the Formgrader screenshots on the nbgrader documentation site show it running in Jupyterlab.  If you are using the "classic" Jupyter notebook user interface, you won't see the JupyterLab panel on the left side of the screen.

Prior to the beginning of a course, IT Services will create a shared nbgrader TA account and provide its login credentials to the instructor. Only the

nbgrader account will have access to nbgrader/formgrader in the course.

To access nbgrader/formgrader from Datahub:

Note: Student usernames will not appear in the "Manage Students" page of Formgrader until they submit assignments.  At that point, only their username (not their first/last names) will be shown.

Creating New Assignments

To create a new assignment in nbgrader, go to the Manage Assignments tab and click Add new assignment... at the bottom of the page.

Developing New Assignments


Note: When naming assignments in nbgrader (not the notebook itself), DO NOT give it an .ipynb extension.  An assignment named "Assignment_1" will create a folder of the same name in your (grader's) home directory.  Within that folder, assignment notebooks will have .ipynb extensions.

Each assignment created with nbgrader has a toolbar for each cell which gives the option to decide how each cell will be graded. To create a toolbar for each cell:

  1. Open the assignment notebook (i.e. Assignment_1.ipynb)
  2. Under View>Cell Toolbar in the drop down tab, select Create Assignment
    • This will format all current and future cells with a toolbar

There are five toolbar options to choose from:

When specifying a cell option, each cell must be given an id and when applicable, how many points the answer is worth. The following sections will go into detail about the different types of toolbar options.

Note: nbgrader has a special syntax lines for cells that begin with ###. When an assignment is released, the region between the special syntax lines are replaced with a code stub, so students are not able to see the solutions that graders have written. If the special syntax is not used, the content of the entire cell will be visible.

Validating Assignments

After the assignment has been created, graders can verify that their solutions are correct and will pass all test cases. To do so, use the Validate button located on the jupyter notebook toolbar. If the solution passes all the tests, a success pop-up will show. Otherwise, a pop-up will show which cells have failed.

Generating and Releasing Assignments


After an assignment has been created, graders can generate the version that students will receive. To generate the student version of the assignment in nbgrader:

  1. Go to the Manage Assignments tab and click the Generate button for the corresponding assignment
  2. If successful, a Success pop-up window containing the log output should appear

After generating the student version of the assignment, graders can preview this version to make sure it looks correct (e.g. hidden tests are indeed hidden from students). There should now be a new folder called release, which has the same structure as source. The release directory contains the release version of the assignment files.

To release the assignment to students, click the Release button. If a grader decides to "un-release" an assignment (e.g. fixing a typo), they may do so by clicking the Release button again (which is now an "x"). 

Note: Students who already downloaded the assignment will still have access to their own copy.

To collect all assignments, click the Collect button, which will display a pop-up window that shows the number of submissions collected.

For more detailed information about managing assignments, check the documentation here.

Grading Assignments


To autograde submissions, click on the number of submissions in the Manage Assignments tab in nbgrader. This will take the grader to a Manage Submissions page with all submissions. The status of each assignment should be needs autograding.

Once the autograding is complete, graders can now see the status of each submission. If there were cells that needed manual grading, the status of the graded assignment will be changed to needs manual grading. Otherwise, the status will be changed to graded.

To manually grade assignments in nbgrader, go to the Manual Grading tab. This will provide an interface for manually grading assignments.

Note: An assignment must be run with the autograder in order to be available for manual grading in nbgrader. This applies to all assignments.

Note: The default assignment file size limit is 100MB.  If some files aren't transferring, open a New->Terminal window as the course grader, and add the following to the /usr/local/etc/jupyter/nbgrader_config.py file: 

c.CourseDirectory.max_file_size = 400000

Giving Feedback

Once assignments have been graded, graders may generate feedback on an assignment to release to the student.

Getting Grades Into Canvas


Manual Upload

To get grades into Canvas, graders can run a command within a JupyterHub terminal to generate a CSV file to upload to Canvas.

  1. In the home directory of the course, select New>Terminal to open a new JupyterHub terminal
  2. Enter the following command: nbgrader export. This will create a CSV file named grades.csv in the home directory
  3. Download this CSV file and navigate to the respective course in Canvas
  4. From the Canvas Home menu, select the Grades tab and click Actions>Import to import grades. This will bring up a menu to select the CSV file with the grades, which should be the grades.csv file 

Once the grades.csv file is imported, the interface will determine whether there are any assignments that are not in Canvas. If so, the grader will be given a dropdown menu asking what "This Assignment Is" in Canvas:

  1. Under the --Choose Assignment-- tab, the grader should select which category fits best for each assignment as well as the amount of points the assignment is worth
  2. Selecting the Continue button will open a menu with all the students and new grades. Any changes in grades (i.e. updating grades) will be highlighted in red
  3. Verify the grades are correct and click Save Changes to process the grades into Canvas and add/update the assignments in the Grades and Assignments tab

Getting First/Last Names into NbGrader DB


As of Spring 2024, we have created the update-nbgrader script which is accessible from dsmlp-login. It will allow you to import the first and last names of your users into the NbGrader DB alongside their usernames so that you can see them from the Formgrader UI.

  1. While logged onto your course's grader account (this will only work on dsmlp-login), run update-nbgrader -c <course-id>. This will generate a CSV in your home directory that you can move to your course's workspace. Please verify the results of the CSV before moving on (such as with cat).
  2. Then you need to start up a notebook in that course, where you can then run nbgrader db student import <course-id>-nbgrader-students.csv.

On dsmlp-login.ucsd.edu:

  1. Login with a grader account for your course.
    • update-nbgrader -c <course_name> will generate a CSV in your course dir containing each of its usernames followed by their First and Last names.

      screenshot of update-nbgrader -c command on terminal
  2. You can then use workspace -l to find your course's location on the filesystem.
  3. In this case, our course is TEST_NBGRADER, and we cd into the dir returned by workspace -l. We then mv the generated CSV into this folder so that we can access it in our JupyterHub pod later.

    screenshot of generating csv using the workspace command and moving it to the right folder

Inside of Course Environment/Notebook:

  1. Launch a pod as a grader for your course. Once you've done this, spawn a terminal window (under New>Terminal) and proceed with the next command.
  2. You can then use nbgrader db student import <CSV_NAME> in the CSV's dir to import students' names into nbgrader.

    screenshot of using nbgrader db student import command in jupyterhub

  3. Nbgrader form grade should now reflect your changes:

    nbgrader manage students section

If you still have questions or need additional assistance, please email datahub@ucsd.edu or visit support.ucsd.edu.