Datahub/DSMLP Grading Tool - nbgrader


nbgrader Overview


Formgrader/nbgrader is a tool for Jupyter notebooks that allows instructors to create, distribute, and grade assignments through the Jupyter notebook interface. Instructors can easily create notebook-based assignments for both coding exercises and written responses. Formgrader/nbgrader also provides a streamlined interface for quickly grading completed assignments.

The following documentation is an overview of the basic concepts of Formgrader/nbgrader on Datahub/DSMLP to assist graders in creating assignments for their courses.

Using formgrader/nbgrader


Accessing nbgrader

Prior to the beginning of a course, ITS will create a shared nbgrader TA account which is provided to the instructor. Only the shared nbgrader account will have access to formgrader/nbgrader in the course.

To access nbgrader from Datahub:

Creating New Assignments

To create a new assignment in nbgrader, go to the Manage Assignments tab and click Add new assignment... at the bottom of the page.


Developing New Assignments


Each assignment created with nbgrader has a toolbar for each cell which gives the option to decide how each cell will be graded. To create a toolbar for each cell:

  1. Open the assignment notebook (i.e. Assignment_1.ipynb)
  2. Under View>Cell Toolbar in the drop down tab, select Create Assignment
    • This will format all current and future cells with a toolbar

There are five toolbar options to choose from:

When specifying a cell option, each cell must be given an id and when applicable, how many points the answer is worth. The following sections will go into detail about the different types of toolbar options.

Note: nbgrader has a special syntax lines for cells that begin with ###. When an assignment is released, the region between the special syntax lines are replaced with a code stub, so students are not able to see the solutions that graders have written. If the special syntax is not used, the content of the entire cell will be visible.


Validating Assignments

After the assignment has been created, graders can verify that their solutions are correct and will pass all test cases. To do so, use the Validate button located on the jupyter notebook toolbar. If the solution passes all the tests, a success pop-up will show. Otherwise, a pop-up will show which cells have failed.


Generating and Releasing Assignments


After an assignment has been created, graders can generate the version that students will receive. To generate the student version of the assignment in nbgrader:

  1. Go to the Manage Assignments tab and click the Generate button for the corresponding assignment
  2. If successful, a Success pop-up window containing the log output should appear

After generating the student version of the assignment, graders can preview this version to make sure it looks correct (e.g. hidden tests are indeed hidden from students). There should now be a new folder called release, which has the same structure as source. The release directory contains the release version of the assignment files.

To release the assignment to students, click the Release button. If a grader decides to "un-release" an assignment (e.g. fixing a typo), they may do so by clicking the Release button again (which is now an "x"). 

Note: Students who already downloaded the assignment will still have access to their own copy.

To collect all assignments, click the Collect button, which will display a pop-up window that shows the number of submissions collected.

For more detailed information about managing assignments, check the documentation here.

Grading Assignments


To autograde submissions, click on the number of submissions in the Manage Assignments tab in nbgrader. This will take the grader to a Manage Submissions page with all submissions. The status of each assignment should be needs autograding.

Once the autograding is complete, graders can now see the status of each submission. If there were cells that needed manual grading, the status of the graded assignment will be changed to needs manual grading. Otherwise, the status will be changed to graded.

To manually grade assignments in nbgrader, go to the Manual Grading tab. This will provide an interface for manually grading assignments.

Note: An assignment must be run with the autograder in order to be available for manual grading in nbgrader. This applies to all assignments.


Giving Feedback

Once assignments have been graded, graders may generate feedback on an assignment to release to the student.


Getting Grades Into Canvas


Automatic Upload via Nb2Canvas Tool

Nb2Canvas is a UCSD created tool that can be used to easily transfer your grades from Formgrader to your Canvas gradebook. Click here for more info on how to setup and use the tool.

Manual Upload

To get grades into Canvas, graders can run a command within a JupyterHub terminal to generate a CSV file to upload to Canvas.

  1. In the home directory of the course, select New>Terminal to open a new JupyterHub terminal
  2. Enter the following command: nbgrader export. This will create a CSV file named grades.csv in the home directory
  3. Download this CSV file and navigate to the respective course in Canvas
  4. From the Canvas Home menu, select the Grades tab and click Actions>Import to import grades. This will bring up a menu to select the CSV file with the grades, which should be the grades.csv file 

Once the grades.csv file is imported, the interface will determine whether there are any assignments that are not in Canvas. If so, the grader will be given a dropdown menu asking what "This Assignment Is" in Canvas:

  1. Under the --Choose Assignment-- tab, the grader should select which category fits best for each assignment as well as the amount of points the assignment is worth
  2. Selecting the Continue button will open a menu with all the students and new grades. Any changes in grades (i.e. updating grades) will be highlighted in red
  3. Verify the grades are correct and click Save Changes to process the grades into Canvas and add/update the assignments in the Grades and Assignments tab


If you still have questions or need additional assistance, please email datahub@ucsd.edu or visit support.ucsd.edu.