Overview

---

nbgrader is a tool for Jupyter notebooks that allows instructors to create, distribute, and grade assignments through the Jupyter notebook interface (Formgrader). Instructors can easily create notebook-based assignments for both coding exercises and written responses. nbgrader also provides a streamlined interface for quickly grading completed assignments.

The following documentation is an overview of the basic concepts of Formgrader/nbgrader on Datahub/DSMLP to assist graders in creating assignments for their courses.

Using nbgrader

---

Accessing nbgrader via its Formgrader user interface

Note: the Formgrader screenshots on the nbgrader documentation site show it running in Jupyterlab.  If your application uses the "classic" Jupyter notebook user interface (or before our 2024.4 images, also known as Notebook 6), you won't see the JupyterLab panel on the left side of the screen.

Prior to the beginning of a course, IT Services will create a shared nbgrader TA account and provide its login credentials to the instructor. Only the

nbgrader account will have access to nbgrader/formgrader in the course.

To access nbgrader/formgrader from Datahub:

• Login in to datahub.ucsd.edu with the shared nbgrader TA account (e.g., username@ucsd.edu) and launch the course environment
• Select the Formgrader tab at the top to open the nbgrader interface, where assignments, grades, and students can be managed

Note: Student usernames will not appear in the "Manage Students" page of Formgrader until they submit assignments.  At that point, only their username (not their first/last names) will be shown.

Creating New Assignments

To create a new assignment in nbgrader, go to the Manage Assignments tab and click Add new assignment... at the bottom of the page.

• This will ask you to provide information about the assignment name and its due date
  
  • The due date is in Coordinated Universal Time (UTC). To offset the due date to Pacific Time (PT)
• Set Due Date to the desired deadline and Timezone as UTC offset to '-0800' for PST ('-0700' for PDT)
• Assignments that are created and developed are saved under the source/{assignment_id}/ directory, where assignment_id is the name of the assignment being created
• Once an assignment is created, navigate the the new assignment folder and create/add the notebook and files needed for the new assignment. Make sure to rename the notebook to the assignment name

Developing New Assignments

---

Note: When naming assignments in nbgrader (not the notebook itself), DO NOT give it an .ipynb extension.  An assignment named "Assignment_1" will create a folder of the same name in your (grader's) home directory.  Within that folder, assignment notebooks will have .ipynb extensions.

Each assignment created with nbgrader has a toolbar for each cell which gives the option to decide how each cell will be graded. To create a toolbar for each cell:

1. Open the assignment notebook (i.e. Assignment_1.ipynb)
2. Under View>Cell Toolbar in the drop down tab, select Create Assignment
   
   • This will format all current and future cells with a toolbar

There are five toolbar options to choose from:

• Manually graded answer
• Manually graded task
• Autograded answer
• Autograder tests
• Read-only

When specifying a cell option, each cell must be given an id and when applicable, how many points the answer is worth. The following sections will go into detail about the different types of toolbar options.

Note: nbgrader has a special syntax lines for cells that begin with ###. When an assignment is released, the region between the special syntax lines are replaced with a code stub, so students are not able to see the solutions that graders have written. If the special syntax is not used, the content of the entire cell will be visible.

• Manually graded answer (Code and Markdown Cells): This option has cells that contains an answer that must be manually graded. It requires the student to perform something in one cell. A common use of this type of cell is for free-response answers (which may interpret the results of code that may have been written or executed in a previous cell).
  
  
• Manually graded task (Code and Markdown Cells): This option has cells that contains an answer that must be manually graded. It requires the student to perform something with cells. A common use of this type of cell is when students need to create several cells on their own. (such as processing data and creating a plot to show their results).
  
  
• Autograded answer (Code Cells): This option has cells that contains an answer to be used for autograding. For these cells, solutions can be specified inline using the special syntax ### BEGIN SOLUTION and ### END SOLUTION, which is then used when performing the autograded tests, A common use of this type of cell is for functions that students are required to write in the cell (typically with the function name given to be used by the autograded tests).
  
  
• Autograder tests (Code Cells): This option has cells that contain tests to be run during the autograding. Test cells should contain assert (or similar) statements. When running the autograding, the cell will pass if no errors are raised, and fail otherwise. Tests in these cells can be hidden using the special syntax ### BEGIN HIDDEN TESTS and ### END HIDDEN TESTS. A common use of this type of cell is to provide tests, either visible and/or hidden, that will grade code from the autograded answer cells.
  
  Note: It is recommended to have at least one test visible in the released assignment or a comment stating there are hidden tests. Otherwise, students will see a blank cell.
  
  
• "Read-only" (Code and Markdown Cells): This option has cells that are read-only and cannot be modified. This indicated by a lock icon on the left side of the cell toolbar. If the read-only cell has been changed in a student's version of the cell, it will be replaced with the original version of the cell. This functionality is important for test cells, which are also marked as read-only, as students cannot modify the autograded tests performed by nbgrader in order to "pass" the tests. 

Validating Assignments

After the assignment has been created, graders can verify that their solutions are correct and will pass all test cases. To do so, use the Validate button located on the jupyter notebook toolbar. If the solution passes all the tests, a success pop-up will show. Otherwise, a pop-up will show which cells have failed.

Generating and Releasing Assignments

---

After an assignment has been created, graders can generate the version that students will receive. To generate the student version of the assignment in nbgrader:

1. Go to the Manage Assignments tab and click the Generate button for the corresponding assignment
2. If successful, a Success pop-up window containing the log output should appear

After generating the student version of the assignment, graders can preview this version to make sure it looks correct (e.g. hidden tests are indeed hidden from students). There should now be a new folder called release, which has the same structure as source. The release directory contains the release version of the assignment files.

To release the assignment to students, click the Release button. If a grader decides to "un-release" an assignment (e.g. fixing a typo), they may do so by clicking the Release button again (which is now an "x"). 

Note: Students who already downloaded the assignment will still have access to their own copy.

To collect all assignments, click the Collect button, which will display a pop-up window that shows the number of submissions collected.

For more detailed information about managing assignments, check the documentation here.

Grading Assignments

---

To autograde submissions, click on the number of submissions in the Manage Assignments tab in nbgrader. This will take the grader to a Manage Submissions page with all submissions. The status of each assignment should be needs autograding.

• To autograde an individual submission, click the Autograde button to grade it
• To autograde all submissions at once for the particular assignment, follow the command line instructions given under the Instructions tab. Once assignments have been graded, they will be located in the autograded directory, which has the same structure as the source directory

Once the autograding is complete, graders can now see the status of each submission. If there were cells that needed manual grading, the status of the graded assignment will be changed to needs manual grading. Otherwise, the status will be changed to graded.

To manually grade assignments in nbgrader, go to the Manual Grading tab. This will provide an interface for manually grading assignments.

Note: An assignment must be run with the autograder in order to be available for manual grading in nbgrader. This applies to all assignments.

Note: The default assignment file size limit is 100MB.  If some files aren't transferring, open a New->Terminal window as the course grader, and add the following to the /usr/local/etc/jupyter/nbgrader_config.py file: 

c.CourseDirectory.max_file_size = 400000

Giving Feedback

Once assignments have been graded, graders may generate feedback on an assignment to release to the student.

• In the Manage Assignments tab in nbgrader, click the Generate Feedback button to create an HTML file that contains the grading breakdown of a students assignment, displaying what was correct and what was incorrect
• Click the Release Feedback button to allow students to receive their grading feedback. The feedback will display what the student got correct/incorrect on their assignment
• If a grader wishes to add comments when grading an assignment, they may go to the Manual Grading tab in nbgrader and select the assignment. This will take the grader to the students submission, where they can now manually add comments to any cell block that the student has used

Getting Grades Into Canvas

---

Manual Upload

To get grades into Canvas, graders can run a command within a JupyterHub terminal to generate a CSV file to upload to Canvas.

1. In the home directory of the course, select New>Terminal to open a new JupyterHub terminal
2. Enter the following command: nbgrader export. This will create a CSV file named grades.csv in the home directory
3. Download this CSV file and navigate to the respective course in Canvas
4. From the Canvas Home menu, select the Grades tab and click Actions>Import to import grades. This will bring up a menu to select the CSV file with the grades, which should be the grades.csv file 

Once the grades.csv file is imported, the interface will determine whether there are any assignments that are not in Canvas. If so, the grader will be given a dropdown menu asking what "This Assignment Is" in Canvas:

1. Under the --Choose Assignment-- tab, the grader should select which category fits best for each assignment as well as the amount of points the assignment is worth
2. Selecting the Continue button will open a menu with all the students and new grades. Any changes in grades (i.e. updating grades) will be highlighted in red
3. Verify the grades are correct and click Save Changes to process the grades into Canvas and add/update the assignments in the Grades and Assignments tab

Getting First/Last Names into NbGrader DB

---

As of Spring 2024, we have created the update-nbgrader script which is accessible from dsmlp-login. It will allow you to import the first and last names of your users into the NbGrader DB alongside their usernames so that you can see them from the Formgrader UI.

1. While logged onto your course's grader account (this will only work on dsmlp-login), run update-nbgrader -c <course-id>. This will generate a CSV in your home directory that you can move to your course's workspace. Please verify the results of the CSV before moving on (such as with cat).
2. Then you need to start up a notebook in that course, where you can then run nbgrader db student import <course-id>-nbgrader-students.csv.

On dsmlp-login.ucsd.edu:

1. Login with a grader account for your course.
   • update-nbgrader -c <course_name> will generate a CSV in your course dir containing each of its usernames followed by their First and Last names.
     
     
2. You can then use workspace -l to find your course's location on the filesystem.
3. In this case, our course is TEST_NBGRADER, and we cd into the dir returned by workspace -l. We then mv the generated CSV into this folder so that we can access it in our JupyterHub pod later.
   
   

Inside of Course Environment/Notebook:

1. Launch a pod as a grader for your course. Once you've done this, spawn a terminal window (under File>New>Terminal) and proceed with the next command.
   
2. You can then use nbgrader db student import <CSV_NAME> in the CSV's dir to import students' names into nbgrader.
   
   
   
   
3. The Nbgrader formgrader should now reflect your changes: