Postman and httr2: baby steps

Learning to write R that talks to Canvas

Setup

Pictures: big and small

The big picture

I want to transact data with the Queensland University of Technology (QUT) Canvas Learning Management System (LMS) via the Canvas LMS representational state transfer (REST) web application programming interface (API).

But more, much more than this, I want my colleagues to be able to enjoy programmatic access to Canvas so that we can spend more time teaching and less time pointing and clicking at a Learning Management System to do repetitive tasks inefficiently.

The small picture

Yesterday, I was kicking ideas around with Dr Jake Bradford about how best to write code to interface with Canvas via its API.

• We figured that there had to be a way to make use of the extensive Canvas LMS API Documentation to automate interface creation.
• With some insightful Googling, Jake navigated to Postman “an API platform for building and using APIs”.
  • It looks very promising!
  • Generate code snippets from API requests looks like its exactly what we need:

Postman can convert an API request into a code snippet, and you can choose the programming language or framework. You can use this generated code snippet in your front-end applications.

Today I am going to give it a go, and with some urgency as we have a bunch of DSB100 marks and feedback to get back to students ASAP.

Getting started with Postman

Postman Beginner’s Course

This course will introduce you to Postman and is suited for beginners. You will learn how to build API requests with Postman, how to inspect responses and create workflows. Postman Beginner’s Course - API Testing

I sign up for a free account, log in,

• open up a workspace
• open up a new tab
• send a made up request: GET https://canvas.qut.edu.au/doc/api/api-docs.json
  • recieve a CORS Error: The request has been blocked because of the CORS policy
• I try to Learn more about troubleshooting API requests and see

If the Postman web app fails to send your request, you may be experiencing a cross-origin resource sharing (CORS) error. Make sure you’re using the best Postman Agent for your request.

I download and install the Desktop Postman Agent, then switch to that agent in the Postman window in Chrome and…

• Blow me down!
  • GET https://canvas.qut.edu.au/doc/api/api-docs.json returns a boatload of json
• Just for laughs, I click Visualise response
  • an AI writes and runs some Javascript (I think)
  • producing a formatted version of the response,
  • a table with two columns: API Path and Description

This is looking very good.

Venturing off on my own

The Beginners Course looks good, but I’m going to see what I can figure out on my own, including how to…

Set my domain and API key

• I read
  • Adding environment variables
  • Understanding variables
• I create a new environment called QUT Canvas, then create
  • a new default variable canvas = https://canvas.qut.edu.au
  • a new secret variable token = <My secret API key>
• In the menu bar, there’s a dropdown menu under the top right corner that allows to set the current Environment
  • I set that to QUT Environment
  • then send the GET request {{canvas}}/doc/api/api-docs.json
  • which returns the desired json

Get info about my courses

• I’m going to try what I did with rcanvas::get_course_list()
• I look up List your courses in the Canvas Courses API
  • I send the query to GET {{canvas}}/api/v1/courses
    • It failes with the message "user authorisation required"
  • I select the Authorisation tab under the query
    • I set Type to Bearer Token and Token to {{token}}, the environment variable I set to <My secret API key>
  • I send the query again
    • Perfect!
  • The Visualize option lists my courses beautifully

Write code snippets in R

I have set the following environment variables

key	value
domain	https://canvas.qut.edu.au
token	<My secret API key>
api	api/v1
canvas	{{domain}}/{{api}}

• In Postman, I go to the query GET {{canvas}}/courses
  • Click on the </> icon at the right
  • Select R-httr as the code language and get the following code
    • I’ve obfuscated my actual secret API key

library(httr)

headers = c(
  'Authorization' = 'Bearer <My secret API key>'
)

res <- VERB("GET", url = "https://canvas.qut.edu.au/api/v1/courses", add_headers(headers))

cat(content(res, 'text'))

The code (appears to) runs perfectly and return the desired json.

• I’d like to wrap that in a function, but first, I need some help with…

Authorization()

my_CanvasAPI_key <- function() Sys.getenv("QUT_CANVAS_KEY")
Authorization    <- function()  c('Authorization' = paste('Bearer', my_CanvasAPI_key))

get_courses()

get_courses <- function(){
  headers = c(Authorization())

  res <- VERB(
    "GET", 
    url = "https://canvas.qut.edu.au/api/v1/courses", 
    add_headers(headers)
  )
  
  cat(content(res, 'text'))
}

Here’s the result of get_courses piped through jsonlite::prettify…

get_courses() |> prettify()

[{"id":4089,"name":"DSB100_21se2  Fundamentals of Data Science","account_id":146,"uuid":"E3dkRXXiASbNn6xANbiwNAQ...

...enrollments_to_course_dates":false}]Error: parse error: premature EOF
                                       
                     (right here) ------^

Hmmm… maybe this requires some pagination: > Requests that return multiple items will be paginated to 10 items by default. You can set a custom per-page amount with the ?per_page parameter.

• Before I dive down the pagination rabbit hole, I add the per_page parameter to the query in Postman and get:

get_courses <- function(){
  headers = c(Authorization())

  res <- VERB(
    "GET", url = "https://canvas.qut.edu.au/api/v1/courses?per_page=200",
    add_headers(headers)
  )
  
  cat(content(res, 'text'))
}

…but that still chokes on an end of file.

• Others have encountered this issue
  • Error parsing JSON file with the jsonlite package explains that response is streaming JSON, and explains how to parse it.

This looks like the kind of thing httr2 is meant to assist with.

Biting the httr2 bullet

Get info about my courses

Let’s try following this httr2 vignette on Wrapping APIs

# Build the request
req <- request("https://canvas.qut.edu.au/api/v1") |>
  req_url_path_append("courses") |>
  req_auth_bearer_token(my_CanvasAPI_key()) 

# Give it a dry run
req |> req_dry_run()

GET /api/v1/courses HTTP/1.1
Host: canvas.qut.edu.au
User-Agent: httr2/0.2.3 r-curl/5.1.0 libcurl/8.3.0
Accept: */*
Accept-Encoding: deflate, gzip
Authorization: <REDACTED>

# Perform it and save the response
req  |> req_perform()    -> resp
resp |> resp_body_json() -> json

The body of the response is lengthy:

json |> toJSON(pretty = TRUE) |> str_length()

[1] 16799

…so here’s a taste of that content

json |> toJSON(pretty = TRUE) |> str_sub(start=1, end=100) |> cat()

[
  {
    "id": [4089],
    "name": ["DSB100_21se2  Fundamentals of Data Science"],
    "account_id"

…and here’s the response status:

resp |> resp_status()

[1] 200

resp |> resp_status_desc()

[1] "OK"

tibble(
  course_id   = map_int(json, "id"),
  course_code = map_chr(json, "course_code"),
  name        = map_chr(json, "name")
)

# A tibble: 10 × 3
   course_id course_code    name                                        
       <int> <chr>          <chr>                                       
 1      4089 DSB100_21se2   DSB100_21se2  Fundamentals of Data Science  
 2     12878 DSB100_22se2   DSB100_22se2  Fundamentals of Data Science  
 3     14567 DSB100_23se2   DSB100_23se2 Fundamentals of Data Science   
 4      2792 EGH400-1_23se1 EGH400-1_23se1 Research Project 1           
 5     14584 EGH400-1_23se2 EGH400-1_23se2 Research Project 1           
 6      2796 EGH400-2_23se1 EGH400-2_23se1 Research Project 2           
 7     14585 EGH400-2_23se2 EGH400-2_23se2 Research Project 2           
 8     14587 EGH408_23se2   EGH408_23se2 Research Project               
 9     14824 IFN657_23se2   IFN657_23se2 Principles of Software Security
10      2927 IFN703_23se1   IFN703_23se1 Advanced Project               

But wait! Is there more?

Also, we can check to see whether there are additional pages of items to be returned. (See Canvas API Pagination, Parse link URL from a response.) In this case, the response from Canvas (resp) contains a link to the url to get the next set of items:

resp |> resp_link_url("next")

[1] "https://canvas.qut.edu.au/api/v1/courses?page=2&per_page=10"

If we request a larger number of items per page…

resp.100 <-
request("https://canvas.qut.edu.au/api/v1") |>
  req_url_path_append("courses") |>
  req_auth_bearer_token(my_CanvasAPI_key()) |>
  req_url_query(per_page = 100) |>
  req_perform()

…all available items can be delivered in the one response, and there is no “next” url:

resp.100 |> resp_link_url("next")

NULL

Getting httr2 to compile paginated responses

The developer of httr2 has thoughtfully provided ways to compile paginated responses… I just have to figure them out. I am going to try to

• compile paginated reponses to a request for course information that returns 10 items at a time
• compare that compilation to a response to a request that returns all items at once

request("https://canvas.qut.edu.au/api/v1") |>
  req_url_path_append("courses") |>
  req_auth_bearer_token(my_CanvasAPI_key()) |>
  req_url_query(per_page = 10) |>
  httr2:::req_paginate_next_url()

A: QUT’s Canvas API documentation as json

• Canvas API
• Submissions API