A quick flyby: Azure Cognitive Services from R

Nothing serious … for now.

A few days ago, I took the Microsoft Azure AI Fundamentals Certification. Honestly, I was really impressed with how one could implement and publish ML and AI workloads such as Classification, Regression, Computer Vision, Natural Language Processing, etc with almost no code 🤯.

Anyway being the hopeless Romantic I am, I found myself exploring the possibility of leveraging some of the customisable, pretrained models in Azure, through R, to do some cool (and eventually useful 😄) AI workloads.

So here we are: my first attempt in using the Microsoft Azure Cognitive Services from R.

Azure Cognitive Services with R

From the Azure Cognitive Services documentation:

Azure Cognitive Services are cloud-based services with REST APIs, client library SDKs, and user interfaces available to help you build cognitive intelligence into your applications. You can add cognitive features to your applications without having artificial intelligence (AI) or data science skills. Cognitive Services comprises various AI services that enable you to build cognitive solutions that can see, hear, speak, understand, and even make decisions.

In this quick walk-through, we’ll explore the Vision cognitive service, in particular: Computer Vision API. The Computer Vision API allows you to analyze content in images and videos.

Let’s get into all things Azure!

Create a Cognitive Services Resource

We’ll start by creating a Cognitive Services resource in our Azure subscription. If you don’t have an Azure subscription, you can create a free Azure account here: https://azure.microsoft.com/en-gb/

Note: If you already have a Cognitive Services resource, we are good to go. Otherwise, follow the steps below to create one.

1. Open the Azure portal at https://portal.azure.com, signing in with your Microsoft account.

2. Click the ＋Create a resource button, search for Cognitive Services, and create a Cognitive Services resource with the following settings:

• Subscription: Your Azure subscription.

• Resource group: Select or create a resource group with a unique name.

• Region: Choose any available region:

• Name: Enter a unique name.

• Pricing tier: S0

• I confirm I have read and understood the notices: Selected.

Just as a reference, the Cognitive Services resource used when writing this post had the following details (ignore the weird names):

Now with that out of the way, let make Azure, AzureR!

AzureR

AzureR is a family of packages that allow you to work with Azure from R. In this section, we’ll set up an endpoint that can be used to communicate with the features offered by the Azure Cognitive Services.

The AzureCognitive::cognitive_endpoint() function allows us to do just this and requires 3 main arguments:

• url: The URL of the endpoint

• service_type: The type (or kind) of service the endpoint will provide. Types of services include ComputerVision, Face, Text, etc. See ?AzureCognitive::cognitive_endpoint() for more details.

• key: A subscription key to authenticate with the endpoint.

These can be easily obtained from the Azure portal on the Keys and Endpoint page of your cognitive service resource as shown below:

An endpoint can be subsequently created as outlined:

• In the Azure portal, on the Keys and Endpoint page for your cognitive service resource, copy Key1 or Key 2 and the Endpoint for your resource and paste them in the code below (within the quotation marks):

# Load required packages
library(AzureRMR) # install.packages("AzureRMR")
library(AzureCognitive) # install.packages("AzureCognitive")

# Create Computer Vision endpoint
endp_cv <- cognitive_endpoint(
  url = "____",
  service_type = "ComputerVision",
  key = "____")

We can of course decide to take the R pill and resurrect the above endpoints as follows:

Note: You will have to modify the subscription name, resource group and cognitive service names to match the ones on the Cognitive Services resource you created earlier.

# Load required packages
library(tidyverse)
library(AzureRMR) # install.packages("AzureRMR")
library(AzureCognitive) # install.packages("AzureCognitive")

# Login to Azure Resource Manager
# uses a web browser and access token to sign in
az <- create_azure_login()

# Obtain subscription for which you have permissions
# to manage Azure resources
sub <- az$get_subscription_by_name("Visual Studio Enterprise Subscription")

# Access the resource group associated with 
# the cognitive services you created earlier
rg <- sub$get_resource_group("keras")

# Retrieve cognitive service
cogsvc <- rg$get_cognitive_service("faciem")

# Obtain Subscription keys
keys <- cogsvc$list_keys()
key_1 <- keys %>%
  as_tibble() %>% 
  pull(value) %>%
  pluck(1)

key_2 <- keys %>% 
  as_tibble() %>%
  pull(value) %>% 
  pluck(2)


## Create Azure cognitive services endpoints ##

# Create Computer Vision endpoint
endp_cv <- cognitive_endpoint(
  url = "https://faciem.cognitiveservices.azure.com/",
  service_type = "ComputerVision",
  key = key_1)
endp_cv

# Create Face endpoint
endp_face <- cognitive_endpoint(
  url = "https://faciem.cognitiveservices.azure.com/",
  service_type = "Face",
  key = key_1)
endp_face

## Azure Cognitive Service endpoint
## Service type: computervision_endpoint 
## Endpoint URL: https://faciem.cognitiveservices.azure.com/vision/v2.0

Now that the endpoint is set up, let’s have some fun!

Computer vision: Does this look like … Hadley 🤔 ?

The Computer Vision API provides us with access to advanced algorithms for processing images and returning information. There are a whole lot of cool things you could do with this API, from extracting handwritten text from images, generating a description of an entire image in human-readable language to measuring compliance with social distancing guidelines!

In this post, we’ll make an API call to analyze an image of Hadley Wickham. Chances are you already know him or have used some of the data science tools he builds such as ggplot2 and dplyr.

Hadley Wickham: Source https://twitter.com/hadleywickham

With the computer vision endpoint set, we just have to make an API call and … done! Let’s take a moment and explain some of arguments in the API call below:

• operation = "analyze": specifies the operation to be performed on a particular image. Depending on the API, examples of operations include analyze, detect, and Optical Character Recognition.

• details = "celebrities": identifies celebrities if detected in the image.

• visualFeatures = "adult,faces,description,tags": A string indicating what visual feature types to return. For instance, Description describes the image content with a complete sentence in supported languages.

Please see the Computer Vision APIfor more details and examples about different operations and features within the computer vision API.

# Specify image link
image_link <- "https://pbs.twimg.com/profile_images/905186381995147264/7zKAG5sY_400x400.jpg"

# Call the computer vision endpoint
call_cognitive_endpoint(
  endpoint = endp_cv,
  operation = "analyze",
  body = list(url = image_link),
  options = list(
    # Request parameters
    details = "celebrities",
    visualFeatures = "adult,faces,description,tags"
  ),
  http_verb = "POST"
)

## $categories
## $categories[[1]]
## $categories[[1]]$name
## [1] "people_portrait"
## 
## $categories[[1]]$score
## [1] 0.7695312
## 
## $categories[[1]]$detail
## $categories[[1]]$detail$celebrities
## $categories[[1]]$detail$celebrities[[1]]
## $categories[[1]]$detail$celebrities[[1]]$name
## [1] "Hadley Wickham"
## 
## $categories[[1]]$detail$celebrities[[1]]$confidence
## [1] 0.9999046
## 
## $categories[[1]]$detail$celebrities[[1]]$faceRectangle
## $categories[[1]]$detail$celebrities[[1]]$faceRectangle$left
## [1] 117
## 
## $categories[[1]]$detail$celebrities[[1]]$faceRectangle$top
## [1] 109
## 
## $categories[[1]]$detail$celebrities[[1]]$faceRectangle$width
## [1] 168
## 
## $categories[[1]]$detail$celebrities[[1]]$faceRectangle$height
## [1] 168
## 
## 
## 
## 
## 
## 
## 
## $adult
## $adult$isAdultContent
## [1] FALSE
## 
## $adult$isRacyContent
## [1] FALSE
## 
## $adult$adultScore
## [1] 0.01674674
## 
## $adult$racyScore
## [1] 0.01742559
## 
## 
## $tags
## $tags[[1]]
## $tags[[1]]$name
## [1] "man"
## 
## $tags[[1]]$confidence
## [1] 0.999713
## 
## 
## $tags[[2]]
## $tags[[2]]$name
## [1] "necktie"
## 
## $tags[[2]]$confidence
## [1] 0.9997081
## 
## 
## $tags[[3]]
## $tags[[3]]$name
## [1] "person"
## 
## $tags[[3]]$confidence
## [1] 0.999671
## 
## 
## $tags[[4]]
## $tags[[4]]$name
## [1] "human face"
## 
## $tags[[4]]$confidence
## [1] 0.9850532
## 
## 
## $tags[[5]]
## $tags[[5]]$name
## [1] "bow"
## 
## $tags[[5]]$confidence
## [1] 0.9765703
## 
## 
## $tags[[6]]
## $tags[[6]]$name
## [1] "wearing"
## 
## $tags[[6]]$confidence
## [1] 0.9742473
## 
## 
## $tags[[7]]
## $tags[[7]]$name
## [1] "smile"
## 
## $tags[[7]]$confidence
## [1] 0.9727616
## 
## 
## $tags[[8]]
## $tags[[8]]$name
## [1] "clothing"
## 
## $tags[[8]]$confidence
## [1] 0.9532967
## 
## 
## $tags[[9]]
## $tags[[9]]$name
## [1] "posing"
## 
## $tags[[9]]$confidence
## [1] 0.8997302
## 
## 
## $tags[[10]]
## $tags[[10]]$name
## [1] "suit"
## 
## $tags[[10]]$confidence
## [1] 0.8962718
## 
## 
## $tags[[11]]
## $tags[[11]]$name
## [1] "indoor"
## 
## $tags[[11]]$confidence
## [1] 0.8934997
## 
## 
## $tags[[12]]
## $tags[[12]]$name
## [1] "tie"
## 
## $tags[[12]]$confidence
## [1] 0.8858114
## 
## 
## $tags[[13]]
## $tags[[13]]$name
## [1] "smiling"
## 
## $tags[[13]]$confidence
## [1] 0.879541
## 
## 
## $tags[[14]]
## $tags[[14]]$name
## [1] "headshot"
## 
## $tags[[14]]$confidence
## [1] 0.7387407
## 
## 
## $tags[[15]]
## $tags[[15]]$name
## [1] "shirt"
## 
## $tags[[15]]$confidence
## [1] 0.7014446
## 
## 
## $tags[[16]]
## $tags[[16]]$name
## [1] "beard"
## 
## $tags[[16]]$confidence
## [1] 0.6043768
## 
## 
## $tags[[17]]
## $tags[[17]]$name
## [1] "purple"
## 
## $tags[[17]]$confidence
## [1] 0.5996998
## 
## 
## $tags[[18]]
## $tags[[18]]$name
## [1] "portrait"
## 
## $tags[[18]]$confidence
## [1] 0.5956337
## 
## 
## $tags[[19]]
## $tags[[19]]$name
## [1] "jacket"
## 
## $tags[[19]]$confidence
## [1] 0.5485496
## 
## 
## $tags[[20]]
## $tags[[20]]$name
## [1] "forehead"
## 
## $tags[[20]]$confidence
## [1] 0.5137761
## 
## 
## $tags[[21]]
## $tags[[21]]$name
## [1] "dressed"
## 
## $tags[[21]]$confidence
## [1] 0.3506995
## 
## 
## 
## $description
## $description$tags
## $description$tags[[1]]
## [1] "man"
## 
## $description$tags[[2]]
## [1] "necktie"
## 
## $description$tags[[3]]
## [1] "person"
## 
## $description$tags[[4]]
## [1] "bow"
## 
## $description$tags[[5]]
## [1] "wearing"
## 
## $description$tags[[6]]
## [1] "clothing"
## 
## $description$tags[[7]]
## [1] "posing"
## 
## $description$tags[[8]]
## [1] "suit"
## 
## $description$tags[[9]]
## [1] "indoor"
## 
## $description$tags[[10]]
## [1] "smiling"
## 
## $description$tags[[11]]
## [1] "shirt"
## 
## $description$tags[[12]]
## [1] "photo"
## 
## $description$tags[[13]]
## [1] "camera"
## 
## $description$tags[[14]]
## [1] "purple"
## 
## $description$tags[[15]]
## [1] "dress"
## 
## $description$tags[[16]]
## [1] "jacket"
## 
## $description$tags[[17]]
## [1] "dressed"
## 
## $description$tags[[18]]
## [1] "standing"
## 
## $description$tags[[19]]
## [1] "neck"
## 
## $description$tags[[20]]
## [1] "glasses"
## 
## $description$tags[[21]]
## [1] "gray"
## 
## $description$tags[[22]]
## [1] "green"
## 
## $description$tags[[23]]
## [1] "old"
## 
## $description$tags[[24]]
## [1] "blue"
## 
## 
## $description$captions
## $description$captions[[1]]
## $description$captions[[1]]$text
## [1] "Hadley Wickham wearing a suit and tie smiling at the camera"
## 
## $description$captions[[1]]$confidence
## [1] 0.9901859
## 
## 
## 
## 
## $faces
## $faces[[1]]
## $faces[[1]]$age
## [1] 34
## 
## $faces[[1]]$gender
## [1] "Male"
## 
## $faces[[1]]$faceRectangle
## $faces[[1]]$faceRectangle$left
## [1] 117
## 
## $faces[[1]]$faceRectangle$top
## [1] 109
## 
## $faces[[1]]$faceRectangle$width
## [1] 168
## 
## $faces[[1]]$faceRectangle$height
## [1] 168
## 
## 
## 
## 
## $requestId
## [1] "4dd756aa-83a5-4d83-9b4b-98287d65bf11"
## 
## $metadata
## $metadata$height
## [1] 400
## 
## $metadata$width
## [1] 400
## 
## $metadata$format
## [1] "Jpeg"

There goes the response of the API call. Not bad, the suit part though… 😄! Probably this indicates that the model associates a tie with a suit.

Let’s put some of the responses into a tibble and display them as a gt table object.

library(gt)
cv_response <- tibble(
  request_parameter = c("categories", "tags", "celebrity_details", "tags", "tags",
"isAdultContent", "isRacyContent", "faces", "description"),
  
  response = c("people_portrait", "human face", "Hadley Wickham", "necktie", "beard",
"FALSE", "FALSE", "estimated age",
"Hadley Wickham wearing a suit and tie smiling at the camera"),
  
  score = c(0.7695312, 0.9850532,
            0.999904632568359, 0.9997081, 0.6043768,
            0.01674674, 0.01742559, 34, 0.9901859) 
)

cv_response %>% 
  gt() %>% 
  tab_header(
    title = md(
    "<img src = 'https://pbs.twimg.com/profile_images/905186381995147264/7zKAG5sY_400x400.jpg'
align = 'center' style = 'height:39px'>
Hadley Wickham wearing a suit and tie smiling at the camera "),
    subtitle = md("Computer Vision API reponse")) %>% 
  cols_align(
    align = "left",
    columns = 1) %>% 
    cols_align(
      align = "center",
      columns = 2) %>% 
    cols_align(
      align = "right",
      columns = 3)

Hadley Wickham wearing a suit and tie smiling at the camera
Computer Vision API reponse
request_parameter	response	score
categories	people_portrait	0.76953120
tags	human face	0.98505320
celebrity_details	Hadley Wickham	0.99990463
tags	necktie	0.99970810
tags	beard	0.60437680
isAdultContent	FALSE	0.01674674
isRacyContent	FALSE	0.01742559
faces	estimated age	34.00000000
description	Hadley Wickham wearing a suit and tie smiling at the camera	0.99018590

Much better! We’ll wrap up this short walk-through at that. We may have only explored a small section of the Computer Vision API (in particular Image Analysis) but this can be extended to other operations such as Optical character recognition or even other cognitive services such as Speech. This means you can even do the learning labs associated with the Microsoft Azure AI Fundamentals Certification in R! Hopefully we’ll get to R some of them soon 😎!

Thanks for reading!

Be sure to check out great blogs, tutorials and other formats of R resources coming out every day at RWeekly.org!

Further reading

• Microsoft Azure AI Fundamentals learning paths

• AzureR: a family of packages for working with Azure from R.

Happy leaRning,

Eric, Gold Microsoft Learn Student Ambassador.