I’ve got a lot on my plate…

Introduction

This vignette walks through how to connect to a visual recognition API, and use its food recognition model to send a batch of image files, and retrieve information on the food present in those images.

Prerequisites

You’ll need to install packages (most are listed at the top of the vignette…some aren’t, which is bad practice). You’ll also need an API key.

This vignette is incomplete, in particular it doesn’t currently save the outputs in any kind of useful format.

Inspiration

Based on https://mran.microsoft.com/snapshot/2016-04-12/web/packages/clarifai/vignettes/using_clarifai.html

Nope, that doesn’t work

Base on http://pablobarbera.com/ECPR-SC104/code/08-clarifai-api.html

If you’d rather Python try https://github.com/robmarkcole/Useful-python/blob/master/Clarifai/Clarafai%207-8-2018.ipynb

In theory you could also use https://github.com/cloudyr/RoogleVision or https://github.com/flovv/Roxford to connect to google and microsoft image services respectively, (and no doubt something for Amazon and IBM’s services too), or hook into one of the specialist food APIs (if that’s your focus).

The code

Ok first we’re going to demo getting info via the API for a single image

Load the libraries

library(httr)

package 㤼㸱httr㤼㸲 was built under R version 3.5.3

We’re going to load the key file.

source("key_files/clarifai_key.R")
#This is a simple script that contains two lines. Save that script, e.g. in a subfolder key_files/   The 'source' line simply loads the keys.
#clarifai_key <- "string here"
#clarifai_id <- "string here"

Now we authenticate using that

#secret_id(c(clarifai_id, clarifai_key)) 
#get_token()  THIS ISNT WORKING IN CURRENT VERSION

#so we'll do it the other way...
apikey <- clarifai_key #just for easy of copying code from elsewhere we'll rename our variable

base_url <- paste0("https://api.clarifai.com/v2/models/","bd367be194cf45149e75f01d59f77ba7","/outputs") #you'll see the weird string, that's for the food one. You could send it to the face detection model instead a403429f2ddf4b49b307e318f00e528b 
#or the general model https://clarifai.com/models/general-image-recognition-model-aaa03c23b3724a16a56b629203edc62c

Form the requests we’re going to make. Note, this is the image we’re sending…to the food API

#We can either type the object in JSON as text (recommended in this case, given its complexity), or create it from within R as a list

requests <- '
  {
    "inputs": [
      {
        "data": {
          "image": {
            "url": "http://i.imgur.com/XmAr3jV.jpg"
          }
        }
      }
    ]
  }'

#or the list way:
req <- list("inputs" = list())
req$inputs[[1]] <- list(data=list(image=list(url = "http://i.imgur.com/XmAr3jV.jpg")))
requests <- rjson::toJSON(req)

And now let’s POST

r

Response [https://api.clarifai.com/v2/models/bd367be194cf45149e75f01d59f77ba7/outputs]
  Date: 2019-04-11 01:22
  Status: 200
  Content-Type: application/json; charset=UTF-8
  Size: 2.22 kB

Ok…so let’s do something with that?

for (result in r$outputs[[1]]$data$concepts){
    message('object: ', result$name, ' -- probability: ',
        result$value)
}

object: lamb -- probability: 0.98912746
object: duck -- probability: 0.84219635
object: tongue -- probability: 0.77590185
object: pork -- probability: 0.77310026
object: sweet -- probability: 0.6828375
object: chicken -- probability: 0.6412517
object: beef -- probability: 0.60754395
object: meat -- probability: 0.5252384
object: venison -- probability: 0.52096176
object: milk -- probability: 0.5084926
object: mutton -- probability: 0.4889681
object: truffle -- probability: 0.4644618
object: bird -- probability: 0.46390593
object: nibble -- probability: 0.45980784
object: ginger -- probability: 0.4556288
object: potato -- probability: 0.428442
object: veal -- probability: 0.40964532
object: grass -- probability: 0.38871378
object: curry -- probability: 0.3645004
object: steak -- probability: 0.35160452

Erm….you’ll see why sending things to the wrong model might be important?

Let’s batch that

Now we want to send a set of images.

First, read a directory of images (from the web or localy)

pic_list <- list.files("food", full.names = T, pattern = ".JPG")
#I would strongly recommend reducing the size of images before proceding!
#There are a bunch of approaches to doing that in R...

If your batch of images are on your machine (rather than on a public folder on the web), you’ll need to convert them to base64. If they’re on the web you can change the code below to use a variant of the url method above.

library(base64enc) #this is bad practice, it should really go at the top

package 㤼㸱base64enc㤼㸲 was built under R version 3.5.2

You can test if you want

#txt2 <- base64encode("C:/blah/blah/food/IMG_6234.JPG")  #give it the path to your image

#or the list way:
# req <- list("inputs" = list())
# req$inputs[[1]] <- list(data=list(image=list(base64 = txt2)))
# requests <- rjson::toJSON(req)
# 
# 
# r <- POST(base_url, 
#     add_headers(
#         "Authorization" = paste0("Key ",apikey),
#         "Content-Type" = "application/json"),
#     body = requests)
# r
# 
# r <- content(r, "parsed")
# 
# for (result in r$outputs[[1]]$data$concepts){
#     message('object: ', result$name, ' -- probability: ',
#         result$value)
# }

And then go through the same as above basically, two options (the second one is better)

Option one: Go through the list using lapply and send each one separately

whats_that_pic <- lapply(pic_list_base64, function(x){  #what we're doing here is going through the list of base64 images you created with a procedure. You could also create a function to do this.
  req <- list("inputs" = list())
  req$inputs[[1]] <- list(data=list(image=list(base64 = x)))  #'x' is the item in the list we're on
  requests <- rjson::toJSON(req)
  
  r <- POST(base_url, 
            add_headers(
              "Authorization" = paste0("Key ",apikey),
              "Content-Type" = "application/json"),
            body = requests)
  
  r <- content(r, "parsed")
  
  whats_that_pic2[length(whats_that_pic2)+1] <<- r
  
  for (result in r$outputs[[1]]$data$concepts){
    message('object: ', result$name, ' -- probability: ',
            result$value)
  }
}
)

number of items to replace is not a multiple of replacement lengthobject: truffle -- probability: 0.9994509
object: chocolate -- probability: 0.9974394
object: sweet -- probability: 0.9622581
object: candy -- probability: 0.94751525
object: cookie -- probability: 0.8903085
object: cake -- probability: 0.8745675
object: cream -- probability: 0.8616116
object: coffee -- probability: 0.7683241
object: milk -- probability: 0.76240355
object: mushroom -- probability: 0.7337792
object: potato -- probability: 0.69337285
object: pastry -- probability: 0.6607497
object: cocoa -- probability: 0.51976323
object: bread -- probability: 0.4994815
object: tuber -- probability: 0.44628826
object: dessert -- probability: 0.4180743
object: milk chocolate -- probability: 0.38415602
object: chips -- probability: 0.38273662
object: hash -- probability: 0.365521
object: dough -- probability: 0.35707122
number of items to replace is not a multiple of replacement lengthobject: cereal -- probability: 0.9909675
object: oatmeal -- probability: 0.97471833
object: granola -- probability: 0.9741415
object: muesli -- probability: 0.9622045
object: milk -- probability: 0.92820925
object: rice -- probability: 0.8862817
object: corn -- probability: 0.8678268
object: sweet -- probability: 0.80016446
object: wheat -- probability: 0.7716304
object: porridge -- probability: 0.73381054
object: cinnamon -- probability: 0.72017
object: oat -- probability: 0.6933167
object: yogurt -- probability: 0.6463264
object: coffee -- probability: 0.5903445
object: banana -- probability: 0.5899875
object: cornflakes -- probability: 0.57549465
object: chocolate -- probability: 0.5632621
object: raisin -- probability: 0.55438405
object: oatmeal cereal -- probability: 0.5541141
object: honey -- probability: 0.5110003
number of items to replace is not a multiple of replacement lengthobject: banana -- probability: 0.9990096
object: sweet -- probability: 0.8639921
object: cream -- probability: 0.8547184
object: dairy product -- probability: 0.8085454
object: chocolate -- probability: 0.75613284
object: strawberry -- probability: 0.7131227
object: milk -- probability: 0.70548236
object: peanut butter -- probability: 0.6488902
object: ice -- probability: 0.6332538
object: yogurt -- probability: 0.5758665
object: pudding -- probability: 0.56315506
object: ice cream -- probability: 0.5539286
object: peanut -- probability: 0.54354995
object: butter -- probability: 0.53937244
object: chips -- probability: 0.4856897
object: syrup -- probability: 0.4630544
object: honey -- probability: 0.43911088
object: berry -- probability: 0.4108279
object: tapioca -- probability: 0.39465758
object: dumpling -- probability: 0.34221533
number of items to replace is not a multiple of replacement lengthobject: apple -- probability: 0.96392536
object: sweet -- probability: 0.94115174
object: vegetable -- probability: 0.93822455
object: pumpkin -- probability: 0.8627503
object: pasture -- probability: 0.83511585
object: pear -- probability: 0.7554394
object: peach -- probability: 0.748327
object: melon -- probability: 0.734571
object: juice -- probability: 0.67565954
object: egg -- probability: 0.6658032
object: squash -- probability: 0.6176888
object: mango -- probability: 0.44059426
object: nectarine -- probability: 0.43110797
object: orange -- probability: 0.39466506
object: potato -- probability: 0.36459038
object: mochi -- probability: 0.35180628
object: cantaloupe -- probability: 0.3383726
object: pepper -- probability: 0.3102156
object: gourd -- probability: 0.29764217
object: persimmon -- probability: 0.29236302
number of items to replace is not a multiple of replacement lengthobject: pasta -- probability: 0.9907646
object: vegetable -- probability: 0.866422
object: noodle -- probability: 0.8506128
object: spaghetti -- probability: 0.8119204
object: sauce -- probability: 0.79027545
object: meat -- probability: 0.7394205
object: broth -- probability: 0.66396284
object: soup -- probability: 0.6317607
object: mushroom -- probability: 0.6186168
object: beef -- probability: 0.57490665
object: pea -- probability: 0.5742003
object: fettuccine -- probability: 0.5588813
object: tortellini -- probability: 0.5226992
object: herb -- probability: 0.5147809
object: legume -- probability: 0.50550234
object: macaroni -- probability: 0.4863832
object: pork -- probability: 0.46765995
object: soy -- probability: 0.45893747
object: dough -- probability: 0.38240322
object: ramen -- probability: 0.3741401
number of items to replace is not a multiple of replacement lengthobject: salad -- probability: 0.99725556
object: spinach -- probability: 0.99677837
object: vegetable -- probability: 0.99643826
object: herb -- probability: 0.9622885
object: lettuce -- probability: 0.9589621
object: arugula -- probability: 0.93880606
object: basil -- probability: 0.9101343
object: oil -- probability: 0.7469167
object: dandelion greens -- probability: 0.69676745
object: cheese -- probability: 0.6604413
object: watercress -- probability: 0.6502621
object: mizuna greens -- probability: 0.57539046
object: olive oil -- probability: 0.5512258
object: new zealand spinach -- probability: 0.5382747
object: miner's lettuce -- probability: 0.46362755
object: komatsuna -- probability: 0.42024606
object: summer purslane -- probability: 0.41810292
object: parmesan -- probability: 0.3799056
object: tatsoi -- probability: 0.3736297
object: yao choy -- probability: 0.32308856
number of items to replace is not a multiple of replacement lengthobject: cream -- probability: 0.9959474
object: yogurt -- probability: 0.9945166
object: milk -- probability: 0.991271
object: dairy -- probability: 0.9457755
object: dairy product -- probability: 0.9393966
object: sour cream -- probability: 0.91296995
object: white sauce -- probability: 0.8943993
object: garlic sauce -- probability: 0.8617635
object: blue cheese -- probability: 0.7913879
object: sweet -- probability: 0.7192614
object: cream cheese -- probability: 0.6670171
object: tzatziki -- probability: 0.6413661
object: cheese -- probability: 0.59471667
object: sauce -- probability: 0.58176863
object: vegetable -- probability: 0.56325936
object: curd -- probability: 0.47145405
object: salad -- probability: 0.42687216
object: coleslaw -- probability: 0.39927113
object: clam chowder -- probability: 0.39023077
object: dill -- probability: 0.37979758
number of items to replace is not a multiple of replacement lengthobject: chips -- probability: 0.90707266
object: chicken -- probability: 0.9052212
object: bacon -- probability: 0.9021294
object: sauce -- probability: 0.84960234
object: potato -- probability: 0.8406528
object: poutine -- probability: 0.8183321
object: cheese -- probability: 0.8039971
object: pork -- probability: 0.7929332
object: cream -- probability: 0.7462302
object: nachos -- probability: 0.72910583
object: beer -- probability: 0.6987153
object: beef -- probability: 0.61058277
object: corn -- probability: 0.6105454
object: steak -- probability: 0.60894316
object: crab -- probability: 0.60693395
object: sweet -- probability: 0.59788543
object: chili -- probability: 0.59686714
object: tacos -- probability: 0.5758448
object: lobster -- probability: 0.57215196
object: lamb -- probability: 0.549315
number of items to replace is not a multiple of replacement lengthobject: banana -- probability: 0.9302554
object: butter -- probability: 0.8768802
object: peanut -- probability: 0.85372376
object: chicken -- probability: 0.76398027
object: gnocchi -- probability: 0.7404165
object: peanut butter -- probability: 0.6769316
object: potato -- probability: 0.6560601
object: coconut -- probability: 0.65582573
object: rice -- probability: 0.6216823
object: cream -- probability: 0.5761461
object: oatmeal -- probability: 0.5478167
object: cheese -- probability: 0.53859895
object: chips -- probability: 0.5125718
object: sweet -- probability: 0.5103579
object: sauce -- probability: 0.50668883
object: garlic -- probability: 0.44809264
object: sausage -- probability: 0.44446173
object: chocolate -- probability: 0.4326467
object: cashew -- probability: 0.4278672
object: milk -- probability: 0.41721702

Option two: More sensible, write a single query that sends a bunch of images in batch https://clarifai.com/developer/guide/inputs#inputs

for (x in length(r$outputs)){
  for (result in r$outputs[[x]]$data$concepts){
    message('object: ', result$name, ' -- probability: ',
            result$value)
  }
}

object: banana -- probability: 0.9302554
object: butter -- probability: 0.8768802
object: peanut -- probability: 0.85372376
object: chicken -- probability: 0.76398027
object: gnocchi -- probability: 0.7404165
object: peanut butter -- probability: 0.6769316
object: potato -- probability: 0.6560601
object: coconut -- probability: 0.65582573
object: rice -- probability: 0.6216823
object: cream -- probability: 0.5761461
object: oatmeal -- probability: 0.5478167
object: cheese -- probability: 0.53859895
object: chips -- probability: 0.5125718
object: sweet -- probability: 0.5103579
object: sauce -- probability: 0.50668883
object: garlic -- probability: 0.44809264
object: sausage -- probability: 0.44446173
object: chocolate -- probability: 0.4326467
object: cashew -- probability: 0.4278672
object: milk -- probability: 0.41721702

How nutritious is my plate?

Ok, then let’s get the nutrition info for one of those images

Inspiration

https://cran.r-project.org/web/packages/NutrienTrackeR/index.html https://github.com/MLH/clarifai-food-nutrition-demo https://stories.mlh.io/watch-what-you-eat-with-clarifai-71185734bc61

Check: https://cran.r-project.org/web/packages/NutrienTrackeR/vignettes/NutrienTrackeR.html looks useful You could also use https://github.com/openfoodfacts/openfoodfacts-python

The code