AWS S3 Tutorial

Amazon S3 - Cloud Object Storage Tutorial

rm(list=ls())
require(aws.s3)
require(aws.signature)

A detailed description of how credentials can be specified is provided at the link below:

browseURL("https://github.com/cloudyr/aws.signature")

The easiest way is to simply set environment variables on the command line prior to starting R or via a .Renviron file, which is to set environment variables in R during startup (see ?Startup).

They can be also set within R:

Sys.setenv("AWS_ACCESS_KEY_ID" = "mykey",
            "AWS_SECRET_ACCESS_KEY" = "mysecretkey",
            "AWS_DEFAULT_REGION" = "us-east-1",
            "AWS_SESSION_TOKEN" = "mytoken")

Access S3 bucketlist

bucketlist()

DOWNLOAD FILES FROM s3

Note: AWS S3 is a flat file store, so there there are just “pseudo-folders” defined by object keys.

# First, we get the address to each json file in our chosen bucket.
files<-get_bucket(bucket = 'BUCKET-NAME', prefix="folder/subfolder/", max=Inf)

# Next, we loop over to save each file in our local machine.
keys <- c() # Empty list
for(i in 1:length(files)) {
  keys[i] <- files[[i]]$Key 
  # Extracting file name to save it with the same name
  save_object(keys[i], file=paste0("~/Downloads/", keysEN[i]), 
              bucket= "NAME-BUCKET", show_progress=F) # Saving each object in chosenfolder
}

SAVING FILES TO s3

# Function to write files from local to s3
writeToS3 = function(file,bucket,filename){
  aws.s3::s3write_using(file, FUN = write.csv,
                        bucket = bucket,
                        object = filename)
}

writeToS3(file, "bucket", "key/filename.csv")