R on Cloud Architectures

Joint Meetup of R Users Group Philippines and AWS Users Group Philippines

2020 July 22

What Am I Currently Doing?

Rowen Remis R. Iral

IT Engineer / Data Science Architect

Architectures

Peritus Knowledge Services Corporation

Peritus

https://www.peritus-services.com

Tech Advisor / Digital Marketing

ZankPOS Enterprises POSNUC

ZankPOS

POSNUC

https://www.zankpos.com https://www.posnuc.com

Tech Advisor

B&B Capital Bull and Bear Capital

• Group of Traders

https://www.facebook.com/BB-Capital-111214880680174/

R on Cloud aRchitectures

The cloud is a useful environment for processing data and storing data.

AWS Cloud

AWS

• ec2 instances (servers)
• RDS for databases
• DynamoDB for streaming data
• EMR - big data storage like hadoop, R jobs

Amazon Web Services

https://aws.amazon.com

Faster Data Processing

Feather Data Format

optimized data format for storing data, lightweight and fast.

• R feather package

Apache Arrow

List of Parallel Processing Packages in R

We can use data store

• ff (basic statistics object)
• ffbase
• ffbase2

And processes in parallel with R

• parallel
• doParallel
• bigmemory
• biganalytics
• bigtabulate

Multi-core and Parallel Programming in R

We can use AWS cloud’s multiprocessor instances or setup some clusters

Parallel Programming in R

• parallel
• segue (uses AWS EMR or ec2 instances and s3 buckets)
• Rmpi (uses OpenMPI)
• pbdMPI (uses OpenMPI)
• snow
• SPRINT
• ROpenCL (CPU and GPU usage)

AWS Lumberyard

Has a snow library not for parallel computing in R but for game effects. Snow, rain and other effects are available if you are a game developer for integration with Twitch and AWS.

Note on Multiple CPUs and it’s Limitations

Divide and conquer has a certain limit depending on the process or computation used.

Amdahl’s Law

You can parallelize, but not all of steps, which will give you a certain minimum time to execute a job. More resources for faster processing.

Gustafson’s law

theoretical speedup in latency of the execution of a task at fixed execution time that can be expected of a system whose resources are improved. More resources for more details, as you can process objects as components of a larger object.

We can parallelize, but there are certain serial process that cannot be run in parallel.

R and Microsoft R Server (former RevoAnalytics R)

• XDF - split data type for storing large data sets
• RevoScaleR functions or the rx functions in Microsoft R - parallel and big data processing

Microsoft R

ContRolling your Server on the Cloud

R has its own package repository called CRAN (Comprehensive R Archive Network)

There is also a package that uses packages on github.

• devtools package

devtools

Genomics and bioinformatics packages

• bioconductor package

Galaxy Server – server for Genomics

bioconductor

R Open Science

• rOpenSci

ROpenSci

Cloud Packages

New cloud packages for R

cloudyr

• remoter package - ssh controls on aws
• cloudyr packages

CloudyR Packages for AWs

• awspack
• aws.s3
• aws.efs
• aws.glacier
• aws.ses
• aws.sns
• aws.sqs
• aws.ec2
• aws.ec2metadata
• aws.lambda
• aws.alexa
• aws.iam
• aws.cloudtrail
• aws.cloudwatch
• aws.kms
• aws.signature
• aws.comprehend
• aws.polly
• aws.transcribe
• aws.translate
• MTurkR
• MTurkRGUI
• aws.code

Securing your RStudio Server on AWS

SSH Tunneling

You only expose SSH port 22 and then tunnel through it where the port 8787 is not exposed publicly but only behind the SSH port 22 tunnel so you can secure the http traffic if https is not available

RStudio Server Security with VPN

VPN - Virtual Private network This also connects remote workers.

VPN keys are needed

Direct Access

Publicly Open Ports are SSH 22 and VPN UDP 1194

Direct Access no Web Server Access

With VPN Security through AWS Ec2 Instance

Can connect to Web Server through VPN

Web Access through VPN

Thank you

Rowen Remis R. Iral

• IT Engineer / Data Science Architect

http://wenup.wordpress.com