RMarkdown

RMarkdown produces high quality documents, reports, presentations and dashboards that are fully reproducible. RMarkdown weaves together narrative text and code from multiple languages (R, Python, SQL, etc.) to produce elegantly formatted static and dynamic outputs in HTML, PDF, MS Word, Beamer, HTML5 slides, Tufte-style handouts, books, dashboards, shiny applications, scientific articles, websites, and more. To install, run the following script in the R Console.

install.packages("rmarkdown") # Make dynamic documents
install.packages("RCurl") # For publishing markdown docs
install.packages("PKI") # For publishing markdown docs
install.packages("rsconnect") # For publishing markdown docs
install.packages("plotly") # For advanced visualizations
install.packages("shiny") # For interactive visualizations

LaTeX

LaTeX is a markup language mainly used to create technical or scientific articles, papers, reports, books or PhD thesis. Pandoc document converter and LaTeX markup language are required to create PDF outputs. To install, connect to the server and enter the following commands.

sudo apt-get install haskell-platform
sudo apt-get install texlive-full
sudo apt-get install texmaker

Troubleshooting

RStudio sessions sometimes stay hanging in a code execution loop. Clearing the workspace will stop the code from running even when restarting the session fails. Note that this will clear all you projects.

sudo rstudio-server active-sessions
sudo rstudio-server force-suspend-session <PID>
sudo rstudio-server active-sessions
su - rstudio  # requires password
whoami
sudo rm -r ~/.rstudio # requires password
exit # user logout

Troubleshooting

To stop a running server, get the Process Identification Number (PID) and then kill the PID.

netstat -tulpn # PID/pyhton will be listed
kill <PID>

If the error conda: command not found or something similiar is encountered.

# TEMPORARY SOLUTION
export PATH=~/anaconda3/bin:$PATH

Docker Cloud on AWS

Docker for AWS is installed with a CloudFormation template that configures Docker in swarm mode, running on instances backed by custom AMIs. It is the easiest way to get started, and requires the least amount of work. All you need to do is run the CloudFormation template, answer “Which SSH key to use?” in the Specify Details section, check the acknowledgment in the Review section, and you are good to go. The CloudFormation template will create everything that you need from scratch: a new VPC, subnets, gateways, and everything else needed in order to run Docker for AWS. Allowing Docker for AWS to create the VPC allows Docker to optimize the environment.

Beware of AWS Free Tier limits. The free tier allotment for Linux and Microsoft Windows EC2 instances is counted separately; you can run 750 hours of a Linux t2.micro or t1.micro instance plus 750 hours of a Windows t2.micro or t1.micro instance each month for the first 12 months. Therefore, launching a swarm with anything more than one manager in the swarm will not fall within the Free Tier. There are also several surprise charges that will accrue if the instance does not meet the Free Tier limits exactly.

Docker Cloud on GCP

There are several options for installing Docker on GCP instances which support nested virtualization. Google Container Engine, powered by Kubernetes, supports the common Docker container format. Compute Engine provides several public VM images that you can use to create instances and run your container workloads. Some of these public VM images have a minimalistic container-optimized operating system that includes newer versions of Docker, rkt, or Kubernetes preinstalled. When you run container workloads on Compute Engine, you have the freedom to employ whatever container technologies and orchestration tools that you need. The below installs Docker on a Ubuntu Zesty 17.04 instance using the Docker repository.

sudo apt-get remove docker docker-engine docker.io
sudo apt-get update
sudo apt-get install apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

(Optional) Verify key has fingerprint 9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88.

sudo apt-key fingerprint 0EBFCD88

Continue the installation process.

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update
sudo apt-get install docker-ce
sudo docker run hello-world

RStudio Server Image

RStudio Server enables you to provide a browser based interface to a version of R running on a remote Linux server. Once the download is finished RStudio-Server will launch invisibly. To connect to it, open web browser and navigate to http://public_dns_name:8787 where public_dns_name is the IPv4 Public IP described above. You will see the RStudio welcome screen. Log in using “rstudio” as the username and password.

sudo docker run -d -p 8787:8787 -e ROOT=TRUE -e USER=rstudio -e PASSWORD=rstudio rocker/rstudio

Anaconda Python Image

Anaconda is the leading open data science platform powered by Python. The open source version of Anaconda is a high performance distribution and includes over 100 of the most popular Python packages for data science. Additionally, it provides access to over 720 Python and R packages that can easily be installed using the conda dependency and environment manager, which is included in Anaconda. You can download and run this image using the following commands:

docker pull continuumio/anaconda3
docker run -i -t continuumio/anaconda3 /bin/bash

Alternatively, you can start a Jupyter Notebook server and interact with Anaconda via your browser:

docker run -i -t -p 8888:8888 continuumio/anaconda3 /bin/bash -c "/opt/conda/bin/conda install jupyter -y --quiet && mkdir /opt/notebooks && /opt/conda/bin/jupyter notebook --notebook-dir=/opt/notebooks --ip='*' --port=8888 --no-browser"

You can then view the Jupyter Notebook by opening http://public_dns_name:8888 in your browser, where public_dns_name is either the AWS IPv4 Public IP described above, or the GCP External IP address for your VM Instance.

DataQuest Data Science Image

Log into Node using SSH (though PuTTy if necessary). Create directory, download Docker Data Science image, and set file permissions.

sudo mkdir -p /home/me/notebooks/
sudo docker run -d -p 8888:8888 -v /home/me/notebooks:/home/ds/notebooks dataquestio/python3-starter
sudo docker ps # to get <container_hash>
sudo docker exec -it <container_hash> bash
sudo chmod 777 /home/ds/notebooks/

Alternatively, permissions can be set from within Jupyter. Open web browser and navigate to http://public_dns_name:8888 where public_dns_name is either the AWS IPv4 Public IP described above, or the GCP External IP address for your VM Instance. This will open the Jupyter interface. Start a “New Terminal” and set file permissions.

sudo chmod 777 /home/ds/notebooks/

pip install --upgrade pip
touch requirements.txt
pip freeze > requirements.txt
pip install -U $(pip freeze | awk '{split($0, a, "=="); print a[1]}')

grep -r 'notebook==\|ipython==\|ipykernel==' requirements.txt

pip install notebook==4.0.6 ipython==4.0.0 ipykernel==4.1.1
pip freeze > requirements.txt

Cloudera Hadoop QuickStart Image

Cloudera CDH (Cloudera’s Distribution Including Apache Hadoop) is to Hadoop, similar to what Anaconda is to Python. Hadoop (part of the Apache project) is a Java-based programming framework that supports the processing and storage of extremely large data sets in a distributed computing environment. A Hadoop Distributed File System (HDFS) stores data in multiple nodes (file systems). The data is accessed through a MapReduce framework that splits the input data set into independent chunks and processes the chunks in a completely parallel manner. The outputs of the maps (parallel outputs) are then sorted, which are then input to the reduce tasks. Hive (for structured data) and Pig (for semi-structured data) can be used to query Hadoop data through MapReduce using SQL. Impala is native to Hadoop and allows users to query Hadoop data directly using SQL. Apache Sqoop is a tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as relational databases. Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. Hue is a web user interface (UI).

sudo docker pull cloudera/quickstart:latest
sudo docker run --hostname=quickstart.cloudera --privileged=true -t -i -d -p 8888:8888 -p 7180:7180 cloudera/quickstart /usr/bin/docker-quickstart /home/cloudera/cloudera-manager --express
sudo docker ps # to get <container_hash>
sudo docker attach <container_hash>

Hue is a web user interface (UI) for Hadoop. After installation you should therefore be able access the server by navigating a web browser http://public_dns_name:8888. Cloudera Manager is an application for managing CDH clusters. After installation you should therefore be able access the server by navigating a web browser http://public_dns_name:7180. For both applications, public_dns_name is either the AWS IPv4 Public IP described above, or the GCP External IP address for your VM Instance. Both Hue and Cloudera Manager will prompt for a username and password, which are both “cloudera” by default.

Note: The QuickStart Docker Container image is no longer updated or maintained. Also, a container dies when you exit the shell, but you can disconnect and leave the container running by typing Ctrl+p followed by Ctrl+q.

Spark, Mesos, Jupyter, Python Image

Apache Spark, an open source framework that combines an engine for distributing programs across clusters of machines with an elegant model for writing programs atop it. Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications or frameworks. With Mesos clusters, any version of Spark drivers and executors can run in Docker containers. Using a Jupyter Notebook makes it easy to write programs to access the Spark clusters. Finally, Python (in the form PySpark) has less of a learning curve and is considered easier to use, less verbose, and more readable than Scala (Spark’s native language).

sudo docker run -d -p 8888:8888 --user root -e GRANT_SUDO=yes jupyter/pyspark-notebook start-notebook.sh --NotebookApp.token=''
sudo docker ps # to get <container_hash>
sudo docker exec -it <container_hash> bash
pip install pyspark --upgrade
pip install findspark --upgrade

Declaring -d will run the container in “detached” mode in the background in lieu of the default foreground mode. This allows the user to continue using the command line. Failing to -d will require a Ctrl+C logout. Not using the ...token='' option will result in a token being assigned automatically and a message similar to the below:

Copy/paste this URL into your browser when you connect for the first time, to login with a token: http://localhost:8888/?token=…

To start using Spark through the Jupyter Notebook interface, open web browser and navigate to http://public_dns_name:8888 where public_dns_name is either the AWS IPv4 Public IP described above, or the GCP External IP address for your VM Instance.

cd ../../usr/local/spark
./bin/pyspark

Remove Exited Containers

sudo docker rm $(sudo docker ps -a -f status=exited -q)

Removing Docker Images

sudo docker images -a # get <image_id>
sudo docker rmi -f <image_id>