Big Data in R - Hadoop

Introduction of Hadoop

Hadoop is an open-source implementation of MapReduce that has gained a lot of traction in the data mining community. Hadoop is distributed with Hadoop Streaming, which allows map/reduce jobs to be written in any language including R

Introduction of Rhipe

RHIPE stands for R and Hadoop Integrated Programming Environment. As mentioned on http://www.datadr.org/, it means “in a moment” in Greek and is a merger of R and Hadoop. It was first developed by Saptarshi Guha for his PhD thesis in the Department of Statistics at Purdue University in 2012. Currently this is carried out by the Department of Statistics team at Purdue University and other active Google discussion groups.

Some Advantage of Rhipe package

• Incorporates an rhlapply function similar to the standard apply variants.
• Uses Google Protocol Buffers.

• Great flexibility in modifying Hadoop parameters.

• Tutorial: http://deltarho.org/docs-RHIPE/

A good example

Here is a good example using Rhipe