Originally published at: https://tutorials.datasciencedojo.com/scale-r-to-big-data-using-hadoop-and-spark/
R is currently one of the most popular data science languages in the world. However, it’s always had constraints around scaling out to big data. What happens when you expand beyond a couple gigabytes of data? You packed up your data and you used something else; Python, Java, or Mahout to name a few. Now it’s possible to stick with R throughout your production analysis all the way to deployment, regardless of the data size.
Companies like Apache, Revolution Analytics, Microsoft, and H20 showed us this year that distributed computing in R is possible. Today we’ll take a look at how we can Scale R to Big Data Using Hadoop and Spark.
In this talk we will show you Microsoft R Server, which is a Hadoop or Spark cluster where R is installed on every computer and is equipped with distributed processing libraries to utilize each and every computer in parallel. We’ll show you how to run your normal native R code via SSH, and how to get an RStudio server up and running on the cluster.
We’ll show you how to wrangle data out of an HDFS and build machine learning models from your large dataset. Then show you how to pack up that model and deploy it to an elastically scaled web service so that anyone may call upon it for predictions and insights.
Outline:
· Setup a Spark cluster with R installed (R server).
· Wrangle data that is inside HDFS using R.
· Build and deploy a machine learning model using R.
Code and Prep Work (if you want to follow along) can be found here
(261)