AWS Big Data Blog

Installing Apache Spark on an Amazon EMR Cluster

Jonathan Fritz is a Senior Product Manager for Amazon EMR ———————– Please note – Amazon EMR now officially supports Spark. For more information about Spark on EMR, visit the Spark on Amazon EMR page or read Intent Media’s guest post on the AWS Big Data Blog about Spark on EMR. ——–————— Over the last five […]

Read More

Deploying Cloudera’s Enterprise Data Hub on AWS

Karthik Krishnan is an AWS Solutions Architect UPDATE April 6, 2015: The newest quickstart reference guide supports Cloudera Director 1.1.0. To manage your cluster with Cloudera Director 1.1.0, refer to the updated reference guide.  Apache Hadoop is an open-source software framework to store and process large scale data-sets.  In this post, we discuss the deployment of […]

Read More

Using Attunity CloudBeam at UMUC to Replicate Data to Amazon RDS and Amazon Redshift

Matt Yanchyshyn is a Principal Solutions Architect at AWS. Brad Helicher, Director of Cloud Business at Attunity, also contributed to this post. Attunity is an APN Big Data Competency Partner. Introduction University of Maryland University College’s mission is to provide a quality education at an affordable cost to busy professionals, mainly adults who are juggling […]

Read More

Ensuring Consistency When Using Amazon S3 and Amazon Elastic MapReduce for ETL Workflows

Jonathan Fritz is a Senior Product Manager for Amazon EMR. AWS Solutions Architect Manjeet Chayel also contributed to this post. The EMR File System (EMRFS) is an implementation of HDFS that allows Amazon Elastic MapReduce (Amazon EMR) clusters to store data on Amazon Simple Storage Service (Amazon S3). Many Amazon EMR customers use it to […]

Read More

Statistical Analysis with Open-Source R and RStudio on Amazon EMR

Markus Schmidberger is a Senior Big Data Consultant for AWS Professional Services Big Data is on every CIO’s mind. It is synonymous with technologies like Hadoop and the ‘NoSQL’ class of databases. Another technology shaking things up in Big Data is R. This blog post describes how to set up R, RHadoop packages and RStudio […]

Read More

Using Amazon EMR with SQL Workbench and other BI Tools

This is a guest post by Kyle Porter, a Sales Engineer at Simba Technologies. Jon Einkauf, a Senior Product Manager for Amazon Elastic MapReduce and AWS Senior Technical Writer Jeff Slone also contributed to this post. —————- Note: Ports have changed on EMR 4.x,. Before walking through this post, please consult the EMR documentation to […]

Read More

Using Amazon EMR and Tableau to Analyze and Visualize Data

Rahul Bhartia is an AWS Solutions Architect Introduction Hadoop provides a great ecosystem of tools for extracting value from data in various formats and sizes. Originally focused on large-batch processing with tools like MapReduce, Pig and Hive, Hadoop now provides many tools for running interactive queries on your data, such as Impala, Drill, and Presto. […]

Read More

Using Amazon Redshift to Analyze Your Elastic Load Balancer Traffic Logs

Biff Gaut is a Solutions Architect with AWS Introduction With the introduction of Elastic Load Balancing (ELB) access logs, administrators have a tremendous amount of data describing all traffic through their ELB. While Amazon Elastic MapReduce (Amazon EMR) and some partner tools are excellent solutions for ongoing, extensive analysis of this traffic, they can require […]

Read More

Getting Started with Amazon EMR Bootstrap Actions

Steve McPherson is a Senior Manager for Amazon Elastic MapReduce Note: This post was updated 2/8/16. The Presto bootstrap action documented in the original post has been deprecated because EMR now offers a Presto-Sandbox as a full-fledged EMR application. For details, see the EMR sandbox.   Amazon Elastic MapReduce (EMR) is a fully managed Hadoop-as-a-service platform […]

Read More

Using AWS for Multi-instance, Multi-part Uploads

James Saull is a Principal Solutions Architect with AWS There are many advantages to using multi-part, multi-instance uploads for large files. First, the throughput is improved because you can upload parts in parallel. Amazon Simple Storage Service (Amazon S3) can store files up to 5TB, yet a single machine with a 1Gbps interface would take […]

Read More