Analyzing Big Data With Aws: Fill & Download for Free

If your goal is to learn about OS, tools, install process then Google “Apache Hadoop” and go from there. There are tons of how to setup Hadoop blogs. Cloudera, MapR, & Hortonworks have packaged distributions that *might* work on your old computer. You'll burn up a lot of time though learning about systems administration.If you really want to get down to analyzing data quickly then take a look at Amazon Web Services and Elastic MapReduce. Amazon and Think Big Analytics offer hands on training courses for the tools and techniques of analyzing Big Data that will get you productive in 3 days.Using the AWS console, you can spin up a cluster of 1-20 nodes and within 5 minutes start playing with Hive, Pig, Hadoop Streaming with R & Python.You'll also want s3cmd.

"Big Data" = less-structured data in the hundreds of terabytes to petabytes. It's scads of data that hasn't been modeled up front to go neatly into tables where it could be conventionally analyzed via a relational database. That data might include social media content, for example, any kind of digitized content, or numerical data that hasn't been prepped for analysis.A cost-effective way to analyze large quantities of any kind of less-structured data was pioneered by Google in the 1990s and early 2000s and later cloned by Yahoo and others as the Apache Hadoop stack. Hadoop uses commodity clusters (i.e., farms of off-the-shelf, Intel-based servers). The data is distributed to each of these servers where it is processed locally at each server node. Historically this processing has been done in batch mode, but Apache Spark now makes near real-time analysis via microbatches possible.These days, you can even use a Raspberry Pi cluster as a Hadoop cluster: Raspberry PI Hadoop Cluster - Jonas WidrikssonBut the easiest way to get started is without an investment in additional hardware, via a cloud service such as Amazon Web Services, Google Compute Cloud, Microsoft Azure. Cloud services are set up to make using a stack such as Hadoop easier.At the base of the Hadoop stack is HDFS, the Hadoop Distributed File System. HDFS provides a useful starting point for a data lake, a single repository for large quantities and varieties of data which we wrote about in 2014: Data lakes and the promise of unsiloed data. See also the interview we did with Mike Lang: Making Hadoop suitable for enterprise data scienceFor more on how large enterprises have been using Hadoop, see Making sense of Big Data.If a data lake sounds like overkill, another more tactical way to analyze Big Data begins with NoSQL or non-relational databases designed for cluster or distributed computing, many of which are open source. See Remapping the database landscape. Some NoSQL databases such as HBase are designed to be used in conjunction with Hadoop, but with the downside of additional complexity.NewSQL or scale-out relational databases can also be used, but you'll have to structure your data up front and adhere to fixed schemas or data descriptions to be able to use them. If your data doesn't change a lot, NewSQL would be workable too. Cloud services such as AWS, GCC or Azure also offer click-to-deploy NoSQL databases such as Mongo, Cassandra, Dynamo or DocumentDB, or click-to-deploy NewSQL should you choose to go that route.Here's a way to think about the various data models in NoSQL and NewSQL databases:It's good to match your particular use case to the most suitable data model. If you're ingesting lots of JSON or XML data objects, perhaps a document store would be best. If it's just skinny tables, a key-value or wide-column store might suit best. Hybrid databases include several different data models, but can be harder to use. You can find case study or use case examples in Remapping the database landscape. Graph databases offer new analytic power, but are somewhat less mature.

Amazon EMR is your go to service for that. Here is a good tutorial about Getting Started: Analyzing Big Data with Amazon EMR.