Time for action – using Hadoop to calculate Pi
We will now use a sample Hadoop program to calculate the value of Pi. Right now, this is primarily to validate the installation and to show how quickly you can get a MapReduce job to execute. Assuming the HADOOP_HOME/bin directory is in your path, type the following commands:
$ Hadoop jar hadoop/hadoop-examples-1.0.4.jar pi 4 1000 Number of Maps = 4 Samples per Map = 1000 Wrote input for Map #0 Wrote input for Map #1 Wrote input for Map #2 Wrote input for Map #3 Starting Job 12/10/26 22:56:11 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 12/10/26 22:56:11 INFO mapred.FileInputFormat: Total input paths to process : 4 12/10/26 22:56:12 INFO mapred.JobClient: Running job: job_local_0001 12/10/26 22:56:12 INFO mapred.FileInputFormat: Total input paths to process : 4 12/10/26 22:56:12 INFO mapred.MapTask: numReduceTasks: 1 … 12/10/26 22:56:14 INFO mapred.JobClient: map 100% reduce 100% 12/10/26 22:56:14 INFO mapred.JobClient: Job complete: job_local_0001 12/10/26 22:56:14 INFO mapred.JobClient: Counters: 13 12/10/26 22:56:14 INFO mapred.JobClient: FileSystemCounters … Job Finished in 2.904 seconds Estimated value of Pi is 3.14000000000000000000 $
What just happened?
There's a lot of information here; even more so when you get the full output on your screen. For now, let's unpack the fundamentals and not worry about much of Hadoop's status output until later in the book. The first thing to clarify is some terminology; each Hadoop program runs as a job that creates multiple tasks to do its work.
Looking at the output, we see it is broadly split into three sections:
- The start up of the job
- The status as the job executes
- The output of the job
In our case, we can see the job creates four tasks to calculate Pi, and the overall job result will be the combination of these subresults. This pattern should sound familiar to the one we came across in Chapter 1, What It's All About; the model is used to split a larger job into smaller pieces and then bring together the results.
The majority of the output will appear as the job is being executed and provide status messages showing progress. On successful completion, the job will print out a number of counters and other statistics. The preceding example is actually unusual in that it is rare to see the result of a MapReduce job displayed on the console. This is not a limitation of Hadoop, but rather a consequence of the fact that jobs that process large data sets usually produce a significant amount of output data that isn't well suited to a simple echoing on the screen.
Congratulations on your first successful MapReduce job!
Three modes
In our desire to get something running on Hadoop, we sidestepped an important issue: in which mode should we run Hadoop? There are three possibilities that alter where the various Hadoop components execute. Recall that HDFS comprises a single NameNode that acts as the cluster coordinator and is the master for one or more DataNodes that store the data. For MapReduce, the JobTracker is the cluster master and it coordinates the work executed by one or more TaskTracker processes. The Hadoop modes deploy these components as follows:
- Local standalone mode: This is the default mode if, as in the preceding Pi example, you don't configure anything else. In this mode, all the components of Hadoop, such as NameNode, DataNode, JobTracker, and TaskTracker, run in a single Java process.
- Pseudo-distributed mode: In this mode, a separate JVM is spawned for each of the Hadoop components and they communicate across network sockets, effectively giving a fully functioning minicluster on a single host.
- Fully distributed mode: In this mode, Hadoop is spread across multiple machines, some of which will be general-purpose workers and others will be dedicated hosts for components, such as NameNode and JobTracker.
Each mode has its benefits and drawbacks. Fully distributed mode is obviously the only one that can scale Hadoop across a cluster of machines, but it requires more configuration work, not to mention the cluster of machines. Local, or standalone, mode is the easiest to set up, but you interact with it in a different manner than you would with the fully distributed mode. In this book, we shall generally prefer the pseudo-distributed mode even when using examples on a single host, as everything done in the pseudo-distributed mode is almost identical to how it works on a much larger cluster.