Hadoop - Word Count Example

Hadoop Version used in the below example is: 1.2.1


WordCount.java


import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

public class WordCount {

    public static void main(String[] args) throws IOException,
            InterruptedException, ClassNotFoundException {

        Path inputPath = new Path(args[0]);
        Path outputDir = new Path(args[1]);

        // Create configuration
        Configuration conf = new Configuration(true);

        // Create job
        Job job = new Job(conf, "WordCount");
        job.setJarByClass(WordCountMapper.class);

        // Setup MapReduce
        job.setMapperClass(WordCountMapper.class);
        job.setReducerClass(WordCountReducer.class);
        job.setNumReduceTasks(1);

        // Specify key / value
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);

        // Input
        FileInputFormat.addInputPath(job, inputPath);
        job.setInputFormatClass(TextInputFormat.class);

        // Output
        FileOutputFormat.setOutputPath(job, outputDir);
        job.setOutputFormatClass(TextOutputFormat.class);

        // Delete output if exists
        FileSystem hdfs = FileSystem.get(conf);
        if (hdfs.exists(outputDir))
            hdfs.delete(outputDir, true);

        // Execute job
        int code = job.waitForCompletion(true) ? 0 : 1;
        System.exit(code);

    }

}


WordCountMapper.java


import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

public class WordCountMapper extends
        Mapper<Object, Text, Text, IntWritable> {

    private final IntWritable ONE = new IntWritable(1);
    private Text word = new Text();

    public void map(Object key, Text value, Context context)
            throws IOException, InterruptedException {

        String[] csv = value.toString().split(" ");
        for (String str : csv) {
            word.set(str);
            context.write(word, ONE);
        }
    }
}


WordCountReducer.java


import java.io.IOException;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

public class WordCountReducer extends
        Reducer<Text, IntWritable, Text, IntWritable> {

    public void reduce(Text text, Iterable<IntWritable> values, Context context)
            throws IOException, InterruptedException {
        int sum = 0;
        for (IntWritable value : values) {
            sum += value.get();
        }
        context.write(text, new IntWritable(sum));
    }
}

Hadoop Administration Practice

chmod -R XXX /...../hadoop_home/

Disable ipv6:

/etc/sysctl.conf

net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1

Add JAVA_HOME and HADOOP_HOME to path:

vi ~/.bash_profile

export JAVA_HOME=/usr/java/jre1.7.0_55/
export HADOOP_HOME=/usr/local/hadoop-1.2.1
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$JAVA_HOME/bin

SSH:
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

To add an IP Address to hosts:
/etc/hosts

192.168.157.166 localhost

To skip input password:

/etc/sudoers

%hadoopgroup ALL(ALL) ALL

%hadoopgroup ALL(ALL) NOPASSWD: ALLL

All Configuration files can be seen under:
/...../hadoop_home/conf//

core-site.xml
hdfs-site.xml
mapred-site.xml
hadoop-evn.sh

start-all.sh and stop-all.sh can be seen under:
/..../hadoop_home/bin/

Hadoop: MapReduce: WordCount: Pseduo Cluster Environment

To list files/folders available under hadoop root directory:
hadoop fs -ls /

Create a file with name xyz.txt:
sudo gedit xyz.txt

Check whether it has created or not(/home/user_name/):
ll   i.e., LL

verify the contents of xyz.txt:
cat xyz.txt

Before writing xyz.txt to HDFS, make sure it doesn't exists in the hadoop root directory:
hadoop fs -ls /

Write xyz.txt to HDFS:
hadoop fs -put xyz.txt /xyz.txt

Now, check, whether xyz.txt exists in hadoop root directory or not:
hadoop fs -ls /

To view files/blocks/locations info use:
hadoop fsck /xyz.txt -files -blocks -locations

Goto the folder structure, where your java programs exists:
cd training_materials/developer/exercises/wordcount/

Compile all your java files, by setting the hadoop-core.jar on classpath:
javac -classpath /usr/lib/hadoop/hadoop-core.jar *.java

Create a jar file out of all compiled java classes(i.e., all .class files):
jar cvf wordcount.jar *.class

Now, run/execute the jar by specifying

1. the .class file name which has got main()
2. the input file/folder name(including path)
3. the output file name(including path)

hadoop jar wordcount.jar WordCount /xyz.txt /xyz_output1

Now, go to /xyz_output1 location:
hadoop fs -ls /xyz_output1
Note: you will see: _SUCCESS, _logs, part_00000. Where as, part_00000 has final reducer results/output.

To view the output:
hadoop fs -cat /xyz_output1/part-00000

To save whatever the commands you entered till now:
cd (means now i am at /home/user_name/)
history > command_history.txt