Saturday, May 3, 2014

Hadoop Administration Practice

chmod -R XXX /...../hadoop_home/

Disable ipv6:

/etc/sysctl.conf

net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1

Add JAVA_HOME and HADOOP_HOME to path:

vi ~/.bash_profile

export JAVA_HOME=/usr/java/jre1.7.0_55/
export HADOOP_HOME=/usr/local/hadoop-1.2.1
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$JAVA_HOME/bin

SSH:
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

To add an IP Address to hosts:
/etc/hosts

192.168.157.166 localhost

To skip input password:

/etc/sudoers

%hadoopgroup ALL(ALL) ALL

%hadoopgroup ALL(ALL) NOPASSWD: ALLL

All Configuration files can be seen under:
/...../hadoop_home/conf//

core-site.xml
hdfs-site.xml
mapred-site.xml
hadoop-evn.sh

start-all.sh and stop-all.sh can be seen under:
/..../hadoop_home/bin/