Hadoop 1.2.1 Installation & Troubleshooting

Installation

 

1. Check the firewalls among servers. ( Recommended to open all ports among servers )
2. Create same account( TEST_ACCOUNT ) to all servers.
3. Create public key of TEST_ACCOUNT for each server. ( ssh-keygen )
4. Authorize public key of TEST_ACCOUNT in master server to each slave server TEST_ACCOUNT so that master account can access each slave servers through ssh( FILE : $HOME/.ssh/authorized_keys ). Make sure the value of chmod of authorized_keys file is “0×600″ or it  doesn’t work.
5. Copy hadoop-1.2.1 to the same directory of each server.
6. Modify  $HADOOP/conf/masters & $HADOOP/conf/slaves to add master and slave servers. Copying modified file to each server is recommended.
   1. SAMPLE : masters
      
      1 2	192.168.0.15 192.168.0.16

      
       
   2. SAMPLE : slaves
      
      1 2 3 4 5

      192.168.0.17 192.168.0.18 192.168.0.19 192.168.0.20 192.168.0.21

      
       
7. Modify /etc/hosts to recognize hostname of each servers(sudo is needed.). Recommended to use same content among servers. I used hostname like “hadoop-master1″ or “hadoop-slave3″.
   1. SAMPLE : /etc/hosts
      
      1 2 3 4 5 6 7 8 9 10

      127.0.0.1   localhost 192.168.0.15 hadoop-master1 192.168.0.16 hadoop-master2 192.168.0.17 hadoop-slave1 192.168.0.18 hadoop-slave2 192.168.0.19 hadoop-slave3 192.168.0.20 hadoop-slave4 192.168.0.21 hadoop-slave5   ::1             localhost6.localdomain6 localhost6

      
       
8. Configure hadoop setting for each server.
9. Run Format($HADOOP/bin/hadoop name node -format) and Start($HADOOP/bin/start-all.sh) at master server.

 

 

Tips

 

Format script

I used two masters with two mounted directories ( /data1 & /data2 ) and five slaves with three mounted directories( /data1 & /data2 & /data3 ). You can change the directory of your own. Locate this script to $HADOOP/bin

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

./stop-all.sh   masters=( 192.168.0.15 192.168.0.16 ) slaves=( 192.168.0.17 192.168.0.18 192.168.19 192.168.20 192.168.21 )   for masterip in ${masters[@]} do     for n in {1..2}     do         echo "ssh $masterip \"rm -rf /data$n/hadoop/*\""         ssh $masterip "rm -rf /data$n/hadoop/*"     done done   for slaveip in ${slaves[@]} do     for n in {1..3}     do         echo "ssh $slaveip \"rm -rf /data$n/hadoop/*\""         ssh $slaveip "rm -rf /data$n/hadoop/*"     done done   ./hadoop namenode -format ./start-all.sh

 

 

Trouble-shootings

 

1. Unregistered data node

• This exception occurred because of different content of $HADOOP/conf/masters and $HADOOP/conf/slaves among servers.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

2014-01-29 04:01:05,868 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hadooptest cause:org.apache.hadoop.hdfs.protocol.UnregisteredDatanodeException: Unregistered data node: xxx.xxx.xxx.xxx:50010 2014-01-29 04:01:05,872 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 9000, call sendHeartbeat(DatanodeRegistration(xxx.xxx.xxx.xxx:50010, storageID=DS-1896089993-xxx.xxx.xxx.xxx-50010-1376501270927, infoPort=50075, ipcPort=50020), 5811631779840, 192512, 5511041220608, 0, 1) from xxx.xxx.xxx.xxx:43160: error: org.apache.hadoop.hdfs.protocol.UnregisteredDatanodeException: Unregistered data node: xxx.xxx.xxx.xxx:50010 org.apache.hadoop.hdfs.protocol.UnregisteredDatanodeException: Unregistered data node: xxx.xxx.xxx.xxx:50010         at org.apache.hadoop.hdfs.server.namenode.NameNode.verifyRequest(NameNode.java:1171)         at org.apache.hadoop.hdfs.server.namenode.NameNode.sendHeartbeat(NameNode.java:1083)         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)         at java.lang.reflect.Method.invoke(Method.java:597)         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)         at java.security.AccessController.doPrivileged(Native Method)         at javax.security.auth.Subject.doAs(Subject.java:396)         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)

 

2. ERROR org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Error getting localhost name. Using ‘localhost’…

• This issue is because the server has localhost in “/etc/hosts” or “/etc/HOSTNAME” or “rc.conf”. There’re several ways to resolve this issues and I choose “removing a line which includes localhost” in hostname file.

 

3. Bad connect ack with firstBadLink as xxx.xxx.xxx:50010

• There are several reasons to cause this issue. In my case, I turned off the firewall ( for example “/etc/init.d/iptable stop” ) to resolve this issue (!!Not recommended because turning off the firewall is dangerous for servers which can be accessed from outside. Refer the following site to add IPs and PORTs to the firewall rule. – http://blog.cloudera.com/blog/2009/08/hadoop-default-ports-quick-reference/ )

1 2 3 4 5 6

14/02/03 17:08:18 INFO hdfs.DFSClient: Exception in createBlockOutputStream xxx.xxx.xxx.xxx:50010 java.io.IOException: Bad connect ack with firstBadLink as xxx.xxx.xxx.xxx:50010 14/02/03 17:08:18 INFO hdfs.DFSClient: Abandoning blk_4689315116139837947_1003 14/02/03 17:08:18 INFO hdfs.DFSClient: Excluding datanode xxx.xxx.xxx.xxx:50010 14/02/03 17:08:24 INFO hdfs.DFSClient: Exception in createBlockOutputStream xxx.xxx.xxx.xxx:50010 java.io.IOException: Bad connect ack with firstBadLink as 211.39.140.240:50010 14/02/03 17:08:24 INFO hdfs.DFSClient: Abandoning blk_-3947753087362711260_1003 14/02/03 17:08:24 INFO hdfs.DFSClient: Excluding datanode xxx.xxx.xxx.xxx:50010