hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sisu Xi <xis...@gmail.com>
Subject Re: hadoop multinode, only master node doing the work
Date Sun, 13 Jul 2014 18:45:58 GMT
I checked the log on the slave node,

it says:

2014-07-13 13:35:37,290 FATAL
org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for
block pool Block pool BP-1408390155-172.16.20.170-1405276491305 (storage id
DS-111686822-127.0.1.1-50010-1405200560127) service to master/
172.16.20.170:54310
java.io.IOException: Incompatible clusterIDs in
/home/xisisu/mydata/hdfs/datanode: namenode clusterID =
CID-5fe42d16-b0cc-4104-b04c-8b4561e907dd; datanode clusterID =
CID-95916f4d-89f1-47e4-a92e-7b88ce97ed8b
        at
org.apache.hadoop.hdfs.server.datanode.DataStorage.doTransition(DataStorage.java:391)
        at
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:191)
        at
org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:219)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:837)
        at
org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:808)
        at
org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:280)
        at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:222)
        at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664)
        at java.lang.Thread.run(Thread.java:744)
2014-07-13 13:35:37,292 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service
for: Block pool BP-1408390155-172.16.20.170-1405276491305 (storage id
DS-111686822-127.0.1.1-50010-1405200560127) service to master/
172.16.20.170:54310
2014-07-13 13:35:37,392 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Removed Block pool
BP-1408390155-172.16.20.170-1405276491305 (storage id
DS-111686822-127.0.1.1-50010-1405200560127)
2014-07-13 13:35:39,392 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: Exiting Datanode
2014-07-13 13:35:39,394 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 0
2014-07-13 13:35:39,395 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at slave-01/127.0.1.1
************************************************************/


I did some google search, it says the problem is with the data dir on the
master node.
I already removed the whole data dir, formated the namenode, and restarted
the hadoop.
But still the same problem.

Sisu




On Sun, Jul 13, 2014 at 12:53 PM, Sisu Xi <xisisu@gmail.com> wrote:

> Hi, Sam:
>
> Thanks for your help! You are right, seems only one node is running.
> Here is the output:
>
> xisisu@slave-01:/usr/local/hadoop$ hadoop dfsadmin -report
> DEPRECATED: Use of this script to execute hdfs command is deprecated.
> Instead use the hdfs command for it.
>
> OpenJDK 64-Bit Server VM warning: You have loaded library
> /usr/local/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled
> stack guard. The VM will try to fix the stack guard now.
> It's highly recommended that you fix the library with 'execstack -c
> <libfile>', or link it with '-z noexecstack'.
> 14/07/13 12:50:14 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Configured Capacity: 14068822016 (13.10 GB)
> Present Capacity: 8425232425 (7.85 GB)
> DFS Remaining: 8423534592 (7.85 GB)
> DFS Used: 1697833 (1.62 MB)
> DFS Used%: 0.02%
> Under replicated blocks: 21
> Blocks with corrupt replicas: 0
> Missing blocks: 0
>
> -------------------------------------------------
> Datanodes available: 1 (1 total, 0 dead)
>
> Live datanodes:
> Name: 172.16.20.170:50010 (master)
> Hostname: master
> Decommission Status : Normal
> Configured Capacity: 14068822016 (13.10 GB)
> DFS Used: 1697833 (1.62 MB)
> Non DFS Used: 5643589591 (5.26 GB)
> DFS Remaining: 8423534592 (7.85 GB)
> DFS Used%: 0.01%
> DFS Remaining%: 59.87%
> Last contact: Sun Jul 13 12:50:13 CDT 2014
>
>
> I get the same output when I run the command on the slave node.
>
>
> Is there anything I am missing in the config file?
>
> Thanks very much!
>
> Sisu
>
>
>
> On Sun, Jul 13, 2014 at 1:36 AM, Kilaru, Sambaiah <
> Sambaiah_Kilaru@intuit.com> wrote:
>
>>  HI Sisu Xi,
>>
>>  On the master node can you check
>> Hadoop dfsadmin –report
>> And listing all the slave nodes or you can check master URL and it should
>> all datanodes listed as slave nodes.
>> Check for RM UI and slave nodes listed there also.
>>
>>  Thanks,
>> Sam
>>
>>   From: Sisu Xi <xisisu@gmail.com>
>> Reply-To: "user@hadoop.apache.org" <user@hadoop.apache.org>
>> Date: Sunday, July 13, 2014 at 11:28 AM
>> To: "user@hadoop.apache.org" <user@hadoop.apache.org>
>> Subject: hadoop multinode, only master node doing the work
>>
>>   Hi, all:
>>
>>  I am new to hadoop. I followed the tutorial on
>>
>> http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/
>>
>>  and installed hadoop 2.2.0 on two 4-core Ubuntu 12.04 machines.
>>
>>  I can start the pi program, however, only the master node is doing the
>> work (I checked the top on each machine).
>> Seems the two nodes are configured correctly, because I can start  the
>> program in the slave node as well, and still only the master node is doing
>> the actual work.
>> I have tried different number of mappers for the pi program, and the
>> results is the same.
>>
>>  Is there anything else I can check?
>>
>>  In the end is my configure file on each host.
>>
>>  Thanks very much!
>>
>>  Sisu
>>
>>  ---------yarn-site.xml-------
>>
>>  <property>
>>   <name>yarn.nodemanager.aux-services</name>
>>   <value>mapreduce_shuffle</value>
>> </property>
>>
>>
>>  <property>
>>   <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
>>   <value>org.apache.hadoop.mapred.ShuffleHandler</value>
>> </property>
>>
>>
>>  <property>
>>   <name>yarn.resourcemanager.address</name>
>>   <value>master:8032</value>
>> </property>
>>
>>  ---------------hdfs-site.xml--------------------
>>
>>  <property>
>>   <name>dfs.replication</name>
>>   <value>2</value>
>> </property>
>>
>>
>>  <property>
>>   <name>dfs.namenode.name.dir</name>
>>   <value>file:/home/xisisu/mydata/hdfs/namenode</value>
>> </property>
>>
>>
>>  <property>
>>   <name>dfs.datanode.data.dir</name>
>>   <value>file:/home/xisisu/mydata/hdfs/datanode</value>
>> </property>
>>
>>
>>  -------------core-site.xml-------------
>>
>>  <property>
>>  <name>fs.default.name</name>
>> <value>hdfs://master:9000</value>
>> </property>
>>
>>  ------------------mapred-site.xml-----------------
>>
>>  <property>
>>   <name>mapreduce.framework.name</name>
>>   <value>yarn</value>
>> </property>
>>
>>  <property>
>>   <name>mapred.job.tracker</name>
>>   <value>master:54311</value>
>>   <description>The host and port that the MapReduce job tracker runs
>>   at.  If "local", then jobs are run in-process as a single map
>>   and reduce task.
>>   </description>
>> </property>
>>
>>
>>
>>
>>  --
>>
>>
>>  *Sisu Xi, PhD Candidate*
>>
>> http://www.cse.wustl.edu/~xis/
>> Department of Computer Science and Engineering
>> Campus Box 1045
>> Washington University in St. Louis
>> One Brookings Drive
>> St. Louis, MO 63130
>>
>
>
>
> --
>
>
> *Sisu Xi, PhD Candidate*
>
> http://www.cse.wustl.edu/~xis/
> Department of Computer Science and Engineering
> Campus Box 1045
> Washington University in St. Louis
> One Brookings Drive
> St. Louis, MO 63130
>



-- 


*Sisu Xi, PhD Candidate*

http://www.cse.wustl.edu/~xis/
Department of Computer Science and Engineering
Campus Box 1045
Washington University in St. Louis
One Brookings Drive
St. Louis, MO 63130

Mime
View raw message