hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vincent Emonet <vincent.emo...@gmail.com>
Subject Hadoop on Safe Mode because Resources are low on NameNode
Date Tue, 26 Aug 2014 15:05:51 GMT
Hello,

We have a 11 nodes Hadoop cluster installed from Hortonworks RPM doc:
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.1/bk_installing_manually_book/content/rpm-chap1.html

The cluster was working fine since it went on Safe Mode during the
execution of a job with this message on the NameNode interface:



*Safe mode is ON. Resources are low on NN. Please add or free up more
resources then turn off safe mode manually. NOTE: If you turn off safe mode
before adding resources, the NN will immediately return to safe mode. Use
"hdfs dfsadmin -safemode leave" to turn safe mode off.*
The error displayed in the job log is:
2014-08-22 08:51:35,446 WARN namenode.NameNodeResourceChecker
(NameNodeResourceChecker.java:isResourceAvailable(89)) - Space available on
volume 'null' is 100720640, which is below the configured reserved amount
104857600 2014-08-22 08:51:35,446 WARN namenode.FSNamesystem
(FSNamesystem.java:run(4042)) - NameNode low on available disk space.
Already in safe mode.

On each node we have 5 hdd used for Hadoop
And we checked the 5 hdd on the namenode are all full (between 95 and 100%)
when the HDFS as still 50% of its capacity available : on the other nodes
the 5 hdd are at 30/40%

So I think this is the cause of the error.

On the NameNode we had some Non HDFS data on 1 hdd, so I deleted them to
have 50% of this hdd available (the 4 others are still between 95 and 100%)
But this didn't resolve the problem
I have also followed the advices found here :
https://issues.apache.org/jira/browse/HDFS-4425
And added the following property to the hdfs-site.xml of the NameNode
(multiplying the default value by 2)
  <property>
     <name>dfs.namenode.resource.du.reserved</name>
       <value>209715200</value>
   </property>

Still impossible to get out of the safe mode and as log as we are in safe
mode we can't delete anything in the HDFS.


Is anyone having a tip about this issue?


Thankfully,

Vincent.

Mime
View raw message