hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vincent Emonet <vincent.emo...@gmail.com>
Subject Hadoop on Safe Mode because Resources are low on NameNode
Date Tue, 26 Aug 2014 15:05:51 GMT

We have a 11 nodes Hadoop cluster installed from Hortonworks RPM doc:

The cluster was working fine since it went on Safe Mode during the
execution of a job with this message on the NameNode interface:

*Safe mode is ON. Resources are low on NN. Please add or free up more
resources then turn off safe mode manually. NOTE: If you turn off safe mode
before adding resources, the NN will immediately return to safe mode. Use
"hdfs dfsadmin -safemode leave" to turn safe mode off.*
The error displayed in the job log is:
2014-08-22 08:51:35,446 WARN namenode.NameNodeResourceChecker
(NameNodeResourceChecker.java:isResourceAvailable(89)) - Space available on
volume 'null' is 100720640, which is below the configured reserved amount
104857600 2014-08-22 08:51:35,446 WARN namenode.FSNamesystem
(FSNamesystem.java:run(4042)) - NameNode low on available disk space.
Already in safe mode.

On each node we have 5 hdd used for Hadoop
And we checked the 5 hdd on the namenode are all full (between 95 and 100%)
when the HDFS as still 50% of its capacity available : on the other nodes
the 5 hdd are at 30/40%

So I think this is the cause of the error.

On the NameNode we had some Non HDFS data on 1 hdd, so I deleted them to
have 50% of this hdd available (the 4 others are still between 95 and 100%)
But this didn't resolve the problem
I have also followed the advices found here :
And added the following property to the hdfs-site.xml of the NameNode
(multiplying the default value by 2)

Still impossible to get out of the safe mode and as log as we are in safe
mode we can't delete anything in the HDFS.

Is anyone having a tip about this issue?



View raw message