hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brahma Reddy Battula <brahmareddy.batt...@huawei.com>
Subject RE: Namenode not able to come out of SAFEMODE
Date Thu, 20 Jul 2017 06:59:42 GMT

Dn1 and Dn2 is re-started successfully..? can you check the DN logs..?

--Brahma Reddy Battula

From: omprakash [mailto:omprakashp@cdac.in]
Sent: 20 July 2017 12:12
To: user@hadoop.apache.org
Subject: Namenode not able to come out of SAFEMODE

Hi all,

I have a setup of 3 node Hadoop cluster(Hadoop-2.8.0). I have deployed 2 namenodes that are
configured in HA mode using QJM. 2 datanodes are configured on the same machine where namenode
are installed. 3rd node is used for quorum purpose only.

Node1 -> nn1, dn1, jn1, zkfc1, zkServer1
Node2 -> nn2, dn2, jn2, zkfc2, zkServer2
Node3 -> jn3,  zkServer3

I stopped the cluster for some reason(power recycled the servers)  and since them I am not
able to start the cluster successfully. After examining the logs I found that the namenodes
are in safe mode and none of them are able to load the block in memory. Below is the status
of namenode from namenode UI.

Safe mode is ON. The reported blocks 0 needs additional 6132675 blocks to reach the threshold
0.9990 of total blocks 6138814. The number of live datanodes 0 has reached the minimum number
0. Safe mode will be turned off automatically once the thresholds have been reached.

61,56,984 files and directories, 61,38,814 blocks = 1,22,95,798 total filesystem object(s).

Heap Memory used 5.6 GB of 7.12 GB Heap Memory. Max Heap Memory is 13.33 GB.

Non Heap Memory used 45.19 MB of 49.75 MB Commited Non Heap Memory. Max Non Heap Memory is
130 MB.

I have tried increasing the HADOOP_HEAPSIZE, increasing the heap size in HADOOP_NAMENODE_OPTS
but no success.
Need help.

Omprakash Paliwal
HPC-Medical and Bioinformatics Applications Group
Centre for Development of Advanced Computing (C-DAC)
Pune University campus,
Maharashtra, India
Contact : +91-20-25704231

[ C-DAC is on Social-Media too. Kindly follow us at:
Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]

This e-mail is for the sole use of the intended recipient(s) and may
contain confidential and privileged information. If you are not the
intended recipient, please contact the sender by reply e-mail and destroy
all copies and the original message. Any unauthorized review, use,
disclosure, dissemination, forwarding, printing or copying of this email
is strictly prohibited and appropriate legal action will be taken.

View raw message