hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sky88088 <sky880883...@hotmail.com>
Subject RE: No edits files in dfs.namenode.edits.dir
Date Tue, 24 May 2016 06:46:35 GMT
Hi, Harsh. 
The "dfs.namenode.edits.dir" from "hdfs getconf" is "file:///data1/hadoop/name", the same
as "dfs.namenode.name.dir".
However, in the NameNode web UI("namenode:8088/conf"), none of "dfs.namenode.edits.dir" or
"dfs.namenode.name.dir" is set. 
In both source, the "hadoop.tmp.dir" is set as "/data1/hadoop/tmp".
It seems strange because "dfs.namenode.name.dir" should be "file://${hadoop.tmp.dir}/dfs/name"
by default, which means "/data1/hadoop/tmp/dfs/name", instead of the path value from "hdfs
However, only "file:///data1/hadoop/name" directionary can be found in my cluster.

I think Ravi's suggestion make sense.  If NameNode actually persisting edits, it could by
found by "lsof -grep edit", right?
By the way, the version of the cluster is hadoop 2.2.0. I have no idea if this information


From: harsh@cloudera.com
Date: Mon, 23 May 2016 03:33:17 +0000
Subject: Re: No edits files in dfs.namenode.edits.dir
To: sky880883368@hotmail.com; gurmukh.dhillon@yahoo.com; ravihadoop@gmail.com
CC: user@hadoop.apache.org

Are you absolutely certain you are looking at the right directory? The NameNode is designed
to crash if it cannot persist edits (transactions) durably.
The "hdfs getconf" utility checks local command classpaths but your service may be running
over a different configuration directory. If you have access to your NameNode web UI, try
visiting its /conf URL and you can see the actual directories configured for the NameNode's
images and edit log.
P.s. The config dfs.namenode.edits.dir usually also defaults to the value of dfs.namenode.name.dir.
On Mon, 23 May 2016 at 11:13 sky88088 <sky880883368@hotmail.com> wrote:

I use "hdfs -getconf" to check all the relate dirs, and still no edits files found.

Thanks Ravi's suggestion. I tried "lsof -grep edit" and "lsof -grep hadoop", but still nothing
meaningful found.

If really no edits persist to disk, any idea about how to fix this problem?

Subject: Re: No edits files in dfs.namenode.edits.dir
To: ravihadoop@gmail.com; sky880883368@hotmail.com
CC: user@hadoop.apache.org
From: gurmukh.dhillon@yahoo.com.INVALID
Date: Sun, 22 May 2016 21:45:50 +1000

    Have you explicitly configured the parameter
    "dfs.namenode.edits.dir" ? else it default to dfs.namenode.name.dir




    On 20/05/16 4:46 AM, Ravi Prakash

      No! You are probably writing the edits file
        somewhere still. An `lsof` on the namenode process may be more
        revealing. Obviously this depends on configuration, but unless
        you have some really crazy settings, I'm pretty sure the edits
        would be persisted to disk.



        On Wed, May 18, 2016 at 2:47 AM,
          sky88088 <sky880883368@hotmail.com>

                I found that there is no edits file in my
                  cluster's dfs.namenode.edits.dir.


                      It is a 20-nodes cluster without secondary
                        namenode, and it runs well for a long while
                        (more than one year).

                    However, it's found that no edits file in the dfs.namenode.edits.dir
                        and the the fsimage file doesn't update. 

                  Is it meaning that
                      all the metadata of name node only stay in
                    the memory?

                  Is there a way to fix it? Is there any
                    configuration to control the persistence?



Thanks and Regards

Gurmukh Singh 		 	   		  
View raw message