hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jiang licht <licht_ji...@yahoo.com>
Subject Re: How often namenode dumps in-memory hdfs meta data to disk (dfs.name.dir)?
Date Fri, 25 Jun 2010 07:24:14 GMT
I guess I need to rephrase my question. First here's what I understand. Basically, there is
a fsimage, which is stored as a file in dfs.name.dir folder(s) and there is also a editlog
which keeps tracking every update to the hdfs, and also saved in dfs.name.dir folder(s). According
to current hdfs documentation, when namenode starts, it update in-memory fsimage with editlog
and truncate old editlog and save the new fsimage to disk and then editlog will only record
changes after this point (still one checkpoint implementation in the latest stable version?).

So, according to this process, meta data (including editlog and fsimage) is read once when
namenode starts; fsimage on the disk (in dfs.name.dir) is written once after namenode starts
and applies the editlog to the one just read from the disk; editlog on the disk (in dfs.name.dir)
is also overwritten (truncated) at this point.

Then, I think that namenode will need to write editlog to disk at a regular interval. Otherwise,
if namenode service is terminated unexpectedly, any change since its startup will get lost.
By the way, for recovery from namenode failure purpose, apparently, the 
whole dfs.name.dir has to be backed up not just fsimage. However, any 
change made to the hdfs after a checkpoint followed by a backup will still get lost if there
is a failure. Considering this, using a list of folders for dfs.name.dir will be the safest
way compared to simply copy the folder over in some manner (e.g. rsync).

Correct me if I misunderstand anything.

So, how often namenode write any meta data/change to dfs.name.dir?



--- On Thu, 6/24/10, Allen Wittenauer <awittenauer@linkedin.com> wrote:

From: Allen Wittenauer <awittenauer@linkedin.com>
Subject: Re: How often namenode dumps in-memory hdfs meta data to disk (dfs.name.dir)?
To: "<common-user@hadoop.apache.org>" <common-user@hadoop.apache.org>
Date: Thursday, June 24, 2010, 8:04 PM

On Jun 24, 2010, at 6:02 PM, jiang licht wrote:

> Is it configurable?

It doesn't.

The only thing the namenode does is update the edits log, and that is continually rolling.

If you want it to dump a new fsimage, then you need to use the secondary namenode to perform
the merge.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message