hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From KrzyCube <yuxh...@gmail.com>
Subject Re: Multi-case "dfs.name.dir"
Date Tue, 26 Jun 2007 02:09:16 GMT

Thanks Doug, thanks Konstantin.

And still i have some question about that .

1. Is there any implements that i can specify one of the "dfs.name.dir"
values a remote directory.
or if i can map a remote fs to local , and direct the valuse to that ? cause
i think it's better for backup namespace data as if the NameNode's hardware

2.Yes , we use the SecondaryNameNode to deal with edits periodically. Then
the "fsimage" which on dish grow big. Then , assume there is a so big
fsimage file in local disk, and try to start the NameNode, now it is trying
to load the fsimage , and i found there is a HashMap called "activeBlocks"
in FSDirectory, i think this is the map for find and delete files in hdfs.
but the fsimage is so large that the memory can not hold it , then what to
do ?  i just not found code deal with that , but i found code deal with
edits file large than 2G size.

Konstantin Shvachko wrote:
> Hi Martin,
> "fsimage" is the file that contains all name space information.
> Name-node stores this information in memory and periodically checkpoints 
> it to disk.
> "edits" contains a log of name space operations that occurred since the 
> last checkpoint.
> So that if the name-node fails or otherwise exits between two 
> checkpoints the
> name-space could be restored by merging the latest image and the edits.
> Property "dfs.name.dir" specifies the directory where fsimage, edits and 
> other name-node
> files are stored.
> You can specify multiple entries (comma delimited) in "dfs.name.dir". 
> Then all name-space
> files will be duplicated in these directories. This is the right way to 
> backup the name-space
> image as a local copy.
> Periodic checkpointing is performed by the secondary name-node. The side 
> effect of the
> checkpoint is that it removes contents of the edits file. Yes you are 
> right the edits file can
> grow big that is why periodic checkpointing is important.
> Does that answer your questions?
> --Konstantin
> Doug Cutting wrote:
>> KrzyCube wrote:
>>>     I found that "File[] editFiles" in FSEditLog.java , then i trace the
>>> call stack and found that it can be configured as multi-case of
>>> "dfs.name.dir" . Is this means the NameNode data can be split into 
>>> pieces or
>>> just set replication as the number of the strings of dirs that 
>>> configured ?
>>>     Is that right the way to backup several copys of the 
>>> editlog+fsimage in
>>> local filesystem ?
>> According to the documentation for dfs.name.dir:
>> http://lucene.apache.org/hadoop/hadoop-default.html#dfs.name.dir
>>   Determines where on the local filesystem the DFS name node should
>>   store the name table. If this is a comma-delimited list of directories
>>   then the name table is replicated in all of the directories, for
>>   redundancy.
>> Note that as of release 0.11 there's support for a secondary namenode, 
>> started with 'bin/hadoop secondarynamenode'.
>> http://issues.apache.org/jira/browse/HADOOP-227#action_12458332
>> This documentation should probably be added to 
>> http://lucene.apache.org/hadoop/hdfs_design.html, or at least to the 
>> wiki...
>> Doug

View this message in context: http://www.nabble.com/Multi-case-%22dfs.name.dir%22-tf3973987.html#a11298991
Sent from the Hadoop Users mailing list archive at Nabble.com.

View raw message