hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-227) Namespace check pointing is not performed until the namenode restarts.
Date Wed, 06 Dec 2006 03:17:24 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-227?page=comments#action_12455851 ] 
Konstantin Shvachko commented on HADOOP-227:

Thanks Philippe, this is a refined effort that uses just the existing code to upload the image
and merge it with the edits.
Unfortunately, it doubles the memory consumption during checkpointing, which is what this
issue all about imo.

> Sounds a lot like a BTree and comes with all of the issues.
It's not a tree, there is no balancing, and I didn't mentioned trees even once.
The only issue I can associate with BTrees is splitting the block into 2.

> Why do we need to do this?
I am advocating to revive Milind's proposal #5 of the initial design.
Our goal is to minimize memory overhead used for checkpointing and to provide uninterrupted
access to the name-node during checkpointing.
We are not considering blocking approaches here so far, which makes minimizing memory our
main requirement.

The copy-on-write approach potentially leads to a linear memory increase and requires additional
name-node data structures.
Proposal #5 is an attempt to separate checkpointing from the name-node regular operation process.
It takes the image file and the edits file and merges them whether the name-node is present
or not.
It does it with lots of IOs BUT in constant space.

I was trying to come up with a simpler algorithm for the stand-alone checkpointing.
It uses more space but does not require external sorting or unintuitive file entry renaming
(as #5).
And it can be adapted to use constant space for the price of more ios.
Giving up IOs imo is the right tradeoff here, since disk is not used by the name-node and
as mostly idle during its regular operation.

> Namespace check pointing is not performed until the namenode restarts.
> ----------------------------------------------------------------------
>                 Key: HADOOP-227
>                 URL: http://issues.apache.org/jira/browse/HADOOP-227
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.2.0
>            Reporter: Konstantin Shvachko
>         Assigned To: Milind Bhandarkar
>         Attachments: patch-async-checkpoints-0.9.0, patch-async-checkpoints-0.9.0, patch-async-checkpoints-0.9.0
> In current implementation when the name node starts, it reads its image file, then
> the edits file, and then saves the updated image back into the image file.
> The image file is never updated after that.
> In order to provide the system reliability reliability the namespace information should
> be check pointed periodically, and the edits file should be kept relatively small.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message