hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Philippe Gassmann (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-227) Namespace check pointing is not performed until the namenode restarts.
Date Tue, 05 Dec 2006 14:12:26 GMT
     [ http://issues.apache.org/jira/browse/HADOOP-227?page=all ]

Philippe Gassmann updated HADOOP-227:
-------------------------------------

    Attachment: patch-async-checkpoints-0.9.0

Here is a patch on the current Hadoop trunk .

This patch do automatic checkpoints without locking the filesystem.

When it is time to do a checkpoint, edit logs stream are closed and new edit logs are opened,
a thread is created that create a fake FSNamesystem that will merge previously written logs
into fsimage. At the end, new edit logs are renamed to their old names.

It  will consume as much memory during the chekpointing as the current running instance of
the FSNamesystem.

The auto checkpointing feature is disabled by default. So applying the patch "as is" is almost
safe. (It does not break current image and logs format and loading philosophy) 

Nonetheless, I can understand that you, the Hadoop dev team,  does not want to integrate this
huge hacky patch as a part of the hadoop distribution...


> Namespace check pointing is not performed until the namenode restarts.
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-227
>                 URL: http://issues.apache.org/jira/browse/HADOOP-227
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.2.0
>            Reporter: Konstantin Shvachko
>         Assigned To: Milind Bhandarkar
>         Attachments: patch-async-checkpoints-0.9.0
>
>
> In current implementation when the name node starts, it reads its image file, then
> the edits file, and then saves the updated image back into the image file.
> The image file is never updated after that.
> In order to provide the system reliability reliability the namespace information should
> be check pointed periodically, and the edits file should be kept relatively small.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message