hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-227) Namespace check pointing is not performed until the namenode restarts.
Date Sat, 16 Dec 2006 05:20:24 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-227?page=comments#action_12458994 ] 
            
dhruba borthakur commented on HADOOP-227:
-----------------------------------------

Konstantin proposal is essentially a push model where the primary Namenode drives the scehduling
policies of the periodic checkpointing. Also, he mentioned about supporting cascading secondaries.

I am going ahead the pull model: the Namenode is a very passive entity as far as periodic
checkpointing is concerned. The scheduling policies are maintained only by the secondary namenode.
The secondary namenode polls the primary periodically (say every 5 minutes) to determine the
size of the current edit log.

The secondary would use HTTP-GET method to transfer fsmage and edits. Al alternative that
was discussed was to use HDFS itself to transfer the image. However using HDFS has the disadvantage
that the secondary would have to poll the primary to determine when the upload to HDFS was
complete (HDFS does not have streaming RPC and has a fixed timeout for an RPC).

> Namespace check pointing is not performed until the namenode restarts.
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-227
>                 URL: http://issues.apache.org/jira/browse/HADOOP-227
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.2.0
>            Reporter: Konstantin Shvachko
>         Assigned To: dhruba borthakur
>         Attachments: patch-async-checkpoints-0.9.0, patch-async-checkpoints-0.9.0, patch-async-checkpoints-0.9.0
>
>
> In current implementation when the name node starts, it reads its image file, then
> the edits file, and then saves the updated image back into the image file.
> The image file is never updated after that.
> In order to provide the system reliability reliability the namespace information should
> be check pointed periodically, and the edits file should be kept relatively small.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message