Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-dev@lucene.apache.org
Message-ID: <20312080.1166246424484.JavaMail.jira@brutus>
Date: Fri, 15 Dec 2006 21:20:24 -0800 (PST)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: hadoop-dev@lucene.apache.org
Subject: [jira] Commented: (HADOOP-227) Namespace check pointing is not
 performed until the namenode restarts.
In-Reply-To: <17890828.1147909205885.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

    [ http://issues.apache.org/jira/browse/HADOOP-227?page=comments#action_12458994 ] 
            
dhruba borthakur commented on HADOOP-227:
-----------------------------------------

Konstantin proposal is essentially a push model where the primary Namenode drives the scehduling policies of the periodic checkpointing. Also, he mentioned about supporting cascading secondaries.

I am going ahead the pull model: the Namenode is a very passive entity as far as periodic checkpointing is concerned. The scheduling policies are maintained only by the secondary namenode. The secondary namenode polls the primary periodically (say every 5 minutes) to determine the size of the current edit log.

The secondary would use HTTP-GET method to transfer fsmage and edits. Al alternative that was discussed was to use HDFS itself to transfer the image. However using HDFS has the disadvantage that the secondary would have to poll the primary to determine when the upload to HDFS was complete (HDFS does not have streaming RPC and has a fixed timeout for an RPC).

> Namespace check pointing is not performed until the namenode restarts.
> ----------------------------------------------------------------------
>
>                 Key: HADOOP-227
>                 URL: http://issues.apache.org/jira/browse/HADOOP-227
>             Project: Hadoop
>          Issue Type: Bug
>          Components: dfs
>    Affects Versions: 0.2.0
>            Reporter: Konstantin Shvachko
>         Assigned To: dhruba borthakur
>         Attachments: patch-async-checkpoints-0.9.0, patch-async-checkpoints-0.9.0, patch-async-checkpoints-0.9.0
>
>
> In current implementation when the name node starts, it reads its image file, then
> the edits file, and then saves the updated image back into the image file.
> The image file is never updated after that.
> In order to provide the system reliability reliability the namespace information should
> be check pointed periodically, and the edits file should be kept relatively small.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira