hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay Radia (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1073) Simpler model for Namenode's fs Image and edit Logs
Date Tue, 13 Apr 2010 21:47:53 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12856636#action_12856636
] 

Sanjay Radia commented on HDFS-1073:
------------------------------------

Todd, thanks for the design. Before you move forward too much on the patch, I would like to
get consensus on the 2 alternate designs. 
Further I think we need to add some items to the design doc. (Given that we may go through
2, 3 versions of the doc would it  be better to attach it rather then post in inline? ).

Please add the following items to your design doc:
* BNN restarts - how does it sync up? What if we have multiple BNNs?
* Checkpoint:
** Concurrent checkpoints (saveImage and checkpointer)
** Checkpoint done in the BNN which is also applying the edits stream to its state - Does
the notion of spooling in the current design change?)
** Explore the notion of having checkpoints done offline - this is not targeted for the next
release but something that
  we may want down the road; we need to evaluate the designs against this. (of course we also
need to evaluate whether or not offline checkpoints are a good idea in the first place.)
* Managing edits and images in an HA environment. Here the idea is to move the image and edits
to shared storage and treat the
  NN as "diskless". This is esp useful for federation when there are mulitple NNs. Moving/writing
the image to shared storage is not difficult and it avoids the need to send the image back
to the primary NN. Moving the edits to share storage is hard because of the latency requirements.
Here book-keeper can come to the rescue; I don't see any other solutions so far.

I am *not* proposing  very detailed design of the above items since we don't have the resources
to do all that. However as we evaluate the 2 alternate design lets use the above items to
guide us.

> Simpler model for Namenode's fs Image and edit Logs 
> ----------------------------------------------------
>
>                 Key: HDFS-1073
>                 URL: https://issues.apache.org/jira/browse/HDFS-1073
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Todd Lipcon
>
> The naming and handling of  NN's fsImage and edit logs can be significantly improved
resulting simpler and more robust code.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message