hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-4539) Streaming Edits to a Standby Name-Node.
Date Wed, 11 Mar 2009 01:04:50 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680719#action_12680719

Konstantin Shvachko commented on HADOOP-4539:

I am going to comment on Suresh's items that are not reflected in the new patch.

Design related:

1. Active node does not turn into Backup node in current design. We plan to have a possibility
to turn a Standby node into Active, yes but this is done by just changing the role of the
node. I think it makes sense to keep BackupNode as a subclass of NameNode because some commands,
those that don't modify the meta-data, will work on the backup node exactly as they do on
the active node. Say you can do {{lsr}} from the backup, but if you try to create a file it
will throw {{SafeModeException}}.

2. EditLogBackupOutputStream failing to write to the backup node is handled by calling {{processIOError()}},
which removes the failed BackupOutputStream from the list of EditLogOutputStream(s) and increments
the checkpointTime for the remaining streams to make the failed image outdated.

Coding related:

1.1. name-node and data-node vs NameNode and DataNode is actually consistent if you look at
other comments. Lets say NameNode and DataNode refer to classes and name-node and data-node
to the abstractions behind them.
1.2. Calling super() in class constructors is unnecessary, but helps me to debug and analyze
the code. You can set a breakpoint, step into, and easily find the code of the super constructor.

2.3. {{UnregisteredNodeException}} Constructor takes DatanodeID and DatanodeInfo as parameters
because this is a constructor specific for data-nodes. We may later need name-node, balancer,
etc specific constructors, which will have their special parameters.

5.1. We do not keep comments for previous versions in the code, they can be retrieved from
svn history. I copied the respective comment from ClientProtocol.

6.1. I think that {{ServerCommand}} actions (static ints) should belong to the commands that
use them. I plan to file an issue to move {{DatanodeCommand}} actions into the respective

7.1. My reasoning is that we have NameNode, DataNode, and now we will have Backup Node and
Checkpoint node. I first thought about naming them "Backup NameNode" and "Checkpoint NameNode"
connotative to SecondaryNameNode. 
But "Backup node" sounds better to me and besides this is what they are - nodes.

8.3. {{dfs.backup.address}} and {{dfs.backup.http.address}} define the addresses which the
backup node starts on its rpc and http servers. You do not need to change the configuration
when the node transitions from backup to active. 
Example. If you start DataNode on port 0, which means on any free port, you do not change
the config file after startup, right? Same with backup/active nodes.
8.9. Backup node does not have backup streams now, so {{errorReport()}} is just harmless.
If we decide to implement chaining edits streams then yes we will need to override {{errorReport()}}
and probably other methods too.
8.10. {{BackupNode.journalSize()}} will report the correct size of its edits file. I don't
think it should throw an exception.
8.11. {{BackupNode.stop()}} should call {{RPC.stopProxy()}} because it needs to stop the name-node
proxy it is connected to as a client.
And the {{NameNode.stop()}} will not take care of that because NameNode does not act as its
own client.

9.1. {{Checkpointer.doCheckPoint()}} first calls {{NameNode.startCheckpoint()}} which in turn
calls {{startJournalSpool()}} from the NameNode itself and does not return until the BackupNode
actually rolls its edits.
9.2. {{convergeJournalSpool()}} contains common logic for Backup and Checkpoint nodes. I agree
the name does not make much sense for the checkpoint node since it does not converge no journals
as it doesn't have any.
But for the backup node its reflects the semantics of the operation.
9.3. {{Checkpointer}} should not be an inner class of the {{BackupNode}}, imo. We have been
through this with FSNamesystem class, had to factor out most of the stuff out of it.
9.4. {{downloadCheckpoint()}} and {{uploadCheckpoint()}} should evelve in the future. Right
now they use {{CheckpointSignature}} for validation, but we may need to include registration
as well.

11.1. {{CheckpointSignature}} currently includes the name-nodes's edits file modification
time only for identification purposes. I planned to create an issue to include there the number
of files in the namepsace at the time when the checkpoint starts.

> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>                 Key: HADOOP-4539
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4539
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: dfs
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.21.0
>         Attachments: BackupNode.patch, BackupNode.patch, BackupNode.patch, image001.gif,
StreamEditsToBN.pdf, StreamEditsToSNN.htm
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN). 
> The long term goal is to make it a warm standby. 
> The purpose of this issue is to provide real time streaming of edits to SNN so that it
contained the up-to-date namespace state.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message