hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-2585) Automatic namespace recovery from the secondary image.
Date Wed, 02 Apr 2008 20:21:24 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-2585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Konstantin Shvachko updated HADOOP-2585:
----------------------------------------

    Attachment: SecondaryStorage.patch

The main features in this patch.
# "-importCheckpoint" argument is introduced for the name-node startup. Started with this
argument 
#- The name-node will try to load image from the directory specified in "fs.checkpoint.dir"
and save it to 
name-node storage directory(s) set in "dfs.name.dir;
#- The name-node will fail if a legal image is contained in "dfs.name.dir";
#- The name-node verifies that the image in "fs.checkpoint.dir" is consistent, but does not
modify it in any way.
# secondary node storage directory structure is standardized to match the primary node directory
structure.
As a consequence we get protection
#- from accidentally starting multiple secondary nodes in the same directory
#- starting primary and secondary in the same directory.
#- The primary can be started directly with the checkpointed storage if necessary.
# The checkpoint directory contains now 2 last images: "current" has the most recent image,
and
"previous.checkpoint" has the previous one, which (as I understand it) was proposed in HADOOP-2987.
# When the checkpoint starts the name-node sends a CheckpointSignature to the secondary which
contains all 
important information about the name space: layout version, namespaceID, creation time, and
length of the edits log file.
Information if verified by both nodes, and serves as a confirmation that a) the image is merged
correctly, and
b) that the name-node is receiving the right image when it downloads it back from the secondary.
We can later extend it this class with fsimage crcs.
# ClientProtocol is changed as a result of that. rollEditLog() returns the signature instead
of just the length of the edits file.
# The primary and Secondary nodes used to have 2 separate servlets for transferring images.
The problem was that the primary and the secondary were the same web-application in the servlet
container.
I registered secondary under "webapps/secondary" and unified the GetImageServlet so that it
is used for both nodes.
# All logic related to image transferring servlet moved from the NameNode class directly to
FSImage.
# I fixed the bug described in HADOOP-3069.
# TestCheckpoint is substantially modified to cover all new cases of failure.
Particularly, I included the test case for HADOOP-3069, which fails with old code and does
not with the new one.
# StringUtils is extended with getStringCollection(String str) method which splits of comma
delimited string 
and returns a collection rather than an array of Strings as getStrings() does.

> Automatic namespace recovery from the secondary image.
> ------------------------------------------------------
>
>                 Key: HADOOP-2585
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2585
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: dfs
>    Affects Versions: 0.15.0
>            Reporter: Konstantin Shvachko
>         Attachments: SecondaryStorage.patch
>
>
> Hadoop has a three way (configuration controlled) protection from loosing the namespace
image.
> # image can be replicated on different hard-drives of the same node;
> # image can be replicated on a nfs mounted drive on an independent node;
> # a stale replica of the image is created during periodic checkpointing and stored on
the secondary name-node.
> Currently during startup the name-node examines all configured storage directories, selects
the
> most up to date image, reads it, merges with the corresponding edits, and writes to the
new image back 
> into all storage directories. Everything is done automatically.
> If due to multiple hardware failures none of those images on mounted hard drives (local
or remote) 
> are available the secondary image although stale (up to one hour old by default) can
be still 
> used in order to recover the majority of the file system data.
> Currently one can reconstruct a valid name-node image from the secondary one manually.
> It would be nice to support an automatic recovery.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message