hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sameer Paranjpye (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-306) Safe mode and name node startup procedures
Date Thu, 24 Aug 2006 20:16:30 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-306?page=comments#action_12430252 ] 
Sameer Paranjpye commented on HADOOP-306:

I wonder if it makes sense to wait for 100% of blocks to be available. It's possible  for
data to go missing because of failed drives, nodes, racks... When some data goes missing it's
usually the case that some other (co-located) data becomes under-replicated and the Namenode
ought to start replicating the under replicated data. Why do we want safe mode?

* To prevent replication thrash when the Namenode starts
* To enable administrators to make the file system read only for diagnosis and debugging

Neither of these require that 100% of blocks are present. Maybe we should have a slightly
lower threshold for blocks or storage ids.

> Safe mode and name node startup procedures
> ------------------------------------------
>                 Key: HADOOP-306
>                 URL: http://issues.apache.org/jira/browse/HADOOP-306
>             Project: Hadoop
>          Issue Type: New Feature
>    Affects Versions: 0.3.2
>            Reporter: Konstantin Shvachko
>         Assigned To: Konstantin Shvachko
>             Fix For: 0.6.0
> This is a proposal to improve DFS cluster startup process.
> The data node startup procedures were described and implemented in HADOOP-124.
> I'm trying to extend them to the name node here.
> The main idea is to introduce safe mode, which can be entered manually for administration
> purposes, or automatically when a configurable threshold of active data nodes is breached,
> or at startup when the node stays in safe mode until the minimal limit of active
> nodes is reached.
> This are high level requirements intended to improve the name node and cluster reliability.
>     = The name node safe mode means that the name node is not changing the state of the
>        file system. Meta data is read-only, and block replication / removal is not taking
>     = In safe mode the name node accepts data node registrations and
>        processes their block reports.
>     = The name node always starts in safe mode and stays safe until the majority
>         (a configurable parameter: safemode.threshold) of data nodes (or blocks?)
>         is reported.
>     = The name node can also fall into safe mode when the number of non-active
>         (heartbeats stopped coming in) data nodes becomes critical.
>     = The startup "silent period", when the name node is in safe mode and is
>         not issuing any block requests to the data nodes, is initially set to a
>         configurable value safemode.timeout.increment. By the end of the timeout
>         the name node checks the safemode.threshold and decides whether to switch
>         to the normal mode or to stay in safe. If the normal mode criteria is not
>         met, then the silent period is extended by incrementing the safemode timeout.
>     = The name node stays in safe mode not longer than a configurable value of
>         safemode.timeout.max, in which case it logs missing data nodes and shuts
>         itself down.
>     = When the name node switches to normal mode it checks whether all required
>         data nodes have actually registered, based on the list of active data storages
>         from the last session. Then it logs missing nodes, if any, and starts
>         replicating and/or deleting blocks as required.
>     = A historical list of data storages (nodes) ever registered with the cluster is
>         persistently stored in the image and log files. The list is used in two ways:
>         a) at startup to verify whether all nodes have registered, and to report
>         missing nodes;
>         b) at runtime if a data node registers with a new storage id the
>         name node verifies that no new blocks are reported from that storage,
>         which would prevent us from accidentally connecting data nodes from a
>         different cluster.
>     = The name node should have an option to run in safe mode. Starting with
>         that option would mean it never leaves safe mode.
>         This is useful for testing the cluster.
>     = Data nodes that can not connect to the name node for a long time (configurable)
>         should shut down themselves.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message