hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-779) Automatic move to safe-mode when cluster size drops
Date Mon, 13 Sep 2010 07:21:36 GMT

    [ https://issues.apache.org/jira/browse/HDFS-779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12908651#action_12908651

Joydeep Sen Sarma commented on HDFS-779:

@Dhruba - i think the real problem is why datanodes are being declared 'dead' in the first
place. if we hadn't declared the nodes dead - there wouldn't be a problem to begin with.

the notion of a fixed timeout is problematic. the configuration of a timeout is based on our
expectations of good behavior. but if large numbers of datanodes exhibit much longer heartbeat
intervals - then we need to reset our expectations. 

Cassandra uses a modified version of accrual failure detector - and i think that's a better
general direction (there's a lot of detail that needs to be worked out though). Speculative
execution is a good point in comparison - there's no fixed timeout to start speculation. instead
the current scheme compares task progress to the mean/stddev of progress rates for related
tasks. For DFS - taking network toptology into account would make sense - all the nodes of
a rack not sending timely hearbeats should be treated differently from the same number of
nodes across different racks not sending timely heartbeats (in that sense datanodes in the
same rack are related as map tasks in the same MR job are).

not declaring nodes dead is much better. it means that the system can keep taking new writes
(potentially) even as admins work out the real problem (as opposed to entering safe mode).
it may also mean that resynchronization between datanodes who report late (but are not declared
dead) and the namenode may be less expensive.

> Automatic move to safe-mode when cluster size drops
> ---------------------------------------------------
>                 Key: HDFS-779
>                 URL: https://issues.apache.org/jira/browse/HDFS-779
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: name-node
>            Reporter: Owen O'Malley
>            Assignee: dhruba borthakur
> As part of looking at using Kerberos, we want to avoid the case where both the primary
(and optional secondary) KDC go offline causing a replication storm as the DataNodes' service
tickets time out and they lose the ability to connect to the NameNode. However, this is a
specific case of a more general problem of loosing too many nodes too quickly. I think we
should have an option to go into safe mode if the cluster size goes down more than N% in terms
of DataNodes.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message