hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "nkeywal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3912) Detecting and avoiding stale datanodes for writing
Date Thu, 13 Sep 2012 20:49:08 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455262#comment-13455262
] 

nkeywal commented on HDFS-3912:
-------------------------------

Some thinking, with an HBase bias:
- if the datanode is too busy and cannot heartbeat in a minute, we will also get timeouts
when writing the blocks (if the datanode is dead: 20s connect timeout. If it's not dead, or
if we had previously a connection, we will fail on the read timeout for the ack, it's around
1 minute by default).
- the recovery is on the critical path, so going to a suspicious node is not something you
want to do.
- things are already quite complicated, so I think I would end up with the same value for
read & write to keep them simple.

Then there is the case when many nodes are staled. I think we're in a really bad shape at
this stage... I feel that just throwing an exception is the best solution. HBase would wait
a few seconds and retry. That's better for the cluster than trying a node that is unlikely
to execute the write. But it's a kind of change vs. today's behavior.

To synthesis, this could make sense imho:
- there are enough fully alive nodes: let's use them, whatever the number of stale nodes.
- there are not enough fully alive nodes, but there are some stale nodes that we could use:
let's use the stale nodes them, at least the behavior will be backward compatible.
- there are not enough live node: as today.

                
> Detecting and avoiding stale datanodes for writing
> --------------------------------------------------
>
>                 Key: HDFS-3912
>                 URL: https://issues.apache.org/jira/browse/HDFS-3912
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Jing Zhao
>            Assignee: Jing Zhao
>
> 1. Make stale timeout adaptive to the number of nodes marked stale in the cluster.
> 2. Consider having a separate configuration for write skipping the stale nodes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message