hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "nkeywal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-3912) Detecting and avoiding stale datanodes for writing
Date Thu, 13 Sep 2012 20:49:08 GMT

    [ https://issues.apache.org/jira/browse/HDFS-3912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455262#comment-13455262

nkeywal commented on HDFS-3912:

Some thinking, with an HBase bias:
- if the datanode is too busy and cannot heartbeat in a minute, we will also get timeouts
when writing the blocks (if the datanode is dead: 20s connect timeout. If it's not dead, or
if we had previously a connection, we will fail on the read timeout for the ack, it's around
1 minute by default).
- the recovery is on the critical path, so going to a suspicious node is not something you
want to do.
- things are already quite complicated, so I think I would end up with the same value for
read & write to keep them simple.

Then there is the case when many nodes are staled. I think we're in a really bad shape at
this stage... I feel that just throwing an exception is the best solution. HBase would wait
a few seconds and retry. That's better for the cluster than trying a node that is unlikely
to execute the write. But it's a kind of change vs. today's behavior.

To synthesis, this could make sense imho:
- there are enough fully alive nodes: let's use them, whatever the number of stale nodes.
- there are not enough fully alive nodes, but there are some stale nodes that we could use:
let's use the stale nodes them, at least the behavior will be backward compatible.
- there are not enough live node: as today.

> Detecting and avoiding stale datanodes for writing
> --------------------------------------------------
>                 Key: HDFS-3912
>                 URL: https://issues.apache.org/jira/browse/HDFS-3912
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Jing Zhao
>            Assignee: Jing Zhao
> 1. Make stale timeout adaptive to the number of nodes marked stale in the cluster.
> 2. Consider having a separate configuration for write skipping the stale nodes.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message