hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yiqun Lin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-13571) Dead DataNode Detector
Date Mon, 15 Jul 2019 15:41:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-13571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16885337#comment-16885337

Yiqun Lin commented on HDFS-13571:

Great, please add the description of the subtasks that you created. I will try to follow
up this.

> Dead DataNode Detector
> ----------------------
>                 Key: HDFS-13571
>                 URL: https://issues.apache.org/jira/browse/HDFS-13571
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>    Affects Versions: 2.4.0, 2.6.0, 3.0.2
>            Reporter: Gang Xie
>            Assignee: Lisheng Sun
>            Priority: Minor
>         Attachments: HDFS-13571-2.6.diff, node status machine.png
> Currently, the information of the dead datanode in DFSInputStream in stored locally.
So, it could not be shared among the inputstreams of the same DFSClient. In our production
env, every days, some datanodes dies with different causes. At this time, after the first
inputstream blocked and detect this, it could share this information to others in the same
DFSClient, thus, the ohter inputstreams are still blocked by the dead node for some time,
which could cause bad service latency.
> To eliminate this impact from dead datanode, we designed a dead datanode detector, which
detect the dead ones in advance, and share this information among all the inputstreams in
the same client. This improvement has being online for some months and works fine.  So,
we decide to port to the 3.0 (the version used in our production env is 2.4 and 2.6).
> I will do the porting work and upload the code later.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message