hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Junping Du (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9178) Slow datanode I/O can cause a wrong node to be marked bad
Date Tue, 02 Feb 2016 18:20:40 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15128715#comment-15128715

Junping Du commented on HDFS-9178:

Thanks [~kihwal] for review the patch! I have commit the patch to branch-2.6.

> Slow datanode I/O can cause a wrong node to be marked bad
> ---------------------------------------------------------
>                 Key: HDFS-9178
>                 URL: https://issues.apache.org/jira/browse/HDFS-9178
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>            Priority: Critical
>             Fix For: 2.7.2, 2.6.4
>         Attachments: 002-HDFS-9178.branch-2.6.patch, HDFS-9178.branch-2.6.patch, HDFS-9178.patch
> When non-leaf datanode in a pipeline is slow on or stuck at disk I/O, the downstream
node can timeout on reading packet since even the heartbeat packets will not be relayed down.
> The packet read timeout is set in {{DataXceiver#run()}}:
> {code}
>   peer.setReadTimeout(dnConf.socketTimeout);
> {code}
> When the downstream node times out and closes the connection to the upstream, the upstream
node's {{PacketResponder}} gets {{EOFException}} and it sends an ack upstream with the downstream
node status set to {{ERROR}}.  This caused the client to exclude the downstream node, even
thought the upstream node was the one got stuck.
> The connection to downstream has longer timeout, so the downstream will always timeout
 first. The downstream timeout is set in {{writeBlock()}}
> {code}
>           int timeoutValue = dnConf.socketTimeout +
>               (HdfsConstants.READ_TIMEOUT_EXTENSION * targets.length);
>           int writeTimeout = dnConf.socketWriteTimeout +
>               (HdfsConstants.WRITE_TIMEOUT_EXTENSION * targets.length);
>           NetUtils.connect(mirrorSock, mirrorTarget, timeoutValue);
>           OutputStream unbufMirrorOut = NetUtils.getOutputStream(mirrorSock,
>               writeTimeout);
> {code}

This message was sent by Atlassian JIRA

View raw message