hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jack levin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6022) Moving deadNodes from being thread local. Improving dead datanode handling in DFSClient
Date Mon, 03 Mar 2014 20:30:28 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918544#comment-13918544

jack levin commented on HDFS-6022:

Colin, thanks for the comments.  12631558/HADOOP-6022.patch is the trunk patch.

>This seems unnecessary to me. We don't need to be constantly trying to connect to dead
nodes. Just expire >them from a Guava cache after a few minutes, like we do in DFSOutputStream

I think the problem with that will be as follows.  Suppose deadNodes list expires a datanode
which is not yet up, (perhaps its down due to power outage for extended period of time), 
all DFSClient input streams will attempt to open tcp connection to dead datanode again causing
a problem, it is specifically problematic for HBASE given that 60 second tcp timeout would
cause things like connections/thread pileups and downtime for HBASE.  DeadNodeVerifier assures
that that does not happen. Guava cache would work, but in our environment it would mean instability
in HBASE while trying to access blocks in semi-permanently dead DN.

I will move deadNodes to ClientContext.



> Moving deadNodes from being thread local. Improving dead datanode handling in DFSClient

> ----------------------------------------------------------------------------------------
>                 Key: HDFS-6022
>                 URL: https://issues.apache.org/jira/browse/HDFS-6022
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>    Affects Versions: 3.0.0, 0.23.9, 0.23.10, 2.2.0, 2.3.0
>            Reporter: Jack Levin
>              Labels: patch
>             Fix For: 3.0.0, 2.3.0
>         Attachments: HADOOP-6022.patch
>   Original Estimate: 0h
>  Remaining Estimate: 0h
> This patch solves an issue of deadNodes list being thread local.  deadNodes list is created
by DFSClient when some problems with write/reading, or contacting datanode exist.  The problem
is that deadNodes is not visible to other DFSInputStream threads, hence every DFSInputStream
ends up building its own deadNodes.  This affect performance of DFSClient to a large degree
especially when a datanode goes completely offline (there is a tcp connect delay experienced
by all DFSInputStream threads affecting performance of the whole cluster).
> This patch moves deadNodes to be global in DFSClient class so that as soon as a single
DFSInputStream thread reports a dead datanode, all other DFSInputStream threads are informed,
negating the need to create their own independent lists (concurrent Map really). 
> Further, a global deadNodes health check manager thread (DeadNodeVerifier) is created
to verify all dead datanodes every 5 seconds, and remove the same list as soon as it is up.
 That thread under normal conditions (deadNodes empty) would be sleeping.  If deadNodes is
not empty, the thread will attempt to open tcp connection every 5 seconds to affected datanodes.
> This patch has a test (TestDFSClientDeadNodes) that is quite simple, since the deadNodes
creation is not affected by the patch, we only test datanode removal from deadNodes by the
health check manager thread.  Test will create a file in dfs minicluster, read from the same
file rapidly, cause datanode to restart, and test is the health check manager thread does
the right thing, removing the alive datanode from the global deadNodes list.

This message was sent by Atlassian JIRA

View raw message