hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5016) Heartbeating thread blocks under some failure conditions leading to loss of datanodes
Date Fri, 26 Jul 2013 01:12:48 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13720329#comment-13720329
] 

Andrew Wang commented on HDFS-5016:
-----------------------------------

Patch looks good, and I agree we can just close out HDFS-4851 after this one goes in.

Some nitty comments:

bq. +          final String msg = "Join on writer thread timedout "
bq. +            String msg = "Responder thread join timedout\n"
Needs a space in "timed out".

{code}
        if (writer.isAlive()) {
          final String msg = "Join on writer thread timedout "
              + writer.toString() + "\n" + StringUtils.getStackTrace(writer);
          DataNode.LOG.warn(msg);
          throw new IOException(msg);
{code}

We probably don't want to stick the entire stack trace in the IOException msg. Same for BlockReceiver.

It's somewhat ambiguous from the log message who's name and trace we're printing here, could
we instead say "Timeout while stopping writer thread <thread name>:" followed by the
stack trace?

For BlockReceiver, how about "Timeout while aborting responder thread <thread name>:"
for consistency? Unsure if you wanted to put the thread name here, since it's missing right
now.
                
> Heartbeating thread blocks under some failure conditions leading to loss of datanodes
> -------------------------------------------------------------------------------------
>
>                 Key: HDFS-5016
>                 URL: https://issues.apache.org/jira/browse/HDFS-5016
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Devaraj Das
>            Assignee: Suresh Srinivas
>            Priority: Blocker
>             Fix For: 2.1.0-beta
>
>         Attachments: HDFS-5016.1.patch, HDFS-5016.2.patch, HDFS-5016.patch, jstack1.txt
>
>
> In the testing of some failure scenarios for HBase MTTR, we have been simulating node
failures via firewalling of nodes (where all communication ports would be firewalled except
ssh's port). We have noticed that when a (data)node is firewalled, we lose certain other datanodes
- those that were involved in some communication with the firewalled node before the latter
was firewalled. Will attach jstack output from one of the lost datanodes. The heartbeating
thread seems to be locked up.
> This jira is to track a fix for the problem.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message