hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6184) Capture NN's thread dump when it fails over
Date Tue, 06 Jan 2015 03:31:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265635#comment-14265635
] 

Hadoop QA commented on HDFS-6184:
---------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12690183/HDFS-6184-2.patch
  against trunk revision 0c4b112.

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 1 new or modified
test files.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

    {color:green}+1 eclipse:eclipse{color}.  The patch built with eclipse:eclipse.

    {color:red}-1 findbugs{color}.  The patch appears to introduce 3 new Findbugs (version
2.0.3) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:green}+1 core tests{color}.  The patch passed unit tests in hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9144//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9144//artifact/patchprocess/newPatchFindbugsWarningshadoop-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/9144//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9144//console

This message is automatically generated.

> Capture NN's thread dump when it fails over
> -------------------------------------------
>
>                 Key: HDFS-6184
>                 URL: https://issues.apache.org/jira/browse/HDFS-6184
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>            Reporter: Ming Ma
>            Assignee: Ming Ma
>         Attachments: HDFS-6184-2.patch, HDFS-6184.patch
>
>
> We have seen several false positives in terms of when ZKFC considers NN to be unhealthy.
Some of these triggers unnecessary failover. Examples,
> 1. SBN checkpoint caused ZKFC's RPC call into NN timeout. The consequence isn't bad;
just that SBN will quit ZK membership and rejoin it later. But it is unnecessary. The reason
is checkpoint acquires NN global write lock and all rpc requests are blocked. Even though
HAServiceProtocol.monitorHealth doesn't need to acquire NN lock; it still needs to user service
rpc queue.
> 2. When ANN is busy, sometimes the global lock can block other requests. ZKFC's RPC call
timeout. This will trigger failover. The question is even if after the failover, the new ANN
might run into similar issue.
> We can increase ZKFC to NN timeout value to mitigate this to some degree. If ZKFC can
be more accurate in judgment if NN is health or not and can predict the failover will help,
that will be useful. For example, we can,
> 1. Have ZKFC made decision based on NN thread dump.
> 2. Have a dedicated rpc pool for ZKFC > NN. Given health check doesn't need to acquire
NN global lock; so it can go through even if NN is doing checkpointing or very busy.
> Any comments?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message