hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9311) Support optional offload of NameNode HA service health checks to a separate RPC server.
Date Thu, 19 May 2016 23:52:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15292373#comment-15292373
] 

Chris Nauroth commented on HDFS-9311:
-------------------------------------

[~magnum], thank you for your comment.  I think we can improve that log message to use the
lifeline address when it's configured.  I filed a follow-up patch on HDFS-10438.

> Support optional offload of NameNode HA service health checks to a separate RPC server.
> ---------------------------------------------------------------------------------------
>
>                 Key: HDFS-9311
>                 URL: https://issues.apache.org/jira/browse/HDFS-9311
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: ha, namenode
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>             Fix For: 2.8.0
>
>         Attachments: HDFS-9311.001.patch, HDFS-9311.002.patch, HDFS-9311.003.patch
>
>
> When a NameNode is overwhelmed with load, it can lead to resource exhaustion of the RPC
handler pools (both client-facing and service-facing).  Eventually, this blocks the health
check RPC issued from ZKFC, which triggers a failover.  Depending on fencing configuration,
the former active NameNode may be killed.  In an overloaded situation, the new active NameNode
is likely to suffer the same fate, because client load patterns don't change after the failover.
 This can degenerate into flapping between the 2 NameNodes without real recovery.  If a NameNode
had been killed by fencing, then it would have to transition through safe mode, further delaying
time to recovery.
> This issue proposes a separate, optional RPC server at the NameNode for isolating the
HA health checks.  These health checks are lightweight operations that do not suffer from
contention issues on the namesystem lock or other shared resources.  Isolating the RPC handlers
is sufficient to avoid this situation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message