hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-7097) Allow block reports to be processed during checkpointing on standby name node
Date Fri, 19 Sep 2014 14:47:34 GMT
Kihwal Lee created HDFS-7097:

             Summary: Allow block reports to be processed during checkpointing on standby
name node
                 Key: HDFS-7097
                 URL: https://issues.apache.org/jira/browse/HDFS-7097
             Project: Hadoop HDFS
          Issue Type: Bug
            Reporter: Kihwal Lee
            Priority: Critical

On a reasonably busy HDFS cluster, there are stream of creates, causing data nodes to generate
incremental block reports.  When a standby name node is checkpointing, RPC handler threads
trying to process a full or incremental block report is blocked on the name system's {{fsLock}},
because the checkpointer acquires the read lock on it.  This can create a serious problem
if the size of name space is big and checkpointing takes a long time.

All available RPC handlers can be tied up very quickly. If you have 100 handlers, it only
takes 34 file creates.  If a separate service RPC port is not used, HA transition will have
to wait in the call queue for minutes. Even if a separate service RPC port is configured,
hearbeats from datanodes will be blocked. A standby NN  with a big name space can lose all
data nodes after checkpointing.  The rpc calls will also be retransmitted by data nodes many
times, filling up the call queue and potentially causing listen queue overflow.

Since block reports are not modifying any state that is being saved to fsimage, I propose
letting them through during checkpointing. 

This message was sent by Atlassian JIRA

View raw message