hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravi Prakash (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-5346) Replication queues should not be initialized in the middle of IBR processing.
Date Wed, 16 Oct 2013 15:15:42 GMT

     [ https://issues.apache.org/jira/browse/HDFS-5346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Ravi Prakash updated HDFS-5346:

    Attachment: HDFS-5346.branch-23.patch

Hmm.... We realized we can set dfs.namenode.replqueue.threshold-pct to 1.0 or even 1.5 to
make sure that only when the NN enters the Safemode extension period are the replication queues
initialized. Thus truncating the patch to include only the optimization for the condition
to not traverse the TreeMap.

> Replication queues should not be initialized in the middle of IBR processing.
> -----------------------------------------------------------------------------
>                 Key: HDFS-5346
>                 URL: https://issues.apache.org/jira/browse/HDFS-5346
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode, performance
>    Affects Versions: 0.23.9, 2.3.0
>            Reporter: Kihwal Lee
>            Assignee: Ravi Prakash
>             Fix For: 2.3.0, 0.23.10
>         Attachments: HDFS-5346.branch-23.patch, HDFS-5346.branch-23.patch, HDFS-5346.patch,
> When initial block reports are being processed, checkMode() is called from incrementSafeBlockCount().
This causes the replication queues to be initialized in the middle of processing a block report
in the IBR processing mode. If there are many block reports waiting to be processed, SafeModeMonitor
won't be able to make name node leave the safe mode soon. It appears that the block report
processing speed degrades considerably during this time. 

This message was sent by Atlassian JIRA

View raw message