hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-8865) Improve quota initialization performance
Date Thu, 06 Aug 2015 18:01:05 GMT
Kihwal Lee created HDFS-8865:
--------------------------------

             Summary: Improve quota initialization performance
                 Key: HDFS-8865
                 URL: https://issues.apache.org/jira/browse/HDFS-8865
             Project: Hadoop HDFS
          Issue Type: Improvement
            Reporter: Kihwal Lee


After replaying edits, the whole file system tree is recursively scanned in order to initialize
the quota. For big name space, this can take a very long time.  Since this is done during
namenode failover, it also affects failover latency.

By using the Fork-Join framework, I was able to greatly reduce the initialization time.  The
following is the test result using the fsimage from one of the big name nodes we have.

|| threads || seconds||
| 1 (existing) | 55|
| 1 (fork-join) | 68 |
| 4 | 16 |
| 8 | 8 |
| 12 | 6 |
| 16 | 5 |
| 20 | 4 |




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message