hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8865) Improve quota initialization performance
Date Thu, 20 Aug 2015 21:48:47 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14705828#comment-14705828
] 

Kihwal Lee commented on HDFS-8865:
----------------------------------

Thanks for the review [~xyao].
1. Numbers for small/medium size name space: I do not have numbers for them, but I expect
them to be better than before. The test cases show it taking a couple of milliseconds for
a very small name space, so the overhead of fork-join doesn't seem to be of too much concern.
For very small namespaces, the ones that take far less than a second to initialize, the multithreaded
initialization cpuld take a bit longer.

2. This involves conversion to slf4j. I started down this path, but ended up needing to update
{{IOUtils}} as well. Since it is not a matter of updating just {{FSImage}}, I won't do it
in this jira. Instead I will simply wrap it with {{isDebugEnabled()}} check for now.

> Improve quota initialization performance
> ----------------------------------------
>
>                 Key: HDFS-8865
>                 URL: https://issues.apache.org/jira/browse/HDFS-8865
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>         Attachments: HDFS-8865.patch, HDFS-8865.v2.checkstyle.patch, HDFS-8865.v2.patch
>
>
> After replaying edits, the whole file system tree is recursively scanned in order to
initialize the quota. For big name space, this can take a very long time.  Since this is done
during namenode failover, it also affects failover latency.
> By using the Fork-Join framework, I was able to greatly reduce the initialization time.
 The following is the test result using the fsimage from one of the big name nodes we have.
> || threads || seconds||
> | 1 (existing) | 55|
> | 1 (fork-join) | 68 |
> | 4 | 16 |
> | 8 | 8 |
> | 12 | 6 |
> | 16 | 5 |
> | 20 | 4 |



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message