hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo (Nicholas), SZE (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6467) Performance improvement for liststatus on directories in hadoop archives.
Date Sun, 14 Feb 2010 06:54:27 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12833521#action_12833521
] 

Tsz Wo (Nicholas), SZE commented on HADOOP-6467:
------------------------------------------------

The patch improved first part a lot.

[Without the patch|https://issues.apache.org/jira/browse/HADOOP-6467?focusedCommentId=12830401&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12830401],
it took ~25 mins to show
{noformat}
10/02/06 00:07:17 INFO input.FileInputFormat: Total input paths to process : 100000
{noformat}
With the patch, it only took 14 seconds.
{noformat}
Sun Feb 14 00:18:08 UTC 2010
10/02/14 00:18:22 INFO input.FileInputFormat: Total input paths to process : 100000
{noformat}


> Performance improvement for liststatus on directories in hadoop archives.
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-6467
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6467
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Mahadev konar
>            Assignee: Mahadev konar
>             Fix For: 0.22.0
>
>         Attachments: Archives_performance.docx, Archives_performance.docx, HADOOP-6467.patch
>
>
> A liststatus call on a directory in hadoop archives leads to ( 2* number of files in
directory) open calls to the namenode. This is very sub optimal and needs to be fixed to make
it performant enough to be used on a daily basis. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message