hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suraj Nayak (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5907) Improve getSplits() performance for fs implementations that can utilize performance gains from recursive listing
Date Wed, 12 Apr 2017 15:49:41 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15966089#comment-15966089
] 

Suraj Nayak commented on MAPREDUCE-5907:
----------------------------------------

That's a good suggestion [~steve_l]. I thought the same. There were a lot of changes and improvements
across frameworks. I'll put next comment on overview of changes what can be done. You can
review it and give your suggestions. Will add the patch to it based on that.


> Improve getSplits() performance for fs implementations that can utilize performance gains
from recursive listing
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5907
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5907
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: client
>    Affects Versions: 2.4.0
>            Reporter: Sumit Kumar
>            Assignee: Sumit Kumar
>              Labels: BB2015-05-TBR
>         Attachments: MAPREDUCE-5907-2.patch, MAPREDUCE-5907-3.patch, MAPREDUCE-5907.patch
>
>
> FileInputFormat (both mapreduce and mapred implementations) use recursive listing while
calculating splits. They however do this by doing listing level by level. That means to discover
files in /foo/bar means they do listing at /foo/bar first to get the immediate children, then
make the same call on all immediate children for /foo/bar to discover their immediate children
and so on. This doesn't scale well for object store based fs implementations like s3 and swift
because every listStatus call ends up being a webservice call to backend. In cases where large
number of files are considered for input, this makes getSplits() call slow. 
> This patch adds a new set of recursive list apis that gives opportunity to the fs implementations
to optimize. The behavior remains the same for other implementations (that is a default implementation
is provided for other fs so they don't have to implement anything new). However for objectstore
based fs implementations it provides a simple change to include recursive flag as true (as
shown in the patch) to improve listing performance.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message