hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3095) Validating input paths and creating splits is slow on S3
Date Mon, 31 Mar 2008 22:52:26 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12583917#action_12583917
] 

Doug Cutting commented on HADOOP-3095:
--------------------------------------

FileInputFormat#getSplits() in trunk still calls FileSystem#exists(), FileSystem#isDirectory(),
FileSystem#getLength() (twice) and FileSystem#getBlockSize() for each input file.  These could
all be eliminated if it used listStatus() instead of listPaths().  The cost of these is negligible
for HDFS currently, since these are all cached in the Path instance.  But once we eliminate
that cache, performance on HDFS will suffer.

FileSystem#getBlockLocations() is also called once per file and would still need to be called,
since FileStatus doesn't (yet) include that information.

Also, do you see a reason for validateInput to continue to exist?  Without the status cache
it too will become very slow, and its purpose is now outdated.

> Validating input paths and creating splits is slow on S3
> --------------------------------------------------------
>
>                 Key: HADOOP-3095
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3095
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: fs, fs/s3
>            Reporter: Tom White
>
> A call to listPaths on S3FileSystem results in an S3 access for each file in the directory
being queried. If the input contains hundreds or thousands of files this is prohibitively
slow. This method is called in FileInputFormat.validateInput and FileInputFormat.getSplits.
This would be easy to fix by overriding listPaths (all four variants) in S3FileSystem to not
use listStatus which creates a FileStatus object for each subpath. However, listPaths is deprecated
in favour of listStatus so this might be OK as a short term measure, but not longer term.
> But it gets worse: FileInputFormat.getSplits goes on to access S3 a further six times
for each input file via these calls:
> 1. fs.isDirectory
> 2. fs.exists
> 3. fs.getLength
> 4. fs.getLength
> 5. fs.exists (from fs.getFileBlockLocations)
> 6. fs.getBlockSize
> So it would be best to change getSplits to use listStatus, and only access S3 once for
each file. (This would help HDFS too.) This change would require some care since FileInputFormat
has a protected method listPaths which subclasses can override (although, in passing I notice
validateInput doesn't use listPaths - is this a bug?).
> For input validation, one approach would be to disable it for S3 by creating a custom
FileInputFormat. In this case, missing files would be detected during split generation. Alternatively,
it may be possible to cache the input paths between validateInput and getSplits.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message