hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suresh Srinivas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9377) FTPFileSystem.listStatus() runs very slow, due to inappropriate call of filePath.makeQualified
Date Thu, 07 Mar 2013 07:42:13 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595648#comment-13595648
] 

Suresh Srinivas commented on HADOOP-9377:
-----------------------------------------

bq. the behavior in the posted patch should be fine.
My cursory look at the patch, I came to the same conclusion. What I did not have time to finish
was, should FTPFileSystem be caching home directory from the time of the creation and not
use remote connection to get that information at all.

Thoughts?
                
> FTPFileSystem.listStatus() runs very slow, due to inappropriate call of filePath.makeQualified
> ----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-9377
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9377
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.3-alpha
>            Reporter: James Yu
>         Attachments: HADOOP-9377.diff
>
>
> FTPFileSystem.listStatus() calls
> getFileStatus(ftpFiles[i], absolute) calls
> new FileStatus(....) calls 
> filePath.makeQualified(...) calls
> fs.getWorkingDirectory() calls
> getHomeDirectory()
> which creates new FTP connection every time, to get the workdingDirectory. this caused
the FTPFileSystem.listStatus() takes long time to run (on average 3-6 seconds per file in
my test).
> I attach a suggestion of fix in FTPFileSystem.java, only 4 lines of change. after the
fix, there's no slowness issue anymore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message