Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-issues@hadoop.apache.org
Date: Thu, 7 Mar 2013 07:00:18 +0000 (UTC)
From: "Andrew Wang (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.12635732.1362635022144.397895.1362639618148@arcas>
In-Reply-To: <JIRA.12635732.1362635022144@arcas>
References: <JIRA.12635732.1362635022144@arcas>
Subject: [jira] [Commented] (HADOOP-9377) FTPFileSystem.listStatus() runs
 very slow, due to inappropriate call of filePath.makeQualified
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-9377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13595619#comment-13595619 ] 

Andrew Wang commented on HADOOP-9377:
-------------------------------------

I looked into this a bit, the slowness is probably because of the call to {{fs#getWorkingDirectory}} inside makeQualified:

{code}
  @Deprecated
  public Path makeQualified(FileSystem fs) {
    return makeQualified(fs.getUri(), fs.getWorkingDirectory());
  }
{code}

Looking in {{FTPFileSystem#getWorkingDirectory}}, this is kind of a bogus call, since it returns the user's home directory, which involves connecting to the FTP server (slow!). So, unless people are depending on their relative paths being qualified with their home dir, the behavior in the posted patch should be fine.
                
> FTPFileSystem.listStatus() runs very slow, due to inappropriate call of filePath.makeQualified
> ----------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-9377
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9377
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.0.3-alpha
>            Reporter: James Yu
>         Attachments: HADOOP-9377.diff
>
>
> FTPFileSystem.listStatus() calls
> getFileStatus(ftpFiles[i], absolute) calls
> new FileStatus(....) calls 
> filePath.makeQualified(...) calls
> fs.getWorkingDirectory() calls
> getHomeDirectory()
> which creates new FTP connection every time, to get the workdingDirectory. this caused the FTPFileSystem.listStatus() takes long time to run (on average 3-6 seconds per file in my test).
> I attach a suggestion of fix in FTPFileSystem.java, only 4 lines of change. after the fix, there's no slowness issue anymore.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira