hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13162) Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs
Date Wed, 18 May 2016 18:41:13 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289553#comment-15289553
] 

Steve Loughran commented on HADOOP-13162:
-----------------------------------------

you should think about including rename here, as on s3 rename is about mkdirs, copies and
deletes. 

consider a sequence like
{code}
mkdir d1/d2/d3/
touch file d1/d2/f.txt

assert is dir d1/d2
assert is dir d1/d2/d3

touch d1/d2/d3/f2.txt
assert is dir d1/d2
assert is dir d1/d2/d3

rename d1/d2/d3 d1/d4
assert is dir d1/d4
assert exists d1/d4/f2.txt

rename d1/d2 s1/d4
assert is dir d1/d4
assert is dir d1/d4/d2
assert exists d1/d4/d2/f2.txt
{code}

> Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs
> -----------------------------------------------------------------------
>
>                 Key: HADOOP-13162
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13162
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>            Reporter: Rajesh Balamohan
>            Priority: Minor
>         Attachments: HADOOP-13162-branch-2-002.patch, HADOOP-13162.001.patch
>
>
> getFileStatus is relatively expensive call and mkdirs invokes it multiple times depending
on how deep the directory structure is. It would be good to reduce the number of getFileStatus
calls in such cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message