hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ping Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
Date Sat, 30 Sep 2017 04:07:02 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16186875#comment-16186875

Ping Liu commented on HADOOP-14600:


Thanks for your detailed code review!  Really appreciate it.

I just finished making recommended changes and attached *HADOOP-14600.003.patch*.

Following are details.

{{Helper.java (new class):}}

For better supporting unit test, I added some more testing mechanism and moved the logic into
a new class called Helper.java.  Now I can not only check permission but also change permission.
I didn't find a place where we can put the utilities.  So just add this one.  In case, if
one want to added other common utility method.  Helper class can be the place.


With this addition, I can improve test by adding testPermission() into TestRawLocalFileSystemContract
where now both loadPermissionInfoByNativeIO() and loadPermissionInfoByNonNativeIO() can be
directly tested as you suggested. I can test it on Linux.  But on Windows, sticky bit change
doesn't take effect.  I guess Windows probably doesn't have sticky bit feature.


Similarly, doStatTest() was simplified by calling the Helper method.

Also improved testStatOnError() by using LambdaTestUtils, improved testMultiThreadedStat()
by using ExecutorService and Future, also adding testMultiThreadedStatOnError().


A minor issue found (loadPermissionInfo()) is that domain returned with group in Windows.
 So we need remove domain.  This is the same as removing domain from domain/user in existing

Lastly, for {{NativeIO.c}}, FindFileOwnerAndPermission is not a MSDN function.  I found it
defined at Line 811 in hadoop-common/src/main/winutils/libwinutils.c.

> LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
> ----------------------------------------------------------------------------------------
>                 Key: HADOOP-14600
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14600
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.7.3
>         Environment: file:// in a dir with many files
>            Reporter: Steve Loughran
>            Assignee: Ping Liu
>         Attachments: HADOOP-14600.001.patch, HADOOP-14600.002.patch, HADOOP-14600.003.patch,
> Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws against the local
FS, because {{FileStatus.getPemissions}} call forces  {{DeprecatedRawLocalFileStatus}} tp
spawn a process to read the real UGI values.
> That is: for every other FS, what's a field lookup or even a no-op, on the local FS it's
a process exec/spawn, with all the costs. This gets expensive if you have many files.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message