hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ping Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
Date Tue, 05 Sep 2017 19:53:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154192#comment-16154192
] 

Ping Liu commented on HADOOP-14600:
-----------------------------------

I couldn't successfully set up a local environment to run test-patch.  So I went to the test
result at https://builds.apache.org/job/PreCommit-HADOOP-Build/13153/testReport/ in the above
table from [~hadoopqa].

I did manual test on all of five tests as follows.

* TestSFTPFileSystem.testStatFile
* TestDNS.testDefaultDnsServer
* TestRaceWhenRelogin.test
* TestKDiag.testKeytabAndPrincipal
* TestKDiag.testFileOutput
* TestKDiag.testLoadResource

But none of the tests hits on the new method *loadPermissionInfoByNativeIO()* in *RawLocalFileSystem*
-- *loadPermissionInfoByNativeIO()* is the new code that swaps the original *_loadPermissionInfo()_*
and is the only change to the previous version.

Additionally, I ran "mvn test -Pnative -Dtest=allNative" on my local environment and found
3 failures and 5 errors.

But they are mainly timed out.  After giving more time, majority of the tests passed.  For
 TestRPCWaitForProxy.testInterruptedWaitForProxy, it's the only one still generating error
after timeout time has been increased.  However, manual test on it didn't hit the break point
in *loadPermissionInfoByNativeIO()* too.

In summary, I didn't find any failed test case for the target new method, *loadPermissionInfoByNativeIO()*.
 Please let me know if this is enough for the verification or there are more tests to run
and how.

CC: [~hadoopqa]



> LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-14600
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14600
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.7.3
>         Environment: file:// in a dir with many files
>            Reporter: Steve Loughran
>            Assignee: Ping Liu
>         Attachments: HADOOP-14600.001.patch, TestRawLocalFileSystemContract.java
>
>
> Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws against the local
FS, because {{FileStatus.getPemissions}} call forces  {{DeprecatedRawLocalFileStatus}} tp
spawn a process to read the real UGI values.
> That is: for every other FS, what's a field lookup or even a no-op, on the local FS it's
a process exec/spawn, with all the costs. This gets expensive if you have many files.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message