hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ping Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
Date Fri, 01 Sep 2017 20:19:02 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16151090#comment-16151090
] 

Ping Liu commented on HADOOP-14600:
-----------------------------------

Now I am trying to do the patch test on my Windows.  It looks like dev-support/bin/test-patch
is a BASH script and cannot be run on Windows.  Is there any guide on how to run it on Windows
or the patch test is not expected to be run on Windows?

CC: [~aw]

> LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
> ----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-14600
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14600
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.7.3
>         Environment: file:// in a dir with many files
>            Reporter: Steve Loughran
>            Assignee: Ping Liu
>         Attachments: HADOOP-14600.001.patch, TestRawLocalFileSystemContract.java
>
>
> Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws against the local
FS, because {{FileStatus.getPemissions}} call forces  {{DeprecatedRawLocalFileStatus}} tp
spawn a process to read the real UGI values.
> That is: for every other FS, what's a field lookup or even a no-op, on the local FS it's
a process exec/spawn, with all the costs. This gets expensive if you have many files.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message