hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14600) LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
Date Thu, 07 Sep 2017 11:56:01 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16156841#comment-16156841

Steve Loughran commented on HADOOP-14600:

bq.  I got lots of cuts and blood with getting mvn test -Dtest=foo running on Window

I like that phrase, I'll have to use it.

If tests are timing out, that's something a (separate) patch should address.

h3. RawLocalFileSystem

L678, 687, 696, 751: need to fallback to non-native code if not present; a lot of client uses
of the hadoop libs don't use the native libs (including all our tests)

we may want to think about actually falling back to the original mechanism if the native IO
Fails, e.g

if (!isPermissionLoaded() && nativeLibPresent) {
  try {
  } catch (NativeIOException ex) {
    LOG.debug("Native call failed", ex)
if (!isPermissionLoaded()) {
  // invoked if 

Probably best to pull this out into some common method "maybeLoadPermissionInfo". 

Looking at the code, I see we don't appear to be doing any synchronization. Really, if a call
is made to get the perm info, it should block other attempts. Making the proposed
{{maybeLoadPermissionInfo()}} synchronized would be the easy way to do this


don't bother with StringifyException. I know a lot of old code uses it, but its best just
to go

throw new RuntimeException(" failure in ..." + e, e)

this adds e.toString() to the exception text, and includes the nested exception there for
code and logs to print.

h3. NativeIO.java 

L451: include the caught exception in the new one

L456. given this adds nothing, just rethrow the caught exception

h3. NativeIO.c

L435. Prefer {{if (path == NULL)}} for consistency.

h3. proposed tests: 

# pass in a null path
# query a nonexistent file

> LocatedFileStatus constructor forces RawLocalFS to exec a process to get the permissions
> ----------------------------------------------------------------------------------------
>                 Key: HADOOP-14600
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14600
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 2.7.3
>         Environment: file:// in a dir with many files
>            Reporter: Steve Loughran
>            Assignee: Ping Liu
>         Attachments: HADOOP-14600.001.patch, TestRawLocalFileSystemContract.java
> Reported in SPARK-21137. a {{FileSystem.listStatus}} call really craws against the local
FS, because {{FileStatus.getPemissions}} call forces  {{DeprecatedRawLocalFileStatus}} tp
spawn a process to read the real UGI values.
> That is: for every other FS, what's a field lookup or even a no-op, on the local FS it's
a process exec/spawn, with all the costs. This gets expensive if you have many files.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message