hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6304) Use java.io.File.set{Readable|Writable|Executable} where possible in RawLocalFileSystem
Date Thu, 20 Jan 2011 00:49:46 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12983960#action_12983960

Todd Lipcon commented on HADOOP-6304:

Hey Arun. MAPREDUCE-2238 was caused by the use of these same APIs in Localizer.PermissionsHandler.setPermissions.
What was happening was something like the following:
- Thread A: setPermissions("/path/to/userlogs", 755)
- Thread B: setPermissions("/path/to/userlogs/attempt_x", 755)

The way they got interleaved was something like:
- B: set userlogs/attempt_x to 000
- A: set userlogs/ to 000
- B: try to restore permissions on attempt_x, but fail since it can't traverse the path
- A: set userlogs/ back to 755
- The attempt_x directory is left at 000 or some other "incomplete" permissions where any
following Hudson runs won't be able to delete it.

This same problem can happen in a real cluster too.

So I think the pattern used in this patch (same as the one in Localizer.PermissionsHandler.setPermissions)
is dangerous because it's not atomic. This was just one manifestation of the issue.

I agree JNI is difficult but this is a very simple use of it. No buffer management, no complicated
errors to handle, no strange system APIs. Let's have the JNI discussion over in HADOOP-7110

> Use java.io.File.set{Readable|Writable|Executable} where possible in RawLocalFileSystem

> ----------------------------------------------------------------------------------------
>                 Key: HADOOP-6304
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6304
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.20.1
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>         Attachments: HADOOP-6304_yhadoop20.patch
> Using java.io.File.set{Readable|Writable|Executable} where possible in RawLocalFileSystem
when g & o perms are same saves a lot of 'fork' system-calls.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message