hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-7110) Implement chmod with JNI
Date Wed, 19 Jan 2011 17:26:49 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12983805#action_12983805

Todd Lipcon commented on HADOOP-7110:

bq. So if it fails right after setting them to 000 it will be hard to remove such file or

It's not due to failure of the process doing the chmod. It's due to other processes or threads
working inside the directory failing because there is a moment in which they can't traverse
the path inside the directory. As we've seen in MR-2238 this race really does happen - we've
got build failures on three separate environments (Apache Hudson, my Hudson, and Greg's build
box) to prove it.

bq. So why don't we just set permissions to the values specified (in the right order, which
is probably important) rather than introducing more JNI code

The API is not flexible enough to do this. If you have a file 755 and want it to be 700, the
only way to get rid of the group/other bits is to first clear those bits for all, then add
them back for user only.

bq. Nobody said we should support posix permissions

Since we now support security, we have to be able to make some files mode 700 and others 755,
etc. There's no way around it. The question here is implementation method, not whether we
should support an API since we've had since 2008. Removing the ability to chmod isn't an option,
considering compatibility.

bq. Your patch is an alternative implementation of HADOOP-6304

And HADOOP-6304 is a sort-of-copy-paste of code originally introduced in MAPREDUCE-842. I
filed this JIRA as part of the fix for MAPREDUCE-2238 (which is caused by the MR code that
has the same problems)

Do you have a particular issue with the JNI code?

> Implement chmod with JNI
> ------------------------
>                 Key: HADOOP-7110
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7110
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: io, native
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>         Attachments: hadoop-7110.txt
> MapReduce is currently using a race-prone workaround to approximate chmod() because forking
chmod is too expensive. This race is causing build failures (and probably task failures too).
We should implement chmod in the NativeIO library so we can have good performance (ie not
fork) and still not be prone to races.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message