hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dave Latham (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-7682) taskTracker could not start because "Failed to set permissions" to "ttprivate to 0700"
Date Thu, 19 Apr 2012 18:42:41 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-7682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13257676#comment-13257676

Dave Latham commented on HADOOP-7682:

Here's my understanding of this issue from digging around a bit in case it's helpful for others.
 I could be mistaken on some of the details.

Some time ago, RawLocalFileSystem.setPermission used to use a shell exec command to fork a
process to alter permissions of a file.  

Somewhere along the 0.20 branch (I believe in the 0.20.security branch) it was decided that
this was too slow, and instead java.io.File.set{Readable|Writable|Executable} should be used
(see HADOOP-6304 for one such issue.  It has a patch with the logic that wound up in FileUtil
though perhaps committed from a different patch).  However the java.io.File methods don't
allow one to directly set all the permissions so the code first has to clear permissions,
then build them up again.  This resulted in two problems.  First, there is a race condition
when the file briefly has no permissions even for the owner (see MAPREDUCE-2238 for more detail).
 Second, Windows doesn't support clearing all permissions on the file (this JIRA).

The first problem was worked around in HADOOP-7110 (HADOOP-7432 backported it to this branch)
by using JNI native code instead.  However, if the native code is not available, then it falls
back to the java.io.File methods.  So, the second problem still remains, that FileUtil.setPermission
(and thus the RawLocalFileSystem setPermission) does not work on Windows because Windows does
not have the native code implementation and also fails the java.io.File fallback.

The issues FKorning ran into in a comment above appear to be wider than this particular JIRA,
though I may have misunderstood what led to his shorn yak.

Windows is listed as a supported platform for Hadoop, and some of our developers use Windows
as a development environment, so it's important for us that hadoop at least functions on Windows,
even if it's not as performant as our production clusters on Linux.  I noted that this is
currently the highest voted open JIRA for hadoop.

In order for it to function on Windows, I added a final fallback in FileUtil.setPermission
to revert to the older behavior of using a shell exec fork for setPermission when the other
methods fail.  Perhaps it will be helpful for others:

  public static void setPermission(File f, FsPermission permission
                                   ) throws IOException {
    FsAction user = permission.getUserAction();
    FsAction group = permission.getGroupAction();
    FsAction other = permission.getOtherAction();

    // use the native/fork if the group/other permissions are different
    // or if the native is available    
    if (group != other || NativeIO.isAvailable()) {
      execSetPermission(f, permission);
	    boolean rv = true;
	    // read perms
	    rv = f.setReadable(group.implies(FsAction.READ), false);
	    checkReturnValue(rv, f, permission);
	    if (group.implies(FsAction.READ) != user.implies(FsAction.READ)) {
	      f.setReadable(user.implies(FsAction.READ), true);
	      checkReturnValue(rv, f, permission);
	    // write perms
	    rv = f.setWritable(group.implies(FsAction.WRITE), false);
	    checkReturnValue(rv, f, permission);
	    if (group.implies(FsAction.WRITE) != user.implies(FsAction.WRITE)) {
	      f.setWritable(user.implies(FsAction.WRITE), true);
	      checkReturnValue(rv, f, permission);
	    // exec perms
	    rv = f.setExecutable(group.implies(FsAction.EXECUTE), false);
	    checkReturnValue(rv, f, permission);
	    if (group.implies(FsAction.EXECUTE) != user.implies(FsAction.EXECUTE)) {
	      f.setExecutable(user.implies(FsAction.EXECUTE), true);
	      checkReturnValue(rv, f, permission);
    catch (IOException ioe)
    	LOG.warn("Java file permissions failed to set " + f + " to " + permission + " falling
back to fork");
    	execSetPermission(f, permission);
> taskTracker could not start because "Failed to set permissions" to "ttprivate to 0700"
> --------------------------------------------------------------------------------------
>                 Key: HADOOP-7682
>                 URL: https://issues.apache.org/jira/browse/HADOOP-7682
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions:,, 1.0.0
>         Environment: OS:WindowsXP SP3 , Filesystem :NTFS, cygwin 1.7.9-1, jdk1.6.0_05
>            Reporter: Magic Xie
> ERROR org.apache.hadoop.mapred.TaskTracker:Can not start task tracker because java.io.IOException:Failed
to set permissions of path:/tmp/hadoop-cyg_server/mapred/local/ttprivate to 0700
>     at org.apache.hadoop.fs.RawLocalFileSystem.checkReturnValue(RawLocalFileSystem.java:525)
>     at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:499)
>     at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:318)
>     at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:183)
>     at org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:635)
>     at org.apache.hadoop.mapred.TaskTracker.(TaskTracker.java:1328)
>     at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3430)
> Since hadoop0.20.203 when the TaskTracker initialize, it checks the permission(TaskTracker
Line 624) of (org.apache.hadoop.mapred.TaskTracker.TT_LOG_TMP_DIR,org.apache.hadoop.mapred.TaskTracker.TT_PRIVATE_DIR,
call setPermission(Line 481) to deal with it, setPermission works fine on *nx, however,it
dose not alway works on windows.
> setPermission call setReadable of Java.io.File in the line 498, but according to the
Table1 below provided by oracle,setReadable(false) will always return false on windows, the
same as setExecutable(false).
> http://java.sun.com/developer/technicalArticles/J2SE/Desktop/javase6/enhancements/
> is it cause the task tracker "Failed to set permissions" to "ttprivate to 0700"?
> Hadoop 0.20.202 works fine in the same environment. 

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message