hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haibo Chen (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-5765) LinuxContainerExecutor creates appcache and its subdirectories with wrong group owner.
Date Fri, 04 Nov 2016 20:10:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15637534#comment-15637534
] 

Haibo Chen edited comment on YARN-5765 at 11/4/16 8:10 PM:
-----------------------------------------------------------

Hey, [~Naganarasimha] Sorry for getting in the way again while you are working to post a patch.
For some reason that I am not aware of, umask(0027) did not work for me. And here is my theory
why it does not work.
Please correct me if I am wrong.

In a nutshell, in my case, the appcache directory is first created with ownership being \{user:
systest, group: hadoop\} and gid set. Then because systest does not belong to group hadoop
in my cluster setup,
chmod(appcache, perm) that runs as user systest will clear the gid. Consequentially, all the
subdirectories and files created under appcache by code that runs as systest will not be owned
by group hadoop any more.
NM that runs as a user that only belongs to hadoop will have no permission to these files/directories.

In my experiment with umask, it seems to me that chmod clears the gid as long as systest does
not belong to hadoop, regardless of the umask. If umask approach has been working for you,
I must have misunderstood
your approach and will verify your patch once it's posted. One alternative I can think of,
is to revert YARN-5287 patch and follow the other approach as you pointed out (explicitly
set umask) to fix YARN-5287. Thought?


was (Author: haibochen):
Hey, [~Naganarasimha] Sorry for getting in the way again while you are working to post a patch.
For some reason that I am not aware of, umask(0027) did not work for me. And here is my theory
why it does not work.
Please correct me if I am wrong.

In a nutshell, in my case, the appcache directory is first created with ownership being {user:
systest, group: hadoop} and gid set. Then because systest does not belong to group hadoop
in my cluster setup,
chmod(appcache, perm) that runs as user systest will clear the gid. Consequentially, all the
subdirectories and files created under appcache by code that runs as systest will not be owned
by group hadoop any more.
NM that runs as a user that only belongs to hadoop will have no permission to these files/directories.

In my experiment with umask, it seems to me that chmod clears the gid as long as systest does
not belong to hadoop, regardless of the umask. If umask approach has been working for you,
I must have misunderstood
your approach and will verify your patch once it's posted. One alternative I can think of,
is to revert YARN-5287 patch and follow the other approach as you pointed out (explicitly
set umask) to fix YARN-5287. Thought?

> LinuxContainerExecutor creates appcache and its subdirectories with wrong group owner.
> --------------------------------------------------------------------------------------
>
>                 Key: YARN-5765
>                 URL: https://issues.apache.org/jira/browse/YARN-5765
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.8.0, 3.0.0-alpha1
>            Reporter: Haibo Chen
>            Assignee: Naganarasimha G R
>            Priority: Blocker
>
> LinuxContainerExecutor creates usercache/\{userId\}/appcache/\{appId\} with wrong group
owner, causing Log aggregation and ShuffleHandler to fail because node manager process does
not have permission to read the files under the directory.
> This can be easily reproduced by enabling LCE and submitting a MR example job as a user
that does not belong to the same group that NM process belongs to. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message