hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohith Sharma K S (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6543) yarn application's privilege is determined by yarn process creator instead of yarn application user.
Date Tue, 02 May 2017 10:06:04 GMT

    [ https://issues.apache.org/jira/browse/YARN-6543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15992668#comment-15992668
] 

Rohith Sharma K S commented on YARN-6543:
-----------------------------------------

This is default behavior of YARN which uses DefaultContainerExeuctor as a default. For achieving
your usecase, you can use LinuxContainerExecutor.
The details about configuring LCE is given in the doc, refer [LCE|http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SecureMode.html#LinuxContainerExecutor].


> yarn application's privilege is determined by yarn process creator instead of yarn application
user.
> ----------------------------------------------------------------------------------------------------
>
>                 Key: YARN-6543
>                 URL: https://issues.apache.org/jira/browse/YARN-6543
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: wuchang
>
> My application is a pyspark application which is impersonated by user 'wuchang'
> My application infomation is :
> {code}
> Application Report : 
>         Application-Id : application_1493004858240_0007
>         Application-Name : livy-session-6
>         Application-Type : SPARK
>         User : wuchang
>         Queue : root.wuchang
>         Start-Time : 1493708942748
>         Finish-Time : 0
>         Progress : 10%
>         State : RUNNING
>         Final-State : UNDEFINED
>         Tracking-URL : http://10.120.241.82:34462
>         RPC Port : 0
>         AM Host : 10.120.241.82
>         Aggregate Resource Allocation : 4369480 MB-seconds, 2131 vcore-seconds
>         Diagnostics :
> {code}
> And the process is :
> {code}
> appuser  25454 25872  0 15:09 ?        00:00:00 bash /data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/default_container_executor.sh
> appuser  25456 25454  0 15:09 ?        00:00:00 /bin/bash -c /home/jdk/bin/java -server
-Xmx1024m -Djava.io.tmpdir=/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/tmp
'-Dspark.ui.port=0' '-Dspark.driver.port=40969' -Dspark.yarn.app.container.log.dir=/home/log/hadoop/logs/userlogs/application_1493004858240_0007/container_1493004858240_0007_01_000004
-XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
spark://CoarseGrainedScheduler@10.120.241.82:40969 --executor-id 2 --hostname 10.120.241.18
--cores 1 --app-id application_1493004858240_0007 --user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/__app__.jar
--user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-api-0.3.0-SNAPSHOT.jar
--user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-rsc-0.3.0-SNAPSHOT.jar
--user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/netty-all-4.0.29.Final.jar
--user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/commons-codec-1.9.jar
--user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-core_2.11-0.3.0-SNAPSHOT.jar
--user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-repl_2.11-0.3.0-SNAPSHOT.jar
1> /home/log/hadoop/logs/userlogs/application_1493004858240_0007/container_1493004858240_0007_01_000004/stdout
2> /home/log/hadoop/logs/userlogs/application_1493004858240_0007/container_1493004858240_0007_01_000004/stderr
> appuser  25468 25456  2 15:09 ?        00:00:09 /home/jdk/bin/java -server -Xmx1024m
-Djava.io.tmpdir=/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/tmp
-Dspark.ui.port=0 -Dspark.driver.port=40969 -Dspark.yarn.app.container.log.dir=/home/log/hadoop/logs/userlogs/application_1493004858240_0007/container_1493004858240_0007_01_000004
-XX:OnOutOfMemoryError=kill %p org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
spark://CoarseGrainedScheduler@10.120.241.82:40969 --executor-id 2 --hostname 10.120.241.18
--cores 1 --app-id application_1493004858240_0007 --user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/__app__.jar
--user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-api-0.3.0-SNAPSHOT.jar
--user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-rsc-0.3.0-SNAPSHOT.jar
--user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/netty-all-4.0.29.Final.jar
--user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/commons-codec-1.9.jar
--user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-core_2.11-0.3.0-SNAPSHOT.jar
--user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-repl_2.11-0.3.0-SNAPSHOT.jar
> appuser  26936 25846  0 15:16 pts/0    00:00:00 grep --color=auto application_1493004858240_0007
> {code}
> The main problem is that the application user is "wuchang" , but the yarn application
is created by my OS super-user "appuser" , so , the privilege becomes the problem. My code
always run as the privilege of appuser instead of "wuchang".
> For example , below is the pyspark code:
> {code}
> import os
> os.system("hadoop fs -rm -r /user/appuser/test.dat")
> {code}
> user "wuchang" should not have privilege to remove the file test.dat which located in
the home directory of appuser. But since the yarn application process is created by "appuser",
it does, although the yarn application user is "wuchang".



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message