hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "wuchang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-6543) yarn application's privilege is determined by yarn process creator instead of yarn application user.
Date Tue, 02 May 2017 08:00:15 GMT
wuchang created YARN-6543:
-----------------------------

             Summary: yarn application's privilege is determined by yarn process creator instead
of yarn application user.
                 Key: YARN-6543
                 URL: https://issues.apache.org/jira/browse/YARN-6543
             Project: Hadoop YARN
          Issue Type: Bug
            Reporter: wuchang


My application is a pyspark application which is impersonated by user 'wuchang'
My application infomation is :
{code}
Application Report : 
        Application-Id : application_1493004858240_0007
        Application-Name : livy-session-6
        Application-Type : SPARK
        User : wuchang
        Queue : root.wuchang
        Start-Time : 1493708942748
        Finish-Time : 0
        Progress : 10%
        State : RUNNING
        Final-State : UNDEFINED
        Tracking-URL : http://10.120.241.82:34462
        RPC Port : 0
        AM Host : 10.120.241.82
        Aggregate Resource Allocation : 4369480 MB-seconds, 2131 vcore-seconds
        Diagnostics :
{code}
And the process is :

{code}
appuser  25454 25872  0 15:09 ?        00:00:00 bash /data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/default_container_executor.sh
appuser  25456 25454  0 15:09 ?        00:00:00 /bin/bash -c /home/jdk/bin/java -server -Xmx1024m
-Djava.io.tmpdir=/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/tmp
'-Dspark.ui.port=0' '-Dspark.driver.port=40969' -Dspark.yarn.app.container.log.dir=/home/log/hadoop/logs/userlogs/application_1493004858240_0007/container_1493004858240_0007_01_000004
-XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
spark://CoarseGrainedScheduler@10.120.241.82:40969 --executor-id 2 --hostname 10.120.241.18
--cores 1 --app-id application_1493004858240_0007 --user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/__app__.jar
--user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-api-0.3.0-SNAPSHOT.jar
--user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-rsc-0.3.0-SNAPSHOT.jar
--user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/netty-all-4.0.29.Final.jar
--user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/commons-codec-1.9.jar
--user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-core_2.11-0.3.0-SNAPSHOT.jar
--user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-repl_2.11-0.3.0-SNAPSHOT.jar
1> /home/log/hadoop/logs/userlogs/application_1493004858240_0007/container_1493004858240_0007_01_000004/stdout
2> /home/log/hadoop/logs/userlogs/application_1493004858240_0007/container_1493004858240_0007_01_000004/stderr
appuser  25468 25456  2 15:09 ?        00:00:09 /home/jdk/bin/java -server -Xmx1024m -Djava.io.tmpdir=/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/tmp
-Dspark.ui.port=0 -Dspark.driver.port=40969 -Dspark.yarn.app.container.log.dir=/home/log/hadoop/logs/userlogs/application_1493004858240_0007/container_1493004858240_0007_01_000004
-XX:OnOutOfMemoryError=kill %p org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
spark://CoarseGrainedScheduler@10.120.241.82:40969 --executor-id 2 --hostname 10.120.241.18
--cores 1 --app-id application_1493004858240_0007 --user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/__app__.jar
--user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-api-0.3.0-SNAPSHOT.jar
--user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-rsc-0.3.0-SNAPSHOT.jar
--user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/netty-all-4.0.29.Final.jar
--user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/commons-codec-1.9.jar
--user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-core_2.11-0.3.0-SNAPSHOT.jar
--user-class-path file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-repl_2.11-0.3.0-SNAPSHOT.jar
appuser  26936 25846  0 15:16 pts/0    00:00:00 grep --color=auto application_1493004858240_0007
{code}

The main problem is that the application user is "wuchang" , but the yarn application is created
by my OS super-user "appuser" , so , the privilege becomes the problem. My code always run
as the privilege of appuser instead of "wuchang".

For example , below is the pyspark code:
{code}
import os
os.system("hadoop fs -rm -r /user/appuser/test.dat")
{code}

user "wuchang" should not have privilege to remove the file test.dat which located in the
home directory of appuser. But since the yarn application process is created by "appuser",
it does, although the yarn application user is "wuchang".




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org


Mime
View raw message