Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Tue, 24 Oct 2017 17:59:00 +0000 (UTC)
From: "Apache Spark (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.13111680.1508849888000.48116.1508867940514@Atlassian.JIRA>
In-Reply-To: <JIRA.13111680.1508849888000@Atlassian.JIRA>
References: <JIRA.13111680.1508849888000@Atlassian.JIRA> <JIRA.13111680.1508849888686@jira-lw-us.apache.org>
Subject: [jira] [Commented] (SPARK-22341) [2.3.0] cannot run Spark on Yarn
 when Yarn impersonation is turned off
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Tue, 24 Oct 2017 17:59:07 -0000


    [ https://issues.apache.org/jira/browse/SPARK-22341?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D162=
17368#comment-16217368 ]=20

Apache Spark commented on SPARK-22341:
--------------------------------------

User 'vanzin' has created a pull request for this issue:
https://github.com/apache/spark/pull/19566

> [2.3.0] cannot run Spark on Yarn when Yarn impersonation is turned off
> ----------------------------------------------------------------------
>
>                 Key: SPARK-22341
>                 URL: https://issues.apache.org/jira/browse/SPARK-22341
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core, YARN
>    Affects Versions: 2.3.0
>            Reporter: Maciej Bry=C5=84ski
>
> I'm trying to run 2.3.0 (from master) on my yarn cluster.
> The result is:
> {code}
> Exception in thread "main" org.apache.hadoop.security.AccessControlExcept=
ion: Permission denied: user=3Dyarn, access=3DEXECUTE, inode=3D"/user/bi/.s=
parkStaging/application_1508815646088_0164":bi:hdfs:drwx------
> =09at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsP=
ermission(FSPermissionChecker.java:271)
> =09at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FS=
PermissionChecker.java:257)
> =09at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTra=
verse(FSPermissionChecker.java:208)
> =09at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPer=
mission(FSPermissionChecker.java:171)
> =09at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission=
(FSNamesystem.java:6795)
> =09at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSN=
amesystem.java:4387)
> =09at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInf=
o(NameNodeRpcServer.java:855)
> =09at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideT=
ranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:8=
35)
> =09at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$=
ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.ja=
va)
> =09at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.c=
all(ProtobufRpcEngine.java:619)
> =09at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
> =09at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2039)
> =09at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2035)
> =09at java.security.AccessController.doPrivileged(Native Method)
> =09at javax.security.auth.Subject.doAs(Subject.java:422)
> =09at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInfor=
mation.java:1628)
> =09at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2033)
> =09at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Metho=
d)
> =09at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstru=
ctorAccessorImpl.java:62)
> =09at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delegatin=
gConstructorAccessorImpl.java:45)
> =09at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> =09at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteEx=
ception.java:106)
> =09at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteE=
xception.java:73)
> =09at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1990)
> =09at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedF=
ileSystem.java:1118)
> =09at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedF=
ileSystem.java:1114)
> =09at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkR=
esolver.java:81)
> =09at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(Distribu=
tedFileSystem.java:1114)
> =09at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$7.apply(App=
licationMaster.scala:219)
> =09at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$7.apply(App=
licationMaster.scala:216)
> =09at scala.Option.foreach(Option.scala:257)
> =09at org.apache.spark.deploy.yarn.ApplicationMaster.<init>(ApplicationMa=
ster.scala:216)
> =09at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMas=
ter.scala:821)
> =09at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMast=
er.scala:842)
> =09at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaste=
r.scala)
> {code}
> I think the problem exists, because I'm not using yarn impersonation whic=
h mean that all jobs on cluster are runned from user yarn.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org