flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dschexna <dsche...@mapr.com>
Subject Flink 1.2 / YARN on Secure MapR Cluster
Date Wed, 08 Mar 2017 22:50:54 GMT
I am attempting to run flink / YARN on a secure MapR 5.2 cluster.  The
cluster is secured using "MapR Native Security", not kerberos.

I did include the MapR zookeeper when building:

opt/apache-maven-3.3.9/bin/mvn clean install -DskipTests -Pvendor-repos
-Dhadoop.version=2.7.0-mapr-1607 -Dzookeeper.version=3.4.5-mapr-1604

When I try to launch yarn-session.sh, I receive the following error:

[mapr@maprdemo flink]$ ./bin/yarn-session.sh -n 1
2017-03-08 14:47:24,223 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: jobmanager.rpc.address, localhost
2017-03-08 14:47:24,224 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: jobmanager.rpc.port, 6123
2017-03-08 14:47:24,224 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: jobmanager.heap.mb, 256
2017-03-08 14:47:24,224 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: taskmanager.heap.mb, 512
2017-03-08 14:47:24,225 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: taskmanager.numberOfTaskSlots, 1
2017-03-08 14:47:24,225 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: taskmanager.memory.preallocate, false
2017-03-08 14:47:24,225 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: parallelism.default, 1
2017-03-08 14:47:24,225 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: jobmanager.web.port, 8081
2017-03-08 14:47:24,292 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: jobmanager.rpc.address, localhost
2017-03-08 14:47:24,292 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: jobmanager.rpc.port, 6123
2017-03-08 14:47:24,293 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: jobmanager.heap.mb, 256
2017-03-08 14:47:24,293 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: taskmanager.heap.mb, 512
2017-03-08 14:47:24,293 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: taskmanager.numberOfTaskSlots, 1
2017-03-08 14:47:24,293 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: taskmanager.memory.preallocate, false
2017-03-08 14:47:24,293 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: parallelism.default, 1
2017-03-08 14:47:24,294 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: jobmanager.web.port, 8081
2017-03-08 14:47:26,107 WARN  org.apache.hadoop.util.NativeCodeLoader                    
 
- Unable to load native-hadoop library for your platform... using
builtin-java classes where applicable
2017-03-08 14:47:26,390 WARN 
org.apache.flink.runtime.security.modules.HadoopModule        - Hadoop
security is enabled but current login user does not have Kerberos
credentials
2017-03-08 14:47:26,390 INFO 
org.apache.flink.runtime.security.modules.HadoopModule        - Hadoop user
set to mapr (auth:CUSTOM)
2017-03-08 14:47:26,431 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: jobmanager.rpc.address, localhost
2017-03-08 14:47:26,431 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: jobmanager.rpc.port, 6123
2017-03-08 14:47:26,431 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: jobmanager.heap.mb, 256
2017-03-08 14:47:26,431 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: taskmanager.heap.mb, 512
2017-03-08 14:47:26,432 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: taskmanager.numberOfTaskSlots, 1
2017-03-08 14:47:26,432 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: taskmanager.memory.preallocate, false
2017-03-08 14:47:26,433 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: parallelism.default, 1
2017-03-08 14:47:26,433 INFO 
org.apache.flink.configuration.GlobalConfiguration            - Loading
configuration property: jobmanager.web.port, 8081
2017-03-08 14:47:26,440 ERROR org.apache.flink.yarn.YarnClusterDescriptor                
 
- Hadoop security is enabled but the login user does not have Kerberos
credentials
Error while deploying YARN cluster: Couldn't deploy Yarn cluster
java.lang.RuntimeException: Couldn't deploy Yarn cluster
	at
org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:421)
	at
org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:620)
	at
org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:476)
	at
org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:473)
	at
org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595)
	at
org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
	at
org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:473)
Caused by: java.lang.RuntimeException: Hadoop security is enabled but the
login user does not have Kerberos credentials
	at
org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:415)

The 'mapr' user does have a ticket:

[mapr@maprdemo flink]$ maprlogin print
Opening keyfile /tmp/maprticket_2000
demo.mapr.com: user = mapr, created = 'Wed Mar 01 07:26:02 PST 2017',
expires = 'Wed Mar 15 08:26:02 PDT 2017', RenewalTill = 'Fri Mar 31 08:26:02
PDT 2017', uid = 2000, gids = 2000, 2001, CanImpersonate = false

I can launch other YARN jobs without error.  Most of the MapR environment
variables are picked up in /opt/mapr/conf/env.sh and
/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop/hadoop-env.sh.

I have set the following environment variables:

export FLINK_HOME=/opt/flink-1.2
export YARN_CONF_DIR=/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop
export HADOOP_HOME=/opt/mapr/hadoop/hadoop-2.7.0
export HADOOP_CONF_DIR=/opt/mapr/hadoop/hadoop-2.7.0/etc/hadoop



--
View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Flink-1-2-YARN-on-Secure-MapR-Cluster-tp16399.html
Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com.

Mime
View raw message