ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zack Marsh (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AMBARI-12403) Unable to submit application to YARN from external client on Kerberos-enabled cluster
Date Mon, 13 Jul 2015 23:33:04 GMT

     [ https://issues.apache.org/jira/browse/AMBARI-12403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Zack Marsh updated AMBARI-12403:
--------------------------------
    Description: 
On an HDP-2.3 cluster, the Edge node has been configured to resemble an external client by
only listing the public IPs of the Hadoop hosts in its /etc/hosts file. 

Prior to enabling Kerberos, I tested running an Oozie job (MR example), launching the Hive
shell and showing tables, and running  the YARN calculate values of Pi example.

Before enabling Kerberos all of these examples succeeded when ran from within the cluster,
and from the external client Edge node.

After enabling Kerberos via the Ambari Wizard none of the examples succeeded due to AMBARI-11054
when ran from any host. After updating the owner of the tdatuser usercache directories the
Oozie example is passing, but launching the Hive shell and the YARN example fail when ran
from the external client with the following error:
{code}
tdatuser@selena7:/root> kinit -kt /etc/security/keytabs/tdatuser.headless.keytab tdatuser
tdatuser@selena7:/root> hive
WARNING: Use "yarn jar" to launch YARN applications.

Logging initialized using configuration in file:/etc/hive/2.3.0.0-2525/0/hive-log4j.properties
Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.TezException:
org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1436575073312_0041
to YARN : Failed to renew token: Kind: TIMELINE_DELEGATION_TOKEN, Service: 10.25.100.118:8188,
Ident: (owner=tdatuser, renewer=yarn, realUser=, issueDate=1436827743392, maxDate=1437432543392,
sequenceNumber=36, masterKeyId=16)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:535)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.tez.dag.api.TezException: org.apache.hadoop.yarn.exceptions.YarnException:
Failed to submit application_1436575073312_0041 to YARN : Failed to renew token: Kind: TIMELINE_DELEGATION_TOKEN,
Service: 10.25.100.118:8188, Ident: (owner=tdatuser, renewer=yarn, realUser=, issueDate=1436827743392,
maxDate=1437432543392, sequenceNumber=36, masterKeyId=16)
        at org.apache.tez.client.TezClient.start(TezClient.java:370)
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:184)
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:116)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:532)
        ... 8 more
Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1436575073312_0041
to YARN : Failed to renew token: Kind: TIMELINE_DELEGATION_TOKEN, Service: 10.25.100.118:8188,
Ident: (owner=tdatuser, renewer=yarn, realUser=, issueDate=1436827743392, maxDate=1437432543392,
sequenceNumber=36, masterKeyId=16)
        at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:271)
        at org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:72)
        at org.apache.tez.client.TezClient.start(TezClient.java:365)
        ... 11 more
{code}

Note: Only the external client host is encountering these failures.



  was:
On an HDP-2.3 cluster, the Edge node has been configured to resemble an external client by
only listing the public IPs of the Hadoop hosts in its /etc/hosts file. 

Prior to enabling Kerberos, I tested running an Oozie job (MR example), launching the Hive
shell and showing tables, and running  the YARN calculate values of Pi example.

Before enabling Kerberos all of these examples succeeded when ran from within the cluster,
and from the external client Edge node.

After enabling Kerberos via the Ambari Wizard none of the examples succeeded due to AMBARI-11054.
After updating the owner of the tdatuser usercache directories the Oozie example is passing,
but launching the Hive shell and the YARN example fail with the following error:
{code}
tdatuser@selena7:/root> kinit -kt /etc/security/keytabs/tdatuser.headless.keytab tdatuser
tdatuser@selena7:/root> hive
WARNING: Use "yarn jar" to launch YARN applications.

Logging initialized using configuration in file:/etc/hive/2.3.0.0-2525/0/hive-log4j.properties
Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.TezException:
org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1436575073312_0041
to YARN : Failed to renew token: Kind: TIMELINE_DELEGATION_TOKEN, Service: 10.25.100.118:8188,
Ident: (owner=tdatuser, renewer=yarn, realUser=, issueDate=1436827743392, maxDate=1437432543392,
sequenceNumber=36, masterKeyId=16)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:535)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.tez.dag.api.TezException: org.apache.hadoop.yarn.exceptions.YarnException:
Failed to submit application_1436575073312_0041 to YARN : Failed to renew token: Kind: TIMELINE_DELEGATION_TOKEN,
Service: 10.25.100.118:8188, Ident: (owner=tdatuser, renewer=yarn, realUser=, issueDate=1436827743392,
maxDate=1437432543392, sequenceNumber=36, masterKeyId=16)
        at org.apache.tez.client.TezClient.start(TezClient.java:370)
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:184)
        at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:116)
        at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:532)
        ... 8 more
Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1436575073312_0041
to YARN : Failed to renew token: Kind: TIMELINE_DELEGATION_TOKEN, Service: 10.25.100.118:8188,
Ident: (owner=tdatuser, renewer=yarn, realUser=, issueDate=1436827743392, maxDate=1437432543392,
sequenceNumber=36, masterKeyId=16)
        at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:271)
        at org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:72)
        at org.apache.tez.client.TezClient.start(TezClient.java:365)
        ... 11 more
{code}





> Unable to submit application to YARN from external client on Kerberos-enabled cluster
> -------------------------------------------------------------------------------------
>
>                 Key: AMBARI-12403
>                 URL: https://issues.apache.org/jira/browse/AMBARI-12403
>             Project: Ambari
>          Issue Type: Bug
>         Environment: sles11sp3
>            Reporter: Zack Marsh
>            Priority: Critical
>
> On an HDP-2.3 cluster, the Edge node has been configured to resemble an external client
by only listing the public IPs of the Hadoop hosts in its /etc/hosts file. 
> Prior to enabling Kerberos, I tested running an Oozie job (MR example), launching the
Hive shell and showing tables, and running  the YARN calculate values of Pi example.
> Before enabling Kerberos all of these examples succeeded when ran from within the cluster,
and from the external client Edge node.
> After enabling Kerberos via the Ambari Wizard none of the examples succeeded due to AMBARI-11054
when ran from any host. After updating the owner of the tdatuser usercache directories the
Oozie example is passing, but launching the Hive shell and the YARN example fail when ran
from the external client with the following error:
> {code}
> tdatuser@selena7:/root> kinit -kt /etc/security/keytabs/tdatuser.headless.keytab tdatuser
> tdatuser@selena7:/root> hive
> WARNING: Use "yarn jar" to launch YARN applications.
> Logging initialized using configuration in file:/etc/hive/2.3.0.0-2525/0/hive-log4j.properties
> Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.TezException:
org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1436575073312_0041
to YARN : Failed to renew token: Kind: TIMELINE_DELEGATION_TOKEN, Service: 10.25.100.118:8188,
Ident: (owner=tdatuser, renewer=yarn, realUser=, issueDate=1436827743392, maxDate=1437432543392,
sequenceNumber=36, masterKeyId=16)
>         at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:535)
>         at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:677)
>         at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:483)
>         at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: org.apache.tez.dag.api.TezException: org.apache.hadoop.yarn.exceptions.YarnException:
Failed to submit application_1436575073312_0041 to YARN : Failed to renew token: Kind: TIMELINE_DELEGATION_TOKEN,
Service: 10.25.100.118:8188, Ident: (owner=tdatuser, renewer=yarn, realUser=, issueDate=1436827743392,
maxDate=1437432543392, sequenceNumber=36, masterKeyId=16)
>         at org.apache.tez.client.TezClient.start(TezClient.java:370)
>         at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:184)
>         at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:116)
>         at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:532)
>         ... 8 more
> Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1436575073312_0041
to YARN : Failed to renew token: Kind: TIMELINE_DELEGATION_TOKEN, Service: 10.25.100.118:8188,
Ident: (owner=tdatuser, renewer=yarn, realUser=, issueDate=1436827743392, maxDate=1437432543392,
sequenceNumber=36, masterKeyId=16)
>         at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:271)
>         at org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:72)
>         at org.apache.tez.client.TezClient.start(TezClient.java:365)
>         ... 11 more
> {code}
> Note: Only the external client host is encountering these failures.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message