hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yibing Shi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-15767) Hive On Spark is not working on secure clusters from Oozie
Date Thu, 13 Jul 2017 12:08:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-15767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16085612#comment-16085612
] 

Yibing Shi commented on HIVE-15767:
-----------------------------------

bq. I think the Spark driver will get the tokens afterwards
I really doubt that Spark driver can do this. In Oozie environment, it is Oozie server that
obtains all the tokens *on behalf of the end user*. When the Hive actions starts a Spark job,
the Spark driver has no access to end user ticket or keytab file. I don't think it can obtain
necessary tokens. 
I believe we should somehow extract all the tokens from existing toke file, and pass it on
to the Spark driver.

> Hive On Spark is not working on secure clusters from Oozie
> ----------------------------------------------------------
>
>                 Key: HIVE-15767
>                 URL: https://issues.apache.org/jira/browse/HIVE-15767
>             Project: Hive
>          Issue Type: Bug
>          Components: Spark
>    Affects Versions: 1.2.1, 2.1.1
>            Reporter: Peter Cseh
>            Assignee: Peter Cseh
>         Attachments: HIVE-15767-001.patch, HIVE-15767-002.patch
>
>
> When a HiveAction is launched form Oozie with Hive On Spark enabled, we're getting errors:
> {noformat}
> Caused by: java.io.IOException: Exception reading file:/yarn/nm/usercache/yshi/appcache/application_1485271416004_0022/container_1485271416004_0022_01_000002/container_tokens
>         at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:188)
>         at org.apache.hadoop.mapreduce.security.TokenCache.mergeBinaryTokens(TokenCache.java:155)
> {noformat}
> This is caused by passing the {{mapreduce.job.credentials.binary}} property to the Spark
configuration in RemoteHiveSparkClient.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message