Mailing-List: contact issues-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Tue, 10 Mar 2015 04:01:39 +0000 (UTC)
From: "Xiaomin Zhang (JIRA)" <jira@apache.org>
To: issues@hive.apache.org
Message-ID: <JIRA.12779965.1425608712000.47151.1425960099125@Atlassian.JIRA>
In-Reply-To: <JIRA.12779965.1425608712000@Atlassian.JIRA>
References: <JIRA.12779965.1425608712000@Atlassian.JIRA>
 <JIRA.12779965.1425608712095@arcas>
Subject: [jira] [Commented] (HIVE-9882) Add jar/file doesn't work with
 yarn-cluster mode [Spark Branch]
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HIVE-9882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14354246#comment-14354246 ] 

Xiaomin Zhang commented on HIVE-9882:
-------------------------------------

It now passed the test, good job!

> Add jar/file doesn't work with yarn-cluster mode [Spark Branch]
> ---------------------------------------------------------------
>
>                 Key: HIVE-9882
>                 URL: https://issues.apache.org/jira/browse/HIVE-9882
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Hive, spark-branch
>    Affects Versions: spark-branch
>            Reporter: Xiaomin Zhang
>            Assignee: Rui Li
>             Fix For: spark-branch
>
>         Attachments: HIVE-9882.1-spark.patch, HIVE-9882.1-spark.patch
>
>
> It seems current fix for HIVE-9425 only uploads the Jar/Files to HDFS, however, they are not accessible by the Driver/Executor.
> I found below in the AM log:
> {noformat}
> 15/02/26 15:10:36 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
> 15/02/26 15:10:36 INFO client.SparkClientUtilities: Added jar[file:/data/hadoop-devel/data/nm/usercache/user/appcache/application_1424933948132_0002/container_1424933948132_0002_01_000001/hdfs:/localhost:8020/tmp/hive/user/47040bca-1da4-49b6-b2c7-69be9bc92855/hive-exec-1.2.0-SNAPSHOT.jar] to classpath.
> 15/02/26 15:10:36 INFO client.SparkClientUtilities: Added jar[file:/data/hadoop-devel/data/nm/usercache/user/appcache/application_1424933948132_0002/container_1424933948132_0002_01_000001/hdfs:/localhost:8020/tmp/hive/user/47040bca-1da4-49b6-b2c7-69be9bc92855/opennlp-maxent-3.0.3.jar] to classpath.
> 15/02/26 15:10:36 INFO client.SparkClientUtilities: Added jar[file:/data/hadoop-devel/data/nm/usercache/user/appcache/application_1424933948132_0002/container_1424933948132_0002_01_000001/hdfs:/localhost:8020/tmp/hive/user/47040bca-1da4-49b6-b2c7-69be9bc92855/bigbenchqueriesmr.jar] to classpath.
> 15/02/26 15:10:36 INFO client.SparkClientUtilities: Added jar[file:/data/hadoop-devel/data/nm/usercache/user/appcache/application_1424933948132_0002/container_1424933948132_0002_01_000001/hdfs:/localhost:8020/tmp/hive/user/47040bca-1da4-49b6-b2c7-69be9bc92855/opennlp-tools-1.5.3.jar] to classpath.
> 15/02/26 15:10:36 INFO client.SparkClientUtilities: Added jar[file:/data/hadoop-devel/data/nm/usercache/user/appcache/application_1424933948132_0002/container_1424933948132_0002_01_000001/hdfs:/localhost:8020/tmp/hive/user/47040bca-1da4-49b6-b2c7-69be9bc92855/jcl-over-slf4j-1.7.5.jar] to classpath.
> 15/02/26 15:10:36 INFO client.RemoteDriver: Failed to run job 6886df05-f430-456c-a0ff-c7621db712d6
> org.apache.hive.com.esotericsoftware.kryo.KryoException: Unable to find class: de.bankmark.bigbench.queries.q10.SentimentUDF 
> {noformat}
> As above shows, the file path which was attempted to add to Classpath is invalid, so actually all uploaded Jars/Files are still not available for Driver/Executor.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)