Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Content-Type: multipart/alternative;
 boundary="===============8184953861940334837=="
MIME-Version: 1.0
Subject: Re: Review Request 30107: HIVE-9410,
 ClassNotFoundException occurs during hive query case execution with UDF
 defined[Spark Branch]
From: "Xuefu Zhang" <xzhang@cloudera.com>
To: "Xuefu Zhang" <xzhang@cloudera.com>
Cc: "chengxiang li" <chengxiang.li@intel.com>, "hive" <dev@hive.apache.org>
Date: Fri, 23 Jan 2015 02:05:31 -0000
Message-ID: <20150123020531.9401.65750@reviews.apache.org>
Auto-Submitted: auto-generated
Sender: "Xuefu Zhang" <noreply@reviews.apache.org>
References: <20150122092336.17164.10586@reviews.apache.org>
In-Reply-To: <20150122092336.17164.10586@reviews.apache.org>
Reply-To: "Xuefu Zhang" <xzhang@cloudera.com>

--===============8184953861940334837==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30107/#review69329
-----------------------------------------------------------


I'm wondering what's the story for Hive CLI. Hive CLI can add jars from local file system. Would this work for Hive on Spark?


ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
<https://reviews.apache.org/r/30107/#comment114004>

    Callers of getBaseWork() will add the jars to the classpath. Why this is necessary? Who are the callers? Any side-effect?


ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java
<https://reviews.apache.org/r/30107/#comment114005>

    So, this is the code that adds the jars to the classpath of the remote driver?
    
    I'm wondering why these jars are necessary in order to deserailize SparkWork.


- Xuefu Zhang


On Jan. 22, 2015, 9:23 a.m., chengxiang li wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30107/
> -----------------------------------------------------------
> 
> (Updated Jan. 22, 2015, 9:23 a.m.)
> 
> 
> Review request for hive and Xuefu Zhang.
> 
> 
> Bugs: HIVE-9410
>     https://issues.apache.org/jira/browse/HIVE-9410
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> The RemoteDriver does not contains added jar in it's classpath, so it would failed to desrialize SparkWork due to NoClassFoundException. For Hive on MR, while use add jar through Hive CLI, Hive add jar into CLI classpath(through thread context classloader) and add it to distributed cache as well. Compare to Hive on MR, Hive on Spark has an extra RemoteDriver componnet, we should add added jar into it's classpath as well.
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java d7cb111 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java 30a00a7 
>   spark-client/src/main/java/org/apache/hive/spark/client/JobContext.java 00aa4ec 
>   spark-client/src/main/java/org/apache/hive/spark/client/JobContextImpl.java 1eb3ff2 
>   spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java 5f9be65 
>   spark-client/src/main/java/org/apache/hive/spark/client/SparkClientUtilities.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/30107/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> chengxiang li
> 
>


--===============8184953861940334837==--