hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "chengxiang li" <chengxiang...@intel.com>
Subject Re: Review Request 27987: HIVE-8833 implement remote spark client
Date Fri, 14 Nov 2014 05:07:09 GMT


> On 十一月 14, 2014, 4:39 a.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java, line
73
> > <https://reviews.apache.org/r/27987/diff/3/?file=763281#file763281line73>
> >
> >     Public or private?

Package scope should be suitable here, as we only allow HiveSparkClientFacotry to create new
RemoteHiveSparkClient instance, it's better not public to others.


> On 十一月 14, 2014, 4:39 a.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java, line
166
> > <https://reviews.apache.org/r/27987/diff/3/?file=763281#file763281line166>
> >
> >     I'm wondering if addFile() should accept uri instead.

I can look at it later, as it's about remote spark context API.


- chengxiang


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27987/#review61398
-----------------------------------------------------------


On 十一月 14, 2014, 3:43 a.m., chengxiang li wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27987/
> -----------------------------------------------------------
> 
> (Updated 十一月 14, 2014, 3:43 a.m.)
> 
> 
> Review request for hive, Rui Li, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-8833
>     https://issues.apache.org/jira/browse/HIVE-8833
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Hive would support submitting spark job through both local spark client and remote spark
client. we should unify the spark client API, and implement remote spark client through Remote
Spark Context.
> 
> 
> Diffs
> -----
> 
>   ql/pom.xml 06d7f27 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClient.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/LocalHiveSparkClient.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkClient.java ee16c9e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 2fea62d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java e3e6d16 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java 51e0510

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobRef.java bf43b6e 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java
d4d14a3 
>   spark-client/src/main/java/org/apache/hive/spark/client/SparkClient.java 8346b28 
>   spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java 5af66ee

> 
> Diff: https://reviews.apache.org/r/27987/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> chengxiang li
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message