hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Szehon Ho" <sze...@cloudera.com>
Subject Re: Review Request 27987: HIVE-8833 implement remote spark client
Date Mon, 17 Nov 2014 22:11:20 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27987/#review61819
-----------------------------------------------------------


Looks mostly good, just some minor nits and basic question as I'm not too familiar.


ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java
<https://reviews.apache.org/r/27987/#comment103720>

    Can we correct this typo while we are in this class?



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java
<https://reviews.apache.org/r/27987/#comment103725>

    Should we put this inside the null-check, to avoid NPE?



spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java
<https://reviews.apache.org/r/27987/#comment103730>

    Sorry for the basic question, what is 'spark' as spark.master signify?


- Szehon Ho


On Nov. 17, 2014, 3:47 a.m., chengxiang li wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27987/
> -----------------------------------------------------------
> 
> (Updated Nov. 17, 2014, 3:47 a.m.)
> 
> 
> Review request for hive, Rui Li, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-8833
>     https://issues.apache.org/jira/browse/HIVE-8833
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Hive would support submitting spark job through both local spark client and remote spark
client. we should unify the spark client API, and implement remote spark client through Remote
Spark Context.
> 
> 
> Diffs
> -----
> 
>   ql/pom.xml 06d7f27 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClient.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/LocalHiveSparkClient.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkClient.java ee16c9e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 2fea62d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java e3e6d16 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java 51e0510

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobRef.java bf43b6e 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java
d4d14a3 
>   spark-client/src/main/java/org/apache/hive/spark/client/SparkClient.java 8346b28 
>   spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java 5af66ee

> 
> Diff: https://reviews.apache.org/r/27987/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> chengxiang li
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message