hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin" <vanzin+...@cloudera.com>
Subject Re: Review Request 27987: HIVE-8833 implement remote spark client
Date Fri, 14 Nov 2014 19:32:21 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27987/#review61479
-----------------------------------------------------------

Ship it!


LGTM, just small nits.


ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClient.java
<https://reviews.apache.org/r/27987/#comment103130>

    nit: space before {
    
    Maybe implement Closeable?



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java
<https://reviews.apache.org/r/27987/#comment103134>

    Use "properties.load(Reader)" instead, so you can force UTF-8 encoding.



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java
<https://reviews.apache.org/r/27987/#comment103135>

    Doesn't this work?
    
    for (Map.Entry<String, String> entry : hiveConf)



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java
<https://reviews.apache.org/r/27987/#comment103137>

    Is Hive still using commons-logging? slf4j makes this much better since it handles format
strings for you...



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java
<https://reviews.apache.org/r/27987/#comment103145>

    Don't you get warnings here since JobHandle needs a type parameter?



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java
<https://reviews.apache.org/r/27987/#comment103148>

    You could use:
    
      new URI(path).getScheme() != null



ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java
<https://reviews.apache.org/r/27987/#comment103150>

    You could use:
    
      new File(path).toURI().toURL()


- Marcelo Vanzin


On Nov. 14, 2014, 3:43 a.m., chengxiang li wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27987/
> -----------------------------------------------------------
> 
> (Updated Nov. 14, 2014, 3:43 a.m.)
> 
> 
> Review request for hive, Rui Li, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-8833
>     https://issues.apache.org/jira/browse/HIVE-8833
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Hive would support submitting spark job through both local spark client and remote spark
client. we should unify the spark client API, and implement remote spark client through Remote
Spark Context.
> 
> 
> Diffs
> -----
> 
>   ql/pom.xml 06d7f27 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClient.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/LocalHiveSparkClient.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkClient.java ee16c9e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 2fea62d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java e3e6d16 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java 51e0510

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobRef.java bf43b6e 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java
d4d14a3 
>   spark-client/src/main/java/org/apache/hive/spark/client/SparkClient.java 8346b28 
>   spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java 5af66ee

> 
> Diff: https://reviews.apache.org/r/27987/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> chengxiang li
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message