hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "chengxiang li" <chengxiang...@intel.com>
Subject Re: Review Request 27987: HIVE-8833 implement remote spark client
Date Mon, 17 Nov 2014 03:47:44 GMT


> On Nov. 14, 2014, 7:32 p.m., Marcelo Vanzin wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClient.java, line 26
> > <https://reviews.apache.org/r/27987/diff/3/?file=763278#file763278line26>
> >
> >     nit: space before {
> >     
> >     Maybe implement Closeable?

fixed.


> On Nov. 14, 2014, 7:32 p.m., Marcelo Vanzin wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java, line
101
> > <https://reviews.apache.org/r/27987/diff/3/?file=763279#file763279line101>
> >
> >     Doesn't this work?
> >     
> >     for (Map.Entry<String, String> entry : hiveConf)

yeah, it works. fixed.


> On Nov. 14, 2014, 7:32 p.m., Marcelo Vanzin wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java, line
107
> > <https://reviews.apache.org/r/27987/diff/3/?file=763279#file763279line107>
> >
> >     Is Hive still using commons-logging? slf4j makes this much better since it handles
format strings for you...

Hive use commons-logging, keep this here.


> On Nov. 14, 2014, 7:32 p.m., Marcelo Vanzin wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java, line
94
> > <https://reviews.apache.org/r/27987/diff/3/?file=763281#file763281line94>
> >
> >     Don't you get warnings here since JobHandle needs a type parameter?

not actually in my Intellij, very strange. fix it.


> On Nov. 14, 2014, 7:32 p.m., Marcelo Vanzin wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java, line 51
> > <https://reviews.apache.org/r/27987/diff/3/?file=763284#file763284line51>
> >
> >     You could use:
> >     
> >       new URI(path).getScheme() != null

fixed.


> On Nov. 14, 2014, 7:32 p.m., Marcelo Vanzin wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java, line 55
> > <https://reviews.apache.org/r/27987/diff/3/?file=763284#file763284line55>
> >
> >     You could use:
> >     
> >       new File(path).toURI().toURL()

fixed.


- chengxiang


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27987/#review61479
-----------------------------------------------------------


On Nov. 14, 2014, 3:43 a.m., chengxiang li wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27987/
> -----------------------------------------------------------
> 
> (Updated Nov. 14, 2014, 3:43 a.m.)
> 
> 
> Review request for hive, Rui Li, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-8833
>     https://issues.apache.org/jira/browse/HIVE-8833
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Hive would support submitting spark job through both local spark client and remote spark
client. we should unify the spark client API, and implement remote spark client through Remote
Spark Context.
> 
> 
> Diffs
> -----
> 
>   ql/pom.xml 06d7f27 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClient.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/LocalHiveSparkClient.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkClient.java ee16c9e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 2fea62d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java e3e6d16 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java 51e0510

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobRef.java bf43b6e 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java
d4d14a3 
>   spark-client/src/main/java/org/apache/hive/spark/client/SparkClient.java 8346b28 
>   spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java 5af66ee

> 
> Diff: https://reviews.apache.org/r/27987/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> chengxiang li
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message