hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "chengxiang li" <chengxiang...@intel.com>
Subject Re: Review Request 27987: HIVE-8833 implement remote spark client
Date Tue, 18 Nov 2014 01:36:25 GMT


> On Nov. 17, 2014, 10:11 p.m., Szehon Ho wrote:
> > spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java, line
193
> > <https://reviews.apache.org/r/27987/diff/4/?file=765595#file765595line193>
> >
> >     Sorry for the basic question, what is 'spark' as spark.master signify?

Spark support different kinds of resource manager, which is configured with spark.master,
such as:
local, local[coreNumber], local-cluster
spark://hostname:port
mesos:://hostname:port
yarn-cluster
yarn-client

spark://hostname:port refer to a standalone spark cluster.


- chengxiang


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27987/#review61819
-----------------------------------------------------------


On Nov. 17, 2014, 3:47 a.m., chengxiang li wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27987/
> -----------------------------------------------------------
> 
> (Updated Nov. 17, 2014, 3:47 a.m.)
> 
> 
> Review request for hive, Rui Li, Szehon Ho, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-8833
>     https://issues.apache.org/jira/browse/HIVE-8833
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Hive would support submitting spark job through both local spark client and remote spark
client. we should unify the spark client API, and implement remote spark client through Remote
Spark Context.
> 
> 
> Diffs
> -----
> 
>   ql/pom.xml 06d7f27 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClient.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/LocalHiveSparkClient.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java PRE-CREATION

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkClient.java ee16c9e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 2fea62d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java e3e6d16 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java 51e0510

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/SparkJobRef.java bf43b6e 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SetSparkReducerParallelism.java
d4d14a3 
>   spark-client/src/main/java/org/apache/hive/spark/client/SparkClient.java 8346b28 
>   spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java 5af66ee

> 
> Diff: https://reviews.apache.org/r/27987/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> chengxiang li
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message