hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Szehon Ho (JIRA)" <>
Subject [jira] [Commented] (HIVE-8795) Switch precommit test from local to local-cluster [Spark Branch]
Date Wed, 12 Nov 2014 19:03:35 GMT


Szehon Ho commented on HIVE-8795:

I dont think its scalable to install Spark on all the build machine because everytime Spark
has any change, we have to recreate the the images every single time.

It might be possible to set SPARK_HOME to the assembly location (which is freshly downloaded
to maven repo anyway, according to JIRA description?).  Do you know if that is the case?

> Switch precommit test from local to local-cluster [Spark Branch]
> ----------------------------------------------------------------
>                 Key: HIVE-8795
>                 URL:
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Xuefu Zhang
>            Assignee: Szehon Ho
>  It seems unlikely that Spark community will provide MRMiniCluster equivalent (SPARK-3691),
and Spark local-cluster was the recommendation. Latest research shows that Spark local-cluster
works with Hive. Therefore, for now, we use Spark local-cluster (instead of current local)
for our precommit test.
> It's previous belived (HIVE-7382) that a Spark installation is required and SPARK_HOME
env variable needs to set. Since Spark pulls in Spark's assembly jar, it's believed now we
only need a few script from Spark installation instead.

This message was sent by Atlassian JIRA

View raw message