hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rui Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-7382) Create a MiniSparkCluster and set up a testing framework [Spark Branch]
Date Thu, 25 Sep 2014 03:45:34 GMT

    [ https://issues.apache.org/jira/browse/HIVE-7382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14147328#comment-14147328
] 

Rui Li commented on HIVE-7382:
------------------------------

Hi [~xuefuz],

Just to clarify, we currently use spark local mode to run the tests. And our goal here is
to use local-cluster mode to run the test, right?
So far I've found, local mode runs all the spark executor, backend, etc. in the same JVM,
while in local-cluster mode, master and workers run in the same JVM and the executors run
in separate JVMs. Local-cluster mode resembles the standalone mode, except that master and
workers run in the same JVM and everything runs on a single machine.
Intuitively, cluster-local mode can catch more errors we may have. But this mode seems to
be intended only for spark tests and not exposed to users. We may need to make sure if we
really want to use it.

> Create a MiniSparkCluster and set up a testing framework [Spark Branch]
> -----------------------------------------------------------------------
>
>                 Key: HIVE-7382
>                 URL: https://issues.apache.org/jira/browse/HIVE-7382
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Xuefu Zhang
>            Assignee: Rui Li
>              Labels: Spark-M1
>
> To automatically test Hive functionality over Spark execution engine, we need to create
a test framework that can execute Hive queries with Spark as the backend. For that, we should
create a MiniSparkCluser for this, similar to other execution engines.
> Spark has a way to create a local cluster with a few processes in the local machine,
each process is a work node. It's fairly close to a real Spark cluster. Our mini cluster can
be based on that.
> For more info, please refer to the design doc on wiki.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message