hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcelo Vanzin (JIRA)" <>
Subject [jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client.[Spark Branch]
Date Tue, 25 Nov 2014 19:16:14 GMT


Marcelo Vanzin commented on HIVE-8836:

I talked briefly with Brock about this, but the main thing here is that, right now, Spark
is not very friendly to applications that are trying to embed it. As you've noticed, the assembly
jar, which contains almost everything you need to run Spark, is not published in maven or
anywhere. And not all artifacts used to build the assembly are published - for example, the
Yarn backend cannot be found anywhere in maven, so without the assembly you cannot submit
jobs to Yarn.

I've suggested it in the past, but I think right now, or until Spark makes itself more friendly
to such use cases, Hive should require a full Spark install to work. If desired we could use
the hacks I added to the remote client to not need the full install for unit tests, but even
those are very limited; it probably only works with a "local" master as some of you may have

> Enable automatic tests with remote spark client.[Spark Branch]
> --------------------------------------------------------------
>                 Key: HIVE-8836
>                 URL:
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Chengxiang Li
>            Assignee: Rui Li
>              Labels: Spark-M3
>         Attachments: HIVE-8836-brock-1.patch, HIVE-8836-brock-2.patch, HIVE-8836-brock-3.patch,
HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch
> In real production environment, remote spark client should be used to submit spark job
for Hive mostly, we should enable automatic test with remote spark client to make sure the
Hive feature workable with it.

This message was sent by Atlassian JIRA

View raw message