hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuefu Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-8836) Enable automatic tests with remote spark client [Spark Branch]
Date Wed, 26 Nov 2014 05:31:14 GMT

    [ https://issues.apache.org/jira/browse/HIVE-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14225779#comment-14225779
] 

Xuefu Zhang commented on HIVE-8836:
-----------------------------------

Many test failures above are shown as plan diff, somehting like: 
{code}
211c211
<         Reducer 2 <- Map 1 (GROUP, 3)
---
>         Reducer 2 <- Map 1 (GROUP, 1)
{code}
This can be explained by the fact that the current algorithm determining the number of reducers
has included the factors such as number of executors in the cluster. Because we switched to
local-cluster with two nodes, the number of reducers is different from previous value. This
is not good, because every time we change the testing cluster, we might have a different plan.

> Enable automatic tests with remote spark client [Spark Branch]
> --------------------------------------------------------------
>
>                 Key: HIVE-8836
>                 URL: https://issues.apache.org/jira/browse/HIVE-8836
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>    Affects Versions: spark-branch
>            Reporter: Chengxiang Li
>            Assignee: Rui Li
>              Labels: Spark-M3
>             Fix For: spark-branch
>
>         Attachments: HIVE-8836.1-spark.patch, HIVE-8836.2-spark.patch, HIVE-8836.3-spark.patch,
HIVE-8836.4-spark.patch, HIVE-8836.5-spark.patch, HIVE-8836.6-spark.patch, HIVE-8836.6-spark.patch
>
>
> In real production environment, remote spark client should be used to submit spark job
for Hive mostly, we should enable automatic test with remote spark client to make sure the
Hive feature workable with it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message