hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-8054) Disable hive.optimize.union.remove when hive.execution.engine=spark [Spark Branch]
Date Mon, 15 Sep 2014 20:02:34 GMT

    [ https://issues.apache.org/jira/browse/HIVE-8054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14134394#comment-14134394
] 

Hive QA commented on HIVE-8054:
-------------------------------



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12668813/HIVE-8054.2-spark.patch

{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 6343 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_fs_default_name2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part13
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation
{noformat}

Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/129/testReport
Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/129/console
Test logs: http://ec2-54-176-176-199.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-129/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12668813

> Disable hive.optimize.union.remove when hive.execution.engine=spark [Spark Branch]
> ----------------------------------------------------------------------------------
>
>                 Key: HIVE-8054
>                 URL: https://issues.apache.org/jira/browse/HIVE-8054
>             Project: Hive
>          Issue Type: Improvement
>          Components: Spark
>            Reporter: Xuefu Zhang
>            Assignee: Na Yang
>              Labels: Spark-M1
>         Attachments: HIVE-8054-spark.patch, HIVE-8054.2-spark.patch
>
>
> Option hive.optimize.union.remove introduced in HIVE-3276 removes union operators from
the operator graph in certain cases as an optimization reduce the number of MR jobs. While
making sense in MR, this optimization is actually harmful to an execution engine such as Spark,
which natives supports union without requiring additional jobs. This is because removing union
operator creates disjointed operator graphs, each graph generating a job, and thus this optimization
requires more jobs to run the query. Not to mention the additional complexity handling linked
FS descriptors.
> I propose that we disable such optimization when the execution engine is Spark.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message