hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <>
Subject [jira] [Commented] (HIVE-9007) Hive may generate wrong plan for map join queries due to IdentityProjectRemover [Spark Branch]
Date Mon, 05 Jan 2015 23:15:36 GMT


Hive QA commented on HIVE-9007:

{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:

{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 7282 tests executed
*Failed tests:*

Test results:
Console output:
Test logs:

Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed

This message is automatically generated.

ATTACHMENT ID: 12690135 - PreCommit-HIVE-SPARK-Build

> Hive may generate wrong plan for map join queries due to IdentityProjectRemover [Spark
> ----------------------------------------------------------------------------------------------
>                 Key: HIVE-9007
>                 URL:
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>    Affects Versions: spark-branch
>            Reporter: Chao
>            Assignee: Szehon Ho
>         Attachments: HIVE-9007-spark.patch, HIVE-9007.2-spark.patch
> HIVE-8435 introduces a new logical optimizer called IdentityProjectRemover, which may
cause map join in spark branch to generate wrong plan.
> Currently, the map join conversion in spark branch first goes through a method {{convertJoinMapJoin}},
which replaces a join op with a mapjoin op, removes RS associated with big table, and keep
RSs for all small tables. Afterwards, in {{SparkReduceSinkMapJoinProc}} it replaces all parent
RSs of the mapjoin op with HTS (note it doesn't check whether the RS belongs to small table
or big table.)
> The issue arises, when IdentityProjectRemover comes into play, which may result into
a situation that a operator tree has two consecutive RSs. Imaging the following example:
> {noformat}
>           Join               MapJoin
>           / \                /   \
>         RS   RS   --->     RS     RS
>        /      \           /         \
>       TS       RS       TS          TS (big table)
>                 \      (small table)
>                  TS
> {noformat}
> In this case, all parents of the mapjoin op will be RS, even the branch for big table!
In {{SparkReduceSinkMapJoinProc}}, they will be replaced with HTS, which is obviously incorrect.

This message was sent by Atlassian JIRA

View raw message