hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jesus Camacho Rodriguez (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-11284) Fix cbo_rp_join0 failure on master
Date Fri, 17 Jul 2015 21:20:04 GMT

    [ https://issues.apache.org/jira/browse/HIVE-11284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14631924#comment-14631924
] 

Jesus Camacho Rodriguez commented on HIVE-11284:
------------------------------------------------

Sure.

==
Background:
When RT is on, HiveJoinToMultiJoinRule is responsible for creating MultiJoin operators out
of Join operators (if possible). But the plan might be the following:
{noformat}
Join(...)
    Project(...)
        Join(...)
            ...
    Filter(...)
        ...
{noformat} 
Thus, first we need to pull up the Project operator(s) through Join(s) using HiveJoinProjectTransposeRule,
to obtain:
{noformat}
Project(...)
    Join(...)
        Join(...)
            ...
    Filter(...)
        ...
{noformat} 
Then, HiveJoinToMultiJoinRule will find both Join operators and merge them. Finally, the Project
operator(s) will be pushed down again throw the tree (this is done by HiveRelTrimmer).

==
Problem:
HIVE-11223 enabled the return path for the cbo_rp_join0.q test.
At the same time, HIVE-11252 changed the application of the rules.
- Before HIVE-11252, HiveJoinToMultiJoinRule was applied first, then HiveJoinProjectTransposeRule
+ HiveJoinToMultiJoinRule. This was merging the operators correctly, because no projects were
found in between the Joins.
- After HIVE-11252, only HiveJoinToMultiJoinRule + HiveJoinProjectTransposeRule are applied
(this is correct). The problem is that 1) HiveJoinProjectTransposeRule.LEFT and HiveJoinProjectTransposeRule.RIGHT
were not enabled, and 2) even if they were enabled, they were not working as expected (more
on this below). That's why the test failed.

==
Solution:
HiveJoinProjectTransposeRule.LEFT and HiveJoinProjectTransposeRule.RIGHT get enabled in the
patch. In addition, JoinProjectTransposeRule was only working on inner joins, but we need
it to work also on outer joins in Hive. That's why I had to bring so much code from the Calcite
superclass (JoinProjectTransposeRule) to the Hive subclass (HiveJoinProjectTransposeRule)
to solve the issue. I plan to create a Calcite patch so we can use JoinProjectTransposeRule
with all kind of Joins. 

> Fix cbo_rp_join0 failure on master
> ----------------------------------
>
>                 Key: HIVE-11284
>                 URL: https://issues.apache.org/jira/browse/HIVE-11284
>             Project: Hive
>          Issue Type: Bug
>          Components: CBO, Query Planning
>    Affects Versions: 2.0.0
>            Reporter: Ashutosh Chauhan
>            Assignee: Jesus Camacho Rodriguez
>         Attachments: HIVE-11284.patch
>
>
> It first failed in this build : http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/4621/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message