hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuefu Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-8412) Make reduce side join work for all join queries [Spark Branch]
Date Thu, 09 Oct 2014 19:41:33 GMT

    [ https://issues.apache.org/jira/browse/HIVE-8412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14165620#comment-14165620
] 

Xuefu Zhang commented on HIVE-8412:
-----------------------------------

Certain configurations exist in many join related q tests to select a particular type of join,
such as hive.auto.convert.sortmerge.join.to.mapjoin. Those configurations have impact on the
operator tree and such manipulation is done at semantic analyzer, which occurs before task
compilation. Since currently not all type of join are supported in Spark. Thus, we like to
fall back these joins to regular, reduce side join. In doing that, we need to turn off these
operator manipulations.

> Make reduce side join work for all join queries [Spark Branch]
> --------------------------------------------------------------
>
>                 Key: HIVE-8412
>                 URL: https://issues.apache.org/jira/browse/HIVE-8412
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Xuefu Zhang
>
> Regardless all these join related optimizations such as map join, bucket join, skewed
join, etc, reduce side join is the fallback. That means, if a join query wasn't taken care
of by any of the optimization, it should work with reduce side join (might in a less optimal
fashion).
> It's found that this isn't case at the moment. For instance, auto_sortmerge_join_1.q
failed to execute on Spark.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message