hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lianhui Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-7384) Research into reduce-side join [Spark Branch]
Date Fri, 22 Aug 2014 03:27:13 GMT

    [ https://issues.apache.org/jira/browse/HIVE-7384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14106407#comment-14106407
] 

Lianhui Wang commented on HIVE-7384:
------------------------------------

i think the thoughts is same as ideas that you said before. like HIVE-7158, that will auto-calculate
the number of reducers based on some input from Hive (upper/lower bound).

> Research into reduce-side join [Spark Branch]
> ---------------------------------------------
>
>                 Key: HIVE-7384
>                 URL: https://issues.apache.org/jira/browse/HIVE-7384
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Xuefu Zhang
>            Assignee: Szehon Ho
>         Attachments: Hive on Spark Reduce Side Join.docx, sales_items.txt, sales_products.txt,
sales_stores.txt
>
>
> Hive's join operator is very sophisticated, especially for reduce-side join. While we
expect that other types of join, such as map-side join and SMB map-side join, will work out
of the box with our design, there may be some complication in reduce-side join, which extensively
utilizes key tag and shuffle behavior. Our design principle prefers to making Hive implementation
work out of box also, which might requires new functionality from Spark. The tasks is to research
into this area, identifying requirements for Spark community and the work to be done on Hive
to make reduce-side join work.
> A design doc might be needed for this. For more information, please refer to the overall
design doc on wiki.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message