hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Szehon Ho (JIRA)" <>
Subject [jira] [Updated] (HIVE-8202) Support SMB Join for Hive on Spark [Spark Branch]
Date Tue, 28 Oct 2014 02:04:34 GMT


Szehon Ho updated HIVE-8202:
    Attachment: HIVE-8202.2-spark.patch

Rebasing + attaching files for another run.  There may have been more check-ins since the
rebase, but see what else fails.

> Support SMB Join for Hive on Spark [Spark Branch]
> -------------------------------------------------
>                 Key: HIVE-8202
>                 URL:
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Xuefu Zhang
>            Assignee: Szehon Ho
>         Attachments: HIVE-8202.1-spark.patch, HIVE-8202.2-spark.patch, Hive on Spark
SMB Join.docx, Hive on Spark SMB Join.pdf
> SMB joins are used wherever the tables are sorted and bucketed. It's a reduce-side join.
The join boils down to just merging the already sorted tables, allowing this operation to
be faster than an ordinary map-join. However, if the tables are partitioned, there could be
a slow down as each mapper would need to get a very small chunk of a partition which has a
single key. Thus, in some scenarios it's beneficial to convert SMB join to SMB map join as
> The task is to research and support the conversion from regular SMB join to SMB map join
for Spark execution engine.

This message was sent by Atlassian JIRA

View raw message