hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suhas Satish (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-8622) Split map-join plan into 2 SparkTasks in 3 stages [Spark Branch]
Date Fri, 07 Nov 2014 19:14:33 GMT

    [ https://issues.apache.org/jira/browse/HIVE-8622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14202496#comment-14202496
] 

Suhas Satish commented on HIVE-8622:
------------------------------------

[~csun] - We already have a mapr of  BaseWork containing the map-join to its parent ReduceSinks.

This exists as {{linkWorkWithReduceSinkMap}} in {{GenSparkProcContext}}

Do you think we can leverage that in some way, or replace the RSs in that Map with the HashTableSinks
that we introduced? It looks like we should still propagate the whole GenSparkProcContext
to the {{SparkMapJoinResolver}} through the SparkCompiler.generateTaskTree(...) and {{SparkCompiler.optimizeTaskPlan}}
 

All the state information stored there will make life a lot easier. 

> Split map-join plan into 2 SparkTasks in 3 stages [Spark Branch]
> ----------------------------------------------------------------
>
>                 Key: HIVE-8622
>                 URL: https://issues.apache.org/jira/browse/HIVE-8622
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Suhas Satish
>            Assignee: Chao
>         Attachments: HIVE-8622.2-spark.patch, HIVE-8622.3-spark.patch, HIVE-8622.patch
>
>
> This is a sub-task of map-join for spark 
> https://issues.apache.org/jira/browse/HIVE-7613
> This can use the baseline patch for map-join
> https://issues.apache.org/jira/browse/HIVE-8616



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message