hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuefu Zhang (JIRA)" <>
Subject [jira] [Commented] (HIVE-7541) Support union all on Spark
Date Thu, 31 Jul 2014 14:07:41 GMT


Xuefu Zhang commented on HIVE-7541:

[~nyang] Thanks for working on this. This task is fairly large, and I think breaking the task
into smaller ones would help in sharing the load and tracking progress. Could you please create
smaller JIRAs for this? I'd image that supporting union would require work in the following

1. SparkCompiler changes: generate a SparkWork that contains UnionWork from logical operator
2. SparkPlan modeling: represent the spark job in terms of a graph (rather than) list of SparkTran
instances. We may need to enhance SparkTran interface. 
3. SparkPlanGenerator: need to generate a plan from SparkWork, which needs to use Spark's
union transformation to achieve the functionality..
4. other earas.

Tez can be a good reference point

Please feel free to create JIRAs for those or other areas.

> Support union all on Spark
> --------------------------
>                 Key: HIVE-7541
>                 URL:
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Xuefu Zhang
>            Assignee: Na Yang
> For union all operator, we will use Spark's union transformation. Refer to the design
doc on wiki for more information.

This message was sent by Atlassian JIRA

View raw message