hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuefu Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HIVE-8220) Refactor multi-insert code such that plan splitting and task generation are modular and reusable [Spark Branch]
Date Fri, 24 Oct 2014 18:03:35 GMT

     [ https://issues.apache.org/jira/browse/HIVE-8220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Xuefu Zhang resolved HIVE-8220.
-------------------------------
    Resolution: Won't Fix

No needed with HIVE-8118.

> Refactor multi-insert code such that plan splitting and task generation are modular and
reusable [Spark Branch]
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-8220
>                 URL: https://issues.apache.org/jira/browse/HIVE-8220
>             Project: Hive
>          Issue Type: Improvement
>          Components: Spark
>            Reporter: Xuefu Zhang
>              Labels: Spark-M1
>
> This is a followup for HIVE-7053. Currently the code to split the operator tree and to
generate tasks is mingled and thus hard to understand and maintain. Logically the two seems
independent. This can be improved by modulizing both. The following might be helpful:
> {code}
> @Override
> protected void generateTaskTree(List<Task<? extends Serializable>> rootTasks,
ParseContext pCtx,
>       List<Task<MoveWork>> mvTask, Set<ReadEntity> inputs, Set<WriteEntity>
outputs)
>       throws SemanticException {
> // 1. Identify if the plan is for multi-insert and split the plan if necessary
> List<Set<Operator>> operatorSets = multiInsertSplit(...);
> // 2. For each operator set, generate a task.
> for (Set<Operator> topOps : operatorSets) {
>   SparkTask task = generateTask(topOps);
>   ...
> }
> // 3. wire up the tasks
> ...
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message