hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-8024) Find out whether it's possible to remove UnionOperator from original operator tree [Spark Branch]
Date Tue, 09 Sep 2014 20:33:31 GMT

    [ https://issues.apache.org/jira/browse/HIVE-8024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14127504#comment-14127504
] 

Chao commented on HIVE-8024:
----------------------------

There's a problem again:

Suppose the operator tree is the following:

{code}
 TS_0    TS_2
   \    /
   UNION_3
     |
    SEL_4
{code}

After removing UNION operator, it will look like this:

{code}
TS_0    TS_2
   \    /
    SEL_4
{code}

(Again, I ignored some operators, but you get the idea.)

Then, we could have MapWork 1 starts from {{TS_0}}, and MapWork 2 starts from {{TS_2}}.
Now, when MapWork 1 initialize itself, it will initialize the operator tree, starting from
{{TS_0}, and go down the tree.
When it gets to {{SEL_4}}, it will not be able to initialize it, because not all of {{SEL_4}}'s
parent are initialize at that point.
Hence, the execution will fail.

> Find out whether it's possible to remove UnionOperator from original operator tree [Spark
Branch]
> -------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-8024
>                 URL: https://issues.apache.org/jira/browse/HIVE-8024
>             Project: Hive
>          Issue Type: Task
>          Components: Spark
>            Reporter: Chao
>            Assignee: Chao
>
> Currently, after operator tree is processed, the generated works with union operators
will go through {{GenSparkUtils::removeUnionOperators}}, which will clone the original operator
plan associated with the work, and remove union operators in it. This caused some issues as
seen, for example, in HIVE-7870. This JIRA is created to find out whether it's possible to
just remove the union operators in the original plan.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message