hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rajesh Balamohan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-17035) Optimizer: Lineage transform() should be invoked after rest of the optimizers are invoked
Date Mon, 17 Jul 2017 03:54:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-17035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Rajesh Balamohan updated HIVE-17035:
------------------------------------
    Status: Open  (was: Patch Available)

> Optimizer: Lineage transform() should be invoked after rest of the optimizers are invoked
> -----------------------------------------------------------------------------------------
>
>                 Key: HIVE-17035
>                 URL: https://issues.apache.org/jira/browse/HIVE-17035
>             Project: Hive
>          Issue Type: Bug
>          Components: Logical Optimizer
>            Reporter: Rajesh Balamohan
>            Assignee: Rajesh Balamohan
>            Priority: Minor
>         Attachments: HIVE-17035.1.patch, HIVE-17035.2.patch, HIVE-17035.3.patch, HIVE-17035.4.patch
>
>
> In a fairly large query which had tens of left join, time taken to create linageInfo
itself took 1500+ seconds. This is due to the fact that the table had lots of columns and
in some processing, it ended up processing 7000+ value columns in {{ReduceSinkLineage}}, though
only 50 columns were projected in the query. 
> It would be good to invoke lineage transform when rest of the optimizers in {{Optimizer}}
are invoked. This would avoid unwanted processing and help in improving the runtime.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message