hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rajesh Balamohan (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HIVE-17035) Optimizer: Lineage transform() should be invoked after rest of the optimizers are invoked
Date Wed, 05 Jul 2017 08:43:00 GMT
Rajesh Balamohan created HIVE-17035:
---------------------------------------

             Summary: Optimizer: Lineage transform() should be invoked after rest of the optimizers
are invoked
                 Key: HIVE-17035
                 URL: https://issues.apache.org/jira/browse/HIVE-17035
             Project: Hive
          Issue Type: Bug
          Components: Logical Optimizer
            Reporter: Rajesh Balamohan
            Priority: Minor


In a fairly large query which had tens of left join, time taken to create linageInfo itself
took 1500+ seconds. This is due to the fact that the table had lots of columns and in some
processing, it ended up having 7000+ value columns in {{ReduceSinkLineage}}. 

It would be good to invoke lineage transform when rest of the optimizers in {{Optimizer}}
are invoked. This would avoid help in improving the runtime.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message