hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Remus Rusanu (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-16757) Use of deprecated {{getRows()}} instead of new {{estimateRowCount(RelMetadataQuery..)}} has serious performance impact
Date Sun, 28 May 2017 20:51:04 GMT

     [ https://issues.apache.org/jira/browse/HIVE-16757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Remus Rusanu updated HIVE-16757:
--------------------------------
    Summary: Use of deprecated {{getRows()}} instead of new {{estimateRowCount(RelMetadataQuery..)}}
has serious performance impact  (was: Use memoization in HiveRelMdRowCount.getRowCount)

> Use of deprecated {{getRows()}} instead of new {{estimateRowCount(RelMetadataQuery..)}}
has serious performance impact
> ----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-16757
>                 URL: https://issues.apache.org/jira/browse/HIVE-16757
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Planning
>            Reporter: Remus Rusanu
>            Assignee: Remus Rusanu
>         Attachments: HIVE-16757.01.patch, HIVE-16757.02.patch
>
>
> Calling Calcite's {{RelMetadataQuery.instance()}} is very expensive because it places
a new memoization cache on the stack. Hidden in the deperecated {{AbstractRelNode.getRows()}}
call is a call to {{instance()}}. In hive we have a number of places where we're calling the
deprecated {{getRows()}} instead of the new API {{estimateRowCount(RelMetadataQuery mq)}}
which accepts the RelMetadataQuery, which most places we actually have it handy to pass. On
looking at the a complex query (49 joins) there are 2995340 calls to {{AbstractRelNode.getRows}},
each one busting the current memoization cache away.
> Was: -On complex queries HiveRelMdRowCount.getRowCount can get called many times. since
it does not memoize its result and the call is recursive, it results in an explosion of calls.
for example a query with 49 joins, during join ordering (LoptOtimizerJoinRule) the HiveRelMdRowCount.getRowCount
gets called 6442 as a top level call, but the recursivity exploded this to 501729 calls. Memoization
of the rezult would stop the recursion early. In my testing this reduced the join reordering
time for said query from 11s to <1s..-
> Note there is no need for {{HiveRelMdRowCount}} memoization because the function is called
in stacks similar to this:
> {code}
> 	at org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdRowCount.getRowCount(HiveRelMdRowCount.java:66)
> 	at GeneratedMetadataHandler_RowCount.getRowCount_$
> 	at GeneratedMetadataHandler_RowCount.getRowCount
> 	at org.apache.calcite.rel.metadata.RelMetadataQuery.getRowCount(RelMetadataQuery.java:204)
> 	at org.apache.calcite.rel.rules.LoptOptimizeJoinRule.swapInputs(LoptOptimizeJoinRule.java:1865)
> 	at org.apache.calcite.rel.rules.LoptOptimizeJoinRule.createJoinSubtree(LoptOptimizeJoinRule.java:1739)
> {code}
> and {{GeneratedMetadataHandler_RowCount.getRowCount}} handles memoization.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message