hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-16757) Use memoization in HiveRelMdRowCount.getRowCount
Date Thu, 25 May 2017 14:30:04 GMT

    [ https://issues.apache.org/jira/browse/HIVE-16757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024793#comment-16024793
] 

Hive QA commented on HIVE-16757:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12869865/HIVE-16757.01.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 10763 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_limit] (batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_limit] (batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[explainuser_1] (batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar] (batchId=152)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cbo_limit] (batchId=138)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/5430/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/5430/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-5430/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12869865 - PreCommit-HIVE-Build

> Use memoization in HiveRelMdRowCount.getRowCount
> ------------------------------------------------
>
>                 Key: HIVE-16757
>                 URL: https://issues.apache.org/jira/browse/HIVE-16757
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Planning
>            Reporter: Remus Rusanu
>            Assignee: Remus Rusanu
>         Attachments: HIVE-16757.01.patch
>
>
> On complex queries HiveRelMdRowCount.getRowCount can get called many times. since it
does not memoize its result and the call is recursive, it results in an explosion of calls.
for example a query with 49 joins, during join ordering (LoptOtimizerJoinRule) the HiveRelMdRowCount.getRowCount
gets called 6442 as a top level call, but the recursivity exploded this to 501729 calls. Memoization
of the rezult would stop the recursion early. In my testing this reduced the join reordering
time for said query from 11s to <1s..



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message