hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-21778) CBO: "Struct is not null" gets evaluated as `nullable` always causing filter miss in the query
Date Thu, 05 Mar 2020 09:52:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-21778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17051958#comment-17051958
] 

Hive QA commented on HIVE-21778:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12995648/HIVE-21778.6.patch

{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 18096 tests executed
*Failed tests:*
{noformat}
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testCancelRenewTokenFlow (batchId=307)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testConnection (batchId=307)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testIsValid (batchId=307)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testIsValidNeg (batchId=307)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeProxyAuth (batchId=307)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeTokenAuth (batchId=307)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testProxyAuth (batchId=307)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testRenewDelegationToken (batchId=307)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=307)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20960/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20960/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20960/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12995648 - PreCommit-HIVE-Build

> CBO: "Struct is not null" gets evaluated as `nullable` always causing filter miss in
the query
> ----------------------------------------------------------------------------------------------
>
>                 Key: HIVE-21778
>                 URL: https://issues.apache.org/jira/browse/HIVE-21778
>             Project: Hive
>          Issue Type: Bug
>          Components: CBO
>    Affects Versions: 4.0.0, 2.3.5
>            Reporter: Rajesh Balamohan
>            Assignee: Vineet Garg
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: HIVE-21778.1.patch, HIVE-21778.2.patch, HIVE-21778.3.patch, HIVE-21778.4.patch,
HIVE-21778.5.patch, HIVE-21778.6.patch, test_null.q, test_null.q.out
>
>          Time Spent: 40m
>  Remaining Estimate: 0h
>
> {noformat}
> drop table if exists test_struct;
> CREATE external TABLE test_struct
> (
>   f1 string,
>   demo_struct struct<f1:string, f2:string, f3:string>,
>   datestr string
> );
> set hive.cbo.enable=true;
> explain select * from etltmp.test_struct where datestr='2019-01-01' and demo_struct is
not null;
> STAGE PLANS:
>   Stage: Stage-0
>     Fetch Operator
>       limit: -1
>       Processor Tree:
>         TableScan
>           alias: test_struct
>           filterExpr: (datestr = '2019-01-01') (type: boolean) <----- Note that demo_struct
filter is not added here
>           Filter Operator
>             predicate: (datestr = '2019-01-01') (type: boolean)
>             Select Operator
>               expressions: f1 (type: string), demo_struct (type: struct<f1:string,f2:string,f3:string>),
'2019-01-01' (type: string)
>               outputColumnNames: _col0, _col1, _col2
>               ListSink
> set hive.cbo.enable=false;
> explain select * from etltmp.test_struct where datestr='2019-01-01' and demo_struct is
not null;
> STAGE PLANS:
>   Stage: Stage-0
>     Fetch Operator
>       limit: -1
>       Processor Tree:
>         TableScan
>           alias: test_struct
>           filterExpr: ((datestr = '2019-01-01') and demo_struct is not null) (type: boolean)
<----- Note that demo_struct filter is added when CBO is turned off
>           Filter Operator
>             predicate: ((datestr = '2019-01-01') and demo_struct is not null) (type:
boolean)
>             Select Operator
>               expressions: f1 (type: string), demo_struct (type: struct<f1:string,f2:string,f3:string>),
'2019-01-01' (type: string)
>               outputColumnNames: _col0, _col1, _col2
>               ListSink
> {noformat}
> In CalcitePlanner::genFilterRelNode, the following code misses to evaluate this filter.

> {noformat}
> RexNode factoredFilterExpr = RexUtil
>           .pullFactors(cluster.getRexBuilder(), convertedFilterExpr);
> {noformat}
> Note that even if we add `demo_struct.f1` it would end up pushing the filter correctly.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message