hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harish Butani (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HIVE-7284) CBO: create Partition Pruning rules in Optiq
Date Tue, 08 Jul 2014 07:29:34 GMT

    [ https://issues.apache.org/jira/browse/HIVE-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054600#comment-14054600
] 

Harish Butani edited comment on HIVE-7284 at 7/8/14 7:27 AM:
-------------------------------------------------------------

Preliminary patch attached. Did some minimal testing:
{code}
create table t1 (key string, value string) partitioned by (ds string);
load data local inpath '../../data/files/kv1.txt' into table t1 partition (ds = '2010');
load data local inpath '../../data/files/kv1.txt' into table t1 partition (ds = '2011');
set hive.stats.dbclass=jdbc:derby;
analyze table t1 partition(ds) compute statistics;
analyze table t1 partition(ds) compute statistics for columns key,value;
set hive.cbo.enable=true;
set hive.stats.fetch.column.stats=true;
select * from t1 where ds='2010';
{code}

The plan after applyPreCBOTransforms has:
- the partitionList set correctly on the RelOptHiveTable
- has the correct Stats on RelOptHiveTable
- has the ds='2010' removed from the FilterRelNode

The missing piece is on the way back to AST(and the reanalysis through SemAly) need to devise
a way to pass the PrunedPartitionList back to SemAly, which then uses it directly. (Note:
the AST genned when going back from Optiq planning is missing the predicate on the partition.)

Also need to do more testing: on datatypes, composite predicates etc. But issues uncovered
by these tests could be broken into separate jiras.


was (Author: rhbutani):
Preliminary patch attached. Did some minimal testing:
{code:sql}
create table t1 (key string, value string) partitioned by (ds string);
load data local inpath '../../data/files/kv1.txt' into table t1 partition (ds = '2010');
load data local inpath '../../data/files/kv1.txt' into table t1 partition (ds = '2011');
set hive.stats.dbclass=jdbc:derby;
analyze table t1 partition(ds) compute statistics;
analyze table t1 partition(ds) compute statistics for columns key,value;
set hive.cbo.enable=true;
set hive.stats.fetch.column.stats=true;
select * from t1 where ds='2010';
{code:sql}

The plan after applyPreCBOTransforms has:
- the partitionList set correctly on the RelOptHiveTable
- has the correct Stats on RelOptHiveTable
- has the ds='2010' removed from the FilterRelNode

The missing piece is on the way back to AST(and the reanalysis through SemAly) need to devise
a way to pass the PrunedPartitionList back to SemAly, which then uses it directly. (Note:
the AST genned when going back from Optiq planning is missing the predicate on the partition.)

Also need to do more testing: on datatypes, composite predicates etc. But issues uncovered
by these tests could be broken into separate jiras.

> CBO: create Partition Pruning rules in Optiq
> --------------------------------------------
>
>                 Key: HIVE-7284
>                 URL: https://issues.apache.org/jira/browse/HIVE-7284
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Harish Butani
>            Assignee: Harish Butani
>         Attachments: HIVE-7284.1.patch
>
>
> Create rules in Optiq that do the job of the PartitionPruner.
> For now we will reuse the logic that evaluates the Partition list from prunedExpr. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message