hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-2036) Update bitmap indexes for automatic usage
Date Mon, 13 Jun 2011 23:05:49 GMT

    [ https://issues.apache.org/jira/browse/HIVE-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048859#comment-13048859
] 

jiraposter@reviews.apache.org commented on HIVE-2036:
-----------------------------------------------------



bq.  On 2011-06-13 22:57:46, John Sichi wrote:
bq.  > ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java, line 114
bq.  > <https://reviews.apache.org/r/857/diff/4/?file=20984#file20984line114>
bq.  >
bq.  >     I don't think this should be necessary.  We just want to propagate the partition
column predicate (whatever it is) from the base table query to the index table query; partition
pruning on the index table query will do the rest of the work.
bq.  >     
bq.  >     In other words, if the original query had
bq.  >     
bq.  >     part_key=<whatever>
bq.  >     
bq.  >     we want to preserve that on the index table query.  That's what the code is
already supposed to be doing before your change; was it not working?
bq.  >

This code is to prevent automatic usage from kicking in if the index has not been built on
the partition specified in the partition predicate. (i.e. if the index has only been built
on partition ds=foo, and the query is select key from src where ds=bar; We do not want to
execute an index query in this case. It seems like adding a test for bitmaps specifically
to mirror index_auto_unused.q(which is where this functionality is tested for Compact indices)
would be a good idea.


- Syed


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/857/#review825
-----------------------------------------------------------


On 2011-06-11 19:05:42, Syed Albiz wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/857/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2011-06-11 19:05:42)
bq.  
bq.  
bq.  Review request for hive and John Sichi.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  Add support for generating index queries to support automatic usage of bitmap indexes.
This required changing the interface to the IndexHandlers to support accepting queries on
multiple indexes. The compact indexes were modified to use this new interface as well, although
no functional changes were made to how they work. Only supports AND predicates right now,
but it should be possibly to extend the BitmapQuery interface defined in this patch to easily
support OR predicates as well. Currently benchmarking these changes on a test cluster.
bq.  
bq.  
bq.  This addresses bug HIVE-2036.
bq.      https://issues.apache.org/jira/browse/HIVE-2036
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 4fba845 
bq.    ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java e5ee183 
bq.    ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java 3caa4cc 
bq.    ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java af9d7b1

bq.    ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapInnerQuery.java PRE-CREATION

bq.    ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapOuterQuery.java PRE-CREATION

bq.    ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapQuery.java PRE-CREATION 
bq.    ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 56e7609

bq.    ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java d64e88b 
bq.    ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
268560d 
bq.    ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java
0873e1a 
bq.    ql/src/test/queries/clientpositive/index_auto_partitioned.q 5f92f04 
bq.    ql/src/test/queries/clientpositive/index_bitmap_auto.q PRE-CREATION 
bq.    ql/src/test/queries/clientpositive/index_bitmap_auto_partitioned.q PRE-CREATION 
bq.    ql/src/test/results/clientpositive/index_auto_partitioned.q.out 05cc84a 
bq.    ql/src/test/results/clientpositive/index_bitmap_auto.q.out PRE-CREATION 
bq.    ql/src/test/results/clientpositive/index_bitmap_auto_partitioned.q.out PRE-CREATION

bq.  
bq.  Diff: https://reviews.apache.org/r/857/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  Passes unit tests, additional testcase to test automatic bitmap indexing index_bitmap_auto.q
was also added to the TestCliDriver suite. Currently benchmarking changes on a test cluster.
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Syed
bq.  
bq.



> Update bitmap indexes for automatic usage
> -----------------------------------------
>
>                 Key: HIVE-2036
>                 URL: https://issues.apache.org/jira/browse/HIVE-2036
>             Project: Hive
>          Issue Type: Improvement
>          Components: Indexing
>    Affects Versions: 0.8.0
>            Reporter: Russell Melick
>            Assignee: Syed S. Albiz
>         Attachments: HIVE-2036.1.patch, HIVE-2036.3.patch
>
>
> HIVE-1644 will provide automatic usage of indexes, and HIVE-1803 adds bitmap index support.
 The bitmap code will need to be extended after it is committed to enable automatic use of
indexing.  Most work will be focused in the BitmapIndexHandler, which needs to generate the
re-entrant QL index query.  There may also be significant work in the IndexPredicateAnalyzer
to support predicates with OR's, instead of just AND's as it is currently.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message