hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-1644) use filter pushdown for automatically accessing indexes
Date Mon, 18 Apr 2011 19:26:05 GMT

    [ https://issues.apache.org/jira/browse/HIVE-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021184#comment-13021184
] 

jiraposter@reviews.apache.org commented on HIVE-1644:
-----------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/558/#review494
-----------------------------------------------------------



conf/hive-default.xml
<https://reviews.apache.org/r/558/#comment1019>

    BTW, these property names should be all-lowercase.



ql/src/java/org/apache/hadoop/hive/ql/Driver.java
<https://reviews.apache.org/r/558/#comment1022>

    When you add an overload, add Javadoc as well (including the new param's meaning).
    



ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java
<https://reviews.apache.org/r/558/#comment1020>

    Could you explain the usage interaction better (along the lines of what I explained in
my review comment)?
    



ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java
<https://reviews.apache.org/r/558/#comment1021>

    You're right.  Either we need to treat them as index columns (so that the predicates on
them will automatically be collected by the predicate analyzer), or we need to explicitly
generate corresponding equality predicates based on the partition values which have already
been identified by partition pruning.
    



ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment1024>

    From an efficiency perspective, you certainly don't want to be doing this over and over
inside the outer for loop; just do it once first outside.
    
    Also, for a table with a huge number of partitions, fetching all of them is a bad idea;
it's better to selectively query the partitions of interest (but batching them if possible).
    



ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment1023>

    This doesn't work because the Partition class does not override the default Java equals
method (which is based on object identity rather than value), and different metastore queries
return different object instances for the same underlying entity.



ql/src/test/queries/clientpositive/index_auto_multiple.q
<https://reviews.apache.org/r/558/#comment1025>

    I don't understand what you mean?  src has two columns, key and value.



ql/src/test/queries/clientpositive/index_auto_unused.q
<https://reviews.apache.org/r/558/#comment1026>

    From the index design doc, there's an optional PARTITION clause when rebuilding an index
which allows you to build just one specific partition, leaving the others unbuilt.  I think
there are some examples in the unit tests.
    
    ALTER INDEX index_name ON table_name [ PARTITION (...) ] REBUILD


- John


On 2011-04-16 06:04:26, Russell Melick wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/558/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2011-04-16 06:04:26)
bq.  
bq.  
bq.  Review request for hive.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  Review request for HIVE-1644.12.patch
bq.  
bq.  
bq.  This addresses bug HIVE-1644.
bq.      https://issues.apache.org/jira/browse/HIVE-1644
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a21f589 
bq.    conf/hive-default.xml c42197f 
bq.    ql/src/java/org/apache/hadoop/hive/ql/Driver.java 14015d0 
bq.    ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 6437385 
bq.    ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b 
bq.    ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d 
bq.    ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f 
bq.    ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java PRE-CREATION

bq.    ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 1f01446

bq.    ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c 
bq.    ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676 
bq.    ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION

bq.    ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2

bq.    ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java
PRE-CREATION 
bq.    ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
PRE-CREATION 
bq.    ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java
PRE-CREATION 
bq.    ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 937a7b3 
bq.    ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f0aca84 
bq.    ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9 
bq.    ql/src/test/queries/clientpositive/index_auto.q PRE-CREATION 
bq.    ql/src/test/queries/clientpositive/index_auto_file_format.q PRE-CREATION 
bq.    ql/src/test/queries/clientpositive/index_auto_multiple.q PRE-CREATION 
bq.    ql/src/test/queries/clientpositive/index_auto_partitioned.q PRE-CREATION 
bq.    ql/src/test/queries/clientpositive/index_auto_unused.q PRE-CREATION 
bq.    ql/src/test/results/clientpositive/index_auto.q.out PRE-CREATION 
bq.    ql/src/test/results/clientpositive/index_auto_file_format.q.out PRE-CREATION 
bq.    ql/src/test/results/clientpositive/index_auto_multiple.q.out PRE-CREATION 
bq.    ql/src/test/results/clientpositive/index_auto_partitioned.q.out PRE-CREATION 
bq.    ql/src/test/results/clientpositive/index_auto_unused.q.out PRE-CREATION 
bq.    ql/src/test/results/clientpositive/index_opt_where.q.out PRE-CREATION 
bq.    ql/src/test/results/clientpositive/index_opt_where_partitioned.q.out PRE-CREATION 
bq.    ql/src/test/results/clientpositive/index_opt_where_simple.q.out PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/558/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Russell
bq.  
bq.



> use filter pushdown for automatically accessing indexes
> -------------------------------------------------------
>
>                 Key: HIVE-1644
>                 URL: https://issues.apache.org/jira/browse/HIVE-1644
>             Project: Hive
>          Issue Type: Improvement
>          Components: Indexing
>    Affects Versions: 0.8.0
>            Reporter: John Sichi
>            Assignee: Russell Melick
>         Attachments: HIVE-1644.1.patch, HIVE-1644.10.patch, HIVE-1644.11.patch, HIVE-1644.12.patch,
HIVE-1644.13.patch, HIVE-1644.14.patch, HIVE-1644.2.patch, HIVE-1644.3.patch, HIVE-1644.4.patch,
HIVE-1644.5.patch, HIVE-1644.6.patch, HIVE-1644.7.patch, HIVE-1644.8.patch, HIVE-1644.9.patch
>
>
> HIVE-1226 provides utilities for analyzing filters which have been pushed down to a table
scan.  The next step is to use these for selecting available indexes and generating access
plans for those indexes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message