hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jiraposter@reviews.apache.org (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-1644) use filter pushdown for automatically accessing indexes
Date Fri, 15 Apr 2011 20:36:05 GMT

    [ https://issues.apache.org/jira/browse/HIVE-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13020422#comment-13020422
] 

jiraposter@reviews.apache.org commented on HIVE-1644:
-----------------------------------------------------


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/558/#review482
-----------------------------------------------------------



ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment982>

    A few comments here.
    
    1) Rather than passing in the entire table scan object and letting the handler set properties
on it, I think we should just have the handler pass back the necessary information (input
format and intermediate file).
    
    2) The generateIndexQuery method's parameter list is growing.  For plugin interfaces,
a good pattern we've been using in other places is to introduce a new context class (say HiveIndexQueryContext)
with getters and setters for the information to be communicated in both directions.  Then
the caller instantiates one of these and passes in an instance.  The plugin reads and writes
to the context.  On return, the caller gets the modified information out.  The main benefit
is that in the future, if we need to pass more information, we just add new members to the
context class, and none of the existing plugin implementations break.
    
    In this case, you could also put the context objects in a map (instead of having to keep
multiple maps indexQueryTasks/additionalInputs etc).
    



ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment983>

    Just put it as a TODO for now; create the followup JIRA issue and reference it in the
TODO.
    



ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
<https://reviews.apache.org/r/558/#comment990>

    Look in Hive.java; there are methods like
    
    public List<Partition> getPartitionsByNames(Table tbl, List<String> partNames)
    
    which look up the actual partitions for a table from the metastore.  You can pass in indexTable.
    



ql/src/test/results/clientpositive/index_opt_where_partitioned.q.out
<https://reviews.apache.org/r/558/#comment991>

    Hmm...what if we could avoid relabeling altogether?  If you look in Driver.java, there's
a method compile which calls TaskFactory.resetId().  This is what causes us to start back
over from 0.
    
    If you add an optional parameter resetTaskIds=true, and then pass false for the Driver
instance used for compiling the reentrant query, that might do it.
    
    
    


- John


On 2011-04-15 08:08:14, Russell Melick wrote:
bq.  
bq.  -----------------------------------------------------------
bq.  This is an automatically generated e-mail. To reply, visit:
bq.  https://reviews.apache.org/r/558/
bq.  -----------------------------------------------------------
bq.  
bq.  (Updated 2011-04-15 08:08:14)
bq.  
bq.  
bq.  Review request for hive.
bq.  
bq.  
bq.  Summary
bq.  -------
bq.  
bq.  Review request for HIVE-1644.12.patch
bq.  
bq.  
bq.  This addresses bug HIVE-1644.
bq.      https://issues.apache.org/jira/browse/HIVE-1644
bq.  
bq.  
bq.  Diffs
bq.  -----
bq.  
bq.    common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a21f589 
bq.    conf/hive-default.xml c42197f 
bq.    ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 6437385 
bq.    ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java c02d90b 
bq.    ql/src/java/org/apache/hadoop/hive/ql/index/AbstractIndexHandler.java dd0186d 
bq.    ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java 411b78f 
bq.    ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 1f01446

bq.    ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 50db44c 
bq.    ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 6162676 
bq.    ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/IndexWhereResolver.java PRE-CREATION

bq.    ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/PhysicalOptimizer.java 0ae9fa2

bq.    ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcCtx.java
PRE-CREATION 
bq.    ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java
PRE-CREATION 
bq.    ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java
PRE-CREATION 
bq.    ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 937a7b3 
bq.    ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f0aca84 
bq.    ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java 73391e9 
bq.    ql/src/test/queries/clientpositive/index_opt_where.q PRE-CREATION 
bq.    ql/src/test/queries/clientpositive/index_opt_where_partitioned.q PRE-CREATION 
bq.    ql/src/test/queries/clientpositive/index_opt_where_simple.q PRE-CREATION 
bq.    ql/src/test/results/clientpositive/index_opt_where.q.out PRE-CREATION 
bq.    ql/src/test/results/clientpositive/index_opt_where_partitioned.q.out PRE-CREATION 
bq.    ql/src/test/results/clientpositive/index_opt_where_simple.q.out PRE-CREATION 
bq.  
bq.  Diff: https://reviews.apache.org/r/558/diff
bq.  
bq.  
bq.  Testing
bq.  -------
bq.  
bq.  
bq.  Thanks,
bq.  
bq.  Russell
bq.  
bq.



> use filter pushdown for automatically accessing indexes
> -------------------------------------------------------
>
>                 Key: HIVE-1644
>                 URL: https://issues.apache.org/jira/browse/HIVE-1644
>             Project: Hive
>          Issue Type: Improvement
>          Components: Indexing
>    Affects Versions: 0.8.0
>            Reporter: John Sichi
>            Assignee: Russell Melick
>         Attachments: HIVE-1644.1.patch, HIVE-1644.10.patch, HIVE-1644.11.patch, HIVE-1644.12.patch,
HIVE-1644.13.patch, HIVE-1644.2.patch, HIVE-1644.3.patch, HIVE-1644.4.patch, HIVE-1644.5.patch,
HIVE-1644.6.patch, HIVE-1644.7.patch, HIVE-1644.8.patch, HIVE-1644.9.patch
>
>
> HIVE-1226 provides utilities for analyzing filters which have been pushed down to a table
scan.  The next step is to use these for selecting available indexes and generating access
plans for those indexes.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message