Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Content-Type: multipart/alternative;
	boundary="===============5520671301017932280=="
MIME-Version: 1.0
Subject: Re: Review Request: HIVE-2036: Update bitmap indexes for automatic
 usage
From: "Syed Albiz" <s.s.albiz@gmail.com>
To: "John Sichi" <jsichi@fb.com>
Date: Tue, 14 Jun 2011 04:05:43 -0000
Message-ID: <20110614040543.13641.50537@reviews.apache.org>
Cc: "Syed Albiz" <s.s.albiz@gmail.com>,"hive" <dev@hive.apache.org>
In-Reply-To: <20110611190542.30868.88881@reviews.apache.org>
References: <20110611190542.30868.88881@reviews.apache.org>

--===============5520671301017932280==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/857/
-----------------------------------------------------------

(Updated 2011-06-14 04:05:43.158797)


Review request for hive and John Sichi.


Changes
-------

Removed redundant check on partition predicate (which is done in IndexWhere=
Processor). The reason this was causing problems was that when the index wa=
s being built, the query generated to build the index was run through the o=
ptimizer and at this stage the optimizer thought that the index was already=
 built and had the partition. A simpler solution is to just disable index q=
uery optimization for building indexes. =


Summary
-------

Add support for generating index queries to support automatic usage of bitm=
ap indexes. This required changing the interface to the IndexHandlers to su=
pport accepting queries on multiple indexes. The compact indexes were modif=
ied to use this new interface as well, although no functional changes were =
made to how they work. Only supports AND predicates right now, but it shoul=
d be possibly to extend the BitmapQuery interface defined in this patch to =
easily support OR predicates as well. Currently benchmarking these changes =
on a test cluster.


This addresses bug HIVE-2036.
    https://issues.apache.org/jira/browse/HIVE-2036


Diffs (updated)
-----

  ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 4fba845 =

  ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexHandler.java e5ee183 =

  ql/src/java/org/apache/hadoop/hive/ql/index/IndexPredicateAnalyzer.java 3=
caa4cc =

  ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.jav=
a af9d7b1 =

  ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapInnerQuery.java =
PRE-CREATION =

  ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapOuterQuery.java =
PRE-CREATION =

  ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapQuery.java PRE-C=
REATION =

  ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.j=
ava 56e7609 =

  ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java d64e88b =

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhere=
Processor.java 268560d =

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhere=
TaskDispatcher.java 0873e1a =

  ql/src/test/queries/clientpositive/index_auto_partitioned.q 5f92f04 =

  ql/src/test/queries/clientpositive/index_bitmap_auto.q PRE-CREATION =

  ql/src/test/queries/clientpositive/index_bitmap_auto_partitioned.q PRE-CR=
EATION =

  ql/src/test/results/clientpositive/index_auto_partitioned.q.out 05cc84a =

  ql/src/test/results/clientpositive/index_bitmap_auto.q.out PRE-CREATION =

  ql/src/test/results/clientpositive/index_bitmap_auto_partitioned.q.out PR=
E-CREATION =


Diff: https://reviews.apache.org/r/857/diff


Testing
-------

Passes unit tests, additional testcase to test automatic bitmap indexing in=
dex_bitmap_auto.q was also added to the TestCliDriver suite. Currently benc=
hmarking changes on a test cluster.


Thanks,

Syed


--===============5520671301017932280==--