hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From prasenjit mukherjee <prasen....@gmail.com>
Subject Fwd: How to avoid a full table scan for column search. ( HIVE+LUCENE)
Date Tue, 16 Mar 2010 12:36:10 GMT
forwarding to hdfs and pig mailing-lists for  responses from wider audience.

---------- Forwarded message ----------
From: prasenjit mukherjee <prasen.bea@gmail.com>
Date: Tue, Mar 16, 2010 at 11:47 AM
Subject: How to avoid a full table scan for column search. ( HIVE+LUCENE)
To: hive-user <hive-user@hadoop.apache.org>

Is there a way to avoid full table scan for an arbitrary where-clause
usage ? partitioning/bucketing makes sense only when you know which
columns will be searched upon. I was wondering if there is any project
which combines the SQL-like features of HIVE and inverted-index like
search-features of LUCENE, and works on cloud. Guess I am asking for
too much :(

I have been using oracle till now and my usage is mainly restricted to
do summation-type queries with some where clause, example being  :
"Select SUM(column1)  where col2='foo' AND col3='bar'".
The output is always some aggregation and where clauses can include
"<, >, =, IN".  I would like to use some kind of distributed
processing to speed up the table generation, search query-time. Hive (
and to some extent Pig ) seems to be the closest tool available to
what I am looking for. I am also exploring hbase, but not sure whether
it will be the right choice for my problem.

Hive can definitely help in parallelizing up the search-processing.
But my main concern is whether hive does ( or plans to do ) any
storage optimization like oracle,lucene ( apart from simple
partitioning/bucketing ).  It seems that all the hadoop-options (
hive,pig,hbase) will have to do  an entire table scan.

Appreciate any suggestions/feedback..


View raw message