lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Drew Kutcharian <d...@venarc.com>
Subject Re: Lucene vs RDBMS indexing at scale
Date Tue, 05 Feb 2013 22:34:35 GMT
The records are mostly logging events where they will have:
1. a timestamp 
2. the type of the event
3. potentially a set of key/value properties

Then I would want to be able to slice and dice the records based on time (required), type
and/or the key/values. 
In addition, I would want to have statistics about the events (i.e. How many of event type
A happened during past 7 days.)

The reason I'm looking into Lucene is mainly because we would like to use ElasticSearch to
ease the scaling of the solution.

HTH,

Drew


On Feb 5, 2013, at 2:29 PM, Stephen Howe <silentgunner@gmail.com> wrote:

> Part of the answer depends on what kind of records you have. For instance,
> are you dealing with a lot of numeric data?
> 
> If you need all those functions and only want to support exact matches and
> basic boolean comparisons, then I'd go with a RDBMS instead of Lucene.
> You'll get better support for the aggregate functions and save yourself the
> overhead of having to worry about tokenization impeding your exact match
> queries.
> 
> 
> On Tue, Feb 5, 2013 at 5:09 PM, Drew Kutcharian <drew@venarc.com> wrote:
> 
>> Hey Guys,
>> 
>> I'm trying to figure out what would be a better approach to indexing when
>> it comes to a large number of records (say 1 billion).
>> 
>> As far as queries:
>> 1. Only support exact matches (a field is equal to some constant value) or
>> range matches (a field is larger/smaller than some constant value) or a
>> combination of both
>> 2. NO need for full-text searching
>> 3. Support for aggregate functions (sum, average, etc.)
>> 
>> Would it be better to use Lucene or an RDBMS to index the data? Are there
>> any good reads on this subject?
>> 
>> -- Drew
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>> 
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message