lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Pilato <da...@pilato.fr>
Subject Re: Lucene vs RDBMS indexing at scale
Date Wed, 06 Feb 2013 06:21:52 GMT
So you probably should ask your question to the Elasticsearch mailing list.
I think that some ES users already scales to x billion docs.

Even if ES is Lucene based, it adds features to scale out (sharding, routing...).

HTH
--
David ;-)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs


Le 5 févr. 2013 à 23:34, Drew Kutcharian <drew@venarc.com> a écrit :

The records are mostly logging events where they will have:
1. a timestamp 
2. the type of the event
3. potentially a set of key/value properties

Then I would want to be able to slice and dice the records based on time (required), type
and/or the key/values. 
In addition, I would want to have statistics about the events (i.e. How many of event type
A happened during past 7 days.)

The reason I'm looking into Lucene is mainly because we would like to use ElasticSearch to
ease the scaling of the solution.

HTH,

Drew


On Feb 5, 2013, at 2:29 PM, Stephen Howe <silentgunner@gmail.com> wrote:

> Part of the answer depends on what kind of records you have. For instance,
> are you dealing with a lot of numeric data?
> 
> If you need all those functions and only want to support exact matches and
> basic boolean comparisons, then I'd go with a RDBMS instead of Lucene.
> You'll get better support for the aggregate functions and save yourself the
> overhead of having to worry about tokenization impeding your exact match
> queries.
> 
> 
> On Tue, Feb 5, 2013 at 5:09 PM, Drew Kutcharian <drew@venarc.com> wrote:
> 
>> Hey Guys,
>> 
>> I'm trying to figure out what would be a better approach to indexing when
>> it comes to a large number of records (say 1 billion).
>> 
>> As far as queries:
>> 1. Only support exact matches (a field is equal to some constant value) or
>> range matches (a field is larger/smaller than some constant value) or a
>> combination of both
>> 2. NO need for full-text searching
>> 3. Support for aggregate functions (sum, average, etc.)
>> 
>> Would it be better to use Lucene or an RDBMS to index the data? Are there
>> any good reads on this subject?
>> 
>> -- Drew
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message