lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alessandro Benedetti <abenede...@apache.org>
Subject Re: one large index vs many small indexes
Date Wed, 11 Nov 2015 17:41:48 GMT
I would add an additional consideration.
Are we aggregating any of the results ?
Is any use case where we aggregate the results from different Indexes into
one response?
In the case we would need to calculate the aggregation overload.

This means that if the Index is not that big, sometimes is better to not
split it across different indexes, because we save lucene side, and we lose
in aggregating results, network traffic etc.

Cheers

On 11 November 2015 at 15:13, Toke Eskildsen <te@statsbiblioteket.dk> wrote:

> Lutz Fechner <Lutz.Fechner@perfect.com> wrote:
> > separated index will allow you split IO load over multiple
> > physical drives as well as use different Analyzers (if your
> > customers are having different content that will make sense).
>
> Other ways to get better IO is RAID, SSD or RAM.
>
> Multiple indexes makes a lot of sense from a functionality point of view
> (logistics, ranking, individualization), but it loses on price/performance
> if most of the data are in use most of the time. It boils down to the
> overhead of running an index.
>
> Discussing this on the abstract level is hard as there are so many
> variables influencing the decision. The quality of our guesswork is
> proportional to the amount of information you give us, Sascha. It would
> help if we knew more, such as
>
> * How many customers?
> * How many customers in a year?
> * How large is the average index data size per customer?
> * How many documents per customer?
> * Are all customer data treated equal or are some of it specialized?
> * Are the sizes fairly uniform or are there a few huge outliers?
> * How often does a customer update the data?
> * How often does a customer issue searches?
> * How many concurrent requests will there be at peak time?
> * Is it okay to have a slow first-search but faster subsequent searches?
>
>
> - Toke Eskildsen
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


-- 
--------------------------

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message