lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Rutherglen" <jason.rutherg...@gmail.com>
Subject Re: Scaling
Date Thu, 17 Jul 2008 13:55:00 GMT
The scaling per machine should be linear.  The overhead from the network is
minimal because the Lucene object sizes are not impacting.  Google mentions
in one of their early white papers on scaling
http://labs.google.com/papers/googlecluster-ieee.pdf that they have sub
indexes which are now popularly called shards over which an individual
thread performs a search over.  Executed in parallel (ParallelMultiSearcher
which does not use thread pooling) the response time will be faster than
using a single thread assuming part of the indexes are in the system cache.
A query is simply an iteration so it is easy to see how parallelization
speeds up response times.  Queries per second should ideally be solved by
adding more hardware with the same indexes on each server.  Then further
dividing these into what can be termed cells which represent different
indexes on sets of servers.

Having a large index on a single machine does not scale well because most of
the index will not be in the system cache.  If the index grows so does the
response time.  Dividing the index up into shards and cells allows for
efficient scaling which is proven at the big G.  It puts more of the total
index in the system cache of many machines.

The general assumption is that hardware is cheap and can be added easily,
search systems can take advantage of this and parallelize as much as
possible, per server, per application.

On Wed, Jul 16, 2008 at 9:41 AM, Karl Wettin <karl.wettin@gmail.com> wrote:

> Is there some sort of a scaling strategies listing available? I think there
> is a Wiki page missing.
>
> What are the typical promblems I'll encounter when distributing the search
> over multiple machines?
>
> Do people split up their index per node or do they use the complete index
> and restrict what part to search in using filters? The latter would be good
> for the scores, right? Then how do I calculate the cost in speed for the
> score with better quality? I mean, splitting the index in two and searching
> on two machines using ParallelMultiSearcher probably means that I'll get
> something like 30% speed improvement and not 100%. Or?
>
> Is there something to win by using multiple threads each restricted to a
> part each of the same index on a single machine, compared to a single
> thread? Or is it all I/O? That would mean there is something to gain if the
> index was on SSD or in RAM, right?
>
>
>      karl
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message