lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Schetinin" <ascheti...@entopia.com>
Subject RE: Distributed Lucene..
Date Mon, 06 Mar 2006 14:37:19 GMT
Hello,

We are implementing a distributed searcher and indexer based on Lucene.
I cannot share its code but I may provide hints basing on our
experience.

What we did basically is having several machines indexing documents and
creating small Lucene indexes. 
We hacked :-) IndexWriter of Lucene to start all segment names with a
prefix unique for each small index part.
Then, when adding it to the actual index, we simply copy the new segment
to the folder with the other segments, and add it in such a way so that
the optimize() function cannot be called.
This way adding a new segment is very unintrusive for the searcher.
Optimization is scheduled to happen at night.

The index is divided into parts located on different physical machines,
and here is where complexity begins.
We do not try normalizing term weights across the machines, assuming
that with large data quantities they will be more or less evenly
distributed. 
But the problem exist, and probably in the future we will think how to
handle it.
The documents are distributed across the machines randomally, and
merging the results becames a little head pain :-)

Best Regards,

Andrew Schetinin


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message