lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Guru Chandar" <>
Subject RE: distributing the indexing process
Date Thu, 30 Jun 2011 09:45:17 GMT
Thanks for the response. The documents are all distinct. My (limited) understanding on partitioning
the indexes will lead to results being different from the case where you have all in one partition,
due to Lucene currently not supporting distributed idf. Is this correct? Is there a way to
make it work seamlessly?


-----Original Message-----
From: Danil Ε’ORIN [] 
Sent: Thursday, June 30, 2011 3:04 PM
Subject: Re: distributing the indexing process

It depends....

If all documents are distinct then, yeah, go for it.

If you have multiple versions of same document in your data and you
only want to index the latest version...then you need a clever way to
split data to make sure that all versions of document will be indexed
on same host, and you won't have duplicates later.

But my biggest concern is: if your index is that big that you need to
index it on different hosts, are you sure you want it to be combine in
a single index?
Maybe it's a good idea to partition it?

On Thu, Jun 30, 2011 at 12:12, Guru Chandar <> wrote:
> If we have to index a lot of documents, is there a way to divide the
> documents into multiple sets and index them on multiple machines in
> parallel, and then merge the resulting indexes back into a single
> machine? If yes, will the result be logically equivalent to indexing all
> the documents on a single machine?
> Thanks,
> -gc

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message