lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Danil Ε’ORIN <torin...@gmail.com>
Subject Re: distributing the indexing process
Date Thu, 30 Jun 2011 09:34:02 GMT
It depends....

If all documents are distinct then, yeah, go for it.

If you have multiple versions of same document in your data and you
only want to index the latest version...then you need a clever way to
split data to make sure that all versions of document will be indexed
on same host, and you won't have duplicates later.

But my biggest concern is: if your index is that big that you need to
index it on different hosts, are you sure you want it to be combine in
a single index?
Maybe it's a good idea to partition it?

On Thu, Jun 30, 2011 at 12:12, Guru Chandar <Guru.Chandar@consona.com> wrote:
>
>
> If we have to index a lot of documents, is there a way to divide the
> documents into multiple sets and index them on multiple machines in
> parallel, and then merge the resulting indexes back into a single
> machine? If yes, will the result be logically equivalent to indexing all
> the documents on a single machine?
>
>
>
> Thanks,
>
> -gc
>
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message