Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of torindan@gmail.com designates
 209.85.216.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <A101F1DEC1952E47BB64C4E78BB1CFC9010FCB34@IDCEXCHVS.made2manage.com>
References: 
 <A101F1DEC1952E47BB64C4E78BB1CFC9010FCB34@IDCEXCHVS.made2manage.com>
From: =?UTF-8?B?RGFuaWwgxaJPUklO?= <torindan@gmail.com>
Date: Thu, 30 Jun 2011 12:34:02 +0300
Message-ID: <BANLkTimqEAuv6qVWVA1N2rVExoWsd+DMpg@mail.gmail.com>
Subject: Re: distributing the indexing process
To: java-user@lucene.apache.org
Content-Type: text/plain; charset=UTF-8

It depends....

If all documents are distinct then, yeah, go for it.

If you have multiple versions of same document in your data and you
only want to index the latest version...then you need a clever way to
split data to make sure that all versions of document will be indexed
on same host, and you won't have duplicates later.

But my biggest concern is: if your index is that big that you need to
index it on different hosts, are you sure you want it to be combine in
a single index?
Maybe it's a good idea to partition it?

On Thu, Jun 30, 2011 at 12:12, Guru Chandar <Guru.Chandar@consona.com> wrote:
>
>
> If we have to index a lot of documents, is there a way to divide the
> documents into multiple sets and index them on multiple machines in
> parallel, and then merge the resulting indexes back into a single
> machine? If yes, will the result be logically equivalent to indexing all
> the documents on a single machine?
>
>
>
> Thanks,
>
> -gc
>
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org