Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Received-SPF: pass (herse.apache.org: domain of fawcett@gmail.com designates
 66.249.82.238 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=gmail.com;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
        b=qZkoCzUU4Adww6sqqVUXaskDveWBFTCSb0u7qlYgkLUDSAA48SDjoHjPyeligVeTWF+3SN92+l8P/cJosFqyOk1VOfykiLrwMSEZYUhz55V9HbeDX8qkZNlsdnbe8x39oaPH5vO8LWqXRzZ9+ivLCE9BvoQezz6ekytHdpk+i8E=
Message-ID: <80bf0fef0610250547v39959afbyddc3baa7f42d854@mail.gmail.com>
Date: Wed, 25 Oct 2006 08:47:24 -0400
From: "John Fawcett" <fawcett@gmail.com>
To: java-dev@lucene.apache.org
Subject: Fwd: Client-Server Lucene - DocumentWriter
In-Reply-To: <653FA7B2D9C77C45812EB4717D558C5201F91FDC@MI8NYCMAIL14.Mi8.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <653FA7B2D9C77C45812EB4717D558C5201F91FDC@MI8NYCMAIL14.Mi8.com>

---------- Forwarded message ----------
From: John Fawcett <John.Fawcett@tamaleresearch.com>
Date: Wed, 25 Oct 2006 08:39:23 -0400
Subject: Client-Server Lucene - DocumentWriter
To: fawcett@gmail.com

Hi,

I have a design challenge in my own application's use of Lucene, which
triggered an idea for distributed Lucene indexing. Below, I've
summarized the design challenge, and then the indexing idea.

My team is working on a client/server application. The server is a
java application, and the client is in C#/.net.

Right now we are adding capability for offline operation of the
client. Search is part of this work, so we have been working with
Lucene.net to port some of our online search capabilities to offline.

The client only holds a subset of the data held on the server, so we'd
like to move a subset of the search index to the client. There are two
types of transfers - bulk and incremental.

Our goal in both is to offload as much work as possible from the
client to the server.

Bulk transfers happen when a client is initializing for offline use,
or resynching after returning to online. In these scenarios we plan to
create a new index on the server, and just send the files to the
client. The client will then have to perform an index merge.

Incremental adds happen when the client application is online. New
documents are transferred to the client asynchronously. Currently, we
are transferring a document's extracted text. However, the client
still has to perform analysis, inversion, and addition to the index.

Looking through the code for the IndexWriter, I found the
DocumentWriter class. DocumentWriter does the inversion and stores it
in a set of integer arrays and an array of "Posting" objects. Looking
through the class, it seems like the inversion info could be
serialized from server to client pretty easily. The serialized data
from DocumentWriter would be a portable "index record" for a single
document.

Our hope is that we can send this index record from the server to the
client. The idea is to reduce the work on the client to be only the
insertion of the inverted document to the local index.

Having a portable index "record" for an individual document seems very
useful - especially for distributed indexing. I can imagine running a
farm of indexers that only invert documents and send them to a set of
search machines that maintain indexes and field search queries.

Is this something that could be added to the Lucene framework? Is the
"search record" data calculated in DocumentWriter in any way dependent
on the contents of the index? Will this actually save us many client
cycles?

Thanks,
fawce

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org