Mailing-List: contact lucy-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: lucy-dev@incubator.apache.org
Received-SPF: neutral (nike.apache.org: 207.173.203.201 is neither permitted
 nor denied by domain of david@kineticode.com)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1082)
From: "David E. Wheeler" <david@kineticode.com>
In-Reply-To: <4D8AA2C2.1040601@peknet.com>
Date: Thu, 24 Mar 2011 14:01:28 -0400
Content-Transfer-Encoding: quoted-printable
Message-Id: <766D9212-041E-461F-9790-1798760EA956@kineticode.com>
References: <0D140267-39B7-4A12-A843-3B74DD5FB64A@kineticode.com>
 <20110323054543.GA28367@rectangular.com>
 <7A1B1BA1-BDA2-4B22-853C-543270C5F9A2@kineticode.com>
 <4D8AA2C2.1040601@peknet.com>
To: lucy-dev@incubator.apache.org
Subject: Re: [lucy-dev] On Transactionality and Performance

On Mar 23, 2011, at 9:47 PM, Peter Karman wrote:

> The index is definitely available for searching while the indexer is =
doing its
> work. The searcher will become stale though, as soon as the =
$indexer->commit()
> is called, and the existing searcher will not have access to the =
recently-added
> segment(s).

Got it.

> Here, for example, is how I manage searchers:
> =
http://cpansearch.perl.org/src/KARMAN/SWISH-Prog-KSx-0.18/lib/SWISH/Prog/K=
Sx/Searcher.pm
>=20
> Note the get_ks() method, which tracks a UUID per index and re-opens a =
new
> searcher whenever the UUID changes.

Hrm. That might be useful. How do I access that from a =
(KinoSearch|Lucy)::Search::IndexSearcher object? Is the UUID updated =
every time the index is changed?

> Marvin's comments about the efficiency of indexers and the advantage =
of
> "batching up" your indexed documents is merely that: an advantage and =
an efficiency.

Sure.

> In my pipeline, I have separate processes that serialize my incoming =
data
> (analogous to unpacking .tar files and converting/normalizing their =
contents
> into something index-able) and the indexers that actually =
parse/tokenize/insert
> those documents. It's up to the searcher(s) (in my case) to detect =
whether they
> should refresh themselves.

That's not a bad idea. I'll have to keep that in mind for the future.

Thanks,

David