lucene-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <>
Subject Re: [PROPOSAL] index server project
Date Mon, 30 Oct 2006 23:53:40 GMT
Yonik Seeley wrote:
> On 10/18/06, Doug Cutting <> wrote:
>> We assume that, within an index, a file with a given name is written
>> only once.
> Is this necessary, and will we need the lockless patch (that avoids
> renaming or rewriting *any* files), or is Lucene's current index
> behavior sufficient?

It's not strictly required, but it would make index synchronization a 
lot simpler.  Yes, I was assuming the lockless patch would be committed 
to Lucene before this project gets very far.  Something more than that 
would be required in order to keep old versions, but this could be as 
simple as a Directory subclass that refuses to remove files for a time.

> The search side seems straightforward enough, but I haven't totally
> figured out how the update side should work.

The master should be out of the loop as much as possible.  One approach 
is that clients randomly assign documents to indexes and send the 
updates directly to the indexing node.  Alternately, clients might index 
locally, then ship the updates to a node packaged as an index.  That was 
the intent of the addIndex method.

> One potental problem is a document overwrite implemented as a delete
> then an add.
> More than one client doing this for the same document could result in
> 0 or 2 documents, instead of 1.  I guess clients will just need to be
> relatively coordinated in their activities.

Good point.  Either the two clients must coordinate, to make sure that 
they're not updating the same document at the same time, or use a 
strategy where updates are routed to the slave that contained the old 
version of the document.  That would require a broadcast query to figure 
out which slave that is.

> It's unfortunate the master needs to be involved on every document add.

That should not normally be the case.  Clients can cache the set of 
writable index locations and directly submit new documents without 
involving the master.

> If deletes were broadcast, and documents could go to any partition,
> that would be one way around it (with the downside of a less powerful
> master that could implement certain distribution policies).
> Another way to lessen the master-in-the-middle cost is to make sure
> one can aggregate small requests:
>    IndexLocation[] getUpdateableIndex(String[] id);

I'd assumed that the updateable version of an index does not move around 
very often.  Perhaps a lease mechanism is required.  For example, a call 
to getUpdateableIndex might be valid for ten minutes.

> We might consider a delete() on the master interface too.  That way it 
> could
>  3) hide the delete policy (broadcast or directl-to-server-that-has-doc)
> 2) potentially do some batching of deletes
> 1) simply do the delete locally if there is a single index partition
> and this is a combination master/searcher

I'm reticent to put any frequently-made call on the master.  I'd prefer 
to keep the master only involved at an executive level, with all 
per-document and per-query traffic going directly from client to slave.

> It seems like the master might want to be involved in commits too, or
> maybe we just rely on the slave to master heartbeat to kick of
> immediately after a commit so that index replication can be initiated?

I like the latter approach.  New versions are only published as 
frequently as clients poll the master for updated IndexLocations. 
Clients keep a cache of both readable and updatable index locations that 
are periodically refreshed.

I was not imagining a real-time system, where the next query after a 
document is added would always include that document.  Is that a 
requirement?  That's harder.

At this point I'm mostly trying to see if this functionality would meet 
the needs of Solr, Nutch and others.

Must we include a notion of document identity and/or document version in 
the mechanism?  Would that facillitate updates and coherency?

In Nutch a typical case is that you have a bunch of URLs with content 
that may-or-may-not have been previously indexed.  The approach I'm 
currently leaning towards is that we'd broadcast the deletions of all of 
these to all slaves, then add index them to randomly assigned indexes. 
In Nutch multiple clients would naturally be coordinated, since each url 
is represented only once in each update cycle.


View raw message