Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: local policy)
From: Daniel Noll <daniel@nuix.com>
Organization: Nuix Pty Ltd
To: java-user@lucene.apache.org
Subject: Re: Document ID shuffling under 2.3.x (on merge?)
Date: Mon, 17 Mar 2008 11:24:42 +1100
User-Agent: KMail/1.9.6 (enterprise 0.20070907.709405)
References: <200803111645.27956.daniel@nuix.com>
 <200803131042.50640.daniel@nuix.com>
 <59BEA59D-D139-441E-8C34-A32C978FA848@mikemccandless.com>
In-Reply-To: <59BEA59D-D139-441E-8C34-A32C978FA848@mikemccandless.com>
MIME-Version: 1.0
Content-Type: text/plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Message-Id: <200803171124.43064.daniel@nuix.com>

On Thursday 13 March 2008 19:46:20 Michael McCandless wrote:
> But, when a normal merge of segments with deletions completes, your
> docIDs will shift.  In trunk we now explicitly compute the docID
> shifting that happens after a merge, because we don't always flush
> pending deletes when flushing added docs, but this is all done
> privately to IndexWriter.

I don't need to worry about deleted documents as such things don't exist in 
our system, hence the optimisation based on document IDs.

> I'm a little confused: you said optimize() introduces the problem,
> but, it sounds like optimize() should be fixing the problem because
> it compacts all docIDs to match what you were "guessing" outside of
> Lucene?  Can you post the full stack trace of the exceptions you're
> hitting?

You're misunderstanding how we're getting the ID, that's all.  We're getting 
it by calling docCount() (after adding) and subtracting 1, which is 
guaranteed to give the right ID at the time of indexing, although of course, 
later is another matter entirely.  We were operating from now out of date 
information which says the IDs don't shift unless you call delete...

Example:

  add document, assume ID 0 (docCount = 1)
  add document, assume ID 1 (docCount = 2)
  add document, FAILS - assumed not added
  re-add document minus reader fields, assume ID 3 (docCount = 4)

So the ID assumptions are correct at this point; when optimize() is called, it 
shifts the third document sucht that it then has ID 2, and our internal 
counts become wrong.

I've backported the expungeDeleted() patch into 2.3 and will be testing it out 
next; seems it will do more or less what we want and merging the deleted 
document should be relatively quick as it will only ever exist in the in 
DocumentsWriter's in-memory buffer.

Daniel

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org