Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: local policy)
From: Daniel Noll <daniel@nuix.com>
Organization: Nuix Pty Ltd
To: java-user@lucene.apache.org
Subject: Re: Document ID shuffling under 2.3.x (on merge?)
Date: Thu, 13 Mar 2008 10:42:50 +1100
User-Agent: KMail/1.9.6 (enterprise 0.20070907.709405)
References: <200803111645.27956.daniel@nuix.com>
 <200803121229.38013.daniel@nuix.com>
 <EFF47371-79C3-44B5-806B-16820ED69602@mikemccandless.com>
In-Reply-To: <EFF47371-79C3-44B5-806B-16820ED69602@mikemccandless.com>
MIME-Version: 1.0
Content-Type: text/plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Message-Id: <200803131042.50640.daniel@nuix.com>

On Wednesday 12 March 2008 19:36:57 Michael McCandless wrote:
> OK, I think very likely this is the issue: when IndexWriter hits an
> exception while processing a document, the portion of the document
> already indexed is left in the index, and then its docID is marked
> for deletion.  You can see these deletions in your infoStream:
>
>    flush 0 buffered deleted terms and 30 deleted docIDs on 20 segments
>
> This means you have deletions in your index, by docID, and so when
> you optimize the docIDs are then compacted.

Aha.  Under 2.2, a failure would result in nothing being added to the text 
index so this would explain the problem.  It would also explain why smaller 
data sets are less likely to cause the problem (it's less likely for there to 
be an error in it.)

Workarounds?
  - flush() after any IOException from addDocument()  (overhead?)
  - use ++ to determine the next document ID instead of
    index.getWriter().docCount()  (out of sync after an error but fixes itself
    on optimize().
  - Use a field for a separate ID (slower later when reading the index)
  - ???

Daniel

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org