lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ian Lea" <ian....@gmail.com>
Subject Re: CorruptIndexException with some versions of java
Date Tue, 18 Mar 2008 13:33:44 GMT
The data is loaded in chunks of up to 100K docs in separate runs of
the program if that helps answer the first question.  All buffers have
default values, docs are small but not tiny, JVM is running with
default settings.

Answers to previous questions, and infostream, will follow once the
job has failed again.


--
Ian.


On Tue, Mar 18, 2008 at 1:24 PM, Michael McCandless
<lucene@mikemccandless.com> wrote:
>
>  One question: do you know whether 67,861 docs "feels like" a newly
>  flushed segment, or, the result of a merge?
>
>  Ie, roughly how many docs are you buffering in IndexWriter before it
>  flushes?  Are they very small documents and your RAM buffer is large?
>
>  Mike
>
>
>
>  Ian Lea wrote:
>
>  > Hi
>  >
>  >
>  > When bulk loading into a new index I'm seeing this exception
>  >
>  > Exception in thread "Thread-1"
>  > org.apache.lucene.index.MergePolicy$MergeException:
>  > org.apache.lucene.index.CorruptIndexException: doc counts differ for
>  > segment _4l: fieldsReader shows 67861 but segmentInfo shows 67862
>  >       at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run
>  > (ConcurrentMergeScheduler.java:271)
>  > Caused by: org.apache.lucene.index.CorruptIndexException: doc counts
>  > differ for segment _4l: fieldsReader shows 67861 but segmentInfo shows
>  > 67862
>  >       at org.apache.lucene.index.SegmentReader.initialize
>  > (SegmentReader.java:313)
>  >       at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:262)
>  >       at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:221)
>  >       at org.apache.lucene.index.IndexWriter.mergeMiddle
>  > (IndexWriter.java:3093)
>  >       at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:2834)
>  >       at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run
>  > (ConcurrentMergeScheduler.java:240)
>  >
>  > when use java version 1.6.0_05-b13 or 1.6.0_04-b12 on linux, with
>  > lucene 2.3.0 or 2.3.1 or lucene-core-2.3-SNAPSHOT from yesterday.
>  >
>  > With java version 1.6.0_03-b05 things work fine.
>  >
>  > The exception happens a few hundred thousand documents into the load.
>  >
>  > A different program updating a different index with different data on
>  > a different server gave a similar error on version 1.6.0_05-b13 and
>  > lucene 2.3.0.
>  >
>  >
>  > Any ideas?  Is this maybe a known issue or am I missing something
>  > obvious?
>  >
>  >
>  >
>  > --
>  > Ian.
>  >
>
>
> > ---------------------------------------------------------------------
>  > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>  > For additional commands, e-mail: java-user-help@lucene.apache.org
>  >
>
>
>  ---------------------------------------------------------------------
>  To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>  For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message