lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Keegan <peterlkee...@gmail.com>
Subject Re: IO exception during merge/optimize
Date Mon, 26 Oct 2009 19:59:24 GMT
On Mon, Oct 26, 2009 at 3:00 PM, Michael McCandless <
lucene@mikemccandless.com> wrote:

> On Mon, Oct 26, 2009 at 2:55 PM, Peter Keegan <peterlkeegan@gmail.com>
> wrote:
> > On Mon, Oct 26, 2009 at 2:50 PM, Michael McCandless <
> > lucene@mikemccandless.com> wrote:
> >
> >> On Mon, Oct 26, 2009 at 10:44 AM, Peter Keegan <peterlkeegan@gmail.com>
> >> wrote:
> >> > Even running in console mode, the exception is difficult to interpret.
> >> > Here's an exception that I think occurred during an add document,
> commit
> >> or
> >> > close:
> >> > doc counts differ for segment _g: field Reader shows 137 but
> segmentInfo
> >> > shows 5777
> >>
> >> That's spooky.  Do you have the full exception for this one?  What IO
> >> system are you running on?  (Is it just a local drive on your windows
> >> computer?) It's almost as if the IO system is not generating an
> >> IOException to Java when disk fills up.
> >>
> >
> > Index and code are all on a local drive. There is no other exception
> coming
> > back - just what I reported.
>
> But, you didn't report a traceback for this first one?
>

Yes, I need to add some more printStackTrace calls.


>
> >> > I ensured that the disk space was low before updating the index.
> >>
> >> You mean, to intentionally test the disk-full case?
> >>
> >
> > Yes, that's right.
>
> OK.  Can you turn on IndexWriter's infoStream, get this disk full /
> corruption to happen again, and post back the resulting output?  Make
> sure your index first passes CheckIndex before starting (so we don't
> begin the test w/ any pre-existing index corruption).
>

Good point about CheckIndex.  I've already found 2 bad ones. I will build
new indexes from scratch. This will take a while.


> >> > On another occasion, the exception was:
> >> > background merge hit exception: _0:C1080260 _1:C139 _2:C123 _3:C107
> >> _4:C126
> >> > _5:C121 _6:C126 _7:C711 _8:C116 into _9 [optimize] [mergeDocStores]
> >>
> >> In this case, the SegmentMerger was trying to open this segment, but
> >> on attempting to read the first int from the fdx (fields index) file
> >> for one of the segments, it hit EOF.
> >>
> >> This is also spooky -- this looks like index corruption, which should
> >> never happen on hitting disk full.
> >>
> >
> > That's what I thought, too. Could Lucene be catching the IOException and
> > turning it into a different exception?
>
> I think that's unlikely, but I guess possible.  We have "disk full"
> tests in the unit tests, that throw an IOException at different times.
>
> What exact windows version are you using?  The local drive is NTFS?
>

Windows Server 2003 Enterprise x64 SP2. Local drive is NTFS


>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message