lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless" <luc...@mikemccandless.com>
Subject Re: background merge hit exception
Date Sat, 03 Jan 2009 13:55:05 GMT
It's remotely possible you hit a JVM bug in the past and that caused the
corruption.
EG there is at least one JVM bug lurking that can affect Lucene (though
apparently with an OS-level fault):
    https://issues.apache.org/jira/browse/LUCENE-1342

I don't know much about Amazon's elastic block store, but presumably it's
unlikely to have undetected IO errors.

Did this corruption happen only once?  (You mentioned hitting dups in the
past... but did you also see corruption too?)

It's very strange that CheckIndex -fix did not resolve the issue.  After
fixing it, if you re-run CheckIndex on the index do you still see that
original one broken segment present?  CheckIndex should have removed
reference to that one segment.

Mike

Brian Whitman <brian@echonest.com> wrote:

> So my apologies for the duplicate comments, I went to go get proof of
> duplicates and was confused as we apparently have duplicates across
> different shards now in our distributed setup (a bug on our end.) I assumed
> when I saw duplicates that it was the same problem as last time. Still
> doesn't help me get at my segment corruption problem, though :(
>
> Michael, in answer to your question: java 1.6 64-bit, debian linux, amazon
> ec2 machine with the index on an elastic block store. No other problems
> with
> that setup for a few months now.
>
> I ran checkindex with -fix on and optimize still throws the same error.
>
>
> On Fri, Jan 2, 2009 at 5:26 PM, Michael McCandless <
> lucene@mikemccandless.com> wrote:
>
> > Also, this (Solr server going down during an add) should not be able to
> > cause this kind of corruption.
> > Mike
> >
> > Yonik Seeley <yseeley@gmail.com> wrote:
> >
> > > On Fri, Jan 2, 2009 at 3:47 PM, Brian Whitman <brian@echonest.com>
> > wrote:
> > > > I will but I bet I can guess what happened -- this index has many
> > > duplicates
> > > > in it as well (same uniqueKey id multiple times) - this happened to
> us
> > > once
> > > > before and it was because the solr server went down during an add.
> > >
> > > That should no longer be possible with Solr 1.3, which uses Lucene for
> > > handling the duplicates in a transactional manner.
> > >
> > > -Yonik
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message