lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: IO exception during merge/optimize
Date Wed, 28 Oct 2009 16:07:32 GMT
That's exactly what oal.util.UnicodeUtils does when convertig UTF-8 to
UTF-16 (which is Java's internal encoding).

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de

> -----Original Message-----
> From: Michael McCandless [mailto:lucene@mikemccandless.com]
> Sent: Wednesday, October 28, 2009 4:25 PM
> To: java-user@lucene.apache.org
> Subject: Re: IO exception during merge/optimize
> 
> Right, I would expect Lucene would silently truncate the term at the
> U+FFFF, and not lead to this odd exception.
> 
> Mike
> 
> On Wed, Oct 28, 2009 at 11:23 AM, Robert Muir <rcmuir@gmail.com> wrote:
> > i might be wrong about this, but recently I intentionally tried to
> create
> > index with terms with U+FFFF to see if it would cause a problem :)
> >
> > the U+FFFF seemed to be discarded completely (maybe at UTF-8 encode
> time)...
> > then again I was using RAMDirectory.
> >
> > On Wed, Oct 28, 2009 at 10:58 AM, Peter Keegan
> <peterlkeegan@gmail.com>wrote:
> >
> >> The only change I made to the source code was the patch for
> >> PayloadNearQuery
> >> (LUCENE-1986).
> >> It's possible that our content contains U+FFFF. I will run in debugger
> and
> >> see.
> >> The data is 'sensitive', so I may not be able to provide a bad segment,
> >> unfortunately.
> >>
> >> Peter
> >>
> >> On Wed, Oct 28, 2009 at 10:43 AM, Michael McCandless <
> >> lucene@mikemccandless.com> wrote:
> >>
> >> > OK... when you exported the sources & built yourself, you didn't make
> >> > any changes, right?
> >> >
> >> > It's really odd how many of the errors are due to the term
> >> > "literals:cfid196$", or some variation (one time with "on" appended,
> >> > another time with "microsoft").  Do you know what documents typically
> >> > contain that term, and what the context is around it?  Maybe try to
> >> > index only those documents and see if this happens?  (It could
> >> > conceivably be caused by bad data, if this is some weird bug).  One
> >> > question: does your content ever use the [invalid] unicode character
> >> > U+FFFF?  (Lucene uses this internally to mark the end of the term).
> >> >
> >> > Would it be possible to zip up all files starting with _1c (should be
> >> > ~22 MB) and post somewhere that I could download?  That's the
> smallest
> >> > of the broken segments I think.
> >> >
> >> > I don't need the full IW output just yet, thanks.
> >> >
> >> > Mike
> >> >
> >> > On Wed, Oct 28, 2009 at 10:21 AM, Peter Keegan
> <peterlkeegan@gmail.com>
> >> > wrote:
> >> > > Yes, I used JDK 1.6.0_16 when running CheckIndex and it reported
> the
> >> same
> >> > > problems when run multiple times.
> >> > >
> >> > >>Also, what does Lucene version "2.9 exported - 2009-10-27 15:31:52"
> >> mean?
> >> > > This appears to be something added by the ant build, since I built
> >> Lucene
> >> > > from the source code.
> >> > >
> >> > > I rebuilt the index using mergeFactor=50, ramBufferSize=200MB,
> >> > > maxBufferedDocs=1000000
> >> > > This produced 49 segments, 9 of which are broken. The broken
> segments
> >> are
> >> > in
> >> > > the latter half, similar to my previous post with 3 segments. Do
> you
> >> > think
> >> > > this could be caused by 'bad' data, for example bad unicode
> characters?
> >> > >
> >> > > Here is the output from CheckIndex:
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >> >
> >> >
> >>
> >
> >
> >
> > --
> > Robert Muir
> > rcmuir@gmail.com
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message