lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Keegan <peterlkee...@gmail.com>
Subject Re: IO exception during merge/optimize
Date Thu, 29 Oct 2009 23:06:11 GMT
Mark,

With 1.9G, I had to increase the JVM heap significantly (to 8G)  to avoid
paging and GC hits. Here is a table comparing indexing times, optimizing
times and peak memory usage as a function of the  RAMBufferSize. This was
run on a 64-bit server with 32GB RAM:

RamSize        Index(min)    Optimize(min)     Max VM
1.9G         24            5                   5G
800M        24            5                  4G

Not much difference. I'll make a couple more runs with lower values.
Btw, the indexing times are really about 5 min. shorter because of some
non-Lucene related delays after the last document.

Peter



On Thu, Oct 29, 2009 at 4:30 PM, Mark Miller <markrmiller@gmail.com> wrote:

> Any chance I could get you to try that again with a buffer of like 800MB
> to a gig and do a comparison?
>
> I've been investigating the returns you get with a larger buffer size.
> It appears to be pretty diminishing returns over 100MB or so - at higher
> than that, I've gotten both slower speeds for some sizes, and larger
> gains for others. But only better by 5-10 docs a second up to a gig. But
> I can't reliably test at over a gig - I have only 4 GB of RAM, and even
> with that, at over a gig it starts to page and the performance gets hit.
> I'd love to see what kind of benefit you see going from around a gig to
> just under 2.
>
> Peter Keegan wrote:
> > Btw, this 2.9 indexer is fast! I indexed 4Gb (1.07 million docs) with
> > optimization in just under 30 min.
> > I used setRAMBufferSizeMB=1.9G
> >
> > Peter
> >
> > On Thu, Oct 29, 2009 at 3:46 PM, Peter Keegan <peterlkeegan@gmail.com
> >wrote:
> >
> >
> >> A handful of the source documents did contain the U+FFFF character. The
> >> patch from *LUCENE-2016<
> https://issues.apache.org/jira/browse/LUCENE-2016>
> >> *fixed the problem.
> >> Thanks Mike!
> >>
> >> Peter
> >>
> >>
> >> On Wed, Oct 28, 2009 at 1:29 PM, Michael McCandless <
> >> lucene@mikemccandless.com> wrote:
> >>
> >>
> >>> Hmm, only a few affected terms, and all this particular
> >>> "literals:cfid196$" term, with optional suffixes.  Really strange.
> >>>
> >>> One things that's odd is the exact term "literals:cfid196$" is printed
> >>> twice, which should never happen (every unique term should be stored
> >>> only once, in the terms dict).
> >>>
> >>> And, otherwise, CheckIndex got through the index just fine.
> >>>
> >>> Try searching a TermQuery with these affected terms and see if it
> >>> succeeds?  If so, maybe trying making an index with one or two of
> >>> them, alone, and see if that index shows the problem?
> >>>
> >>> OK I'm attaching more mods.  Can you re-run your CheckIndex?  It will
> >>> produce an enormous amount of output, but if you can excise the few
> >>> lines around when that warning comes out & post back that'd be great.
> >>>
> >>> Mike
> >>>
> >>> On Wed, Oct 28, 2009 at 12:23 PM, Peter Keegan <peterlkeegan@gmail.com
> >
> >>> wrote:
> >>>
> >>>> Just to be safe, I ran with the official jar file from one of the
> >>>>
> >>> mirrors
> >>>
> >>>> and reproduced the problem.
> >>>> The debug session is not showing any characters = '\uffff' (checking
> >>>>
> >>> this in
> >>>
> >>>> Tokenizer).
> >>>> The output from the modified CheckIndex follows. There are only a few
> >>>>
> >>> terms
> >>>
> >>>> with the inconsistency. They are all legitimate terms from the app's
> >>>> context. With this info, I might be able to isolate the source
> >>>>
> >>> documents.
> >>>
> >>>> What should I be looking for when they are indexed?
> >>>>
> >>>> CheckInput output:
> >>>>
> >>>> Opening index @
> >>>>
> >>> D:\mnsavs\lresumes1\lresumes1.luc\lresumes1.search.main.4
> >>>
> >>>> Segments file=segments_2 numSegments=3 version=FORMAT_DIAGNOSTICS
> >>>>
> >>> [Lucene
> >>>
> >>>> 2.9]
> >>>>  1 of 3: name=_0 docCount=413585
> >>>>    compound=false
> >>>>    hasProx=true
> >>>>    numFiles=8
> >>>>    size (MB)=1,148.817
> >>>>    diagnostics = {os.version=5.2, os=Windows 2003,
> lucene.version=2.9.0
> >>>> 817268P - 2009-09-21 10:25:09, source=flush, os.arch=amd64,
> >>>> java.version=1.6.0_16, java.vendor=Sun Microsystems Inc.}
> >>>>    docStoreOffset=0
> >>>>    docStoreSegment=_0
> >>>>    docStoreIsCompoundFile=false
> >>>>    no deletions
> >>>>    test: open reader.........OK
> >>>>    test: fields..............OK [33 fields]
> >>>>    test: field norms.........OK [33 fields]
> >>>>    test: terms, freq, prox...OK [7704753 terms; 180326717 terms/docs
> >>>>
> >>> pairs;
> >>>
> >>>> 340244234 tokens]
> >>>>    test: stored fields.......OK [1240755 total field count; avg 3
> fields
> >>>> per doc]
> >>>>    test: term vectors........OK [0 total vector count; avg 0 term/freq
> >>>> vector fields per doc]
> >>>>
> >>>>  2 of 3: name=_1 docCount=359068
> >>>>    compound=false
> >>>>    hasProx=true
> >>>>    numFiles=8
> >>>>    size (MB)=1,125.161
> >>>>    diagnostics = {os.version=5.2, os=Windows 2003,
> lucene.version=2.9.0
> >>>> 817268P - 2009-09-21 10:25:09, source=flush, os.arch=amd64,
> >>>> java.version=1.6.0_16, java.vendor=Sun Microsystems Inc.}
> >>>>    docStoreOffset=413585
> >>>>    docStoreSegment=_0
> >>>>    docStoreIsCompoundFile=false
> >>>>    no deletions
> >>>>    test: open reader.........OK
> >>>>    test: fields..............OK [33 fields]
> >>>>    test: field norms.........OK [33 fields]
> >>>>    test: terms, freq, prox...WARNING: term  literals:cfid196$
> docFreq=43
> >>>>
> >>> !=
> >>>
> >>>> num docs seen 4 + num docs deleted 0
> >>>> WARNING: term  literals:cfid196$ docFreq=1 != num docs seen 4 + num
> docs
> >>>> deleted 0
> >>>> WARNING: term  literals:cfid196$ docFreq=1 != num docs seen 4 + num
> docs
> >>>> deleted 0
> >>>> WARNING: term  literals:cfid196$commandant docFreq=1 != num docs seen
> 9
> >>>>
> >>> +
> >>>
> >>>> num docs deleted 0
> >>>> WARNING: term  literals:cfid196$on docFreq=3178 != num docs seen 1 +
> num
> >>>> docs deleted 0
> >>>> OK [7137621 terms; 179101847 terms/docs pairs; 346076058 tokens]
> >>>>    test: stored fields.......OK [1077204 total field count; avg 3
> fields
> >>>> per doc]
> >>>>    test: term vectors........OK [0 total vector count; avg 0 term/freq
> >>>> vector fields per doc]
> >>>>
> >>>>  3 of 3: name=_2 docCount=304849
> >>>>    compound=false
> >>>>    hasProx=true
> >>>>    numFiles=8
> >>>>    size (MB)=962.004
> >>>>    diagnostics = {os.version=5.2, os=Windows 2003,
> lucene.version=2.9.0
> >>>> 817268P - 2009-09-21 10:25:09, source=flush, os.arch=amd64,
> >>>> java.version=1.6.0_16, java.vendor=Sun Microsystems Inc.}
> >>>>    docStoreOffset=772653
> >>>>    docStoreSegment=_0
> >>>>    docStoreIsCompoundFile=false
> >>>>    no deletions
> >>>>    test: open reader.........OK
> >>>>    test: fields..............OK [33 fields]
> >>>>    test: field norms.........OK [33 fields]
> >>>>    test: terms, freq, prox...WARNING: term  contents:? docFreq=1 !=
> num
> >>>> docs seen 246 + num docs deleted 0
> >>>> WARNING: term  literals:cfid196$ docFreq=45 != num docs seen 4 + num
> >>>>
> >>> docs
> >>>
> >>>> deleted 0
> >>>> WARNING: term  literals:cfid196$ docFreq=1 != num docs seen 4 + num
> docs
> >>>> deleted 0
> >>>> WARNING: term  literals:cfid196$cashier docFreq=1 != num docs seen 37
> +
> >>>>
> >>> num
> >>>
> >>>> docs deleted 0
> >>>> WARNING: term  literals:cfid196$interrogation docFreq=181 != num docs
> >>>>
> >>> seen 1
> >>>
> >>>> + num docs deleted 0
> >>>> WARNING: term  literals:cfid196$leader docFreq=1 != num docs seen 353
> +
> >>>>
> >>> num
> >>>
> >>>> docs deleted 0
> >>>> WARNING: term  literals:cfid196$microsoft docFreq=3114 != num docs
> seen
> >>>>
> >>> 1 +
> >>>
> >>>> num docs deleted 0
> >>>> WARNING: term  literals:cfid196$nt docFreq=200 != num docs seen 1 +
> num
> >>>>
> >>> docs
> >>>
> >>>> deleted 0
> >>>> OK [6497769 terms; 145296880 terms/docs pairs; 293458734 tokens]
> >>>>    test: stored fields.......OK [914547 total field count; avg 3
> fields
> >>>>
> >>> per
> >>>
> >>>> doc]
> >>>>    test: term vectors........OK [0 total vector count; avg 0 term/freq
> >>>> vector fields per doc]
> >>>>
> >>>> No problems were detected with this index.
> >>>>
> >>>> Peter
> >>>>
> >>>>
> >>>> On Wed, Oct 28, 2009 at 11:29 AM, Michael McCandless <
> >>>> lucene@mikemccandless.com> wrote:
> >>>>
> >>>>
> >>>>> On Wed, Oct 28, 2009 at 10:58 AM, Peter Keegan <
> peterlkeegan@gmail.com
> >>>>>
> >>>>> wrote:
> >>>>>
> >>>>>> The only change I made to the source code was the patch for
> >>>>>>
> >>>>> PayloadNearQuery
> >>>>>
> >>>>>> (LUCENE-1986).
> >>>>>>
> >>>>> That patch certainly shouldn't lead to this.
> >>>>>
> >>>>>
> >>>>>> It's possible that our content contains U+FFFF. I will run in
> >>>>>>
> >>> debugger
> >>>
> >>>>> and
> >>>>>
> >>>>>> see.
> >>>>>>
> >>>>> OK may as well check just so we cover all possibilities.
> >>>>>
> >>>>>
> >>>>>> The data is 'sensitive', so I may not be able to provide a bad
> >>>>>>
> >>> segment,
> >>>
> >>>>>> unfortunately.
> >>>>>>
> >>>>> OK, maybe we can modify your CheckIndex instead.  Let's start with
> >>>>> this, which prints a warning whenever the docFreq differs but
> >>>>> otherwise continues (vs throwing RuntimeException).  I'm curious
how
> >>>>> many terms show this, and whether the TermEnum keeps working after
> >>>>> this term that has different docFreq:
> >>>>>
> >>>>> Index: src/java/org/apache/lucene/index/CheckIndex.java
> >>>>> ===================================================================
> >>>>> --- src/java/org/apache/lucene/index/CheckIndex.java    (revision
> >>>>>
> >>> 829889)
> >>>
> >>>>> +++ src/java/org/apache/lucene/index/CheckIndex.java    (working
> copy)
> >>>>> @@ -672,8 +672,8 @@
> >>>>>         }
> >>>>>
> >>>>>         if (freq0 + delCount != docFreq) {
> >>>>> -          throw new RuntimeException("term " + term + " docFreq="
+
> >>>>> -                                     docFreq + " != num docs seen
"
> +
> >>>>> freq0 + " + num docs deleted " + delCount);
> >>>>> +          System.out.println("WARNING: term  " + term + " docFreq="
> +
> >>>>> +                             docFreq + " != num docs seen " + freq0
> +
> >>>>> " + num docs deleted " + delCount);
> >>>>>         }
> >>>>>       }
> >>>>>
> >>>>> Mike
> >>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>>>
> >>>>>
> >>>>>
> >>
> >
> >
>
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message