lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shai Erera" <ser...@gmail.com>
Subject Re: Explanation on RAMBufferSizeMB
Date Fri, 21 Mar 2008 17:40:09 GMT
What do you mean by "does your test do any merging"?
All I do is create IndexWriter w/ the RAM and MBD settings as I've described
before. Then I just call addDocument. At the end I call optimize() (it is a
one time created index, after that I need it optimized for search).
I guess Lucene performs segment merging in between, although I think I
noticed that before optimize is called, there were 10 segments (CFS files)
of size 200MB each, which matches the default merge factor setting to 10 (if
I'm not mistaken). Does it mean that no segments were merged during the
indexing process?

Shai


On 3/21/08, Michael McCandless <lucene@mikemccandless.com> wrote:
>
>
> Shai Erera wrote:
> > Besides the content field, everything is stored, so that may
> > explain the
> > large CFS files.
>
> You could run w/o CFS turned on and then look at the size of fdt/fdx
> to see if this explains the size.
>
> > Regarding the RAM-usage performance, I tried setting to 128, 256
> > and 512,
> > all gave the same time measurements (give or take ~5%) as the MBD
> > (set to
> > 10,000) run. I think it needs further investigation. Was it tested
> > before? I
> > mean - has someone tried to set RAM to 2GB for example and noticed
> > a major
> > performance improvement (as I'd expect)?
>
> Yes, in LUCENE-843 Doron did some tests and found similar tail-off, I
> think above 32 or 64 MB RAM buffer. I also saw the same thing in my
> tests.
>
> However one thing to remember is: a larger RAM buffer will result in
> less merging.  But, with CMS and a machine with good enough
> concurrency, net run time may be unaffected by this benefit even
> though net resource consumption is.  This makes it hard to test
> speedups in merging :)  Does your test do any merging at all?
>
> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>


-- 
Regards,

Shai Erera

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message