lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun Kumar K <arunk...@gmail.com>
Subject Re: "4.1 consuming more memory than 3.0.2 while Indexing"
Date Mon, 01 Apr 2013 12:21:18 GMT
Hi Adrien,

I have seen memory usage using linux command top for RES memory & i have
used SimpleFSDirectory for indexing (NOT FSDirectory which i guess will
take it's default as MMapDir)  in both versions.

Does 4.1 still take MMapDir as default ?


Arun


On Mon, Apr 1, 2013 at 5:38 PM, Adrien Grand <jpountz@gmail.com> wrote:

> On Mon, Apr 1, 2013 at 1:56 PM, Arun Kumar K <arunk786@gmail.com> wrote:
> > Hi Guys,
>
> Hi,
>
> > I have been finding out the heap space requirement for indexing and
> > searching with 3.0.2 vs 4.1 (with BlockPostings Format).
> >
> > I have a 2GB index with 1 million docs with around 42 fields with 40
> fields
> > being random strings.
> >
> > I have seen that memory for search has reduced by 5X with 4.1 (with
> > BlockPostings Format) but the memory usage during indexing with 4.1 is
> > around 800MB~1.7 GB whereas for 3.0.2 it is 300~600MB.
> > But indexing time is almost same with both versions.
>
> How did you measure memory usage ? Operating systems may report high
> memory usage because MMapDirectory became the new default directory
> implementation and memory-mapped files are taken into account in the
> virtual memory of your processes. See
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
> for more information.
>
> --
> Adrien
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message