lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Lea <ian....@gmail.com>
Subject Re: Excessive mem usage with 32-bit app, on 64-bit server
Date Thu, 22 Nov 2012 14:12:25 GMT
> 1. Does memory usage go up with multiple simultaneous searches - does
> it need to load the data structures multiple times?

Lucene loads some stuff into RAM, but just once rather than for each
search.  But there will of course be memory used for each search, more
concurrent searches will use more.

> 2. Has this got anything to do with a 32-bit app on a 64-bit architecture?

No idea on that, but if so you might have thought it would be an all
or nothing problem, unrelated to concurrent searches.

> 3. Normally, after a search query is executed, when does the memory
> used by the result get free again? Is it after an idle period or when
> the JVM hits memory usage limits or what?

Up to the JVM garbage collector.

> 4. Could this be caused due to a memory leak in our code? Any "common
> mistakes" that we could check first?

Yes, it certainly could be a problem with your code.  Are you hanging
on to result lists or related structures longer than necessary?
Running a memory profiler would surely help.


There are useful for info on memory usage:
http://blog.mikemccandless.com/2010/07/lucenes-ram-usage-for-searching.html
and http://searchhub.org/2011/09/14/estimating-memory-and-storage-for-lucenesolr/


--
Ian.


On Thu, Nov 22, 2012 at 1:16 PM, Antony Joseph <antonyjoseph89@gmail.com> wrote:
>
> Hi,
>
> We have a 32-bit python app which uses pylucence and bundles the JRE with
> it.
>
> This is installed on a 64-bit Windows 2012 Server with 16GB of RAM.
> The heap settings are: -Xms=64Mb and -Xmx=1024Mb
>
> It's a web app and works fine when a user or two connect. But if there
> are a few more users, the memory usage increases and at some point
> there is an "Out of memory error".
>
> By and large, a single index searcher is used and shared amongst
> multiple queries.
>
> While checking the Net someone said that with Lucene there is a lot of
> memory usage behind the scene (it needs to load some data structures
> into memory, etc.) - the questions that we have now include:
>
> 1. Does memory usage go up with multiple simultaneous searches - does
> it need to load the data structures multiple times?
> 2. Has this got anything to do with a 32-bit app on a 64-bit architecture?
> 3. Normally, after a search query is executed, when does the memory
> used by the result get free again? Is it after an idle period or when
> the JVM hits memory usage limits or what?
> 4. Could this be caused due to a memory leak in our code? Any "common
> mistakes" that we could check first?
>
> Thanks,
>
> Antony
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message