lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Bamford <chris.bamf...@talktalk.net>
Subject Re: Memory question
Date Wed, 16 May 2012 16:00:52 GMT

 Thanks everyone. Looks like I have lots of reading to do :-)

 

 

-----Original Message-----
From: Nader, John P <john.nader@cengage.com>
To: java-user@lucene.apache.org <java-user@lucene.apache.org>
Sent: Wed, 16 May 2012 16:27
Subject: Re: Memory question


Another good link is
http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html,
which also includes details on iCMS, which is the Incremental Mode for CMS.

On 5/15/12 6:32 PM, "Lutz Fechner" <LFechner@hubwoo.com> wrote:

>CMS is the concurrent mark sweep garbage collector. Instead of waiting
>for the memory to fill up and being collected for memory to be freed up
>again, it runs concurrently while the app threads are running. Usually
>the JVM would call a full stop the world collection. All threads would
>be on hold until GC is finished. CMS would prevent lengthy stops from
>happening in trade for constant micro stops.
>
>This usually is a good option for apps that are sensitive (from a user
>experience) from hanging during GC time.
>
>See http://docs.oracle.com/javase/6/docs/technotes/guides/vm/cms-6.html
>
>Best Regards
>
>Lutz
>
>-----Original Message-----
>From: Chris Bamford [mailto:chris.bamford@talktalk.net]
>Sent: Dienstag, 15. Mai 2012 16:38
>To: java-user@lucene.apache.org
>Subject: Re: Memory question
>
>
> Hi John,
>
>Very interesting, thanks for the detailed explanation.   It certainly
>sounds like the same symptoms!
>Can I please clarify a couple of things ?
>
>- I have googled CMS / iCMS as I wasn't familiar with those acronyms
>(apart from 'code management system' and that didn't sound right!)
>  and am I right in thinking that it is some sort of monitoring code
>pulled into your server via a jar? (I'm confused why it would have its'
>own
>  GC cycle...)
>- So are you suggesting I play with my own JVM's (Sun/Oracle) parameters
>to achieve a similar effect ?
>
>Thanks again,
>
>- Chris
>
>
> 
>
> 
>
>-----Original Message-----
>From: Nader, John P <john.nader@cengage.com>
>To: java-user@lucene.apache.org <java-user@lucene.apache.org>
>Sent: Tue, 15 May 2012 21:12
>Subject: Re: Memory question
>
>
>We've encountered this issue and came up with a fairly good approach to
>address it.
>
>We are on Lucene 3.0.2 with Java 1.6.0_29.  Our indices are about 35GB
>in
>size.  Our JVM runs at 20GB of heap, with about 12GB of steady usage.
>Our
>server has 32GB total.
>
>What would happen in our case is that the linux would page in more and
>more of the memory mapped index files into memory, forcing idle portions
>of the JVM heap to be swapped out.  This was not an issue until our CMS
>GC
>kicked in.  This would force swapping in of all JVM memory to collect
>unused references.  I/O wait would shoot up and performance would
>suffer.
>Yes, even CMS can kill performance is you are swapping.  The tell-tale
>sign was a spike in inbound swap at the start of CMS.
>
>In our case, we addressed the situation using iCMS, which is Incremental
>CMS.  This takes the mark phase (and sweep too?) and does it
>continuously
>with a configurable duty cycle.  The result was that swapping was
>smoothed
>out to be a small stead drag on the system instead of a hard spike.
>There
>was an small loss in performance, but a big gain in stability.
>
>This tuning may be an option for you.  BTW, pmap will give you statistic
>on total file size and how much is resident.  The java heap shows up in
>pmap as well on linux, so you can determine how much of that is in
>memory
>as well.
>
>John
>
>
>
>On 5/15/12 3:38 PM, "Chris Bamford" <chris.bamford@talktalk.net> wrote:
>
>>Thanks Uwe. 
>>
>>What I'd like to understand is the implications of this on a server
>which
>>opens a large number of indexes over a long period. Will this non-heap
>>memory continue to grow? Will gc be effective at spotting it and
>>releasing it via references in the heap?
>>
>> I had an instance yesterday where a server swapped itself to a
>>standstill and had to be restarted. The load average was through the
>roof
>>and I am trying to understand why. One of my recent changes is updating
>>from 2.3 to 3.6, so naturally I am keen to know the impact of the mmap
>>stuff which is now standard under the covers.
>>
>>My server caches indexsearchers and then closes them based on how full
>>the heap is getting. My worry is that if the bulk of the memory is
>being
>>allocated outside the Jvm, how can I make sensible decisions?
>>
>>Thanks for any pointers / info.
>>
>>Chris
>>
>>
>>
>>-----Original Message-----
>>From: uwe@thetaphi.de
>>To: java-user@lucene.apache.org
>>Sent: Tue, 15 May 2012 18:10
>>Subject: RE: Memory question
>>
>>
>>
>>It mmaps the files into virtual memory if it runs on a 64 bit JVM.
>Because
>>of that you see the mmapped CFS files. This is outside Java Heap and is
>>all
>>*virtual* no RAM is explicitely occupied except the O/S cache.
>>
>>-----
>>Uwe Schindler
>>H.-H.-Meier-Allee 63, D-28213 Bremen
>>http://www.thetaphi.de
>>eMail: uwe@thetaphi.de
>>
>>> -----Original Message-----
>>> From: Chris Bamford [mailto:chris.bamford@talktalk.net]
>>> Sent: Tuesday, May 15, 2012 4:47 PM
>>> To: java-user@lucene.apache.org
>>> Subject: Memory question
>>> 
>>> Hi
>>> 
>>> Can anyone tell me what happens to the memory when Lucene opens an
>>>index?
>>> Is it loaded into the JVM's heap or is it mapped into virtual memory
>>outside of
>>> it?
>>> I am running on Linux and if I use pmap on the PID of my JVM, I can
>see
>>lots of
>>> entries for index cfs files.
>>> 
>>> Does this mean that indexes are mapped into non-heap memory?  If so,
>how
>>> can I monitor the space my process is using if I cache open
>>IndexSearchers?
>>> 
>>> The details are:
>>> 
>>> Sun 64-bit JVM on Linux.
>>> Lucene 3.6 running in 2.3 compatibility mode (as we are in the in the
>>process of
>>> a migration to 3.6)
>>> 
>>> Thanks,
>>> 
>>> - Chris
>>
>>---------------------------------------------------------------------
>>To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> 
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


 

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message