jakarta-jcs-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mpcoo...@lineone.net
Subject RE: Out of memory problem
Date Sun, 09 May 2004 22:03:40 GMT
This sounds plausible.

Running the profiler on our production machine for 30 minutes showed heavy
memory spikes but no leakage when the memory shrinker was enabled (After
a successfull GC it always seemed to go back to a normal level). I also
checked the memory shrinker code manually,  I didn't dig too deep into the
spooling but suspected this might be causing problems simply because the
rest of the memory shrinker code was so simple.

I would have liked to run the profiler for a longer period but Jprofiler
I was using is not stable enough to leave unattended on a production system
(it crashes the jvm sometimes). So I was unable to confirm if it was a slow
leak or sudden spike. I have seen other libraries OutOfMemory due to heap
usage spikes where we previously suspected a leak, so it's not uncommon
(in an XML processing library).

Good luck anyway,
Matt.

>-- Original Message --
>Reply-To: "Turbine JCS Users List" <turbine-jcs-user@jakarta.apache.org>
>Date: Sun, 09 May 2004 15:15:05 -0500
>From: Aaron Smuts <aasmuts@wisc.edu>
>Subject: RE: Out of memory problem
>To: 'Turbine JCS Users List' <turbine-jcs-user@jakarta.apache.org>
>
>
>I've been profiling JCS for days.  I still have more work to do, but I
>have a few ideas. 
>
>The shrinker does not maintain references to any objects.  However,
>there is a potential issue involved in using the shrinker, namely, it
>causes sudden spikes in memory usage.  
>
>The main purpose of the shrinker is to find elements that have exceeded
>their maximum idle memory time and then to spool these to disk.  It also
>checks for expired elements in the memory cache and removes them.  
>
>When the shrinker finds elements to spool it initiates the spooling
>process.  This invovles putting the items in purgatory and then they are
>queued to be written for disk.  The disk writing can take a while if you
>have thousands of items.  The shrinker will try to spool every item in
>memory that has exceeded the maxmemory time.  Basically, this can result
>in thousands of items in the queue which results in a memory spike that
>can exceed the maximum heap size.  All my tests indicate that the memory
>is reclaimed.
>
>I will add a configuration option to set the maximum number of items to
>spool per run of the shrinker.  This way the size of the spike can be
>minimized.  
>
>There is a potential memory issue invovled in using the indexed disk
>cache.  The disk cache stores the keys in memory.  If an expired element
>is pulled from the disk cache or an item is removed from the cache, the
>key gets removed from the set held in memory by the disk cache.
>However, if you create millions of entries in the cache that you never
>use, many of these get pushed onto the disk, and they don't get removed,
>the in memory key size of the disk cache can grow indefinitely.  
>
>This would most likely be the result of improper cache usage, since you
>shouldn't cache something that won't likely be used again.  However,
>there is a high usage scenario that could result in massive disk key
>memory usage.  Let's assume that you cache objects that might not get
>used very frequently, and you set them to expire after a short duration,
>let's say 30 seconds.  If that item gets spooled to disk within 30
>seconds, because either you have a small max memory size or extremely
>high usage, and there is never a get for that item, then the key will
>sit in disk memory forever or until a get.  If there is a get it will be
>found on disk, checked for expiration, and then removed, but if there is
>no get, it will just hang around.  
>
>For now, the way to prevent this is to increase the memory size and
>shrink the max life time.  Alternatively, you can disable the disk cache
>if most of your items are of this type.  
>
>In the future some sort of disk cleanup should occur, or the key sotrage
>for the disk cache could use a LRU algorithm that would limit it size.
>I think this is actually the best option.  I will do something like this
>soon.
>
>I can't find any actual memory leaks, only the spike from the shrinker
>and the potential key bloat in the disk cache.
>
>I'm still hunting for more.
>
>Aaron  
>
>
>
>
>> -----Original Message-----
>> From: Estefano Eduardo [mailto:eduardo.estefano@siemens.com]
>> Sent: Friday, May 07, 2004 2:43 AM
>> To: Turbine JCS Users List
>> Subject: RE: Out of memory problem
>> 
>> We also started getting out of memory errors. Same situation, only
>after
>> we started using the jcs library (I have latest copy from main trunk).
>> 
>> We did not pin point the error yet though. It did not happen straight
>> away, only after the server was running for a few days. The solution
>is
>> to restart the server at the moment. We only use memory cache and we
>do
>> use schrinkers on some regions.
>> 
>> -----Original Message-----
>> From: Aaron Smuts [mailto:aasmuts@wisc.edu]
>> Sent: Thursday, 06 May, 2004 19:52
>> To: 'Turbine JCS Users List'
>> Subject: RE: Out of memory problem
>> 
>> 
>> Matthew informs me that the problem seems to go away with the shrinker
>> off.
>> 
>> This suggests a problem with the memory shrinker.  It could be that
>the
>> thread is maintaining a reference to the keys, the spooling from the
>> shrinker is problematic, or there is a problem generating the list of
>> keys.
>> 
>> I don't think any of these things is happening, so I'm sort of at a
>loss
>> for an explanation.
>> 
>> I think the shrinker just sleeps for an interval.  Perhaps it would be
>> better to use a timer and start the shrinker and then let it die.  One
>> concern with this is that we wouldn't want more than one shrinker
>> running per region at any given time.  This could happen say if the
>> number of elements was huge and the interval between shrinks was very
>> small.  I can solve that by passing in a lock that will prevent a
>second
>> shrinker from running.
>> 
>> Hmmn.  I'll look into it again.
>> 
>> Aaron
>> 
>> > -----Original Message-----
>> > From: Aaron Smuts [mailto:aasmuts@wisc.edu]
>> > Sent: Thursday, May 06, 2004 9:43 AM
>> > To: 'Turbine JCS Users List'
>> > Subject: RE: Out of memory problem
>> >
>> > Send your cache.ccf and let me know how much memory you've allocated
>> for
>> > the jvm.
>> >
>> > Aaron
>> >
>> > > -----Original Message-----
>> > > From: Parvathi Rajaraman [mailto:Parvathi.Rajaraman@metcash.com]
>> > > Sent: Thursday, May 06, 2004 1:53 AM
>> > > To: 'turbine-jcs-user@jakarta.apache.org'
>> > > Subject: Out of memory problem
>> > >
>> > > Hi,
>> > >
>> > > I have recently started using JCS implementation for Caching in my
>> > > project. I have noticed that every few minutes, I keep getting a
>> > > out-of
>> memory
>> > > error
>> > > and this has happened only after I have started using JCS. So is
>> this
>> > > something to do with the way I have configured it. Can you help me
>> > out.
>> > >
>> > > Regds
>> > > Parvathi
>> > >
>> > >
>> ---------------------------------------------------------------------
>> > > To unsubscribe, e-mail:
>> > turbine-jcs-user-unsubscribe@jakarta.apache.org
>> > > For additional commands, e-mail:
>> > turbine-jcs-user-help@jakarta.apache.org
>> >
>> >
>> >
>---------------------------------------------------------------------
>> > To unsubscribe, e-mail:
>> turbine-jcs-user-unsubscribe@jakarta.apache.org
>> > For additional commands, e-mail:
>> turbine-jcs-user-help@jakarta.apache.org
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>turbine-jcs-user-unsubscribe@jakarta.apache.org
>> For additional commands, e-mail:
>> turbine-jcs-user-help@jakarta.apache.org
>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail:
>turbine-jcs-user-unsubscribe@jakarta.apache.org
>> For additional commands, e-mail:
>turbine-jcs-user-help@jakarta.apache.org
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
>For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org
>



---------------------------------------------------------------------
To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org


Mime
View raw message