jakarta-jcs-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Smuts <aasm...@wisc.edu>
Subject RE: Out of memory problem
Date Sun, 09 May 2004 20:15:05 GMT
I've been profiling JCS for days.  I still have more work to do, but I
have a few ideas. 

The shrinker does not maintain references to any objects.  However,
there is a potential issue involved in using the shrinker, namely, it
causes sudden spikes in memory usage.  

The main purpose of the shrinker is to find elements that have exceeded
their maximum idle memory time and then to spool these to disk.  It also
checks for expired elements in the memory cache and removes them.  

When the shrinker finds elements to spool it initiates the spooling
process.  This invovles putting the items in purgatory and then they are
queued to be written for disk.  The disk writing can take a while if you
have thousands of items.  The shrinker will try to spool every item in
memory that has exceeded the maxmemory time.  Basically, this can result
in thousands of items in the queue which results in a memory spike that
can exceed the maximum heap size.  All my tests indicate that the memory
is reclaimed.

I will add a configuration option to set the maximum number of items to
spool per run of the shrinker.  This way the size of the spike can be
minimized.  

There is a potential memory issue invovled in using the indexed disk
cache.  The disk cache stores the keys in memory.  If an expired element
is pulled from the disk cache or an item is removed from the cache, the
key gets removed from the set held in memory by the disk cache.
However, if you create millions of entries in the cache that you never
use, many of these get pushed onto the disk, and they don't get removed,
the in memory key size of the disk cache can grow indefinitely.  

This would most likely be the result of improper cache usage, since you
shouldn't cache something that won't likely be used again.  However,
there is a high usage scenario that could result in massive disk key
memory usage.  Let's assume that you cache objects that might not get
used very frequently, and you set them to expire after a short duration,
let's say 30 seconds.  If that item gets spooled to disk within 30
seconds, because either you have a small max memory size or extremely
high usage, and there is never a get for that item, then the key will
sit in disk memory forever or until a get.  If there is a get it will be
found on disk, checked for expiration, and then removed, but if there is
no get, it will just hang around.  

For now, the way to prevent this is to increase the memory size and
shrink the max life time.  Alternatively, you can disable the disk cache
if most of your items are of this type.  

In the future some sort of disk cleanup should occur, or the key sotrage
for the disk cache could use a LRU algorithm that would limit it size.
I think this is actually the best option.  I will do something like this
soon.

I can't find any actual memory leaks, only the spike from the shrinker
and the potential key bloat in the disk cache.

I'm still hunting for more.

Aaron  




> -----Original Message-----
> From: Estefano Eduardo [mailto:eduardo.estefano@siemens.com]
> Sent: Friday, May 07, 2004 2:43 AM
> To: Turbine JCS Users List
> Subject: RE: Out of memory problem
> 
> We also started getting out of memory errors. Same situation, only
after
> we started using the jcs library (I have latest copy from main trunk).
> 
> We did not pin point the error yet though. It did not happen straight
> away, only after the server was running for a few days. The solution
is
> to restart the server at the moment. We only use memory cache and we
do
> use schrinkers on some regions.
> 
> -----Original Message-----
> From: Aaron Smuts [mailto:aasmuts@wisc.edu]
> Sent: Thursday, 06 May, 2004 19:52
> To: 'Turbine JCS Users List'
> Subject: RE: Out of memory problem
> 
> 
> Matthew informs me that the problem seems to go away with the shrinker
> off.
> 
> This suggests a problem with the memory shrinker.  It could be that
the
> thread is maintaining a reference to the keys, the spooling from the
> shrinker is problematic, or there is a problem generating the list of
> keys.
> 
> I don't think any of these things is happening, so I'm sort of at a
loss
> for an explanation.
> 
> I think the shrinker just sleeps for an interval.  Perhaps it would be
> better to use a timer and start the shrinker and then let it die.  One
> concern with this is that we wouldn't want more than one shrinker
> running per region at any given time.  This could happen say if the
> number of elements was huge and the interval between shrinks was very
> small.  I can solve that by passing in a lock that will prevent a
second
> shrinker from running.
> 
> Hmmn.  I'll look into it again.
> 
> Aaron
> 
> > -----Original Message-----
> > From: Aaron Smuts [mailto:aasmuts@wisc.edu]
> > Sent: Thursday, May 06, 2004 9:43 AM
> > To: 'Turbine JCS Users List'
> > Subject: RE: Out of memory problem
> >
> > Send your cache.ccf and let me know how much memory you've allocated
> for
> > the jvm.
> >
> > Aaron
> >
> > > -----Original Message-----
> > > From: Parvathi Rajaraman [mailto:Parvathi.Rajaraman@metcash.com]
> > > Sent: Thursday, May 06, 2004 1:53 AM
> > > To: 'turbine-jcs-user@jakarta.apache.org'
> > > Subject: Out of memory problem
> > >
> > > Hi,
> > >
> > > I have recently started using JCS implementation for Caching in my
> > > project. I have noticed that every few minutes, I keep getting a
> > > out-of
> memory
> > > error
> > > and this has happened only after I have started using JCS. So is
> this
> > > something to do with the way I have configured it. Can you help me
> > out.
> > >
> > > Regds
> > > Parvathi
> > >
> > >
> ---------------------------------------------------------------------
> > > To unsubscribe, e-mail:
> > turbine-jcs-user-unsubscribe@jakarta.apache.org
> > > For additional commands, e-mail:
> > turbine-jcs-user-help@jakarta.apache.org
> >
> >
> >
---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> turbine-jcs-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> turbine-jcs-user-help@jakarta.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
turbine-jcs-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
> turbine-jcs-user-help@jakarta.apache.org
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
turbine-jcs-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
turbine-jcs-user-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org


Mime
View raw message