jakarta-jcs-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Smuts <aasm...@wisc.edu>
Subject RE: Out of memory problem
Date Mon, 10 May 2004 02:54:24 GMT
Your wrapper is sort of unnecessary, but if you want it, you don't need
to store the regions in your own map.  JCS does it for you.

Just call JCS.getInstance( String region ) 

JCS should also manage the stats.  I need to make sure the stats
functionality is still accessible.

The saveDoman and loadDomain methods are not necessary.  The JCS disk
cache will do this on startup.  If you just want to dump some stuff into
it, then you could do that, but the dumpmap methods have been removed
from the access classes.  You'd have to get regions via the
CacheManager, but I don't recommend this.  

You've also added a layer of synonyms that can make things confusing.
You call regions domains and get lookup.  Indirection with synonyms can
make the code hard to maintain.  

Cheers,

Aaron

> -----Original Message-----
> From: Parvathi Rajaraman [mailto:Parvathi.Rajaraman@metcash.com]
> Sent: Sunday, May 09, 2004 9:27 PM
> To: 'Turbine JCS Users List'
> Subject: RE: Out of memory problem
> 
> Hi Aaron,
> 
> I am just running a local weblogic instance for my development
purposes. I
> have started working on this new project which requires caching of
data.
> Hence I wrote a wrapper that sort of delegates all the calls to the
actual
> JCS implementation. I just copied whatever was available in the sample
to
> start with.
> 
> My version of implementation looks like this
> 
> 	/**
> 	 * Static variable to hold Log reference
> 	 */
> 
> 	private Log log =
LogFactory.getLog(CacheServiceDefaultImpl.class);
> 
> 	/**
> 	 * This attribute needs to be static the reason being all
instances
> of PE from the pool
> 	 * have to share the same concrete cache
> 	 */
> 
>     private static CacheStatisticsImpl stats = new
CacheStatisticsImpl();
> 
>     /**
>      * Declare a static synchronized HashMap that will be the concrete
> cache
> and will be shared globally
>      */
> 
> 	private static Map cacheInstances =
Collections.synchronizedMap(new
> HashMap());
> 
> 
>     public void cacheObject(String domain, Object key, Object value) {
>     	try{
>     		JCS cacheInstance = getDomain(domain);
> 			cacheInstance.put(key,value);
>     	}catch(CacheException e){
> 			String errorString = "Caught CacheException
while
> trying to cache an object for domain "+domain;
> 			log.error(errorString, e);
> 			throw new SystemException(errorString, e);
>     	}
>     }
> 
>     public Object lookup(String domain, Object key) {
>         stats.incrementHitCount();
>         Object cacheValue = null;
> 		JCS cacheInstance = getDomain(domain);
> 		cacheValue=cacheInstance.get(key);
> 		if(cacheValue == null)
> 			stats.incrementFailCount();
> 		return 	cacheValue;
>     }
> 
>     public void remove(String domain, Object key) {
> 		try{
> 			JCS cacheInstance = getDomain(domain);
> 			cacheInstance.remove(key);
> 		}catch(CacheException e){
> 			String errorString = "Caught CacheException
while
> trying to remove an object for domain "+domain;
> 			log.error(errorString, e);
> 			throw new SystemException(errorString, e);
> 		}
>     }
> 
>     public void clearDomain(String domain) {
> 		try{
> 			JCS cacheInstance = getDomain(domain);
> 			cacheInstance.remove();
> 		}catch(CacheException e){
> 			String errorString = "Caught CacheException
while
> trying to clear the domain "+domain;
> 			log.error(errorString, e);
> 			throw new SystemException(errorString, e);
> 		}
>     }
> 
>     public void saveDomainCache(String domain, ObjectOutputStream
> objectOutputStream) {
> 		//TODO:To be implemented
>     }
> 
>     public void loadDomainCache(String domain, ObjectInputStream ois)
{
> 		//TODO:To be implemented
>     }
> 
>     public CacheStatistics getCacheStatistics() {
>         return null;
>     }
> 
> 	private JCS getDomain(String domain){
> 		JCS cacheInstance = null;
> 		try{
> 			cacheInstance = (JCS)
cacheInstances.get(domain);
> 			//If the cacheInstance in not available then
create
> one
> 			if(cacheInstance == null){
> 				cacheInstance = JCS.getInstance(domain);
> 				//Add to the JCS pool
>
cacheInstances.put(domain,cacheInstance);
> 			}
> 		}catch(CacheException e){
> 			String errorString = "Caught CacheException
while
> trying to retrieve cached for domain "+domain;
> 			log.error(errorString, e);
> 			throw new SystemException(errorString, e);
> 		}
> 		return cacheInstance;
> 	}
> 
> and the test case which actually tests this looks like this
> 
> 	public void testCacheObject() throws Exception{
> 		PricingEngine pe = null;
> 		try {
> 			pe =
> PricingEngineFactory.getInstance().getPricingEngine();
> 			assertNotNull("Pricing engine instance should
not be
> null", pe);
> 
> 			CacheService cacheService = pe.serviceCache();
> 			assertNotNull("Cache service should not be
null",
> cacheService);
> 
> 			String key="Request1";
> 			String value="Response1";
> 			String domain =REQUESTDOMAIN;
> 			String storedValue=null;
> 
> 			cacheService.cacheObject(domain,key,value);
> 			storedValue = (String)
> cacheService.lookup(domain,key);
> 
> 			assertNotNull("The object was not
> cached",storedValue);
> 			assertEquals("The stored value is not retrieved
for
> the key",true,value.equals(storedValue));
> 
> 		} finally {
> 			if (pe != null) {
> 				pe.close();
> 			}
> 		}
> 	}
> 
> Thats all I m doing at this stage.
> 
> Hope this helps you to know if I have done something wrong
> 
> Thanks
> Parvathi
> 
> -----Original Message-----
> From: Aaron Smuts [mailto:aasmuts@wisc.edu]
> Sent: Monday, 10 May 2004 12:22 PM
> To: 'Turbine JCS Users List'
> Subject: RE: Out of memory problem
> 
> 
> That's the entire cache.ccf?
> 
> Half a gig isn't much ram to run weblogic in!
> 
> I'd like to know more about how you are caching.
> 
> 
> Aaron
> 
> 
> > -----Original Message-----
> > From: Parvathi Rajaraman [mailto:Parvathi.Rajaraman@metcash.com]
> > Sent: Sunday, May 09, 2004 5:42 PM
> > To: 'Turbine JCS Users List'
> > Subject: RE: Out of memory problem
> >
> > Hi,
> >
> > The cache.ccf file looks like this
> >
> > ********************
> > # JCS Basic Config, just a simple memory only cache
> >
> > jcs.default=
> >
>
jcs.default.cacheattributes=org.apache.jcs.engine.CompositeCacheAttribut
> es
> > jcs.default.cacheattributes.MaxObjects=1000
> >
>
jcs.default.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memory
> .l
> > ru
> > .LRUMemoryCache
> > **********************
> >
> > I have just written a very basic implementation as we are still in
the
> > intial stages of our project and trying to build the prototype.All I
> did
> > to
> > use caching was to write a test case, that caches 2 string. My JVM
> > allotted
> > is 256MB to 512MB. At the moment whenever it occurs i just restart
my
> > weblogic server and it goes on well for a few hours.
> >
> > Thanks-
> > Parvathi
> >
> > -----Original Message-----
> > From: mpcooke3@lineone.net [mailto:mpcooke3@lineone.net]
> > Sent: Monday, 10 May 2004 8:04 AM
> > To: Turbine JCS Users List
> > Subject: RE: Out of memory problem
> >
> >
> > This sounds plausible.
> >
> > Running the profiler on our production machine for 30 minutes showed
> heavy
> > memory spikes but no leakage when the memory shrinker was enabled
> (After
> > a successfull GC it always seemed to go back to a normal level). I
> also
> > checked the memory shrinker code manually,  I didn't dig too deep
into
> the
> > spooling but suspected this might be causing problems simply because
> the
> > rest of the memory shrinker code was so simple.
> >
> > I would have liked to run the profiler for a longer period but
> Jprofiler
> > I was using is not stable enough to leave unattended on a production
> > system
> > (it crashes the jvm sometimes). So I was unable to confirm if it was
a
> > slow
> > leak or sudden spike. I have seen other libraries OutOfMemory due to
> heap
> > usage spikes where we previously suspected a leak, so it's not
> uncommon
> > (in an XML processing library).
> >
> > Good luck anyway,
> > Matt.
> >
> > >-- Original Message --
> > >Reply-To: "Turbine JCS Users List"
> <turbine-jcs-user@jakarta.apache.org>
> > >Date: Sun, 09 May 2004 15:15:05 -0500
> > >From: Aaron Smuts <aasmuts@wisc.edu>
> > >Subject: RE: Out of memory problem
> > >To: 'Turbine JCS Users List' <turbine-jcs-user@jakarta.apache.org>
> > >
> > >
> > >I've been profiling JCS for days.  I still have more work to do,
but
> I
> > >have a few ideas.
> > >
> > >The shrinker does not maintain references to any objects.  However,
> > >there is a potential issue involved in using the shrinker, namely,
it
> > >causes sudden spikes in memory usage.
> > >
> > >The main purpose of the shrinker is to find elements that have
> exceeded
> > >their maximum idle memory time and then to spool these to disk.  It
> also
> > >checks for expired elements in the memory cache and removes them.
> > >
> > >When the shrinker finds elements to spool it initiates the spooling
> > >process.  This invovles putting the items in purgatory and then
they
> are
> > >queued to be written for disk.  The disk writing can take a while
if
> you
> > >have thousands of items.  The shrinker will try to spool every item
> in
> > >memory that has exceeded the maxmemory time.  Basically, this can
> result
> > >in thousands of items in the queue which results in a memory spike
> that
> > >can exceed the maximum heap size.  All my tests indicate that the
> memory
> > >is reclaimed.
> > >
> > >I will add a configuration option to set the maximum number of
items
> to
> > >spool per run of the shrinker.  This way the size of the spike can
be
> > >minimized.
> > >
> > >There is a potential memory issue invovled in using the indexed
disk
> > >cache.  The disk cache stores the keys in memory.  If an expired
> element
> > >is pulled from the disk cache or an item is removed from the cache,
> the
> > >key gets removed from the set held in memory by the disk cache.
> > >However, if you create millions of entries in the cache that you
> never
> > >use, many of these get pushed onto the disk, and they don't get
> removed,
> > >the in memory key size of the disk cache can grow indefinitely.
> > >
> > >This would most likely be the result of improper cache usage, since
> you
> > >shouldn't cache something that won't likely be used again.
However,
> > >there is a high usage scenario that could result in massive disk
key
> > >memory usage.  Let's assume that you cache objects that might not
get
> > >used very frequently, and you set them to expire after a short
> duration,
> > >let's say 30 seconds.  If that item gets spooled to disk within 30
> > >seconds, because either you have a small max memory size or
extremely
> > >high usage, and there is never a get for that item, then the key
will
> > >sit in disk memory forever or until a get.  If there is a get it
will
> be
> > >found on disk, checked for expiration, and then removed, but if
there
> is
> > >no get, it will just hang around.
> > >
> > >For now, the way to prevent this is to increase the memory size and
> > >shrink the max life time.  Alternatively, you can disable the disk
> cache
> > >if most of your items are of this type.
> > >
> > >In the future some sort of disk cleanup should occur, or the key
> sotrage
> > >for the disk cache could use a LRU algorithm that would limit it
> size.
> > >I think this is actually the best option.  I will do something like
> this
> > >soon.
> > >
> > >I can't find any actual memory leaks, only the spike from the
> shrinker
> > >and the potential key bloat in the disk cache.
> > >
> > >I'm still hunting for more.
> > >
> > >Aaron
> > >
> > >
> > >
> > >
> > >> -----Original Message-----
> > >> From: Estefano Eduardo [mailto:eduardo.estefano@siemens.com]
> > >> Sent: Friday, May 07, 2004 2:43 AM
> > >> To: Turbine JCS Users List
> > >> Subject: RE: Out of memory problem
> > >>
> > >> We also started getting out of memory errors. Same situation,
only
> > >after
> > >> we started using the jcs library (I have latest copy from main
> trunk).
> > >>
> > >> We did not pin point the error yet though. It did not happen
> straight
> > >> away, only after the server was running for a few days. The
> solution
> > >is
> > >> to restart the server at the moment. We only use memory cache and
> we
> > >do
> > >> use schrinkers on some regions.
> > >>
> > >> -----Original Message-----
> > >> From: Aaron Smuts [mailto:aasmuts@wisc.edu]
> > >> Sent: Thursday, 06 May, 2004 19:52
> > >> To: 'Turbine JCS Users List'
> > >> Subject: RE: Out of memory problem
> > >>
> > >>
> > >> Matthew informs me that the problem seems to go away with the
> shrinker
> > >> off.
> > >>
> > >> This suggests a problem with the memory shrinker.  It could be
that
> > >the
> > >> thread is maintaining a reference to the keys, the spooling from
> the
> > >> shrinker is problematic, or there is a problem generating the
list
> of
> > >> keys.
> > >>
> > >> I don't think any of these things is happening, so I'm sort of at
a
> > >loss
> > >> for an explanation.
> > >>
> > >> I think the shrinker just sleeps for an interval.  Perhaps it
would
> be
> > >> better to use a timer and start the shrinker and then let it die.
> One
> > >> concern with this is that we wouldn't want more than one shrinker
> > >> running per region at any given time.  This could happen say if
the
> > >> number of elements was huge and the interval between shrinks was
> very
> > >> small.  I can solve that by passing in a lock that will prevent a
> > >second
> > >> shrinker from running.
> > >>
> > >> Hmmn.  I'll look into it again.
> > >>
> > >> Aaron
> > >>
> > >> > -----Original Message-----
> > >> > From: Aaron Smuts [mailto:aasmuts@wisc.edu]
> > >> > Sent: Thursday, May 06, 2004 9:43 AM
> > >> > To: 'Turbine JCS Users List'
> > >> > Subject: RE: Out of memory problem
> > >> >
> > >> > Send your cache.ccf and let me know how much memory you've
> allocated
> > >> for
> > >> > the jvm.
> > >> >
> > >> > Aaron
> > >> >
> > >> > > -----Original Message-----
> > >> > > From: Parvathi Rajaraman
> [mailto:Parvathi.Rajaraman@metcash.com]
> > >> > > Sent: Thursday, May 06, 2004 1:53 AM
> > >> > > To: 'turbine-jcs-user@jakarta.apache.org'
> > >> > > Subject: Out of memory problem
> > >> > >
> > >> > > Hi,
> > >> > >
> > >> > > I have recently started using JCS implementation for Caching
in
> my
> > >> > > project. I have noticed that every few minutes, I keep
getting
> a
> > >> > > out-of
> > >> memory
> > >> > > error
> > >> > > and this has happened only after I have started using JCS. So
> is
> > >> this
> > >> > > something to do with the way I have configured it. Can you
help
> me
> > >> > out.
> > >> > >
> > >> > > Regds
> > >> > > Parvathi
> > >> > >
> > >> > >
> > >>
> ---------------------------------------------------------------------
> > >> > > To unsubscribe, e-mail:
> > >> > turbine-jcs-user-unsubscribe@jakarta.apache.org
> > >> > > For additional commands, e-mail:
> > >> > turbine-jcs-user-help@jakarta.apache.org
> > >> >
> > >> >
> > >> >
> >
>---------------------------------------------------------------------
> > >> > To unsubscribe, e-mail:
> > >> turbine-jcs-user-unsubscribe@jakarta.apache.org
> > >> > For additional commands, e-mail:
> > >> turbine-jcs-user-help@jakarta.apache.org
> > >>
> > >>
> > >>
> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail:
> > >turbine-jcs-user-unsubscribe@jakarta.apache.org
> > >> For additional commands, e-mail:
> > >> turbine-jcs-user-help@jakarta.apache.org
> > >>
> > >>
> > >>
> ---------------------------------------------------------------------
> > >> To unsubscribe, e-mail:
> > >turbine-jcs-user-unsubscribe@jakarta.apache.org
> > >> For additional commands, e-mail:
> > >turbine-jcs-user-help@jakarta.apache.org
> > >
> > >
> >
>---------------------------------------------------------------------
> > >To unsubscribe, e-mail:
> turbine-jcs-user-unsubscribe@jakarta.apache.org
> > >For additional commands, e-mail:
> turbine-jcs-user-help@jakarta.apache.org
> > >
> >
> >
> >
> >
---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> turbine-jcs-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> turbine-jcs-user-help@jakarta.apache.org
> >
> >
---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> turbine-jcs-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> turbine-jcs-user-help@jakarta.apache.org
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
turbine-jcs-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
turbine-jcs-user-help@jakarta.apache.org
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
turbine-jcs-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
turbine-jcs-user-help@jakarta.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org


Mime
View raw message