jakarta-jcs-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Parvathi Rajaraman <Parvathi.Rajara...@metcash.com>
Subject RE: Out of memory problem
Date Mon, 10 May 2004 02:26:44 GMT
Hi Aaron,

I am just running a local weblogic instance for my development purposes. I
have started working on this new project which requires caching of data.
Hence I wrote a wrapper that sort of delegates all the calls to the actual
JCS implementation. I just copied whatever was available in the sample to
start with.

My version of implementation looks like this

	/**
	 * Static variable to hold Log reference
	 */
	
	private Log log = LogFactory.getLog(CacheServiceDefaultImpl.class);
	
	/**
	 * This attribute needs to be static the reason being all instances
of PE from the pool
	 * have to share the same concrete cache 
	 */
	
    private static CacheStatisticsImpl stats = new CacheStatisticsImpl();
    
    /**
     * Declare a static synchronized HashMap that will be the concrete cache
and will be shared globally
     */

	private static Map cacheInstances = Collections.synchronizedMap(new
HashMap());

    
    public void cacheObject(String domain, Object key, Object value) {
    	try{
    		JCS cacheInstance = getDomain(domain);
			cacheInstance.put(key,value);	
    	}catch(CacheException e){
			String errorString = "Caught CacheException while
trying to cache an object for domain "+domain;
			log.error(errorString, e);
			throw new SystemException(errorString, e);
    	}
    }

    public Object lookup(String domain, Object key) {
        stats.incrementHitCount();
        Object cacheValue = null;
		JCS cacheInstance = getDomain(domain);
		cacheValue=cacheInstance.get(key);
		if(cacheValue == null)
			stats.incrementFailCount();
		return 	cacheValue;
    }

    public void remove(String domain, Object key) {
		try{
			JCS cacheInstance = getDomain(domain);
			cacheInstance.remove(key);	
		}catch(CacheException e){
			String errorString = "Caught CacheException while
trying to remove an object for domain "+domain;
			log.error(errorString, e);
			throw new SystemException(errorString, e);
		}
    }

    public void clearDomain(String domain) {
		try{
			JCS cacheInstance = getDomain(domain);
			cacheInstance.remove();	
		}catch(CacheException e){
			String errorString = "Caught CacheException while
trying to clear the domain "+domain;
			log.error(errorString, e);
			throw new SystemException(errorString, e);
		}
    }

    public void saveDomainCache(String domain, ObjectOutputStream
objectOutputStream) {
		//TODO:To be implemented
    }

    public void loadDomainCache(String domain, ObjectInputStream ois) {
		//TODO:To be implemented
    }

    public CacheStatistics getCacheStatistics() {
        return null;
    }

	private JCS getDomain(String domain){
		JCS cacheInstance = null;
		try{
			cacheInstance = (JCS) cacheInstances.get(domain);
			//If the cacheInstance in not available then create
one
			if(cacheInstance == null){
				cacheInstance = JCS.getInstance(domain);
				//Add to the JCS pool
				cacheInstances.put(domain,cacheInstance);
			}
		}catch(CacheException e){
			String errorString = "Caught CacheException while
trying to retrieve cached for domain "+domain;
			log.error(errorString, e);
			throw new SystemException(errorString, e);
		}
		return cacheInstance;
	}

and the test case which actually tests this looks like this

	public void testCacheObject() throws Exception{
		PricingEngine pe = null;
		try {
			pe =
PricingEngineFactory.getInstance().getPricingEngine();
			assertNotNull("Pricing engine instance should not be
null", pe);

			CacheService cacheService = pe.serviceCache();
			assertNotNull("Cache service should not be null",
cacheService);
			
			String key="Request1";
			String value="Response1";
			String domain =REQUESTDOMAIN;
			String storedValue=null;
						
			cacheService.cacheObject(domain,key,value);
			storedValue = (String)
cacheService.lookup(domain,key);
			
			assertNotNull("The object was not
cached",storedValue);
			assertEquals("The stored value is not retrieved for
the key",true,value.equals(storedValue));
			
		} finally {
			if (pe != null) {
				pe.close();
			}
		}
	}

Thats all I m doing at this stage.

Hope this helps you to know if I have done something wrong

Thanks
Parvathi

-----Original Message-----
From: Aaron Smuts [mailto:aasmuts@wisc.edu]
Sent: Monday, 10 May 2004 12:22 PM
To: 'Turbine JCS Users List'
Subject: RE: Out of memory problem


That's the entire cache.ccf?

Half a gig isn't much ram to run weblogic in!

I'd like to know more about how you are caching.


Aaron


> -----Original Message-----
> From: Parvathi Rajaraman [mailto:Parvathi.Rajaraman@metcash.com]
> Sent: Sunday, May 09, 2004 5:42 PM
> To: 'Turbine JCS Users List'
> Subject: RE: Out of memory problem
> 
> Hi,
> 
> The cache.ccf file looks like this
> 
> ********************
> # JCS Basic Config, just a simple memory only cache
> 
> jcs.default=
>
jcs.default.cacheattributes=org.apache.jcs.engine.CompositeCacheAttribut
es
> jcs.default.cacheattributes.MaxObjects=1000
>
jcs.default.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memory
.l
> ru
> .LRUMemoryCache
> **********************
> 
> I have just written a very basic implementation as we are still in the
> intial stages of our project and trying to build the prototype.All I
did
> to
> use caching was to write a test case, that caches 2 string. My JVM
> allotted
> is 256MB to 512MB. At the moment whenever it occurs i just restart my
> weblogic server and it goes on well for a few hours.
> 
> Thanks-
> Parvathi
> 
> -----Original Message-----
> From: mpcooke3@lineone.net [mailto:mpcooke3@lineone.net]
> Sent: Monday, 10 May 2004 8:04 AM
> To: Turbine JCS Users List
> Subject: RE: Out of memory problem
> 
> 
> This sounds plausible.
> 
> Running the profiler on our production machine for 30 minutes showed
heavy
> memory spikes but no leakage when the memory shrinker was enabled
(After
> a successfull GC it always seemed to go back to a normal level). I
also
> checked the memory shrinker code manually,  I didn't dig too deep into
the
> spooling but suspected this might be causing problems simply because
the
> rest of the memory shrinker code was so simple.
> 
> I would have liked to run the profiler for a longer period but
Jprofiler
> I was using is not stable enough to leave unattended on a production
> system
> (it crashes the jvm sometimes). So I was unable to confirm if it was a
> slow
> leak or sudden spike. I have seen other libraries OutOfMemory due to
heap
> usage spikes where we previously suspected a leak, so it's not
uncommon
> (in an XML processing library).
> 
> Good luck anyway,
> Matt.
> 
> >-- Original Message --
> >Reply-To: "Turbine JCS Users List"
<turbine-jcs-user@jakarta.apache.org>
> >Date: Sun, 09 May 2004 15:15:05 -0500
> >From: Aaron Smuts <aasmuts@wisc.edu>
> >Subject: RE: Out of memory problem
> >To: 'Turbine JCS Users List' <turbine-jcs-user@jakarta.apache.org>
> >
> >
> >I've been profiling JCS for days.  I still have more work to do, but
I
> >have a few ideas.
> >
> >The shrinker does not maintain references to any objects.  However,
> >there is a potential issue involved in using the shrinker, namely, it
> >causes sudden spikes in memory usage.
> >
> >The main purpose of the shrinker is to find elements that have
exceeded
> >their maximum idle memory time and then to spool these to disk.  It
also
> >checks for expired elements in the memory cache and removes them.
> >
> >When the shrinker finds elements to spool it initiates the spooling
> >process.  This invovles putting the items in purgatory and then they
are
> >queued to be written for disk.  The disk writing can take a while if
you
> >have thousands of items.  The shrinker will try to spool every item
in
> >memory that has exceeded the maxmemory time.  Basically, this can
result
> >in thousands of items in the queue which results in a memory spike
that
> >can exceed the maximum heap size.  All my tests indicate that the
memory
> >is reclaimed.
> >
> >I will add a configuration option to set the maximum number of items
to
> >spool per run of the shrinker.  This way the size of the spike can be
> >minimized.
> >
> >There is a potential memory issue invovled in using the indexed disk
> >cache.  The disk cache stores the keys in memory.  If an expired
element
> >is pulled from the disk cache or an item is removed from the cache,
the
> >key gets removed from the set held in memory by the disk cache.
> >However, if you create millions of entries in the cache that you
never
> >use, many of these get pushed onto the disk, and they don't get
removed,
> >the in memory key size of the disk cache can grow indefinitely.
> >
> >This would most likely be the result of improper cache usage, since
you
> >shouldn't cache something that won't likely be used again.  However,
> >there is a high usage scenario that could result in massive disk key
> >memory usage.  Let's assume that you cache objects that might not get
> >used very frequently, and you set them to expire after a short
duration,
> >let's say 30 seconds.  If that item gets spooled to disk within 30
> >seconds, because either you have a small max memory size or extremely
> >high usage, and there is never a get for that item, then the key will
> >sit in disk memory forever or until a get.  If there is a get it will
be
> >found on disk, checked for expiration, and then removed, but if there
is
> >no get, it will just hang around.
> >
> >For now, the way to prevent this is to increase the memory size and
> >shrink the max life time.  Alternatively, you can disable the disk
cache
> >if most of your items are of this type.
> >
> >In the future some sort of disk cleanup should occur, or the key
sotrage
> >for the disk cache could use a LRU algorithm that would limit it
size.
> >I think this is actually the best option.  I will do something like
this
> >soon.
> >
> >I can't find any actual memory leaks, only the spike from the
shrinker
> >and the potential key bloat in the disk cache.
> >
> >I'm still hunting for more.
> >
> >Aaron
> >
> >
> >
> >
> >> -----Original Message-----
> >> From: Estefano Eduardo [mailto:eduardo.estefano@siemens.com]
> >> Sent: Friday, May 07, 2004 2:43 AM
> >> To: Turbine JCS Users List
> >> Subject: RE: Out of memory problem
> >>
> >> We also started getting out of memory errors. Same situation, only
> >after
> >> we started using the jcs library (I have latest copy from main
trunk).
> >>
> >> We did not pin point the error yet though. It did not happen
straight
> >> away, only after the server was running for a few days. The
solution
> >is
> >> to restart the server at the moment. We only use memory cache and
we
> >do
> >> use schrinkers on some regions.
> >>
> >> -----Original Message-----
> >> From: Aaron Smuts [mailto:aasmuts@wisc.edu]
> >> Sent: Thursday, 06 May, 2004 19:52
> >> To: 'Turbine JCS Users List'
> >> Subject: RE: Out of memory problem
> >>
> >>
> >> Matthew informs me that the problem seems to go away with the
shrinker
> >> off.
> >>
> >> This suggests a problem with the memory shrinker.  It could be that
> >the
> >> thread is maintaining a reference to the keys, the spooling from
the
> >> shrinker is problematic, or there is a problem generating the list
of
> >> keys.
> >>
> >> I don't think any of these things is happening, so I'm sort of at a
> >loss
> >> for an explanation.
> >>
> >> I think the shrinker just sleeps for an interval.  Perhaps it would
be
> >> better to use a timer and start the shrinker and then let it die.
One
> >> concern with this is that we wouldn't want more than one shrinker
> >> running per region at any given time.  This could happen say if the
> >> number of elements was huge and the interval between shrinks was
very
> >> small.  I can solve that by passing in a lock that will prevent a
> >second
> >> shrinker from running.
> >>
> >> Hmmn.  I'll look into it again.
> >>
> >> Aaron
> >>
> >> > -----Original Message-----
> >> > From: Aaron Smuts [mailto:aasmuts@wisc.edu]
> >> > Sent: Thursday, May 06, 2004 9:43 AM
> >> > To: 'Turbine JCS Users List'
> >> > Subject: RE: Out of memory problem
> >> >
> >> > Send your cache.ccf and let me know how much memory you've
allocated
> >> for
> >> > the jvm.
> >> >
> >> > Aaron
> >> >
> >> > > -----Original Message-----
> >> > > From: Parvathi Rajaraman
[mailto:Parvathi.Rajaraman@metcash.com]
> >> > > Sent: Thursday, May 06, 2004 1:53 AM
> >> > > To: 'turbine-jcs-user@jakarta.apache.org'
> >> > > Subject: Out of memory problem
> >> > >
> >> > > Hi,
> >> > >
> >> > > I have recently started using JCS implementation for Caching in
my
> >> > > project. I have noticed that every few minutes, I keep getting
a
> >> > > out-of
> >> memory
> >> > > error
> >> > > and this has happened only after I have started using JCS. So
is
> >> this
> >> > > something to do with the way I have configured it. Can you help
me
> >> > out.
> >> > >
> >> > > Regds
> >> > > Parvathi
> >> > >
> >> > >
> >>
---------------------------------------------------------------------
> >> > > To unsubscribe, e-mail:
> >> > turbine-jcs-user-unsubscribe@jakarta.apache.org
> >> > > For additional commands, e-mail:
> >> > turbine-jcs-user-help@jakarta.apache.org
> >> >
> >> >
> >> >
> >---------------------------------------------------------------------
> >> > To unsubscribe, e-mail:
> >> turbine-jcs-user-unsubscribe@jakarta.apache.org
> >> > For additional commands, e-mail:
> >> turbine-jcs-user-help@jakarta.apache.org
> >>
> >>
> >>
---------------------------------------------------------------------
> >> To unsubscribe, e-mail:
> >turbine-jcs-user-unsubscribe@jakarta.apache.org
> >> For additional commands, e-mail:
> >> turbine-jcs-user-help@jakarta.apache.org
> >>
> >>
> >>
---------------------------------------------------------------------
> >> To unsubscribe, e-mail:
> >turbine-jcs-user-unsubscribe@jakarta.apache.org
> >> For additional commands, e-mail:
> >turbine-jcs-user-help@jakarta.apache.org
> >
> >
> >---------------------------------------------------------------------
> >To unsubscribe, e-mail:
turbine-jcs-user-unsubscribe@jakarta.apache.org
> >For additional commands, e-mail:
turbine-jcs-user-help@jakarta.apache.org
> >
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
turbine-jcs-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
turbine-jcs-user-help@jakarta.apache.org
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
turbine-jcs-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
turbine-jcs-user-help@jakarta.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org


Mime
View raw message