Return-Path: 
 <turbine-jcs-user-return-470-apmail-jakarta-turbine-jcs-user-archive=jakarta.apache.org@jakarta.apache.org>
Delivered-To: apmail-jakarta-turbine-jcs-user-archive@www.apache.org
Received: (qmail 48038 invoked from network); 11 Sep 2003 16:40:29 -0000
Received: from daedalus.apache.org (HELO mail.apache.org) (208.185.179.12)
  by minotaur-2.apache.org with SMTP; 11 Sep 2003 16:40:29 -0000
Received: (qmail 99031 invoked by uid 500); 11 Sep 2003 16:39:56 -0000
Delivered-To: apmail-jakarta-turbine-jcs-user-archive@jakarta.apache.org
Received: (qmail 98990 invoked by uid 500); 11 Sep 2003 16:39:55 -0000
Mailing-List: contact turbine-jcs-user-help@jakarta.apache.org; run by ezmlm
Precedence: bulk
List-Unsubscribe: <mailto:turbine-jcs-user-unsubscribe@jakarta.apache.org>
List-Subscribe: <mailto:turbine-jcs-user-subscribe@jakarta.apache.org>
List-Help: <mailto:turbine-jcs-user-help@jakarta.apache.org>
List-Post: <mailto:turbine-jcs-user@jakarta.apache.org>
List-Id: "Turbine JCS Users List" <turbine-jcs-user.jakarta.apache.org>
Reply-To: "Turbine JCS Users List" <turbine-jcs-user@jakarta.apache.org>
Delivered-To: mailing list turbine-jcs-user@jakarta.apache.org
Received: (qmail 98846 invoked from network); 11 Sep 2003 16:39:54 -0000
Received: from unknown (HELO web41402.mail.yahoo.com) (66.218.93.68)
  by daedalus.apache.org with SMTP; 11 Sep 2003 16:39:54 -0000
Message-ID: <20030911163957.74649.qmail@web41402.mail.yahoo.com>
Received: from [207.108.42.8] by web41402.mail.yahoo.com via HTTP;
 Thu, 11 Sep 2003 09:39:57 PDT
Date: Thu, 11 Sep 2003 09:39:57 -0700 (PDT)
From: Daniel Rosenbaum <drosenbaum@yahoo.com>
Subject: RE: Memory test
To: Turbine JCS Users List <turbine-jcs-user@jakarta.apache.org>
In-Reply-To: <000001c37875$5bb214d0$07c09792@realpulse.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N
X-Spam-Rating: minotaur-2.apache.org 1.6.2 0/1000/N

Setting the memory shrinker to false had no effect, memory was
still leaking.

When I switched off the aux disk cache though memory is very
stable.  This seems to indicate a problem with the disk cache.

Daniel

--- Aaron Smuts <aasmuts@wisc.edu> wrote:
> If turning off the memory shrinker doesn't work, try not using
> a disk
> cache at all.  
> 
> I want to isolate the issue.
> 
> Aaron
> 
> > -----Original Message-----
> > From: Daniel Rosenbaum [mailto:drosenbaum@yahoo.com]
> > Sent: Thursday, September 11, 2003 9:39 AM
> > To: Turbine JCS Users List
> > Subject: RE: Memory test
> > 
> > As a follow up, I let my test run for a while and after
> about 5
> > hours my JVM crashed with a java.lang.OutOfMemoryError,
> after
> > putting somewhere bewtween 3087000-3088000 items into the
> cache.
> >  My system has 512 megs of ram.  Following are the last few
> > lines of my log:
> > 
> > Sep 10 21:00:35 2003: i=3086000
> > Sep 10 21:00:35 2003: avail = 2757728
> > Sep 10 21:00:35 2003: total = 524025856
> > Sep 10 21:00:35 2003: [GC
> > Sep 10 21:00:35 2003:  509217K->507306K(511744K)
> > Sep 10 21:00:35 2003: , 0.0695549 secs]
> > Sep 10 21:00:35 2003:
> > Sep 10 21:00:36 2003: [GC
> > Sep 10 21:00:36 2003:  509354K->507447K(511744K)
> > Sep 10 21:00:36 2003: , 0.0713898 secs]
> > Sep 10 21:00:36 2003:
> > Sep 10 21:00:37 2003: [Full GC
> > Sep 10 21:17:39 2003:  509495K->507573K(511744K)
> > Sep 10 21:17:39 2003: , 1019.7951857 secs]
> > Sep 10 21:17:39 2003:
> > Sep 10 21:17:41 2003: i=3087000
> > Sep 10 21:17:44 2003: avail = 3737904
> > Sep 10 21:17:44 2003: total = 524025856
> > Sep 10 21:17:45 2003: [Full GC
> > Sep 10 21:30:07 2003:  511743K->509584K(511744K)
> > Sep 10 21:30:07 2003: , 739.8379643 secs]
> > Sep 10 21:30:07 2003:
> > Sep 10 21:30:07 2003: [Full GC
> > Sep 10 21:43:08 2003:  509584K->509584K(511744K)
> > Sep 10 21:43:08 2003: , 798.5084294 secs]
> > Sep 10 21:43:08 2003:
> > Sep 10 21:43:09 2003: java.lang.OutOfMemoryError
> > Sep 10 21:43:12 2003: 	<<no stack trace available>>
> > 
> > Notice especially the Full GCs, which took 739 and 798
> seconds
> > to complete.  You could see there was probably alot of
> virtual
> > memory swapping at that point, the system trying to stay
> alive
> > until it quit.
> > 
> > Seems there is a memory leak somewhere, doesn't it?
> > 
> > I wonder if it has anything to do with the IndexedDiskCache
> file
> > itself.  The documentation says "Depending on the key size,
> > 500,000 disk entries will probably only require about 1 MB
> of
> > memory."  This has not been my experience though with my
> test.
> > According to the logs I posted before, 1000 elements between
> > i=2000 and i=3000 required 4341272-3843352=497920, or an
> average
> > of 497.92 bytes per element. (I looked for a log entry in
> which
> > there was no GC between two log writings and the total
> memory
> > had the same number).  Later on, I had this:
> > 
> > Sep 10 18:49:52 2003: i=1531000
> > Sep 10 18:49:52 2003: avail = 90172696
> > Sep 10 18:49:52 2003: total = 382328832
> > Sep 10 18:49:53 2003: [GC
> > Sep 10 18:49:53 2003:  286864K->284953K(373368K)
> > Sep 10 18:49:53 2003: , 0.0060882 secs]
> > Sep 10 18:49:53 2003:
> > Sep 10 18:49:54 2003: [GC
> > Sep 10 18:49:54 2003:  287001K->285095K(373368K)
> > Sep 10 18:49:54 2003: , 0.0062273 secs]
> > Sep 10 18:49:54 2003:
> > Sep 10 18:49:54 2003: i=1532000
> > Sep 10 18:49:54 2003: avail = 89255520
> > Sep 10 18:49:54 2003: total = 382328832
> > 
> > So between i=1531000 and i=1532000, there was
> > 90172696-89255520=917176, or an average of 917.176 bytes per
> > item!  Grew almost 2x!  This is even with two GCs between
> them.
> > 
> > The bottom line is, something is definately eating up
> memory.
> > 
> > Daniel
> > 
> > 
> > --- Daniel Rosenbaum <drosenbaum@yahoo.com> wrote:
> > > Thanks Wayne and Amy for your help.
> > >
> > > Firstly with the -verbose:gc option, I modified my code as
> > > follows:
> > >
> > > for(int i=0; i < 200000000; i++) {
> > >   Vector vec = new Vector();
> > >   testCache.put(key+i, vec);
> > >   if (i%1000==0) {
> > >     System.out.println("i=" + i);
> > >     System.out.println("avail = " +
> > > Runtime.getRuntime().freeMemory());
> > >     System.out.println("total = " +
> > > Runtime.getRuntime().totalMemory());
> > >   }
> > > }
> > >
> > > so info would be dumped out every 1000 elements.  Partial
> > > results are:
> > >
> > > Sep 10 10:22:46 2003: i=0
> > > Sep 10 10:22:46 2003: avail = 500208
> > > Sep 10 10:22:46 2003: total = 7782400
> > > Sep 10 10:22:46 2003: [GC
> > > Sep 10 10:22:46 2003:  7431K->7237K(7856K)
> > > Sep 10 10:22:46 2003: , 0.0086215 secs]
> > > Sep 10 10:22:46 2003:
> > > Sep 10 10:22:46 2003: i=1000
> > > Sep 10 10:22:46 2003: avail = 443920
> > > Sep 10 10:22:46 2003: total = 8044544
> > > Sep 10 10:22:46 2003: [Full GC
> > > Sep 10 10:22:47 2003:  7749K->5441K(9840K)
> > > Sep 10 10:22:47 2003: , 0.1197119 secs]
> > > Sep 10 10:22:47 2003:
> > > Sep 10 10:22:47 2003: i=2000
> > > Sep 10 10:22:47 2003: avail = 4341272
> > > Sep 10 10:22:47 2003: total = 10076160
> > > Sep 10 10:22:47 2003: i=3000
> > > Sep 10 10:22:47 2003: avail = 3843352
> > > Sep 10 10:22:47 2003: total = 10076160
> > > Sep 10 10:22:47 2003: [GC
> > > Sep 10 10:22:47 2003:  6145K->5949K(9840K)
> > > Sep 10 10:22:47 2003: , 0.0081893 secs]
> > > Sep 10 10:22:47 2003:
> > > Sep 10 10:22:47 2003: i=4000
> > > Sep 10 10:22:47 2003: avail = 3570144
> > > Sep 10 10:22:47 2003: total = 10076160
> > > Sep 10 10:22:47 2003: [GC
> > > Sep 10 10:22:47 2003:  6653K->6451K(9840K)
> > > Sep 10 10:22:47 2003: , 0.0078136 secs]
> > > Sep 10 10:22:47 2003:
> > > Sep 10 10:22:47 2003: i=5000
> > > Sep 10 10:22:47 2003: avail = 3255256
> > > Sep 10 10:22:47 2003: total = 10076160
> > > Sep 10 10:22:47 2003: i=6000
> > > Sep 10 10:22:47 2003: avail = 2781920
> > > Sep 10 10:22:47 2003: total = 10076160
> > > Sep 10 10:22:47 2003: [GC
> > > Sep 10 10:22:47 2003:  7155K->6964K(9840K)
> > > Sep 10 10:22:47 2003: , 0.0088930 secs]
> > > Sep 10 10:22:47 2003:
> > > Sep 10 10:22:47 2003: i=7000
> > > Sep 10 10:22:47 2003: avail = 2503968
> > > Sep 10 10:22:47 2003: total = 10076160
> > > Sep 10 10:22:47 2003: [GC
> > > Sep 10 10:22:47 2003:  7668K->7465K(9840K)
> > > Sep 10 10:22:47 2003: , 0.0070875 secs]
> > >
> > > So we see that the GC messages are appearing.  Yet as you
> > > could
> > > see memory keeps going down.
> > >
> > > Afterward I put in a 10ms delay between puts:
> > >
> > > for(int i=0; i < 200000000; i++) {
> > >   Vector vec = new Vector();
> > >   testCache.put(key+i, vec);
> > >   if (i%1000==0) {
> > >     System.out.println("i=" + i);
> > >     System.out.println("avail = " +
> > > Runtime.getRuntime().freeMemory());
> > >     System.out.println("total = " +
> > > Runtime.getRuntime().totalMemory());
> > >   }
> > >   Thread.sleep(10);
> > > }
> > >
> > > Now it is going much slower but memory is still noticably
> > > still
> > > going down.
> > >
> > > Any other ideas?
> > >
> > > Thanks,
> > > Daniel
> > >
> > >
> > >
> > >
> > >
> > >
> > > --- "Young, Wayne" <WYoung@metasolv.com> wrote:
> > > > Do you know if garbage collections are happening? Try
> > > running
> > > > with the
> > > > -verbose:gc option. It could be that the objects ARE
> being
> > > > dereferenced, but
> > > > just aren't being collected before your loop ends.
> > > >
> > > > Wayne
> > > >
> > > > -----Original Message-----
> > > > From: Daniel Rosenbaum [mailto:drosenbaum@yahoo.com]
> > > > Sent: Tuesday, September 09, 2003 8:26 PM
> > > > To: turbine-jcs-user@jakarta.apache.org
> > > > Subject: Memory test
> > > >
> > > >
> > > > Hi all,
> > > >
> > > > I am testing JCS to see how it handles memory.  The
> > > following
> > > > is
> > > > a snippet of my code and config file:
> > > >
> > > > for(int i=0; i < 200000000; i++) {
> > > >   Vector vec = new Vector();
> > > >   testCache.put("testkey"+i, vec);
> > > > }
> > > >
> > > > # DEFAULT CACHE REGION (memory cache)
> > > > jcs.default=DC
> > > >
> > >
> >
>
jcs.default.cacheattributes=org.apache.jcs.engine.CompositeCacheAttribut
> es
> > > > jcs.default.cacheattributes.MaxObjects=2000
> > > >
> > >
> >
>
jcs.default.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memory
> .l
> > ru
> > > > .LRUMemoryCache
> > > > jcs.default.cacheattributes.UseMemoryShrinker=true
> > > > jcs.default.cacheattributes.ShrinkerIntervalSeconds=60
> > > >
> > >
> >
>
jcs.default.elementattributes=org.apache.jcs.engine.ElementAttributes
> > > > jcs.default.elementattributes.IsEternal=false
> > > > jcs.default.elementattributes.MaxLifeSeconds=43200
> > > > jcs.default.elementattributes.IsSpool=true
> > > > jcs.default.elementattributes.IsRemote=false
> > > > jcs.default.elementattributes.IsLateral=false
> > > >
> > > > #Auxiliary CACHE (disk cache)
> > > >
> > >
> >
>
jcs.auxiliary.DC=org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCacheF
> ac
> > to
> > > > ry
> > > >
> > >
> >
>
jcs.auxiliary.DC.attributes=org.apache.jcs.auxiliary.disk.indexed.Indexe
> dD
> > is
> > > > kCacheAttributes
> > > > jcs.auxiliary.DC.attributes.DiskPath=jcscache
> > > >
> > > > jcs.region.testCache=DC
> > > >
> > >
> >
>
jcs.region.testCache.cacheattributes=org.apache.jcs.engine.CompositeCach
> eA
> > tt
> > > > ributes
> > > > jcs.region.testCache.cacheattributes.MaxObjects=50
> > > >
> > >
> >
>
jcs.region.testCache.cacheattributes.MemoryCacheName=org.apache.jcs.engi
> ne
> > .m
> > > > emory.lru.LRUMemoryCache
> > > >
> jcs.region.testCache.cacheattributes.UseMemoryShrinker=true
> > > >
> > >
> >
>
jcs.region.testCache.cacheattributes.ShrinkerIntervalSeconds=60
> > > >
> > >
> >
>
jcs.region.testCache.elementattributes=org.apache.jcs.engine.ElementAttr
> ib
> > ut
> > > > es
> > > > jcs.region.testCache.elementattributes.IsEternal=false
> > > >
> jcs.region.testCache.elementattributes.MaxLifeSeconds=43200
> > > > jcs.region.testCache.elementattributes.IdleTime=43200
> > > > jcs.region.testCache.elementattributes.IsSpool=true
> > > > jcs.region.testCache.elementattributes.IsRemote=false
> > > > jcs.region.testCache.elementattributes.IsLateral=false
> > > >
> > > > When I run this code the memory on my system goes down
> very
> > > > quickly.  I expected that after the first 50 elements it
> > > would
> > > > start putting items on disk and removing them from
> memory
> > > but
> > > > this does not seem to be happening.  I see the file on
> disk
> > > > getting increasingly bigger but memory still keeps
> going,
> > > > leading me to believe that the elements are not getting
> > > > garbage
> > > > collected or was never freed.  Am I doing something
> > > > incorrectly?
> > > >
> > > > Thanks,
> > > > Daniel
> > > >
> > > >
> > > > __________________________________
> > > > Do you Yahoo!?
> > > > Yahoo! SiteBuilder - Free, easy-to-use web site design
> > > > software
> > > > http://sitebuilder.yahoo.com
> > > >
> > > >
> > >
> >
>
---------------------------------------------------------------------
> > > > To unsubscribe, e-mail:
> > > > turbine-jcs-user-unsubscribe@jakarta.apache.org
> > > > For additional commands, e-mail:
> > > > turbine-jcs-user-help@jakarta.apache.org
> > > >
> > >
> > >
> > > __________________________________
> > > Do you Yahoo!?
> > > Yahoo! SiteBuilder - Free, easy-to-use web site design
> > > software
> > > http://sitebuilder.yahoo.com
> > >
> > >
> >
>
---------------------------------------------------------------------
> > > To unsubscribe, e-mail:
> > > turbine-jcs-user-unsubscribe@jakarta.apache.org
> > > For additional commands, e-mail:
> > > turbine-jcs-user-help@jakarta.apache.org
> > >
> > 
> > 
> > __________________________________
> > Do you Yahoo!?
> > Yahoo! SiteBuilder - Free, easy-to-use web site design
> software
> > http://sitebuilder.yahoo.com
> > 
> >
>
---------------------------------------------------------------------
> > To unsubscribe, e-mail:
> turbine-jcs-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> turbine-jcs-user-help@jakarta.apache.org
> 
> 
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> turbine-jcs-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
> turbine-jcs-user-help@jakarta.apache.org
> 


__________________________________
Do you Yahoo!?
Yahoo! SiteBuilder - Free, easy-to-use web site design software
http://sitebuilder.yahoo.com

---------------------------------------------------------------------
To unsubscribe, e-mail: turbine-jcs-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: turbine-jcs-user-help@jakarta.apache.org