cocoon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gerhard Froehlich" <>
Subject RE: Store Janitor Hangs System
Date Sun, 27 Jan 2002 16:55:16 GMT
Ok guys,
I fixed that bug more or less quick and dirty:

The loop which removes objects out of the stores
is only iterated once, now. I also removed the
call of the gc in the loop. The next Thread
iteration calls the gc next:

run() {
  if(memoryIsLow) {
    call gc

    while(memoryIsStillLow && Iteration < 1) {
     	free some objects ot of the reg. stores
      count iteration +1
  thread sleeps

Not very creative, I know. 

"Eagles may soar, but weasels don't get 
sucked into jet engines.
(Todd C. Somers)"

>-----Original Message-----
>From: Gerhard Froehlich []
>Sent: Sunday, January 27, 2002 2:30 PM
>Subject: RE: Store Janitor Hangs System
>>From: Stefano Mazzocchi []
>>Gerhard Froehlich wrote:
>>> Hi,
>>> >From: Peter Hargreaves []
>>> >
>>> >Hi Gerhard and Team,
>>> >
>>> >I'm posting in this group now I think I found a design problem with
>>> >store-janitor.
>>> Slowly but constant this Store issues are a PITA for me :-/. I don't know
>>> why this is in Java that difficult to control a little bit memory.
>>> That's the price for automatic Memory Management!
>>> (Nothing against you Peter!)
>>Eh, if you look at the VM problems in Linux 2.4 you'll see that this
>>realm is a complex one itself and now just because it's java (sure, java
>>doesn't help you when it hides everything from you, like in this case).
>Complex is the point.
>>> >I think store-janitor works fine when it can free sufficient memory by
>>> >calling the finalize and garbage collection. But if it needs to free the
>>> >store it locks up!
>>> >
>>> >The reason it appears to lock up (I think), is because it calls the
>>> >garbage collector too often - i.e. every time it frees an item from a
>>> >store. Once called the store-janitor will free an item and call the
>>> >garbage collector, repeating both until it has freed sufficient memory.
>>> >If one item frees 4.44k and the garbage collector takes 2844ms then the
>>> >time taken to free 20m will be 3.5 hours - during which time the system
>>> >appears to be locked!
>>Ouch! that hurts!
>Yep, I still recover from that ;).
>>> >The following is a brief summary of my analysis:
>>> >
>>> >With the following settings:
>>> >Xms=0m
>>> >Xmx=140m
>>> >heapsize =100000000
>>> >freememory=20000000
>>> >
>>> >Using Optimizeit I watched my java heap 'total' and 'in-use' growing.
>>> >When total heapsize gets to 100m the store-janitor kicks in and frees a
>>> >chunk of memory. This repeats every ten seconds but each time it frees
>>> >less memory until it is no longer able to free memory and locks up. When
>>> >locked up the garbage collector is running about 99% of the time.
>>That's *very* bad!
>>> >Looking at the store-janitor code and my debug log I can see exactly
>>> >what happens when it locks.
>>> >
>>> >I notice, examining that, if the garbage
>>> >collector alone manages to clear sufficient memory, no store items are
>>> >cleared (as expected). This is possibly all that is happening when
>>> >store-janitor appears to work! However if that is not sufficient the
>>> >following loop is started.
>>> >
>>> >                 synchronized (this) {
>>> >                     while (this.memoryLow() &&
>>> >this.getStoreList().size() > 0) {
>>> >                         this.freeMemory();
>>> >                     }
>>> >                     this.resetIndex();
>>> >                 }
>>> >
>>> >this.memoryLow() tests to see if sufficient is free and prints Debug info.
>>> >this.freeMemory() removes an item, runs garbage collection, and prints
>>> >Debug info.
>>> >
>>> > From my log files I can see this loop being executed. Each time round
>>> >the loop I can see how much memory is freed and how long it takes.
>>> >Typically it can take seconds to free one item which can be only a few
>>> >kb. So, in my case with 'freememory' set to 20m, it is likely to take a
>>> >few hours to free sufficient memory!!
>>> >
>>> >I've noticed when running Optimizeit that although garbage collection
>>> >can be quick, it sometimes takes a few seconds. If it has just been run,
>>> >and you force it to run again, then it seems to take longer. If it can't
>>> >find much to freeup, it seems to do a more time consuming search for
>>> >garbage.
>>> This all makes sense too me.
>>> Then we need a limit how often the GC can be called. Maybe 100 loops
>>> should be enough. 
>>I don't get it: why more than one loop? I mean, if you call the GC, it
>>should free all the memory that it can... doing any more loops at that
>>time, will not change anything.
>Maybe I did something wrong. But the idea was that:
>while(MemoryIsLow) {
>  free objects out of the store //physicall
>  run gc
>I run the GC after each "free objects" to clean away
>does not needed objects!
>The problem is now, that when he doesn't has anything
>to free from the registered stores, he runs the GC 
>all the time. And then we have this endless loop!
>Therefore I should check if the Stores are empty. If they
>are empty and there is nothing to free anymore we should 
>leave that loop.
>>> After that it waits as configured (threadinterval)
>>> before the next execution.
>>> Maybe there is a more elegent solution.
>>Well, this is a very difficult realm, as I said, but sure, we can start
>>a redesign phase on this.
>We should!
>>> I actually develop in jakarta-commons a store (again) with Weak References.
>>> This is a much better solution. No Threads, no GC calls. When it is ready
>>> to introduce I'll explain you!
>>Be careful when going down this road: the JLS doesn't say anything about
>>'how' the weak references should be cleaned. And Java 1.3 implements it
>>in the most memory efficient possible way: if the GC gets a weak
>>reference, all of them are removed.
>>Now, imagine the case: you have spent a couple of hours building up and
>>tuning your cache using highly complex statistical analysis... but you
>>saved them into a store that is implemented using weak references.
>>You hit the heap ceiling and the GC kicks in.
>>If finds one of those weak references, but it doesn't have the ability
>>to ask you (well, the store) about 'which is weaker', so, wooosh,
>>everything is gone.
>>Sure, you probably collect 40Mb or heap, but then your perceived
>>performance drops and your cache system works in a 'saw-like' figure
>>cache size
>> ^ 
>> |     /|   /|
>> |    / |  / |
>> |   /  | /  |
>> |  /   |/   |/
>> +-----------------> time
>>        ^
>>    GC kicks in 
>>and since the cache size is linearly proportional with the perceived
>>efficiency of the cache, Cocoon will appear to be 'fast at times' (to be
>>kind) or 'awefully bumping' (to be more reasonable).
>>The solution I thought for this problem is simple but I don't know if it
>>works (never tested it real life):
>> 1) partition the memory store into 'clusters' which are connected using
>>hard references.
>That is the idea of jakarta-commons simplestore. To build up a store
>> 2) these 'clusters' are the real containers of the stored entities and
>>such entities are weak-referenced from the 'clusters'
>>The idea is to have something like this:
>> store +==> cluster +---> entity
>>       |            +---> entity
>>       |            +---> entity
>>       |
>>       +==> cluster +---> entity
>>                    +---> entity
>>                    +---> entity
>> ===> hard reference
>> ---> soft reference
>>and the key concept is that, whenever the GC needs to free some memory
>>that is occupied by weak references, the avalanche cleanup will be
>>'stopped' at a cluster level, saving the other clusters.
>>Now, I'm not sure this works, but if it does, we have a reasonable
>>solution to the problem.
>We had the same idea and we are working on that.
>==> jakarta-commons-sandbox/simplestore
>"simple" to integrate "complex" to code ;-).
>>> Sorry for all that crap with Store and everything.
>>Don't worry, it takes time to do it right but I'm sure we'll get there.
>>Probably we need to discuss its design *before* implementing it :)
>Indeed ;-).
>   Gerhard
>"The most important question when any new 
>computer architecture is introduced is 'So what?'"
>To unsubscribe, e-mail:
>For additional commands, email:

To unsubscribe, e-mail:
For additional commands, email:

View raw message