Mailing-List: contact cocoon-dev-help@xml.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cocoon-dev@xml.apache.org
From: "Gerhard Froehlich" <g-froehlich@gmx.de>
To: <cocoon-dev@xml.apache.org>
Subject: RE: Store Janitor Hangs System
Date: Sun, 27 Jan 2002 17:55:16 +0100
Message-ID: <LDEKKGCNJLEPFDOEIHFDKEPMCOAA.g-froehlich@gmx.de>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
Importance: Normal
In-Reply-To: <LDEKKGCNJLEPFDOEIHFDEEPKCOAA.g-froehlich@gmx.de>

Ok guys,
I fixed that bug more or less quick and dirty:

The loop which removes objects out of the stores
is only iterated once, now. I also removed the
call of the gc in the loop. The next Thread
iteration calls the gc next:

run() {
  if(memoryIsLow) {
    call gc

    while(memoryIsStillLow && Iteration < 1) {
     	free some objects ot of the reg. stores
      count iteration +1
    }
  }
  thread sleeps
}

Not very creative, I know. 

  Gerhard
 
"Eagles may soar, but weasels don't get 
sucked into jet engines.
(Todd C. Somers)"


>-----Original Message-----
>From: Gerhard Froehlich [mailto:g-froehlich@gmx.de]
>Sent: Sunday, January 27, 2002 2:30 PM
>To: cocoon-dev@xml.apache.org
>Subject: RE: Store Janitor Hangs System
>
>
>Hi,
>
>>From: Stefano Mazzocchi [mailto:stefano@apache.org]
>>
>>Gerhard Froehlich wrote:
>>> 
>>> Hi,
>>> 
>>> >From: Peter Hargreaves [mailto:pdh@totalise.co.uk]
>>> >
>>> >Hi Gerhard and Team,
>>> >
>>> >I'm posting in this group now I think I found a design problem with
>>> >store-janitor.
>>> 
>>> Slowly but constant this Store issues are a PITA for me :-/. I don't know
>>> why this is in Java that difficult to control a little bit memory.
>>> That's the price for automatic Memory Management!
>>> (Nothing against you Peter!)
>>
>>Eh, if you look at the VM problems in Linux 2.4 you'll see that this
>>realm is a complex one itself and now just because it's java (sure, java
>>doesn't help you when it hides everything from you, like in this case).
>
>Complex is the point.
>
>>> >I think store-janitor works fine when it can free sufficient memory by
>>> >calling the finalize and garbage collection. But if it needs to free the
>>> >store it locks up!
>>> >
>>> >The reason it appears to lock up (I think), is because it calls the
>>> >garbage collector too often - i.e. every time it frees an item from a
>>> >store. Once called the store-janitor will free an item and call the
>>> >garbage collector, repeating both until it has freed sufficient memory.
>>> >If one item frees 4.44k and the garbage collector takes 2844ms then the
>>> >time taken to free 20m will be 3.5 hours - during which time the system
>>> >appears to be locked!
>>
>>Ouch! that hurts!
>
>Yep, I still recover from that ;).
>
>>> >The following is a brief summary of my analysis:
>>> >
>>> >With the following settings:
>>> >Xms=0m
>>> >Xmx=140m
>>> >heapsize =100000000
>>> >freememory=20000000
>>> >
>>> >Using Optimizeit I watched my java heap 'total' and 'in-use' growing.
>>> >When total heapsize gets to 100m the store-janitor kicks in and frees a
>>> >chunk of memory. This repeats every ten seconds but each time it frees
>>> >less memory until it is no longer able to free memory and locks up. When
>>> >locked up the garbage collector is running about 99% of the time.
>>
>>That's *very* bad!
>
>Indeed!
>
>>> >Looking at the store-janitor code and my debug log I can see exactly
>>> >what happens when it locks.
>>> >
>>> >I notice, examining Store-janitorImpl.run() that, if the garbage
>>> >collector alone manages to clear sufficient memory, no store items are
>>> >cleared (as expected). This is possibly all that is happening when
>>> >store-janitor appears to work! However if that is not sufficient the
>>> >following loop is started.
>>> >
>>> >                 synchronized (this) {
>>> >                     while (this.memoryLow() &&
>>> >this.getStoreList().size() > 0) {
>>> >                         this.freeMemory();
>>> >                     }
>>> >                     this.resetIndex();
>>> >                 }
>>> >
>>> >this.memoryLow() tests to see if sufficient is free and prints Debug info.
>>> >this.freeMemory() removes an item, runs garbage collection, and prints
>>> >Debug info.
>>> >
>>> > From my log files I can see this loop being executed. Each time round
>>> >the loop I can see how much memory is freed and how long it takes.
>>> >Typically it can take seconds to free one item which can be only a few
>>> >kb. So, in my case with 'freememory' set to 20m, it is likely to take a
>>> >few hours to free sufficient memory!!
>>> >
>>> >I've noticed when running Optimizeit that although garbage collection
>>> >can be quick, it sometimes takes a few seconds. If it has just been run,
>>> >and you force it to run again, then it seems to take longer. If it can't
>>> >find much to freeup, it seems to do a more time consuming search for
>>> >garbage.
>>> 
>>> This all makes sense too me.
>>> Then we need a limit how often the GC can be called. Maybe 100 loops
>>> should be enough. 
>>
>>I don't get it: why more than one loop? I mean, if you call the GC, it
>>should free all the memory that it can... doing any more loops at that
>>time, will not change anything.
>
>Maybe I did something wrong. But the idea was that:
>
>while(MemoryIsLow) {
>  free objects out of the store //physicall
>  run gc
>}
>
>I run the GC after each "free objects" to clean away
>does not needed objects!
>
>The problem is now, that when he doesn't has anything
>to free from the registered stores, he runs the GC 
>all the time. And then we have this endless loop!
>
>Therefore I should check if the Stores are empty. If they
>are empty and there is nothing to free anymore we should 
>leave that loop.
>
>>> After that it waits as configured (threadinterval)
>>> before the next execution.
>>> 
>>> Maybe there is a more elegent solution.
>>
>>Well, this is a very difficult realm, as I said, but sure, we can start
>>a redesign phase on this.
>
>We should!
>
>>> I actually develop in jakarta-commons a store (again) with Weak References.
>>> This is a much better solution. No Threads, no GC calls. When it is ready
>>> to introduce I'll explain you!
>>
>>Be careful when going down this road: the JLS doesn't say anything about
>>'how' the weak references should be cleaned. And Java 1.3 implements it
>>in the most memory efficient possible way: if the GC gets a weak
>>reference, all of them are removed.
>>
>>Now, imagine the case: you have spent a couple of hours building up and
>>tuning your cache using highly complex statistical analysis... but you
>>saved them into a store that is implemented using weak references.
>>
>>You hit the heap ceiling and the GC kicks in.
>>
>>If finds one of those weak references, but it doesn't have the ability
>>to ask you (well, the store) about 'which is weaker', so, wooosh,
>>everything is gone.
>>
>>Sure, you probably collect 40Mb or heap, but then your perceived
>>performance drops and your cache system works in a 'saw-like' figure
>>
>>
>>cache size
>> ^ 
>> |     /|   /|
>> |    / |  / |
>> |   /  | /  |
>> |  /   |/   |/
>> +-----------------> time
>>        ^
>>    GC kicks in 
>>
>>and since the cache size is linearly proportional with the perceived
>>efficiency of the cache, Cocoon will appear to be 'fast at times' (to be
>>kind) or 'awefully bumping' (to be more reasonable).
>>
>>The solution I thought for this problem is simple but I don't know if it
>>works (never tested it real life):
>>
>> 1) partition the memory store into 'clusters' which are connected using
>>hard references.
>
>That is the idea of jakarta-commons simplestore. To build up a store
>chain!
>
>> 2) these 'clusters' are the real containers of the stored entities and
>>such entities are weak-referenced from the 'clusters'
>>
>>The idea is to have something like this:
>>
>> store +==> cluster +---> entity
>>       |            +---> entity
>>       |            +---> entity
>>       |
>>       +==> cluster +---> entity
>>                    +---> entity
>>                    +---> entity
>>
>>where:
>> 
>> ===> hard reference
>> ---> soft reference
>>
>>and the key concept is that, whenever the GC needs to free some memory
>>that is occupied by weak references, the avalanche cleanup will be
>>'stopped' at a cluster level, saving the other clusters.
>>
>>Now, I'm not sure this works, but if it does, we have a reasonable
>>solution to the problem.
>
>We had the same idea and we are working on that.
>==> jakarta-commons-sandbox/simplestore
>"simple" to integrate "complex" to code ;-).
>
>>> Sorry for all that crap with Store and everything.
>>
>>Don't worry, it takes time to do it right but I'm sure we'll get there.
>>
>>Probably we need to discuss its design *before* implementing it :)
>
>Indeed ;-).
>
>   Gerhard
> 
>"The most important question when any new 
>computer architecture is introduced is 'So what?'"
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
>For additional commands, email: cocoon-dev-help@xml.apache.org
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org