cocoon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Hargreaves <...@totalise.co.uk>
Subject Re: Store Janitor Hangs System
Date Sun, 27 Jan 2002 16:02:17 GMT
Hi Folks,

Its great to see such enthusiasm - thanks for listening. Just got a
coupe of comments and a simple idea that might help towards the solution
- if you scroll down.

Gerhard Froehlich wrote:

 > Hi,
 >
 >  >From: Stefano Mazzocchi [mailto:stefano@apache.org]
 >  >
 >  >Gerhard Froehlich wrote:
 >  >>
 >  >> Hi,
 >  >>
 >  >> >From: Peter Hargreaves [mailto:pdh@totalise.co.uk]
 >  >> >
 >  >> >Hi Gerhard and Team,
 >  >> >
 >  >> >I'm posting in this group now I think I found a design problem with
 >  >> >store-janitor.
 >  >>
 >  >> Slowly but constant this Store issues are a PITA for me :-/. I don't
 > know
 >  >> why this is in Java that difficult to control a little bit memory.
 >  >> That's the price for automatic Memory Management!
 >  >> (Nothing against you Peter!)
 >  >
 >  >Eh, if you look at the VM problems in Linux 2.4 you'll see that this
 >  >realm is a complex one itself and now just because it's java (sure, java
 >  >doesn't help you when it hides everything from you, like in this case).
 >
 > Complex is the point.
 >
 >  >> >I think store-janitor works fine when it can free sufficient memory by
 >  >> >calling the finalize and garbage collection. But if it needs to
 > free the
 >  >> >store it locks up!
 >  >> >
 >  >> >The reason it appears to lock up (I think), is because it calls the
 >  >> >garbage collector too often - i.e. every time it frees an item from a
 >  >> >store. Once called the store-janitor will free an item and call the
 >  >> >garbage collector, repeating both until it has freed sufficient
 > memory.
 >  >> >If one item frees 4.44k and the garbage collector takes 2844ms then
 > the
 >  >> >time taken to free 20m will be 3.5 hours - during which time the
 > system
 >  >> >appears to be locked!
 >  >
 >  >Ouch! that hurts!
 >
 > Yep, I still recover from that ;).
 >
 >  >> >The following is a brief summary of my analysis:
 >  >> >
 >  >> >With the following settings:
 >  >> >Xms=0m
 >  >> >Xmx=140m
 >  >> >heapsize =100000000
 >  >> >freememory=20000000
 >  >> >
 >  >> >Using Optimizeit I watched my java heap 'total' and 'in-use' growing.
 >  >> >When total heapsize gets to 100m the store-janitor kicks in and
 > frees a
 >  >> >chunk of memory. This repeats every ten seconds but each time it frees
 >  >> >less memory until it is no longer able to free memory and locks up.
 > When
 >  >> >locked up the garbage collector is running about 99% of the time.
 >  >
 >  >That's *very* bad!
 >
 > Indeed!
 >
 >  >> >Looking at the store-janitor code and my debug log I can see exactly
 >  >> >what happens when it locks.
 >  >> >
 >  >> >I notice, examining Store-janitorImpl.run() that, if the garbage
 >  >> >collector alone manages to clear sufficient memory, no store items are
 >  >> >cleared (as expected). This is possibly all that is happening when
 >  >> >store-janitor appears to work! However if that is not sufficient the
 >  >> >following loop is started.
 >  >> >
 >  >> >                 synchronized (this) {
 >  >> >                     while (this.memoryLow() &&
 >  >> >this.getStoreList().size() > 0) {
 >  >> >                         this.freeMemory();
 >  >> >                     }
 >  >> >                     this.resetIndex();
 >  >> >                 }
 >  >> >
 >  >> >this.memoryLow() tests to see if sufficient is free and prints
 > Debug info.
 >  >> >this.freeMemory() removes an item, runs garbage collection, and prints
 >  >> >Debug info.
 >  >> >
 >  >> > From my log files I can see this loop being executed. Each time round
 >  >> >the loop I can see how much memory is freed and how long it takes.
 >  >> >Typically it can take seconds to free one item which can be only a few
 >  >> >kb. So, in my case with 'freememory' set to 20m, it is likely to
 > take a
 >  >> >few hours to free sufficient memory!!
 >  >> >
 >  >> >I've noticed when running Optimizeit that although garbage collection
 >  >> >can be quick, it sometimes takes a few seconds. If it has just been
 > run,
 >  >> >and you force it to run again, then it seems to take longer. If it
 > can't
 >  >> >find much to freeup, it seems to do a more time consuming search for
 >  >> >garbage.
 >  >>
 >  >> This all makes sense too me.
 >  >> Then we need a limit how often the GC can be called. Maybe 100 loops
 >  >> should be enough.
 >  >
 >  >I don't get it: why more than one loop? I mean, if you call the GC, it
 >  >should free all the memory that it can... doing any more loops at that
 >  >time, will not change anything.
 >
 > Maybe I did something wrong. But the idea was that:
 >
 > while(MemoryIsLow) {
 >   free objects out of the store //physicall
 >   run gc
 > }
 >
 > I run the GC after each "free objects" to clean away
 > does not needed objects!
 >
 > The problem is now, that when he doesn't has anything
 > to free from the registered stores, he runs the GC
 > all the time. And then we have this endless loop!
 >
 > Therefore I should check if the Stores are empty. If they
 > are empty and there is nothing to free anymore we should
 > leave that loop.


Is this where you mean:

    while (this.memoryLow() && this.getStoreList().size() > 0) {
        this.freeMemory();
    }

I assumed that "this.getStoreList().size()" returned 0 when all the
stores were empty. If this is not the case (as you suggest) then it
looks like a permanent loop. I have seen it come out of this loop
however. Perhaps another thread freed up memory letting the gc become
successful at breaking out of the loop through "this.memoryLow()".

I saw it break out of the loop when I had the jvm -Xnoclassgc option
set. With this option classes that haven't been used for a while get
dropped from memory don't they? So, if stuck in a loop, it won't be long
before there are a lot of classes that haven't been used for a while!
When I saw it come out, it was spectacular - a sudden big drop in memory
in use!


 >
 >  >> After that it waits as configured (threadinterval)
 >  >> before the next execution.
 >  >>
 >  >> Maybe there is a more elegent solution.


Hava you considered a sort of binary approach to emptying stores?

while (MemoryIsLow) {
     free half of the items in the next store
     run gc
}


 >  >
 >  >Well, this is a very difficult realm, as I said, but sure, we can start
 >  >a redesign phase on this.
 >
 > We should!
 >
 >  >> I actually develop in jakarta-commons a store (again) with Weak
 > References.
 >  >> This is a much better solution. No Threads, no GC calls. When it is
 > ready
 >  >> to introduce I'll explain you!
 >  >
 >  >Be careful when going down this road: the JLS doesn't say anything about
 >  >'how' the weak references should be cleaned. And Java 1.3 implements it
 >  >in the most memory efficient possible way: if the GC gets a weak
 >  >reference, all of them are removed.
 >  >
 >  >Now, imagine the case: you have spent a couple of hours building up and
 >  >tuning your cache using highly complex statistical analysis... but you
 >  >saved them into a store that is implemented using weak references.
 >  >
 >  >You hit the heap ceiling and the GC kicks in.
 >  >
 >  >If finds one of those weak references, but it doesn't have the ability
 >  >to ask you (well, the store) about 'which is weaker', so, wooosh,
 >  >everything is gone.
 >  >
 >  >Sure, you probably collect 40Mb or heap, but then your perceived
 >  >performance drops and your cache system works in a 'saw-like' figure
 >  >
 >  >
 >  >cache size
 >  > ^
 >  > |     /|   /|
 >  > |    / |  / |
 >  > |   /  | /  |
 >  > |  /   |/   |/
 >  > +-----------------> time
 >  >        ^
 >  >    GC kicks in
 >  >
 >  >and since the cache size is linearly proportional with the perceived
 >  >efficiency of the cache, Cocoon will appear to be 'fast at times' (to be
 >  >kind) or 'awefully bumping' (to be more reasonable).
 >  >
 >  >The solution I thought for this problem is simple but I don't know if it
 >  >works (never tested it real life):
 >  >
 >  > 1) partition the memory store into 'clusters' which are connected using
 >  >hard references.
 >
 > That is the idea of jakarta-commons simplestore. To build up a store
 > chain!
 >
 >  > 2) these 'clusters' are the real containers of the stored entities and
 >  >such entities are weak-referenced from the 'clusters'
 >  >
 >  >The idea is to have something like this:
 >  >
 >  > store +==> cluster +---> entity
 >  >       |            +---> entity
 >  >       |            +---> entity
 >  >       |
 >  >       +==> cluster +---> entity
 >  >                    +---> entity
 >  >                    +---> entity
 >  >
 >  >where:
 >  >
 >  > ===> hard reference
 >  > ---> soft reference
 >  >
 >  >and the key concept is that, whenever the GC needs to free some memory
 >  >that is occupied by weak references, the avalanche cleanup will be
 >  >'stopped' at a cluster level, saving the other clusters.
 >  >
 >  >Now, I'm not sure this works, but if it does, we have a reasonable
 >  >solution to the problem.
 >
 > We had the same idea and we are working on that.
 > ==> jakarta-commons-sandbox/simplestore
 > "simple" to integrate "complex" to code ;-).
 >
 >  >> Sorry for all that crap with Store and everything.
 >  >
 >  >Don't worry, it takes time to do it right but I'm sure we'll get there.
 >  >
 >  >Probably we need to discuss its design *before* implementing it :)
 >
 > Indeed ;-).
 >
 >    Gerhard
 >
 > "The most important question when any new
 > computer architecture is introduced is 'So what?'"
 >

Peter.



---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org


Mime
View raw message