cocoon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gerhard Froehlich" <>
Subject RE: StoreJanitor new calculation [was: Re: Store Janitor Hangs System]
Date Mon, 04 Feb 2002 18:45:57 GMT

>From: Peter Hargreaves []
>Hi Gerhard,
>I've tested your latest Store Janitor improvements (using Optimizit and 
>your debug messages) and although its a very great improvement, I've 
>found some other problems :-(


>As my Optimizit trial was running out I decided to do some urgent work 
>on the Checker and now have a solution which appears to work well :-)
>Here is a summary of the problems with ideas for correcting them (some 
>are nitty gritty, and some you already know):
>*) When demand for memory is high, it is possible to run out of 
>java.memory before the store has been emptied. Might it be possible to 
>make sure the stores are empty before the memory can run out?

Then we don't need the StoreJanitor, or?

>*) When memory is low, there is a risk of running out of it before the 
>next check. Why not set the interval short enough to ensure there is no 
>risk of running out of memory before the next check?

Be careful with Threads. When they run in a to short interval they'll
kill your performance!

>*) Calling the gc once every check interval seriously undermines system 
>performance because it can take several seconds to complete, which is 
>comparable to the check interval. Why not call it only when some items 
>are released from a store? Why not trust it and use its characteristics 
>to advantage?

That's a contradiction to previous point, I guess. On the one hand you want
to increase the the iteration of the Thread on the other hand you want to
decrease the GC calls.

>*) When several instances of Cocoon are running (independent .WAR apps 
>in the same servlet container), the calls to gc become very dominant. 
>Would this affect the strategy for store-janitor settings?
>*) If the percent to reduce storage is set to 10%, it fails to remove 
>any when the number of items are below 10. The number of items to be 
>removed needs rounding upwards. Why not remove a fixed number of items 
>instead of a percentage? (My idea and now I think it was wrong!!)

Au, my embarrsing math capabilities strikes agein ;).

>*) It is difficult to get a good picture of what is happening by 
>observing the debug output, so it is difficult to know if it is working 
>and difficult use the debug output to optimise choice of janitor 
>settings. Might it be possible to make this easy?

Yes and no. That's always a problem in Cocoon. But you can define your
own log target (own file) in the logkit.xconf for the StoreJanitor. Then
you get rid of the noise.

>The proposed, implemented and tested solution to the above works as follows:
>1) Monitor the rate at which memory is consumed during each check 
>interval. Remember the biggest rate that has been found so far, to use 
>it later. My system logs this at 2.5 megabytes per second.


>2) If heapIsBig() && freeIsLow(), then attempt to free storage.
>2.1) 'heapIsBig' means that the total memory has exceeded the parameter 
>2.2) 'freeIsLow' means the free memory is less than the parameter 'minfree'.
>2.3) Attempt to remove from the stores, the number of items specified by 
>the parameter 'reduceby'.
>2.4) To free storage, start at the next store and remove items, moving 
>to the next again if necessary, until the specified number of items have 
>been removed or all the stores are empty.
>2.5) Then call the garbage collector, but only if some memory items were 

Ok, that's nice.

>3) Sleep for an interval half that in which the memory could run out.
>3.1) If the remaining heap could be more than half filled during the 
>interval specified by the 'maxsleep' parameter (at the max rate of 
>memory consumption) then sleep for the time it would take to only half 
>fill (freememory/maxrate)/2.  Otherwise sleep for the max sleep interval.

Damn, that's sounds great ;). Your biggest unknown variable is the 
"could be". But I think taking the last biggest consumption is not that
bad. But how about taking the average of all calls. That would be more exact
then the last biggest value. 

>The effect of the store-janitor parameters in cocoon.xconf are now 
>slightly changed, so I've changed the names of the parameters to reflect 
>this. For the moment the names are changed only inside 
> If the solution were to be adopted then the names 
>in cocoon.xconf would also change. The changes, with notes, are as follows:

Naming was indeed crap.

>'maxheap' (was 'heapsize')  The maximum total heap size below which 
>stores are guaranteed to remain intact.
>'minfree' (was 'freememory') The minimum free heap required for stores 
>to remain intact. I have found this should be set to less than 10% of 
>avaliable memory, so that the jvm will respond to low memory by 
>allocating more heap (if its available), before stores are likely to be 
>reduced. Default value might be 2000000. Setting it to zero or very low 
>will effectively disable store reduction.
>'maxsleep' (was 'cleanupthreadinterval') The maximum interval between 
>checks. Should be short enough to ensure that 'rate of change' data is 
>collected before free memory runs low. Suggest 5 secs as default.
>'reduceby' (was 'percent_to_free') is the number of items to be removed 
>>from the stores, on each attempt to reduce the stores. Each  successfull 
>attempt to reduce stores results in a call to the garbage collector 
>taking several seconds. This limits the rate a which the stores can be 
>emptied. Removing several items at a time compensates for this. Using 
>the debug output, choose a number of items which results in an increase 
>of free memory between successive store reductions. Best tested with a 
>sudden high demand for memory when the stores are full. In practice this 
>setting is not critical. Suggested default is 10 items. Ideally 
>'reduceby' would be in bytes!
>Setting 'minfree' and 'reduceby' both too low can intensify gc activity 
>if sudden high demand requires all stores to be completely freed and can 
>result in memory running out before all the stores are free. Verify 
>using debug output.
>'priority' remains unchanged.
>The above solution is implemented and tested and it works very well 
>indeed for me. The parameters do not seem to be at all critical - thank 
>goodness! I see no reason why the default settings would need to be 
>changed if more memory were available. I have attached the 
> so others can test it. I find I can get a very 
>good picture of what is happening by using my editor's repeat find 
>command on the debug output in the core.log.000001 file. Optimizeit is 
>no longer necessary to prove that it works effectively.

Just define as mentioned above you own log target for the store.janitor.
Take a look into logkit.xconf in the WEB-INF dir!

>Beware! - there might still be things I don't know about that I have not 
>take into account.
>Ideally - low memory should be detected by some sort of interrupt or 
>exception rather than by polling.

That's correct. But here java is a big white spot.

>If you are subsequently interested in committing these changes into 
>Cocoon and you want me to do some more work to conform to design 
>practice etc., then please let me know.

Yep of course. But I switch to my picky mode now. Please reformat your
code to our coding standards.
- 4 indents, no tabs.
- if(foo)
     this.isBad = true;
That's not good coding style. Use brackets for each condition or block.
- Make it more readable

Then I will commit your stuff into the Scratchpad area.
(That's the normal way this days)

What do you think!

>Very humble appologies, Gerhard (and others), for intruding into your 
>code. If you like my changes, I hope you feel that I have released your 
>brilliance (not undermined it). I feel that by far the biggest part of 
>any credit should very definately still be yours (I only fixed the 
>checker) :-).

Wah, no apologizes please! I always wished, that somebody start to refactor
this idea, because I stuck in the moment.

A DAMN CATHEDRAL. We need more people sending patches. Make yourself heard here!!


My parents have been visiting me for a few days. 
I just dropped them off at the airport. 
They leave tomorrow. 

To unsubscribe, e-mail:
For additional commands, email:

View raw message