cocoon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Hargreaves <>
Subject Re: StoreJanitor new calculation [was: Re: Store Janitor Hangs System]
Date Mon, 04 Feb 2002 17:38:48 GMT
Hi Gerhard,

I've tested your latest Store Janitor improvements (using Optimizit and 
your debug messages) and although its a very great improvement, I've 
found some other problems :-(

As my Optimizit trial was running out I decided to do some urgent work 
on the Checker and now have a solution which appears to work well :-)

Here is a summary of the problems with ideas for correcting them (some 
are nitty gritty, and some you already know):

*) When demand for memory is high, it is possible to run out of 
java.memory before the store has been emptied. Might it be possible to 
make sure the stores are empty before the memory can run out?

*) When memory is low, there is a risk of running out of it before the 
next check. Why not set the interval short enough to ensure there is no 
risk of running out of memory before the next check?

*) Calling the gc once every check interval seriously undermines system 
performance because it can take several seconds to complete, which is 
comparable to the check interval. Why not call it only when some items 
are released from a store? Why not trust it and use its characteristics 
to advantage?

*) When several instances of Cocoon are running (independent .WAR apps 
in the same servlet container), the calls to gc become very dominant. 
Would this affect the strategy for store-janitor settings?

*) If the percent to reduce storage is set to 10%, it fails to remove 
any when the number of items are below 10. The number of items to be 
removed needs rounding upwards. Why not remove a fixed number of items 
instead of a percentage? (My idea and now I think it was wrong!!)

*) It is difficult to get a good picture of what is happening by 
observing the debug output, so it is difficult to know if it is working 
and difficult use the debug output to optimise choice of janitor 
settings. Might it be possible to make this easy?

The proposed, implemented and tested solution to the above works as follows:

1) Monitor the rate at which memory is consumed during each check 
interval. Remember the biggest rate that has been found so far, to use 
it later. My system logs this at 2.5 megabytes per second.

2) If heapIsBig() && freeIsLow(), then attempt to free storage.

2.1) 'heapIsBig' means that the total memory has exceeded the parameter 

2.2) 'freeIsLow' means the free memory is less than the parameter 'minfree'.

2.3) Attempt to remove from the stores, the number of items specified by 
the parameter 'reduceby'.

2.4) To free storage, start at the next store and remove items, moving 
to the next again if necessary, until the specified number of items have 
been removed or all the stores are empty.

2.5) Then call the garbage collector, but only if some memory items were 

3) Sleep for an interval half that in which the memory could run out.

3.1) If the remaining heap could be more than half filled during the 
interval specified by the 'maxsleep' parameter (at the max rate of 
memory consumption) then sleep for the time it would take to only half 
fill (freememory/maxrate)/2.  Otherwise sleep for the max sleep interval.

The effect of the store-janitor parameters in cocoon.xconf are now 
slightly changed, so I've changed the names of the parameters to reflect 
this. For the moment the names are changed only inside If the solution were to be adopted then the names 
in cocoon.xconf would also change. The changes, with notes, are as follows:

'maxheap' (was 'heapsize')  The maximum total heap size below which 
stores are guaranteed to remain intact.

'minfree' (was 'freememory') The minimum free heap required for stores 
to remain intact. I have found this should be set to less than 10% of 
avaliable memory, so that the jvm will respond to low memory by 
allocating more heap (if its available), before stores are likely to be 
reduced. Default value might be 2000000. Setting it to zero or very low 
will effectively disable store reduction.

'maxsleep' (was 'cleanupthreadinterval') The maximum interval between 
checks. Should be short enough to ensure that 'rate of change' data is 
collected before free memory runs low. Suggest 5 secs as default.

'reduceby' (was 'percent_to_free') is the number of items to be removed 
from the stores, on each attempt to reduce the stores. Each  successfull 
attempt to reduce stores results in a call to the garbage collector 
taking several seconds. This limits the rate a which the stores can be 
emptied. Removing several items at a time compensates for this. Using 
the debug output, choose a number of items which results in an increase 
of free memory between successive store reductions. Best tested with a 
sudden high demand for memory when the stores are full. In practice this 
setting is not critical. Suggested default is 10 items. Ideally 
'reduceby' would be in bytes!

Setting 'minfree' and 'reduceby' both too low can intensify gc activity 
if sudden high demand requires all stores to be completely freed and can 
result in memory running out before all the stores are free. Verify 
using debug output.

'priority' remains unchanged.

The above solution is implemented and tested and it works very well 
indeed for me. The parameters do not seem to be at all critical - thank 
goodness! I see no reason why the default settings would need to be 
changed if more memory were available. I have attached the so others can test it. I find I can get a very 
good picture of what is happening by using my editor's repeat find 
command on the debug output in the core.log.000001 file. Optimizeit is 
no longer necessary to prove that it works effectively.

Beware! - there might still be things I don't know about that I have not 
take into account.
Ideally - low memory should be detected by some sort of interrupt or 
exception rather than by polling.

If you are subsequently interested in committing these changes into 
Cocoon and you want me to do some more work to conform to design 
practice etc., then please let me know.

Very humble appologies, Gerhard (and others), for intruding into your 
code. If you like my changes, I hope you feel that I have released your 
brilliance (not undermined it). I feel that by far the biggest part of 
any credit should very definately still be yours (I only fixed the 
checker) :-).

Feedback very welcome please.

Pete H. wrote:

> Hi Team,
> I hope I fixed the problem now!
> I changed the StoreJanitorImpl, I try to explain in
> prosa:
> <not to serious>
> Cocoon reaches 1 million request per second and
> memory is getting dangerously low. Time for the
> StoreJanitor to kick in and do something. First
> he forces the GC. Damn, memory is still low,
> we have to remove object from the Store to get
> some bytes free. Panic, which Store shall we take?
> Ahh index is at -1 so we take the first at index 0 
> in the StoreArray. Gosh this one is pretty full how
> much objects shall we kick out? Stupid question,
> take 10% from all as configured. Chacka, did ya see
> them fly?
> Next time we take the next store on index 1.
> Yeah great work dudes, take a rest.
> </not to serious>
> Test it!
>   Gerhard
> ------------------------------------------------
> If patience is a virtue, and ignorance is bliss,
> you can have a pretty good life if you're stupid
> and willing to wait.
> ------------------------------------------------
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, email:

View raw message