Mailing-List: contact cocoon-dev-help@xml.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cocoon-dev@xml.apache.org
From: "Gerhard Froehlich" <g-froehlich@gmx.de>
To: <cocoon-dev@xml.apache.org>
Subject: RE: StoreJanitor new calculation [was: Re: Store Janitor Hangs
 System]
Date: Mon, 4 Feb 2002 19:45:57 +0100
Message-ID: <LDEKKGCNJLEPFDOEIHFDIEFPDAAA.g-froehlich@gmx.de>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: 7bit
In-Reply-To: <3C5EC728.1010402@netscape.net>
Importance: Normal

Peter,

>From: Peter Hargreaves [mailto:pdhargr@netscape.net]
>
>Hi Gerhard,
>
>I've tested your latest Store Janitor improvements (using Optimizit and 
>your debug messages) and although its a very great improvement, I've 
>found some other problems :-(

Damn...

>As my Optimizit trial was running out I decided to do some urgent work 
>on the Checker and now have a solution which appears to work well :-)
>
>Here is a summary of the problems with ideas for correcting them (some 
>are nitty gritty, and some you already know):
>
>*) When demand for memory is high, it is possible to run out of 
>java.memory before the store has been emptied. Might it be possible to 
>make sure the stores are empty before the memory can run out?

Then we don't need the StoreJanitor, or?

>*) When memory is low, there is a risk of running out of it before the 
>next check. Why not set the interval short enough to ensure there is no 
>risk of running out of memory before the next check?

Be careful with Threads. When they run in a to short interval they'll
kill your performance!

>*) Calling the gc once every check interval seriously undermines system 
>performance because it can take several seconds to complete, which is 
>comparable to the check interval. Why not call it only when some items 
>are released from a store? Why not trust it and use its characteristics 
>to advantage?

That's a contradiction to previous point, I guess. On the one hand you want
to increase the the iteration of the Thread on the other hand you want to
decrease the GC calls.

>*) When several instances of Cocoon are running (independent .WAR apps 
>in the same servlet container), the calls to gc become very dominant. 
>Would this affect the strategy for store-janitor settings?
>
>*) If the percent to reduce storage is set to 10%, it fails to remove 
>any when the number of items are below 10. The number of items to be 
>removed needs rounding upwards. Why not remove a fixed number of items 
>instead of a percentage? (My idea and now I think it was wrong!!)

Au, my embarrsing math capabilities strikes agein ;).

>*) It is difficult to get a good picture of what is happening by 
>observing the debug output, so it is difficult to know if it is working 
>and difficult use the debug output to optimise choice of janitor 
>settings. Might it be possible to make this easy?

Yes and no. That's always a problem in Cocoon. But you can define your
own log target (own file) in the logkit.xconf for the StoreJanitor. Then
you get rid of the noise.

>The proposed, implemented and tested solution to the above works as follows:
>
>1) Monitor the rate at which memory is consumed during each check 
>interval. Remember the biggest rate that has been found so far, to use 
>it later. My system logs this at 2.5 megabytes per second.

Understand.

>2) If heapIsBig() && freeIsLow(), then attempt to free storage.
>
>2.1) 'heapIsBig' means that the total memory has exceeded the parameter 
>'maxheap'.
>
>2.2) 'freeIsLow' means the free memory is less than the parameter 'minfree'.
>
>2.3) Attempt to remove from the stores, the number of items specified by 
>the parameter 'reduceby'.
>
>2.4) To free storage, start at the next store and remove items, moving 
>to the next again if necessary, until the specified number of items have 
>been removed or all the stores are empty.
>
>2.5) Then call the garbage collector, but only if some memory items were 
>freed.

Ok, that's nice.

>3) Sleep for an interval half that in which the memory could run out.
>
>3.1) If the remaining heap could be more than half filled during the 
>interval specified by the 'maxsleep' parameter (at the max rate of 
>memory consumption) then sleep for the time it would take to only half 
>fill (freememory/maxrate)/2.  Otherwise sleep for the max sleep interval.

Damn, that's sounds great ;). Your biggest unknown variable is the 
"could be". But I think taking the last biggest consumption is not that
bad. But how about taking the average of all calls. That would be more exact
then the last biggest value. 

>The effect of the store-janitor parameters in cocoon.xconf are now 
>slightly changed, so I've changed the names of the parameters to reflect 
>this. For the moment the names are changed only inside 
>StoreJanitorImpl.java. If the solution were to be adopted then the names 
>in cocoon.xconf would also change. The changes, with notes, are as follows:

Naming was indeed crap.

>'maxheap' (was 'heapsize')  The maximum total heap size below which 
>stores are guaranteed to remain intact.
>
>'minfree' (was 'freememory') The minimum free heap required for stores 
>to remain intact. I have found this should be set to less than 10% of 
>avaliable memory, so that the jvm will respond to low memory by 
>allocating more heap (if its available), before stores are likely to be 
>reduced. Default value might be 2000000. Setting it to zero or very low 
>will effectively disable store reduction.
>
>'maxsleep' (was 'cleanupthreadinterval') The maximum interval between 
>checks. Should be short enough to ensure that 'rate of change' data is 
>collected before free memory runs low. Suggest 5 secs as default.
>
>'reduceby' (was 'percent_to_free') is the number of items to be removed 
>>from the stores, on each attempt to reduce the stores. Each  successfull 
>attempt to reduce stores results in a call to the garbage collector 
>taking several seconds. This limits the rate a which the stores can be 
>emptied. Removing several items at a time compensates for this. Using 
>the debug output, choose a number of items which results in an increase 
>of free memory between successive store reductions. Best tested with a 
>sudden high demand for memory when the stores are full. In practice this 
>setting is not critical. Suggested default is 10 items. Ideally 
>'reduceby' would be in bytes!
>
>Setting 'minfree' and 'reduceby' both too low can intensify gc activity 
>if sudden high demand requires all stores to be completely freed and can 
>result in memory running out before all the stores are free. Verify 
>using debug output.
>
>'priority' remains unchanged.
>
>The above solution is implemented and tested and it works very well 
>indeed for me. The parameters do not seem to be at all critical - thank 
>goodness! I see no reason why the default settings would need to be 
>changed if more memory were available. I have attached the 
>StoreJanitorImpl.java so others can test it. I find I can get a very 
>good picture of what is happening by using my editor's repeat find 
>command on the debug output in the core.log.000001 file. Optimizeit is 
>no longer necessary to prove that it works effectively.

Just define as mentioned above you own log target for the store.janitor.
Take a look into logkit.xconf in the WEB-INF dir!

>Beware! - there might still be things I don't know about that I have not 
>take into account.
>Ideally - low memory should be detected by some sort of interrupt or 
>exception rather than by polling.

That's correct. But here java is a big white spot.

>If you are subsequently interested in committing these changes into 
>Cocoon and you want me to do some more work to conform to design 
>practice etc., then please let me know.

Yep of course. But I switch to my picky mode now. Please reformat your
code to our coding standards.
- 4 indents, no tabs.
- if(foo)
     this.isBad = true;
That's not good coding style. Use brackets for each condition or block.
- Make it more readable

Then I will commit your stuff into the Scratchpad area.
(That's the normal way this days)

What do you think!

>Very humble appologies, Gerhard (and others), for intruding into your 
>code. If you like my changes, I hope you feel that I have released your 
>brilliance (not undermined it). I feel that by far the biggest part of 
>any credit should very definately still be yours (I only fixed the 
>checker) :-).

Wah, no apologizes please! I always wished, that somebody start to refactor
this idea, because I stuck in the moment.

PLEASE PEOPLE DON'T APLOGIZES FOR SENDING PATCHES. THAT'S OSS HERE AND NOT
A DAMN CATHEDRAL. We need more people sending patches. Make yourself heard here!!

  ~Gerhard

+----------------------------------------------+
My parents have been visiting me for a few days. 
I just dropped them off at the airport. 
They leave tomorrow. 
+----------------------------------------------+


---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org