cocoon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Hargreaves <pdha...@netscape.net>
Subject Re: StoreJanitor new calculation [was: Re: Store Janitor Hangs System]
Date Tue, 05 Feb 2002 00:36:13 GMT
Hi Gerhard,

Great to hear that I've not been wasting my time :-)

Please find attached the updated code.

g-froehlich@gmx.de wrote:

> Peter,
> 
> 
>>From: Peter Hargreaves [mailto:pdhargr@netscape.net]
>>
>>Hi Gerhard,
>>
>>I've tested your latest Store Janitor improvements (using Optimizit and 
>>your debug messages) and although its a very great improvement, I've 
>>found some other problems :-(
>>
> 
> Damn...
> 
> 
>>As my Optimizit trial was running out I decided to do some urgent work 
>>on the Checker and now have a solution which appears to work well :-)
>>
>>Here is a summary of the problems with ideas for correcting them (some 
>>are nitty gritty, and some you already know):
>>
>>*) When demand for memory is high, it is possible to run out of 
>>java.memory before the store has been emptied. Might it be possible to 
>>make sure the stores are empty before the memory can run out?
>>
> 
> Then we don't need the StoreJanitor, or?
> 
> 
>>*) When memory is low, there is a risk of running out of it before the 
>>next check. Why not set the interval short enough to ensure there is no 
>>risk of running out of memory before the next check?
>>
> 
> Be careful with Threads. When they run in a to short interval they'll
> kill your performance!
> 
> 
>>*) Calling the gc once every check interval seriously undermines system 
>>performance because it can take several seconds to complete, which is 
>>comparable to the check interval. Why not call it only when some items 
>>are released from a store? Why not trust it and use its characteristics 
>>to advantage?
>>
> 
> That's a contradiction to previous point, I guess. On the one hand you want
> to increase the the iteration of the Thread on the other hand you want to
> decrease the GC calls.


Ah, but only call the GC if stores are reduced - I think you got it 
lower down the page.


> 
> 
>>*) When several instances of Cocoon are running (independent .WAR apps 
>>in the same servlet container), the calls to gc become very dominant. 
>>Would this affect the strategy for store-janitor settings?
>>
>>*) If the percent to reduce storage is set to 10%, it fails to remove 
>>any when the number of items are below 10. The number of items to be 
>>removed needs rounding upwards. Why not remove a fixed number of items 
>>instead of a percentage? (My idea and now I think it was wrong!!)
>>
> 
> Au, my embarrsing math capabilities strikes agein ;).
> 
> 
>>*) It is difficult to get a good picture of what is happening by 
>>observing the debug output, so it is difficult to know if it is working 
>>and difficult use the debug output to optimise choice of janitor 
>>settings. Might it be possible to make this easy?
>>
> 
> Yes and no. That's always a problem in Cocoon. But you can define your
> own log target (own file) in the logkit.xconf for the StoreJanitor. Then
> you get rid of the noise.


I suspected something like this might be possible - sounds great!


> 
> 
>>The proposed, implemented and tested solution to the above works as follows:
>>
>>1) Monitor the rate at which memory is consumed during each check 
>>interval. Remember the biggest rate that has been found so far, to use 
>>it later. My system logs this at 2.5 megabytes per second.
>>
> 
> Understand.
> 
> 
>>2) If heapIsBig() && freeIsLow(), then attempt to free storage.
>>
>>2.1) 'heapIsBig' means that the total memory has exceeded the parameter 
>>'maxheap'.
>>
>>2.2) 'freeIsLow' means the free memory is less than the parameter 'minfree'.
>>
>>2.3) Attempt to remove from the stores, the number of items specified by 
>>the parameter 'reduceby'.
>>
>>2.4) To free storage, start at the next store and remove items, moving 
>>to the next again if necessary, until the specified number of items have 
>>been removed or all the stores are empty.
>>
>>2.5) Then call the garbage collector, but only if some memory items were 
>>freed.
>>
> 
> Ok, that's nice.
> 
> 
>>3) Sleep for an interval half that in which the memory could run out.
>>
>>3.1) If the remaining heap could be more than half filled during the 
>>interval specified by the 'maxsleep' parameter (at the max rate of 
>>memory consumption) then sleep for the time it would take to only half 
>>fill (freememory/maxrate)/2.  Otherwise sleep for the max sleep interval.
>>
> 
> Damn, that's sounds great ;). Your biggest unknown variable is the 
> "could be". But I think taking the last biggest consumption is not that
> bad. But how about taking the average of all calls. That would be more exact
> then the last biggest value.


You got straight to the weakest point didn't you? My fear is that a 
single rogue ridiculously high value might get stuck in there - I think 
that's possibly what you are getting at.

Can't average all the rates, because you need only the periods where 
memory is trying to increase as fast as it can. Which ones are these?

Could possibly, average any new highest value with the current highest. 
Or average two of the current one with the new one (i.e. give a greater 
weight to the current one).

So, for the moment I've changed it to average the new highest rate with 
the old - instead of just replacing it. I'm still thinking about this one.

 
> 
>>The effect of the store-janitor parameters in cocoon.xconf are now 
>>slightly changed, so I've changed the names of the parameters to reflect 
>>this. For the moment the names are changed only inside 
>>StoreJanitorImpl.java. If the solution were to be adopted then the names 
>>in cocoon.xconf would also change. The changes, with notes, are as follows:
>>
> 
> Naming was indeed crap.
> 
> 
>>'maxheap' (was 'heapsize')  The maximum total heap size below which 
>>stores are guaranteed to remain intact.
>>
>>'minfree' (was 'freememory') The minimum free heap required for stores 
>>to remain intact. I have found this should be set to less than 10% of 
>>avaliable memory, so that the jvm will respond to low memory by 
>>allocating more heap (if its available), before stores are likely to be 
>>reduced. Default value might be 2000000. Setting it to zero or very low 
>>will effectively disable store reduction.
>>
>>'maxsleep' (was 'cleanupthreadinterval') The maximum interval between 
>>checks. Should be short enough to ensure that 'rate of change' data is 
>>collected before free memory runs low. Suggest 5 secs as default.
>>
>>'reduceby' (was 'percent_to_free') is the number of items to be removed 
>>>from the stores, on each attempt to reduce the stores. Each  successfull 
>>attempt to reduce stores results in a call to the garbage collector 
>>taking several seconds. This limits the rate a which the stores can be 
>>emptied. Removing several items at a time compensates for this. Using 
>>the debug output, choose a number of items which results in an increase 
>>of free memory between successive store reductions. Best tested with a 
>>sudden high demand for memory when the stores are full. In practice this 
>>setting is not critical. Suggested default is 10 items. Ideally 
>>'reduceby' would be in bytes!
>>
>>Setting 'minfree' and 'reduceby' both too low can intensify gc activity 
>>if sudden high demand requires all stores to be completely freed and can 
>>result in memory running out before all the stores are free. Verify 
>>using debug output.
>>
>>'priority' remains unchanged.
>>
>>The above solution is implemented and tested and it works very well 
>>indeed for me. The parameters do not seem to be at all critical - thank 
>>goodness! I see no reason why the default settings would need to be 
>>changed if more memory were available. I have attached the 
>>StoreJanitorImpl.java so others can test it. I find I can get a very 
>>good picture of what is happening by using my editor's repeat find 
>>command on the debug output in the core.log.000001 file. Optimizeit is 
>>no longer necessary to prove that it works effectively.
>>
> 
> Just define as mentioned above you own log target for the store.janitor.
> Take a look into logkit.xconf in the WEB-INF dir!
> 
> 
>>Beware! - there might still be things I don't know about that I have not 
>>take into account.
>>Ideally - low memory should be detected by some sort of interrupt or 
>>exception rather than by polling.
>>
> 
> That's correct. But here java is a big white spot.
> 
> 
>>If you are subsequently interested in committing these changes into 
>>Cocoon and you want me to do some more work to conform to design 
>>practice etc., then please let me know.
>>
> 
> Yep of course. But I switch to my picky mode now. Please reformat your
> code to our coding standards.
> - 4 indents, no tabs.
> - if(foo)
>      this.isBad = true;
> That's not good coding style. Use brackets for each condition or block.
> - Make it more readable


O.K. done that. It now makes a thrilling read! I'm not sure about a 
sequel though.


> 
> Then I will commit your stuff into the Scratchpad area.
> (That's the normal way this days)
> 


Peter.

Mime
View raw message