cocoon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gerhard Froehlich" <g-froehl...@gmx.de>
Subject RE: StoreJanitor new calculation [was: Re: Store Janitor Hangs System]
Date Thu, 07 Feb 2002 23:40:49 GMT
Hi,
applied to the scratchpad. Own config is missing.
I have to go to bed now ;)...

TIA
  ~Gerhard
 
"Sorry, but my karma just ran over your dogma."

>-----Original Message-----
>From: Peter Hargreaves [mailto:pdhargr@netscape.net]
>Sent: Tuesday, February 05, 2002 1:36 AM
>To: cocoon-dev@xml.apache.org
>Subject: Re: StoreJanitor new calculation [was: Re: Store Janitor Hangs
>System]
>
>
>Hi Gerhard,
>
>Great to hear that I've not been wasting my time :-)
>
>Please find attached the updated code.
>
>g-froehlich@gmx.de wrote:
>
>> Peter,
>> 
>> 
>>>From: Peter Hargreaves [mailto:pdhargr@netscape.net]
>>>
>>>Hi Gerhard,
>>>
>>>I've tested your latest Store Janitor improvements (using Optimizit and 
>>>your debug messages) and although its a very great improvement, I've 
>>>found some other problems :-(
>>>
>> 
>> Damn...
>> 
>> 
>>>As my Optimizit trial was running out I decided to do some urgent work 
>>>on the Checker and now have a solution which appears to work well :-)
>>>
>>>Here is a summary of the problems with ideas for correcting them (some 
>>>are nitty gritty, and some you already know):
>>>
>>>*) When demand for memory is high, it is possible to run out of 
>>>java.memory before the store has been emptied. Might it be possible to 
>>>make sure the stores are empty before the memory can run out?
>>>
>> 
>> Then we don't need the StoreJanitor, or?
>> 
>> 
>>>*) When memory is low, there is a risk of running out of it before the 
>>>next check. Why not set the interval short enough to ensure there is no 
>>>risk of running out of memory before the next check?
>>>
>> 
>> Be careful with Threads. When they run in a to short interval they'll
>> kill your performance!
>> 
>> 
>>>*) Calling the gc once every check interval seriously undermines system 
>>>performance because it can take several seconds to complete, which is 
>>>comparable to the check interval. Why not call it only when some items 
>>>are released from a store? Why not trust it and use its characteristics 
>>>to advantage?
>>>
>> 
>> That's a contradiction to previous point, I guess. On the one hand you want
>> to increase the the iteration of the Thread on the other hand you want to
>> decrease the GC calls.
>
>
>Ah, but only call the GC if stores are reduced - I think you got it 
>lower down the page.
>
>
>> 
>> 
>>>*) When several instances of Cocoon are running (independent .WAR apps 
>>>in the same servlet container), the calls to gc become very dominant. 
>>>Would this affect the strategy for store-janitor settings?
>>>
>>>*) If the percent to reduce storage is set to 10%, it fails to remove 
>>>any when the number of items are below 10. The number of items to be 
>>>removed needs rounding upwards. Why not remove a fixed number of items 
>>>instead of a percentage? (My idea and now I think it was wrong!!)
>>>
>> 
>> Au, my embarrsing math capabilities strikes agein ;).
>> 
>> 
>>>*) It is difficult to get a good picture of what is happening by 
>>>observing the debug output, so it is difficult to know if it is working 
>>>and difficult use the debug output to optimise choice of janitor 
>>>settings. Might it be possible to make this easy?
>>>
>> 
>> Yes and no. That's always a problem in Cocoon. But you can define your
>> own log target (own file) in the logkit.xconf for the StoreJanitor. Then
>> you get rid of the noise.
>
>
>I suspected something like this might be possible - sounds great!
>
>
>> 
>> 
>>>The proposed, implemented and tested solution to the above works as follows:
>>>
>>>1) Monitor the rate at which memory is consumed during each check 
>>>interval. Remember the biggest rate that has been found so far, to use 
>>>it later. My system logs this at 2.5 megabytes per second.
>>>
>> 
>> Understand.
>> 
>> 
>>>2) If heapIsBig() && freeIsLow(), then attempt to free storage.
>>>
>>>2.1) 'heapIsBig' means that the total memory has exceeded the parameter 
>>>'maxheap'.
>>>
>>>2.2) 'freeIsLow' means the free memory is less than the parameter 'minfree'.
>>>
>>>2.3) Attempt to remove from the stores, the number of items specified by 
>>>the parameter 'reduceby'.
>>>
>>>2.4) To free storage, start at the next store and remove items, moving 
>>>to the next again if necessary, until the specified number of items have 
>>>been removed or all the stores are empty.
>>>
>>>2.5) Then call the garbage collector, but only if some memory items were 
>>>freed.
>>>
>> 
>> Ok, that's nice.
>> 
>> 
>>>3) Sleep for an interval half that in which the memory could run out.
>>>
>>>3.1) If the remaining heap could be more than half filled during the 
>>>interval specified by the 'maxsleep' parameter (at the max rate of 
>>>memory consumption) then sleep for the time it would take to only half 
>>>fill (freememory/maxrate)/2.  Otherwise sleep for the max sleep interval.
>>>
>> 
>> Damn, that's sounds great ;). Your biggest unknown variable is the 
>> "could be". But I think taking the last biggest consumption is not that
>> bad. But how about taking the average of all calls. That would be more exact
>> then the last biggest value.
>
>
>You got straight to the weakest point didn't you? My fear is that a 
>single rogue ridiculously high value might get stuck in there - I think 
>that's possibly what you are getting at.
>
>Can't average all the rates, because you need only the periods where 
>memory is trying to increase as fast as it can. Which ones are these?
>
>Could possibly, average any new highest value with the current highest. 
>Or average two of the current one with the new one (i.e. give a greater 
>weight to the current one).
>
>So, for the moment I've changed it to average the new highest rate with 
>the old - instead of just replacing it. I'm still thinking about this one.
>
> 
>> 
>>>The effect of the store-janitor parameters in cocoon.xconf are now 
>>>slightly changed, so I've changed the names of the parameters to reflect 
>>>this. For the moment the names are changed only inside 
>>>StoreJanitorImpl.java. If the solution were to be adopted then the names 
>>>in cocoon.xconf would also change. The changes, with notes, are as follows:
>>>
>> 
>> Naming was indeed crap.
>> 
>> 
>>>'maxheap' (was 'heapsize')  The maximum total heap size below which 
>>>stores are guaranteed to remain intact.
>>>
>>>'minfree' (was 'freememory') The minimum free heap required for stores 
>>>to remain intact. I have found this should be set to less than 10% of 
>>>avaliable memory, so that the jvm will respond to low memory by 
>>>allocating more heap (if its available), before stores are likely to be 
>>>reduced. Default value might be 2000000. Setting it to zero or very low 
>>>will effectively disable store reduction.
>>>
>>>'maxsleep' (was 'cleanupthreadinterval') The maximum interval between 
>>>checks. Should be short enough to ensure that 'rate of change' data is 
>>>collected before free memory runs low. Suggest 5 secs as default.
>>>
>>>'reduceby' (was 'percent_to_free') is the number of items to be removed 
>>>>>from the stores, on each attempt to reduce the stores. Each  successfull

>>>attempt to reduce stores results in a call to the garbage collector 
>>>taking several seconds. This limits the rate a which the stores can be 
>>>emptied. Removing several items at a time compensates for this. Using 
>>>the debug output, choose a number of items which results in an increase 
>>>of free memory between successive store reductions. Best tested with a 
>>>sudden high demand for memory when the stores are full. In practice this 
>>>setting is not critical. Suggested default is 10 items. Ideally 
>>>'reduceby' would be in bytes!
>>>
>>>Setting 'minfree' and 'reduceby' both too low can intensify gc activity 
>>>if sudden high demand requires all stores to be completely freed and can 
>>>result in memory running out before all the stores are free. Verify 
>>>using debug output.
>>>
>>>'priority' remains unchanged.
>>>
>>>The above solution is implemented and tested and it works very well 
>>>indeed for me. The parameters do not seem to be at all critical - thank 
>>>goodness! I see no reason why the default settings would need to be 
>>>changed if more memory were available. I have attached the 
>>>StoreJanitorImpl.java so others can test it. I find I can get a very 
>>>good picture of what is happening by using my editor's repeat find 
>>>command on the debug output in the core.log.000001 file. Optimizeit is 
>>>no longer necessary to prove that it works effectively.
>>>
>> 
>> Just define as mentioned above you own log target for the store.janitor.
>> Take a look into logkit.xconf in the WEB-INF dir!
>> 
>> 
>>>Beware! - there might still be things I don't know about that I have not 
>>>take into account.
>>>Ideally - low memory should be detected by some sort of interrupt or 
>>>exception rather than by polling.
>>>
>> 
>> That's correct. But here java is a big white spot.
>> 
>> 
>>>If you are subsequently interested in committing these changes into 
>>>Cocoon and you want me to do some more work to conform to design 
>>>practice etc., then please let me know.
>>>
>> 
>> Yep of course. But I switch to my picky mode now. Please reformat your
>> code to our coding standards.
>> - 4 indents, no tabs.
>> - if(foo)
>>      this.isBad = true;
>> That's not good coding style. Use brackets for each condition or block.
>> - Make it more readable
>
>
>O.K. done that. It now makes a thrilling read! I'm not sure about a 
>sequel though.
>
>
>> 
>> Then I will commit your stuff into the Scratchpad area.
>> (That's the normal way this days)
>> 
>
>
>Peter.
>


---------------------------------------------------------------------
To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org
For additional commands, email: cocoon-dev-help@xml.apache.org


Mime
View raw message