Return-Path: Delivered-To: apmail-xml-cocoon-dev-archive@xml.apache.org Received: (qmail 32511 invoked by uid 500); 5 Feb 2002 00:35:32 -0000 Mailing-List: contact cocoon-dev-help@xml.apache.org; run by ezmlm Precedence: bulk list-help: list-unsubscribe: list-post: Reply-To: cocoon-dev@xml.apache.org Delivered-To: mailing list cocoon-dev@xml.apache.org Received: (qmail 32496 invoked from network); 5 Feb 2002 00:35:32 -0000 Message-ID: <3C5F28FD.1010801@netscape.net> Date: Tue, 05 Feb 2002 00:36:13 +0000 From: Peter Hargreaves User-Agent: Mozilla/5.0 (Windows; U; WinNT4.0; en-US; rv:0.9.2) Gecko/20010726 Netscape6/6.1 X-Accept-Language: en-us MIME-Version: 1.0 To: cocoon-dev@xml.apache.org Subject: Re: StoreJanitor new calculation [was: Re: Store Janitor Hangs System] References: Content-Type: multipart/mixed; boundary="------------080800070701050208060703" X-Mailer: Unknown (No Version) X-Spam-Rating: daedalus.apache.org 1.6.2 0/1000/N --------------080800070701050208060703 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Hi Gerhard, Great to hear that I've not been wasting my time :-) Please find attached the updated code. g-froehlich@gmx.de wrote: > Peter, > > >>From: Peter Hargreaves [mailto:pdhargr@netscape.net] >> >>Hi Gerhard, >> >>I've tested your latest Store Janitor improvements (using Optimizit and >>your debug messages) and although its a very great improvement, I've >>found some other problems :-( >> > > Damn... > > >>As my Optimizit trial was running out I decided to do some urgent work >>on the Checker and now have a solution which appears to work well :-) >> >>Here is a summary of the problems with ideas for correcting them (some >>are nitty gritty, and some you already know): >> >>*) When demand for memory is high, it is possible to run out of >>java.memory before the store has been emptied. Might it be possible to >>make sure the stores are empty before the memory can run out? >> > > Then we don't need the StoreJanitor, or? > > >>*) When memory is low, there is a risk of running out of it before the >>next check. Why not set the interval short enough to ensure there is no >>risk of running out of memory before the next check? >> > > Be careful with Threads. When they run in a to short interval they'll > kill your performance! > > >>*) Calling the gc once every check interval seriously undermines system >>performance because it can take several seconds to complete, which is >>comparable to the check interval. Why not call it only when some items >>are released from a store? Why not trust it and use its characteristics >>to advantage? >> > > That's a contradiction to previous point, I guess. On the one hand you want > to increase the the iteration of the Thread on the other hand you want to > decrease the GC calls. Ah, but only call the GC if stores are reduced - I think you got it lower down the page. > > >>*) When several instances of Cocoon are running (independent .WAR apps >>in the same servlet container), the calls to gc become very dominant. >>Would this affect the strategy for store-janitor settings? >> >>*) If the percent to reduce storage is set to 10%, it fails to remove >>any when the number of items are below 10. The number of items to be >>removed needs rounding upwards. Why not remove a fixed number of items >>instead of a percentage? (My idea and now I think it was wrong!!) >> > > Au, my embarrsing math capabilities strikes agein ;). > > >>*) It is difficult to get a good picture of what is happening by >>observing the debug output, so it is difficult to know if it is working >>and difficult use the debug output to optimise choice of janitor >>settings. Might it be possible to make this easy? >> > > Yes and no. That's always a problem in Cocoon. But you can define your > own log target (own file) in the logkit.xconf for the StoreJanitor. Then > you get rid of the noise. I suspected something like this might be possible - sounds great! > > >>The proposed, implemented and tested solution to the above works as follows: >> >>1) Monitor the rate at which memory is consumed during each check >>interval. Remember the biggest rate that has been found so far, to use >>it later. My system logs this at 2.5 megabytes per second. >> > > Understand. > > >>2) If heapIsBig() && freeIsLow(), then attempt to free storage. >> >>2.1) 'heapIsBig' means that the total memory has exceeded the parameter >>'maxheap'. >> >>2.2) 'freeIsLow' means the free memory is less than the parameter 'minfree'. >> >>2.3) Attempt to remove from the stores, the number of items specified by >>the parameter 'reduceby'. >> >>2.4) To free storage, start at the next store and remove items, moving >>to the next again if necessary, until the specified number of items have >>been removed or all the stores are empty. >> >>2.5) Then call the garbage collector, but only if some memory items were >>freed. >> > > Ok, that's nice. > > >>3) Sleep for an interval half that in which the memory could run out. >> >>3.1) If the remaining heap could be more than half filled during the >>interval specified by the 'maxsleep' parameter (at the max rate of >>memory consumption) then sleep for the time it would take to only half >>fill (freememory/maxrate)/2. Otherwise sleep for the max sleep interval. >> > > Damn, that's sounds great ;). Your biggest unknown variable is the > "could be". But I think taking the last biggest consumption is not that > bad. But how about taking the average of all calls. That would be more exact > then the last biggest value. You got straight to the weakest point didn't you? My fear is that a single rogue ridiculously high value might get stuck in there - I think that's possibly what you are getting at. Can't average all the rates, because you need only the periods where memory is trying to increase as fast as it can. Which ones are these? Could possibly, average any new highest value with the current highest. Or average two of the current one with the new one (i.e. give a greater weight to the current one). So, for the moment I've changed it to average the new highest rate with the old - instead of just replacing it. I'm still thinking about this one. > >>The effect of the store-janitor parameters in cocoon.xconf are now >>slightly changed, so I've changed the names of the parameters to reflect >>this. For the moment the names are changed only inside >>StoreJanitorImpl.java. If the solution were to be adopted then the names >>in cocoon.xconf would also change. The changes, with notes, are as follows: >> > > Naming was indeed crap. > > >>'maxheap' (was 'heapsize') The maximum total heap size below which >>stores are guaranteed to remain intact. >> >>'minfree' (was 'freememory') The minimum free heap required for stores >>to remain intact. I have found this should be set to less than 10% of >>avaliable memory, so that the jvm will respond to low memory by >>allocating more heap (if its available), before stores are likely to be >>reduced. Default value might be 2000000. Setting it to zero or very low >>will effectively disable store reduction. >> >>'maxsleep' (was 'cleanupthreadinterval') The maximum interval between >>checks. Should be short enough to ensure that 'rate of change' data is >>collected before free memory runs low. Suggest 5 secs as default. >> >>'reduceby' (was 'percent_to_free') is the number of items to be removed >>>from the stores, on each attempt to reduce the stores. Each successfull >>attempt to reduce stores results in a call to the garbage collector >>taking several seconds. This limits the rate a which the stores can be >>emptied. Removing several items at a time compensates for this. Using >>the debug output, choose a number of items which results in an increase >>of free memory between successive store reductions. Best tested with a >>sudden high demand for memory when the stores are full. In practice this >>setting is not critical. Suggested default is 10 items. Ideally >>'reduceby' would be in bytes! >> >>Setting 'minfree' and 'reduceby' both too low can intensify gc activity >>if sudden high demand requires all stores to be completely freed and can >>result in memory running out before all the stores are free. Verify >>using debug output. >> >>'priority' remains unchanged. >> >>The above solution is implemented and tested and it works very well >>indeed for me. The parameters do not seem to be at all critical - thank >>goodness! I see no reason why the default settings would need to be >>changed if more memory were available. I have attached the >>StoreJanitorImpl.java so others can test it. I find I can get a very >>good picture of what is happening by using my editor's repeat find >>command on the debug output in the core.log.000001 file. Optimizeit is >>no longer necessary to prove that it works effectively. >> > > Just define as mentioned above you own log target for the store.janitor. > Take a look into logkit.xconf in the WEB-INF dir! > > >>Beware! - there might still be things I don't know about that I have not >>take into account. >>Ideally - low memory should be detected by some sort of interrupt or >>exception rather than by polling. >> > > That's correct. But here java is a big white spot. > > >>If you are subsequently interested in committing these changes into >>Cocoon and you want me to do some more work to conform to design >>practice etc., then please let me know. >> > > Yep of course. But I switch to my picky mode now. Please reformat your > code to our coding standards. > - 4 indents, no tabs. > - if(foo) > this.isBad = true; > That's not good coding style. Use brackets for each condition or block. > - Make it more readable O.K. done that. It now makes a thrilling read! I'm not sure about a sequel though. > > Then I will commit your stuff into the Scratchpad area. > (That's the normal way this days) > Peter. --------------080800070701050208060703 Content-Type: text/plain; name="StoreJanitorImpl.java" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="StoreJanitorImpl.java" /***************************************************************************** * Copyright (C) The Apache Software Foundation. All rights reserved. * * ------------------------------------------------------------------------- * * This software is published under the terms of the Apache Software License * * version 1.1, a copy of which has been included with this distribution in * * the LICENSE file. * *****************************************************************************/ package org.apache.cocoon.components.store; import org.apache.avalon.framework.activity.Startable; import org.apache.avalon.framework.configuration.Configurable; import org.apache.avalon.framework.configuration.Configuration; import org.apache.avalon.framework.configuration.ConfigurationException; import org.apache.avalon.framework.logger.AbstractLoggable; import org.apache.avalon.framework.parameters.Parameters; import org.apache.avalon.framework.thread.ThreadSafe; import java.util.ArrayList; import java.util.Enumeration; import java.util.Iterator; /** * This class is a implentation of a StoreJanitor. Store classes * can register to the StoreJanitor. When memory is too low, * the StoreJanitor frees the registered caches until memory is normal. * * @author Christian Schmitt * @author Gerhard Froehlich * @author Peter Royal * @author Peter Hargreaves */ public class StoreJanitorImpl extends AbstractLoggable implements StoreJanitor, Configurable, ThreadSafe, Runnable, Startable { private int freememory = -1; private int heapsize = -1; private int cleanupthreadinterval = -1; private int priority = -1; private Runtime jvm; private ArrayList storelist; private int index = -1; private static boolean doRun = false; private int m_percent; /** * Initialize the StoreJanitorImpl. * A few options can be used : *
    *
  • freememory = how many bytes shall be always free in the jvm
    • *
  • heapsize = max. size of jvm memory consumption
    • *
  • cleanupthreadinterval = how often (sec) shall run the cleanup thread
    • *
  • threadpriority = priority of the thread (1-10). (Default: 10)
    • *
* * @param the Configuration of the application * @exception ConfigurationException */ public void configure(Configuration conf) throws ConfigurationException { if (this.getLogger().isDebugEnabled()) { this.getLogger().debug("Configure StoreJanitorImpl"); } this.setJVM(Runtime.getRuntime()); Parameters params = Parameters.fromConfiguration(conf); this.setFreememory(params.getParameterAsInteger("freememory",1000000)); this.setHeapsize(params.getParameterAsInteger("heapsize",60000000)); this.setCleanupthreadinterval(params.getParameterAsInteger("cleanupthreadinterval",10)); this.setPriority(params.getParameterAsInteger( "threadpriority", Thread.currentThread().getPriority())); this.m_percent = params.getParameterAsInteger( "percent_to_free",10); if ((this.getFreememory() < 1)) { throw new ConfigurationException("StoreJanitorImpl freememory parameter has to be greater then 1"); } if ((this.getHeapsize() < 1)) { throw new ConfigurationException("StoreJanitorImpl heapsize parameter has to be greater then 1"); } if ((this.getCleanupthreadinterval() < 1)) { throw new ConfigurationException("StoreJanitorImpl cleanupthreadinterval parameter has to be greater then 1"); } if ((this.getPriority() < 1)) { throw new ConfigurationException("StoreJanitorImpl threadpriority has to be greater then 1"); } if ((this.m_percent > 100 && this.m_percent < 1)) { throw new ConfigurationException("StoreJanitorImpl percent_to_free, has to be between 1 and 100"); } this.setStoreList(new ArrayList()); } public void start() { doRun = true; Thread checker = new Thread(this); if (this.getLogger().isDebugEnabled()) { this.getLogger().debug("Intializing checker thread"); } checker.setPriority(this.getPriority()); checker.setDaemon(true); checker.setName("checker"); checker.start(); } public void stop() { doRun = false; } private long inUseAfter; // Remember while sleeping. private long sleepPeriod; // Remember while sleeping. private long maxRateOfChange; // Remember while sleeping. /** * The "checker" thread checks if items should be removed from the stores. */ public void run() { // Initialise for a safe first calculation of rate of change. inUseAfter = inUseNow(); sleepPeriod = Long.MAX_VALUE; maxRateOfChange = 1; while (doRun) { // Monitor the rate of change of heap in use. long changeWhileSleeping = inUseNow() - inUseAfter; long rateOfChange = longDiv(changeWhileSleeping, sleepPeriod); // bpms (same as or kbps). if (maxRateOfChange < rateOfChange) { maxRateOfChange = (maxRateOfChange + rateOfChange)/2; } // Output debug info. debug("Waking after " + sleepPeriod + "ms, in use change " + changeWhileSleeping + " to " + inUseNow() + ", rate " + rateOfChange + "kb/sec, max rate " + maxRateOfChange + "kb/sec"); debug("maxHeap=" + getMaxHeap() + ", totalHeap=" + jvm.totalMemory() + ", heapIsBig=" + heapIsBig()); debug("minFree=" + getMinFree() + ", freeHeap=" + jvm.freeMemory() + ", freeIsLow=" + freeIsLow()); // If the heap is big, and the free memory is low. if (heapIsBig() && freeIsLow()) { synchronized (this) { attemptToFreeStorage(); } } // Remember memory in use before sleeping in order to calc slope when waking. inUseAfter = inUseNow(); // If time to half fill could be less than max sleep, then sleep for half min time to fill (& remember it to calc slope next time). sleepPeriod = minTimeToFill()/2 < getMaxSleep() ? minTimeToFill()/2 : getMaxSleep(); debug("Store checker going to sleep for " + sleepPeriod + "ms, (max sleep=" + getMaxSleep() + "ms), with memory in use=" + inUseAfter); try { Thread.currentThread().sleep(sleepPeriod); } catch (InterruptedException ignore) {} } } /** * Starting at the next store, removes items, moving to the next again if necessary, until the specified number of items have been removed or all the stores are empty. */ private void attemptToFreeStorage() { int storeListSize = getStoreList().size(); int remove = getReduceBy(); debug("number of stores is " + storeListSize + ", number of items to be removed is " + remove + ", if possible!"); incIndex(); for (int cnt = 0; cnt < storeListSize; incIndex(), cnt++) { // Look in all stores if necessary. if ((remove = reduceStoreBy(remove)) == 0 ) { break; // Keep looking till all items removed, } } // or all stores are empty. if ( remove < getReduceBy() ) { // If items were removed call garbage collector. long gcTime = System.currentTimeMillis(); jvm.gc(); gcTime = System.currentTimeMillis() - gcTime; debug("items removed, so collecting garbage - took " + gcTime + "ms"); debug("minFree=" + getMinFree() + ", freeHeap=" + jvm.freeMemory() + ", freeIsLow=" + freeIsLow()); } } /** * Increment the store index. * * @return the incremented index. */ private void incIndex() { if (++index >= getStoreList().size()) { index = 0; } } /** * Reduce the current store by the number of items specified, if possible. * * @param remove The number of items to be removed. * @return the remaining count of items, that could not be removed. */ private int reduceStoreBy(int remove) { Store store = (Store)storelist.get(index); int sizeBefore = countSize(store); for (int i = 0; i < sizeBefore & remove > 0; i++, remove--) { store.free(); } int sizeAfter = countSize(store); debug("store index=" + index + ", size before=" + sizeBefore + ", size after=" + sizeAfter + ", removed=" + (sizeBefore - sizeAfter)); return remove; } /** * To check if total memory is big enough to allow stores to be reduced. * * @return true if big enough. */ private boolean heapIsBig() { return jvm.totalMemory() > getMaxHeap(); } /** * To check if free memory is small enough to start reducing stores. * * @return true if small enough. */ private boolean freeIsLow() { return jvm.freeMemory() < getMinFree(); } /** * To calculate the minimum time in which the memory could be filled at the maximum rate of use. * * @return the minimum time to fill. */ private long minTimeToFill() { return longDiv(jvm.freeMemory(), maxRateOfChange); } /** * Long division, guarding agains accidental divide by zero. * * @return the result of division. */ private long longDiv(long top, long bottom) { try { return top / bottom; } catch (Exception e) { return top > 0 ? Long.MAX_VALUE : Long.MIN_VALUE; } } /** * Calculate the jvm memory in use now. * * @return memory in use. */ private long inUseNow() { return jvm.totalMemory() - jvm.freeMemory(); } /** * Count the size of a store. * * @return the size of the store. */ private int countSize(Store store) { int size = 0; Enumeration enum = store.keys(); while(enum.hasMoreElements()) { size++; enum.nextElement(); } return size; } // Renaming of parameters due to new functionality. private int getMaxHeap() { return heapsize; } private int getMinFree() { return freememory; } private int getMaxSleep() { return cleanupthreadinterval*1000; } private int getReduceBy() { return m_percent; } /** * Shorten the call to print a debug message. */ private void debug(String message) { if (this.getLogger().isDebugEnabled()) { this.getLogger().debug(message); } } /** * This method register the stores * * @param the store to be registered */ public void register(Store store) { this.getStoreList().add(store); if (this.getLogger().isDebugEnabled()) { this.getLogger().debug("Registering store instance"); this.getLogger().debug("Size of StoreJanitor now:" + this.getStoreList().size()); } } /** * This method unregister the stores * * @param the store to be unregistered */ public void unregister(Store store) { this.getStoreList().remove(store); if (this.getLogger().isDebugEnabled()) { this.getLogger().debug("Unregister store instance"); this.getLogger().debug("Size of StoreJanitor now:" + this.getStoreList().size()); } } /** * This method return a java.util.Iterator of every registered stores * * The iterators returned is fail-fast: if list is structurally * modified at any time after the iterator is created, in any way, the * iterator will throw a ConcurrentModificationException. Thus, in the * face of concurrent modification, the iterator fails quickly and * cleanly, rather than risking arbitrary, non-deterministic behavior at * an undetermined time in the future. * * @return a java.util.Iterator */ public Iterator iterator() { return this.getStoreList().iterator(); } private int getFreememory() { return freememory; } private void setFreememory(int _freememory) { this.freememory = _freememory; } private int getHeapsize() { return this.heapsize; } private void setHeapsize(int _heapsize) { this.heapsize = _heapsize; } private int getCleanupthreadinterval() { return this.cleanupthreadinterval; } private void setCleanupthreadinterval(int _cleanupthreadinterval) { this.cleanupthreadinterval = _cleanupthreadinterval; } private int getPriority() { return this.priority; } private void setPriority(int _priority) { this.priority = _priority; } private Runtime getJVM() { return this.jvm; } private void setJVM(Runtime _runtime) { this.jvm = _runtime; } private ArrayList getStoreList() { return this.storelist; } private void setStoreList(ArrayList _storelist) { this.storelist = _storelist; } } --------------080800070701050208060703 Content-Type: text/plain; charset=us-ascii --------------------------------------------------------------------- To unsubscribe, e-mail: cocoon-dev-unsubscribe@xml.apache.org For additional commands, email: cocoon-dev-help@xml.apache.org --------------080800070701050208060703--