Return-Path: Delivered-To: apmail-jakarta-jcs-dev-archive@www.apache.org Received: (qmail 31353 invoked from network); 19 Aug 2006 23:24:17 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur.apache.org with SMTP; 19 Aug 2006 23:24:17 -0000 Received: (qmail 32443 invoked by uid 500); 19 Aug 2006 23:24:17 -0000 Delivered-To: apmail-jakarta-jcs-dev-archive@jakarta.apache.org Received: (qmail 32363 invoked by uid 500); 19 Aug 2006 23:24:17 -0000 Mailing-List: contact jcs-dev-help@jakarta.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "JCS Developers List" Delivered-To: mailing list jcs-dev@jakarta.apache.org Received: (qmail 32352 invoked by uid 500); 19 Aug 2006 23:24:17 -0000 Delivered-To: apmail-jakarta-jcs-commits@jakarta.apache.org Received: (qmail 32349 invoked by uid 500); 19 Aug 2006 23:24:17 -0000 Delivered-To: apmail-jakarta-jcs-cvs@jakarta.apache.org Received: (qmail 32346 invoked by uid 99); 19 Aug 2006 23:24:17 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 Aug 2006 16:24:17 -0700 X-ASF-Spam-Status: No, hits=-9.4 required=10.0 tests=ALL_TRUSTED,NO_REAL_NAME X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: local policy) Received: from [140.211.166.113] (HELO eris.apache.org) (140.211.166.113) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 Aug 2006 16:24:15 -0700 Received: by eris.apache.org (Postfix, from userid 65534) id 361531A981A; Sat, 19 Aug 2006 16:23:55 -0700 (PDT) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r432898 - in /jakarta/jcs/trunk/src: java/org/apache/jcs/auxiliary/disk/ java/org/apache/jcs/auxiliary/disk/indexed/ java/org/apache/jcs/utils/struct/ test-conf/ test/org/apache/jcs/auxiliary/disk/indexed/ test/org/apache/jcs/utils/struct/ Date: Sat, 19 Aug 2006 23:23:53 -0000 To: jcs-cvs@jakarta.apache.org From: asmuts@apache.org X-Mailer: svnmailer-1.0.8 Message-Id: <20060819232355.361531A981A@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Author: asmuts Date: Sat Aug 19 16:23:52 2006 New Revision: 432898 URL: http://svn.apache.org/viewvc?rev=432898&view=rev Log: The optimization patch broke the recyle bin. I fixed it and added two new unit tests to verify that it is being used properly. I added a bytes free counter and unit tests to verify that is works properly. I added the counter value to the stats. Eventually the optimization should be driven by this number. I added a new load test for sustained testing of the disk cache. I added a new config property called OptimizeOnShutdown which allows you to turn off the shutdown optimization. I added two new simple unit tests for the sorted pref array. I made some minor formatting changes. Added: jakarta/jcs/trunk/src/test-conf/TestDiskCacheSteadyLoad.ccf jakarta/jcs/trunk/src/test/org/apache/jcs/auxiliary/disk/indexed/IndexedDiskCacheSteadyLoadTest.java Modified: jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/AbstractDiskCache.java jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/LRUMapJCS.java jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/indexed/IndexedDiskCache.java jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/indexed/IndexedDiskCacheAttributes.java jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/indexed/IndexedDiskElementDescriptor.java jakarta/jcs/trunk/src/java/org/apache/jcs/utils/struct/LRUMap.java jakarta/jcs/trunk/src/test-conf/TestDiskCacheDefragPerformance.ccf jakarta/jcs/trunk/src/test/org/apache/jcs/auxiliary/disk/indexed/IndexDiskCacheUnitTest.java jakarta/jcs/trunk/src/test/org/apache/jcs/utils/struct/SortedPrefArrayUnitTest.java Modified: jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/AbstractDiskCache.java URL: http://svn.apache.org/viewvc/jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/AbstractDiskCache.java?rev=432898&r1=432897&r2=432898&view=diff ============================================================================== --- jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/AbstractDiskCache.java (original) +++ jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/AbstractDiskCache.java Sat Aug 19 16:23:52 2006 @@ -94,8 +94,8 @@ */ protected int purgHits = 0; - // we lock here, so that we cannot get an update after a remove all - // an individual removeal locks the item. + // we lock here, so that we cannot get an update after a remove all. + // an individual removal locks the item. private WriterPreferenceReadWriteLock removeAllLock = new WriterPreferenceReadWriteLock(); // ----------------------------------------------------------- constructors Modified: jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/LRUMapJCS.java URL: http://svn.apache.org/viewvc/jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/LRUMapJCS.java?rev=432898&r1=432897&r2=432898&view=diff ============================================================================== --- jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/LRUMapJCS.java (original) +++ jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/LRUMapJCS.java Sat Aug 19 16:23:52 2006 @@ -53,6 +53,5 @@ log.debug( "Removing key [" + key + "] from key store, value [" + value + "]" ); log.debug( "Key store size [" + this.size() + "]" ); } - } } Modified: jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/indexed/IndexedDiskCache.java URL: http://svn.apache.org/viewvc/jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/indexed/IndexedDiskCache.java?rev=432898&r1=432897&r2=432898&view=diff ============================================================================== --- jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/indexed/IndexedDiskCache.java (original) +++ jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/indexed/IndexedDiskCache.java Sat Aug 19 16:23:52 2006 @@ -97,6 +97,9 @@ private int startupSize = 0; + // the number of bytes free on disk. + private long bytesFree = 0; + /** * Use this lock to synchronize reads and writes to the underlying storage mechansism. */ @@ -112,19 +115,18 @@ super( cattr ); String rootDirName = cattr.getDiskPath(); - maxKeySize = cattr.getMaxKeySize(); + this.maxKeySize = cattr.getMaxKeySize(); - // TODO: These should be separate configuration attributes - isRealTimeOptimizationEnabled = cattr.getOptimizeAtRemoveCount() > 0; - isShutdownOptimizationEnabled = cattr.getOptimizeAtRemoveCount() >= 0; + this.isRealTimeOptimizationEnabled = cattr.getOptimizeAtRemoveCount() > 0; + this.isShutdownOptimizationEnabled = cattr.isOptimizeOnShutdown(); this.cattr = cattr; this.logCacheName = "Region [" + getCacheName() + "] "; this.fileName = getCacheName(); - rafDir = new File( rootDirName ); - rafDir.mkdirs(); + this.rafDir = new File( rootDirName ); + this.rafDir.mkdirs(); if ( log.isInfoEnabled() ) { @@ -133,9 +135,9 @@ try { - dataFile = new IndexedDisk( new File( rafDir, fileName + ".data" ) ); + this.dataFile = new IndexedDisk( new File( rafDir, fileName + ".data" ) ); - keyFile = new IndexedDisk( new File( rafDir, fileName + ".key" ) ); + this.keyFile = new IndexedDisk( new File( rafDir, fileName + ".key" ) ); // If the key file has contents, try to initialize the keys // from it. In no keys are loaded reset the data file. @@ -429,6 +431,9 @@ } else { + // we need this to compare in the recycle bin + ded = new IndexedDiskElementDescriptor( dataFile.length(), data.length ); + if ( doRecycle ) { IndexedDiskElementDescriptor rep = (IndexedDiskElementDescriptor) recycle @@ -438,6 +443,7 @@ ded = rep; ded.len = data.length; recycleCnt++; + this.adjustBytesFree( ded, false ); if ( log.isDebugEnabled() ) { log.debug( logCacheName + "using recycled ded " + ded.pos + " rep.len = " + rep.len @@ -446,12 +452,6 @@ } } - // We couldn't replace or update anything so it is ok to create a new one. - if ( ded == null ) - { - ded = new IndexedDiskElementDescriptor( dataFile.length(), data.length ); - } - // Put it in the map keyHash.put( ce.getKey(), ded ); @@ -653,6 +653,7 @@ addToRecycleBin( ded ); iter.remove(); removed = true; + // TODO this needs to update the rmove count separately } } } @@ -703,8 +704,12 @@ reset(); } - // this increments the removecount - doOptimizeRealTime(); + // this increments the removecount. + // there is no reason to call this if an item was not removed. + if ( removed ) + { + doOptimizeRealTime(); + } return removed; } @@ -882,7 +887,10 @@ try { - log.debug( logCacheName + "Closing files, base filename: " + fileName ); + if ( log.isDebugEnabled() ) + { + log.debug( logCacheName + "Closing files, base filename: " + fileName ); + } dataFile.close(); dataFile = null; keyFile.close(); @@ -900,22 +908,27 @@ } /** - * Add descfriptor to recycle bin. + * Add descriptor to recycle bin if it is not null. Adds the length of the item to the bytes + * free. *

* @param ded */ private void addToRecycleBin( IndexedDiskElementDescriptor ded ) { - if ( doRecycle ) + // reuse the spot + if ( ded != null ) { - // reuse the spot - if ( ded != null ) + this.adjustBytesFree( ded, true ); + + if ( doRecycle ) { + recycle.add( ded ); if ( log.isDebugEnabled() ) { log.debug( logCacheName + "recycled ded" + ded ); } + } } } @@ -1047,6 +1060,7 @@ { // RESTORE NORMAL OPERATION removeCount = 0; + bytesFree = 0; initRecycleBin(); queuedPutList.clear(); queueInput = false; @@ -1161,6 +1175,60 @@ } /** + * Returns the size of the recyclebin in number of elements. + *

+ * @return The number of items in the bin. + */ + protected int getRecyleBinSize() + { + return this.recycle.size(); + } + + /** + * Returns the number of times we have used spots from the recycle bin. + *

+ * @return The number of spots used. + */ + protected int getRecyleCount() + { + return this.recycleCnt; + } + + /** + * Returns the number of bytes that are free. When an item is removed, its length is recorded. + * When a spot is used form the recycle bin, the length of the item stored is recorded. + *

+ * @return The number bytes free on the disk file. + */ + protected long getBytesFree() + { + return this.bytesFree; + } + + /** + * To subtract you can pass in false for add.. + *

+ * @param ded + * @param add + */ + private synchronized void adjustBytesFree( IndexedDiskElementDescriptor ded, boolean add ) + { + if ( ded != null ) + { + int amount = ded.len + IndexedDisk.RECORD_HEADER; + + if ( add ) + { + this.bytesFree += amount; + } + else + { + this.bytesFree -= amount; + } + } + } + + /** * This is for debugging and testing. *

* @return the length of the data file. @@ -1284,6 +1352,11 @@ } se = new StatElement(); + se.setName( "Bytes Free" ); + se.setData( "" + this.bytesFree ); + elems.add( se ); + + se = new StatElement(); se.setName( "Optimize Operation Count" ); se.setData( "" + this.removeCount ); elems.add( se ); @@ -1299,6 +1372,11 @@ elems.add( se ); se = new StatElement(); + se.setName( "Recycle Bin Size" ); + se.setData( "" + this.recycle.size() ); + elems.add( se ); + + se = new StatElement(); se.setName( "Startup Size" ); se.setData( "" + this.startupSize ); elems.add( se ); @@ -1399,7 +1477,6 @@ doOptimizeRealTime(); } - } /** Modified: jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/indexed/IndexedDiskCacheAttributes.java URL: http://svn.apache.org/viewvc/jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/indexed/IndexedDiskCacheAttributes.java?rev=432898&r1=432897&r2=432898&view=diff ============================================================================== --- jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/indexed/IndexedDiskCacheAttributes.java (original) +++ jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/indexed/IndexedDiskCacheAttributes.java Sat Aug 19 16:23:52 2006 @@ -38,6 +38,11 @@ // default to -1, i.e., don't optimize until shutdown private int optimizeAtRemoveCount = -1; + /** Should we optimize on shutdown. */ + public static final boolean DEFAULT_OPTIMIZE_ON_SHUTDOWN = true; + + private boolean optimizeOnShutdown = DEFAULT_OPTIMIZE_ON_SHUTDOWN; + /** * Constructor for the DiskCacheAttributes object */ @@ -118,6 +123,22 @@ } /** + * @param optimizeOnShutdown The optimizeOnShutdown to set. + */ + public void setOptimizeOnShutdown( boolean optimizeOnShutdown ) + { + this.optimizeOnShutdown = optimizeOnShutdown; + } + + /** + * @return Returns the optimizeOnShutdown. + */ + public boolean isOptimizeOnShutdown() + { + return optimizeOnShutdown; + } + + /** * Returns a copy of the attributes. *

* @return AuxiliaryCacheAttributes @@ -150,6 +171,7 @@ str.append( "\n maxRecycleBinSize = " + maxRecycleBinSize ); str.append( "\n optimizeAtRemoveCount = " + optimizeAtRemoveCount ); str.append( "\n shutdownSpoolTimeLimit = " + shutdownSpoolTimeLimit ); + str.append( "\n optimizeOnShutdown = " + optimizeOnShutdown ); return str.toString(); } Modified: jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/indexed/IndexedDiskElementDescriptor.java URL: http://svn.apache.org/viewvc/jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/indexed/IndexedDiskElementDescriptor.java?rev=432898&r1=432897&r2=432898&view=diff ============================================================================== --- jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/indexed/IndexedDiskElementDescriptor.java (original) +++ jakarta/jcs/trunk/src/java/org/apache/jcs/auxiliary/disk/indexed/IndexedDiskElementDescriptor.java Sat Aug 19 16:23:52 2006 @@ -18,7 +18,6 @@ public class IndexedDiskElementDescriptor implements Serializable, Comparable { - private static final long serialVersionUID = -3029163572847659450L; /** Position of the cache data entry on disk. */ Modified: jakarta/jcs/trunk/src/java/org/apache/jcs/utils/struct/LRUMap.java URL: http://svn.apache.org/viewvc/jakarta/jcs/trunk/src/java/org/apache/jcs/utils/struct/LRUMap.java?rev=432898&r1=432897&r2=432898&view=diff ============================================================================== --- jakarta/jcs/trunk/src/java/org/apache/jcs/utils/struct/LRUMap.java (original) +++ jakarta/jcs/trunk/src/java/org/apache/jcs/utils/struct/LRUMap.java Sat Aug 19 16:23:52 2006 @@ -19,14 +19,14 @@ import org.apache.jcs.engine.stats.behavior.IStats; /** - * This is a simple LRUMap. It implements most of the map methods. It is not - * recommended that you use any but put, get, remove, and clear. + * This is a simple LRUMap. It implements most of the map methods. It is not recommended that you + * use any but put, get, remove, and clear. *

- * Children can implement the processRemovedLRU method if they want to handle - * the removal of the lest recently used item. + * Children can implement the processRemovedLRU method if they want to handle the removal of the + * lest recently used item. *

- * This class was abstracted out of the LRU Memory cache. Put, remove, and get - * should be thread safe. It uses a hashtable and our own double linked list. + * This class was abstracted out of the LRU Memory cache. Put, remove, and get should be thread + * safe. It uses a hashtable and our own double linked list. *

* Locking is done on the instance. *

@@ -56,8 +56,8 @@ private int chunkSize = 1; /** - * This creates an unbounded version. Setting the max objects will result in - * spooling on subsequent puts. + * This creates an unbounded version. Setting the max objects will result in spooling on + * subsequent puts. *

* @param maxObjects */ @@ -124,8 +124,7 @@ } /** - * This is an expensive operation that determines if the object supplied is - * mapped to any key. + * This is an expensive operation that determines if the object supplied is mapped to any key. *

* @see java.util.Map#containsValue(java.lang.Object) */ @@ -162,10 +161,12 @@ } /** - * This returns a set of entries. Our LRUMapEntry is used since the value - * stored in the underlying map is a node in the double linked list. We - * wouldn't want to return this to the client, so we construct a new entry - * with the payload of the node. + * This returns a set of entries. Our LRUMapEntry is used since the value stored in the + * underlying map is a node in the double linked list. We wouldn't want to return this to the + * client, so we construct a new entry with the payload of the node. + *

+ * TODO we should return out own set wrapper, so we can avoid the extra object creation if it + * isn't necessary. *

* @see java.util.Map#entrySet() */ @@ -238,10 +239,9 @@ } /** - * This gets an element out of the map without adjusting it's posisiton in - * the LRU. In other words, this does not count as being used. If the - * element is the last item in the list, it will still be the last itme in - * the list. + * This gets an element out of the map without adjusting it's posisiton in the LRU. In other + * words, this does not count as being used. If the element is the last item in the list, it + * will still be the last itme in the list. *

* @param key * @return Object @@ -397,8 +397,7 @@ * Adds a new node to the start of the link list. *

* @param key - * @param val - * The feature to be added to the First + * @param val The feature to be added to the First */ private synchronized void addFirst( Object key, Object val ) { @@ -451,8 +450,8 @@ } /** - * Checks to see if all the items that should be in the cache are. Checks - * consistency between List and map. + * Checks to see if all the items that should be in the cache are. Checks consistency between + * List and map. */ protected void verifyCache() { @@ -570,8 +569,7 @@ } /** - * This is called when an item is removed from the LRU. We just log some - * information. + * This is called when an item is removed from the LRU. We just log some information. *

* Children can implement this method for special behavior. * @param key @@ -587,11 +585,9 @@ } /** - * The chunk size is the number of items to remove when the max is reached. - * By default it is 1. + * The chunk size is the number of items to remove when the max is reached. By default it is 1. *

- * @param chunkSize - * The chunkSize to set. + * @param chunkSize The chunkSize to set. */ public void setChunkSize( int chunkSize ) { Modified: jakarta/jcs/trunk/src/test-conf/TestDiskCacheDefragPerformance.ccf URL: http://svn.apache.org/viewvc/jakarta/jcs/trunk/src/test-conf/TestDiskCacheDefragPerformance.ccf?rev=432898&r1=432897&r2=432898&view=diff ============================================================================== --- jakarta/jcs/trunk/src/test-conf/TestDiskCacheDefragPerformance.ccf (original) +++ jakarta/jcs/trunk/src/test-conf/TestDiskCacheDefragPerformance.ccf Sat Aug 19 16:23:52 2006 @@ -15,3 +15,4 @@ jcs.auxiliary.DC.attributes.MaxPurgatorySize=5000 jcs.auxiliary.DC.attributes.OptimizeAtRemoveCount=5000 jcs.auxiliary.DC.attributes.MaxRecycleBinSize=5000 + Added: jakarta/jcs/trunk/src/test-conf/TestDiskCacheSteadyLoad.ccf URL: http://svn.apache.org/viewvc/jakarta/jcs/trunk/src/test-conf/TestDiskCacheSteadyLoad.ccf?rev=432898&view=auto ============================================================================== --- jakarta/jcs/trunk/src/test-conf/TestDiskCacheSteadyLoad.ccf (added) +++ jakarta/jcs/trunk/src/test-conf/TestDiskCacheSteadyLoad.ccf Sat Aug 19 16:23:52 2006 @@ -0,0 +1,18 @@ +# Java Caching System configuration file + +# DEFAULT CACHE REGION +jcs.default=DC +jcs.default.cacheattributes=org.apache.jcs.engine.CompositeCacheAttributes +jcs.default.cacheattributes.MaxObjects=100 +jcs.default.cacheattributes.MemoryCacheName=org.apache.jcs.engine.memory.lru.LRUMemoryCache +jcs.default.cacheattributes.DiskUsagePatternName=UPDATE + + +# AVAILABLE AUXILIARY CACHES +jcs.auxiliary.DC=org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCacheFactory +jcs.auxiliary.DC.attributes=org.apache.jcs.auxiliary.disk.indexed.IndexedDiskCacheAttributes +jcs.auxiliary.DC.attributes.DiskPath=target/test-sandbox/steady-load +jcs.auxiliary.DC.attributes.maxKeySize=1000 +jcs.auxiliary.DC.attributes.MaxPurgatorySize=1000 +jcs.auxiliary.DC.attributes.OptimizeAtRemoveCount=20000 +jcs.auxiliary.DC.attributes.MaxRecycleBinSize=10000 Modified: jakarta/jcs/trunk/src/test/org/apache/jcs/auxiliary/disk/indexed/IndexDiskCacheUnitTest.java URL: http://svn.apache.org/viewvc/jakarta/jcs/trunk/src/test/org/apache/jcs/auxiliary/disk/indexed/IndexDiskCacheUnitTest.java?rev=432898&r1=432897&r2=432898&view=diff ============================================================================== --- jakarta/jcs/trunk/src/test/org/apache/jcs/auxiliary/disk/indexed/IndexDiskCacheUnitTest.java (original) +++ jakarta/jcs/trunk/src/test/org/apache/jcs/auxiliary/disk/indexed/IndexDiskCacheUnitTest.java Sat Aug 19 16:23:52 2006 @@ -252,4 +252,161 @@ assertEquals( "Wrong file size", expectedSize, resultSize ); } + /** + * Verify that items are added to the recyle bin on removal. + *

+ * @throws IOException + * @throws InterruptedException + */ + public void testRecyleBinSize() + throws IOException, InterruptedException + { + // SETUP + int numberToInsert = 20; + + IndexedDiskCacheAttributes cattr = new IndexedDiskCacheAttributes(); + cattr.setCacheName( "testRecyleBinSize" ); + cattr.setDiskPath( "target/test-sandbox/UnitTest" ); + cattr.setMaxRecycleBinSize( numberToInsert ); + cattr.setOptimizeAtRemoveCount( numberToInsert ); + cattr.setMaxKeySize( numberToInsert * 2 ); + cattr.setMaxPurgatorySize( numberToInsert ); + IndexedDiskCache disk = new IndexedDiskCache( cattr ); + + int bytes = 24; + ICacheElement[] elements = DiskTestObjectUtil.createCacheElementsWithTestObjects( numberToInsert, bytes, cattr + .getCacheName() ); + + for ( int i = 0; i < elements.length; i++ ) + { + disk.doUpdate( elements[i] ); + } + + Thread.yield(); + Thread.sleep( 100 ); + Thread.yield(); + + // remove half + int numberToRemove = elements.length / 2; + for ( int i = 0; i < numberToRemove; i++ ) + { + disk.doRemove( elements[i].getKey() ); + } + + // verify that the recyle bin has the correct amount. + assertEquals( "The recycle bin should have the number removed.", numberToRemove, disk.getRecyleBinSize() ); + } + + /** + * Verify that items of the same size use recyle bin spots. Setup the receyle bin by removing + * some items. Add some of the same size. Verify that the recyle count is the number added. + *

+ * @throws IOException + * @throws InterruptedException + */ + public void testRecyleBinUsage() + throws IOException, InterruptedException + { + // SETUP + int numberToInsert = 20; + + IndexedDiskCacheAttributes cattr = new IndexedDiskCacheAttributes(); + cattr.setCacheName( "testRecyleBinUsage" ); + cattr.setDiskPath( "target/test-sandbox/UnitTest" ); + cattr.setMaxRecycleBinSize( numberToInsert ); + cattr.setOptimizeAtRemoveCount( numberToInsert ); + cattr.setMaxKeySize( numberToInsert * 2 ); + cattr.setMaxPurgatorySize( numberToInsert ); + IndexedDiskCache disk = new IndexedDiskCache( cattr ); + + // we will reuse these + int bytes = 24; + ICacheElement[] elements = DiskTestObjectUtil.createCacheElementsWithTestObjects( numberToInsert, bytes, cattr + .getCacheName() ); + + // Add some to the disk + for ( int i = 0; i < elements.length; i++ ) + { + disk.doUpdate( elements[i] ); + } + + Thread.yield(); + Thread.sleep( 100 ); + Thread.yield(); + + // remove half of those added + int numberToRemove = elements.length / 2; + for ( int i = 0; i < numberToRemove; i++ ) + { + disk.doRemove( elements[i].getKey() ); + } + + // verify that the recyle bin has the correct amount. + assertEquals( "The recycle bin should have the number removed.", numberToRemove, disk.getRecyleBinSize() ); + + // add half as many as we removed. These should all use spots in the recycle bin. + int numberToAdd = numberToRemove / 2; + for ( int i = 0; i < numberToAdd; i++ ) + { + disk.doUpdate( elements[i] ); + } + + // verify that we used the correct number of spots + assertEquals( "The recycle bin should have the number removed." + disk.getStats(), numberToAdd, disk.getRecyleCount() ); + } + + /** + * Verify that the data size is as expected after a remove and after a put that should use the spots. + *

+ * @throws IOException + * @throws InterruptedException + */ + public void testBytesFreeSize() + throws IOException, InterruptedException + { + // SETUP + IndexedDiskCacheAttributes cattr = new IndexedDiskCacheAttributes(); + cattr.setCacheName( "testBytesFreeSize" ); + cattr.setDiskPath( "target/test-sandbox/UnitTest" ); + IndexedDiskCache disk = new IndexedDiskCache( cattr ); + + int numberToInsert = 20; + int bytes = 24; + ICacheElement[] elements = DiskTestObjectUtil.createCacheElementsWithTestObjects( numberToInsert, bytes, cattr + .getCacheName() ); + + for ( int i = 0; i < elements.length; i++ ) + { + disk.doUpdate( elements[i] ); + } + + Thread.yield(); + Thread.sleep( 100 ); + Thread.yield(); + + // remove half of those added + int numberToRemove = elements.length / 2; + for ( int i = 0; i < numberToRemove; i++ ) + { + disk.doRemove( elements[i].getKey() ); + } + + long expectedSize = DiskTestObjectUtil.totalSize( elements, numberToRemove ); + long resultSize = disk.getBytesFree(); + + System.out.println( "testBytesFreeSize stats " + disk.getStats() ); + + assertEquals( "Wrong bytes free size" + disk.getStats(), expectedSize, resultSize ); + + // add half as many as we removed. These should all use spots in the recycle bin. + int numberToAdd = numberToRemove / 2; + for ( int i = 0; i < numberToAdd; i++ ) + { + disk.doUpdate( elements[i] ); + } + + long expectedSize2 = DiskTestObjectUtil.totalSize( elements, numberToAdd ); + long resultSize2 = disk.getBytesFree(); + assertEquals( "Wrong bytes free size" + disk.getStats(), expectedSize2, resultSize2 ); + } } Added: jakarta/jcs/trunk/src/test/org/apache/jcs/auxiliary/disk/indexed/IndexedDiskCacheSteadyLoadTest.java URL: http://svn.apache.org/viewvc/jakarta/jcs/trunk/src/test/org/apache/jcs/auxiliary/disk/indexed/IndexedDiskCacheSteadyLoadTest.java?rev=432898&view=auto ============================================================================== --- jakarta/jcs/trunk/src/test/org/apache/jcs/auxiliary/disk/indexed/IndexedDiskCacheSteadyLoadTest.java (added) +++ jakarta/jcs/trunk/src/test/org/apache/jcs/auxiliary/disk/indexed/IndexedDiskCacheSteadyLoadTest.java Sat Aug 19 16:23:52 2006 @@ -0,0 +1,130 @@ +package org.apache.jcs.auxiliary.disk.indexed; + +import java.text.DecimalFormat; +import java.util.Random; + +import junit.framework.TestCase; + +import org.apache.jcs.JCS; +import org.apache.jcs.utils.timing.ElapsedTimer; + +/** + * This allows you to put thousands of large objects into the disk cache and to force removes to + * trigger optimizations along the way. + *

+ * @author Aaron Smuts + */ +public class IndexedDiskCacheSteadyLoadTest + extends TestCase +{ + private static final String LOG_DIVIDER = "---------------------------"; + + private static Runtime rt = Runtime.getRuntime(); + + private static DecimalFormat format = new DecimalFormat( "#,###" ); + + /** + * Insert 2000 wait 1 second, repeat. Average 1000 / sec. + *

+ * @throws Exception + */ + public void testRunSteadyLoadTest() + throws Exception + { + JCS.setConfigFilename( "/TestDiskCacheSteadyLoad.ccf" ); + + System.out.println( "runSteadyLoadTest" ); + + logMemoryUsage(); + + int numPerRun = 200; + long pauseBetweenRuns = 1000; + int runCount = 0; + int runs = 1000; + int upperKB = 50; + + JCS jcs = JCS.getInstance( ( numPerRun / 2 ) + "aSecond" ); + + ElapsedTimer timer = new ElapsedTimer(); + int numToGet = numPerRun * ( runs / 10 ); + for ( int i = 0; i < numToGet; i++ ) + { + jcs.get( String.valueOf( i ) ); + } + System.out.println( LOG_DIVIDER ); + System.out.println( "After getting " + numToGet ); + System.out.println( "Elapsed " + timer.getElapsedTimeString() ); + logMemoryUsage(); + + jcs.clear(); + Thread.sleep( 3000 ); + System.out.println( LOG_DIVIDER ); + System.out.println( "Start putting" ); + + long totalSize = 0; + int totalPut = 0; + + Random random = new Random( 89 ); + while ( runCount < runs ) + { + runCount++; + for ( int i = 0; i < numPerRun; i++ ) + { + // 1/2 upper to upperKB-4 KB + int kiloBytes = Math.max( upperKB / 2, random.nextInt( upperKB ) ); + int bytes = ( kiloBytes ) * 1024; + totalSize += bytes; + totalPut++; + DiskTestObject object = new DiskTestObject( new Integer( i ), new byte[bytes] ); + jcs.put( String.valueOf( totalPut ), object ); + } + + // remove half of those inserted the previous run + if ( runCount > 1 ) + { + for ( int j = ( ( totalPut - numPerRun ) - ( numPerRun / 2 ) ); j < ( totalPut - numPerRun ); j++ ) + { + jcs.remove( String.valueOf( j ) ); + } + } + + Thread.sleep( pauseBetweenRuns ); + if ( runCount % 1 == 0 ) + { + System.out.println( LOG_DIVIDER ); + System.out.println( "Elapsed " + timer.getElapsedTimeString() ); + System.out.println( "Run count: " + runCount + " Average size: " + ( totalSize / totalPut ) + "\n" + + jcs.getStats() ); + logMemoryUsage(); + } + } + + Thread.sleep( 3000 ); + System.out.println( jcs.getStats() ); + logMemoryUsage(); + + Thread.sleep( 10000 ); + System.out.println( jcs.getStats() ); + logMemoryUsage(); + + System.gc(); + Thread.sleep( 3000 ); + System.gc(); + System.out.println( jcs.getStats() ); + logMemoryUsage(); + } + + /** + * Logs the memory usage. + */ + private static void logMemoryUsage() + { + long byte2MB = 1024 * 1024; + long total = rt.totalMemory() / byte2MB; + long free = rt.freeMemory() / byte2MB; + long used = total - free; + System.out.println( LOG_DIVIDER ); + System.out.println( "Memory:" + " Used:" + format.format( used ) + "MB" + " Free:" + format.format( free ) + + "MB" + " Total:" + format.format( total ) + "MB" ); + } +} Modified: jakarta/jcs/trunk/src/test/org/apache/jcs/utils/struct/SortedPrefArrayUnitTest.java URL: http://svn.apache.org/viewvc/jakarta/jcs/trunk/src/test/org/apache/jcs/utils/struct/SortedPrefArrayUnitTest.java?rev=432898&r1=432897&r2=432898&view=diff ============================================================================== --- jakarta/jcs/trunk/src/test/org/apache/jcs/utils/struct/SortedPrefArrayUnitTest.java (original) +++ jakarta/jcs/trunk/src/test/org/apache/jcs/utils/struct/SortedPrefArrayUnitTest.java Sat Aug 19 16:23:52 2006 @@ -400,4 +400,55 @@ + " Instead we got " + taken, taken ); } + /** + * Try taking an item equal to the greatest. Make the last two the same size + */ + public void testEqualToGreatest_LastTwoSameSize() + { + int maxSize = 3; + + SortedPreferentialArray array = new SortedPreferentialArray( maxSize ); + // array.setPreferLarge( false ); + array.setPreferLarge( true ); + String[] elem = { "01", "02", "03", "03" }; + + // put more than the max in a random order + for ( int i = 0; i < elem.length; i++ ) + { + array.add( elem[i] ); + System.out.println( array.dumpArray() ); + } + + // DO WORK + Comparable taken = array.takeNearestLargerOrEqual( "03" ); + System.out.println( "testEqualToGreatest_LastTwoSameSize" + array.dumpArray() ); + + assertNotNull( "We should have something since the largest element was equal to what we asked for.", taken ); + } + + /** + * Try taking an item equal to the greatest. The second to last should be smaller. This verifies the most basic funtionality. + */ + public void testEqualToGreatest() + { + int maxSize = 3; + + SortedPreferentialArray array = new SortedPreferentialArray( maxSize ); + // array.setPreferLarge( false ); + array.setPreferLarge( true ); + String[] elem = { "01", "02", "03" }; + + // put more than the max in a random order + for ( int i = 0; i < elem.length; i++ ) + { + array.add( elem[i] ); + System.out.println( array.dumpArray() ); + } + + // DO WORK + Comparable taken = array.takeNearestLargerOrEqual( "03" ); + System.out.println( "testEqualToGreatest" + array.dumpArray() ); + + assertNotNull( "We should have something since the largest element was equal to what we asked for.", taken ); + } } --------------------------------------------------------------------- To unsubscribe, e-mail: jcs-dev-unsubscribe@jakarta.apache.org For additional commands, e-mail: jcs-dev-help@jakarta.apache.org