db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David W. Van Couvering" <David.Vancouver...@Sun.COM>
Subject Re: svn commit: r384605 - in /db/derby/code/trunk/java: engine/org/apache/derby/iapi/services/memory/ engine/org/apache/derby/impl/jdbc/ engine/org/apache/derby/jdbc/ testing/org/apache/derbyTesting/functionTests/tests/memory/
Date Thu, 09 Mar 2006 21:41:10 GMT
This is great, Dan, it's great to get some extra robustness into the 
system.

I understand why you made the exception a static. As it stands in your 
code the original stack trace is lost, so how will I know who is 
throwing the static NO_MEM exception?  As someone trying to debug the 
situation, for example, I wouldn't know if the NO_MEM exception was 
thrown in EmbedConnection or in one of two places in InternalDriver.

Wouldn't it make more sense to create a separate static instance for 
each situation?  I understand that this could be a bit expensive in 
terms of footprint, but I'm worried about debugging, determining what 
particular code-path caused the out-of-memory situation.

David

djd@apache.org wrote:
> Author: djd
> Date: Thu Mar  9 12:29:47 2006
> New Revision: 384605
> 
> URL: http://svn.apache.org/viewcvs?rev=384605&view=rev
> Log:
> DERBY-444 Handle out of memory errors when embedded opening connections. Provide some
> initial framework that sets low memory water marks for an operation and fails that operation
> until free memory is above that mark or five seconds have elapsed.
> See java/engine/org/apache/derby/iapi/services/memory/LowMemory.java for more details.
> 
> Added:
>     db/derby/code/trunk/java/engine/org/apache/derby/iapi/services/memory/
>     db/derby/code/trunk/java/engine/org/apache/derby/iapi/services/memory/LowMemory.java
  (with props)
> Modified:
>     db/derby/code/trunk/java/engine/org/apache/derby/impl/jdbc/EmbedConnection.java
>     db/derby/code/trunk/java/engine/org/apache/derby/jdbc/InternalDriver.java
>     db/derby/code/trunk/java/testing/org/apache/derbyTesting/functionTests/tests/memory/ConnectionHandling.java
> 
> Added: db/derby/code/trunk/java/engine/org/apache/derby/iapi/services/memory/LowMemory.java
> URL: http://svn.apache.org/viewcvs/db/derby/code/trunk/java/engine/org/apache/derby/iapi/services/memory/LowMemory.java?rev=384605&view=auto
> ==============================================================================
> --- db/derby/code/trunk/java/engine/org/apache/derby/iapi/services/memory/LowMemory.java
(added)
> +++ db/derby/code/trunk/java/engine/org/apache/derby/iapi/services/memory/LowMemory.java
Thu Mar  9 12:29:47 2006
> @@ -0,0 +1,137 @@
> +/*
> +
> +   Derby - Class org.apache.derby.iapi.services.memory.LowMemory
> +
> +   Copyright 2005, 2006 The Apache Software Foundation or its licensors, as applicable.
> +
> +   Licensed under the Apache License, Version 2.0 (the "License");
> +   you may not use this file except in compliance with the License.
> +   You may obtain a copy of the License at
> +
> +      http://www.apache.org/licenses/LICENSE-2.0
> +
> +   Unless required by applicable law or agreed to in writing, software
> +   distributed under the License is distributed on an "AS IS" BASIS,
> +   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> +   See the License for the specific language governing permissions and
> +   limitations under the License.
> +
> + */
> +
> +package org.apache.derby.iapi.services.memory;
> +
> +/**
> + * Methods to aid classes recover from OutOfMemoryErrors by denying
> + * or reducing service rather than a complete shutdown of the JVM.
> + * It's intended that classes use to functionality to allow then to
> + * deny service when memory is low to allow the JVM to recover,
> + * rather than start new operations that are probably doomed to
> + * failure due to the low memory.
> + * <P>
> + * Expected usage is one instance of this class per major logical
> + * operation, e.g. creating a connection, preparing a statement,
> + * adding an entry to a specific cache etc.
> + * <BR>
> + * The logical operation would call isLowMemory() before starting
> + * the operation, and thrown a static exception if it returns true.
> + * <BR>
> + * If during the operation an OutOfMemoryException is thrown the
> + * operation would call setLowMemory() and throw its static exception
> + * representing low memory.
> + * <P>
> + * Future enhancments could be a callback mechanism for modules
> + * where they register they can reduce memory usage on a low
> + * memory situation. These callbacks would be triggered by
> + * a call to setLowMemory. For example the page cache could
> + * reduce its current size by 10% in a low memory situation.
> + * 
> + */
> +public class LowMemory {
> +
> +    /**
> +     * Free memory seen when caller indicated an out of
> +     * memory situation. Becomes a low memory watermark
> +     * for five seconds that causes isLowMemory to return
> +     * true if free memory is lower than this value.
> +     * This allows the JVM a chance to recover memory
> +     * rather than start new operations that are probably
> +     * doomed to failure due to the low memory.
> +     * 
> +     */
> +    private long lowMemory;
> +    
> +    /**
> +     * Time in ms corresponding to System.currentTimeMillis() when
> +     * lowMemory was set.
> +     */
> +    private long whenLowMemorySet;
> +    
> +    /**
> +     * Set a low memory watermark where the owner of this object just hit an
> +     * OutOfMemoryError. The caller is assumed it has just freed up any
> +     * references it obtained during the operation, so that the freeMemory call
> +     * as best as it can reflects the memory before the action that caused the
> +     * OutOfMemoryError, not part way through the action.
> +     * 
> +     */
> +    public void setLowMemory() {
> +        
> +        // Can read lowMemory unsynchronized, worst
> +        // case is that we force extra garbage collection.
> +        if (lowMemory == 0L) {
> +            
> +            // The caller tried to dereference any objects it
> +            // created during its instantation. Try to garbage
> +            // collect these so that we can a best-guess effort
> +            // at the free memory before the overall operation we are
> +            // failing on occurred. Of course in active multi-threading
> +            // systems we run the risk that some other thread just freed
> +            // up some memory that throws off our calcuation. This is
> +            // avoided by clearing lowMemory some time later on an
> +            // isLowMemory() call.
> +            for (int i = 0; i < 5; i++) {
> +                System.gc();
> +                System.runFinalization();
> +                try {
> +                    Thread.sleep(50L);
> +                } catch (InterruptedException e) {
> +                }
> +            }
> +        }
> +        synchronized (this) {
> +            if (lowMemory == 0L) {
> +                lowMemory = Runtime.getRuntime().freeMemory();
> +                whenLowMemorySet = System.currentTimeMillis();
> +            }
> +        }
> +    }
> +
> +    /**
> +     * Return true if a low memory water mark has been set and the current free
> +     * memory is lower than it. Otherwise return false.
> +     */
> +    public boolean isLowMemory() {
> +        synchronized (this) {
> +            long lm = lowMemory;
> +            if (lm == 0)
> +                return false;
> +            
> +            if (Runtime.getRuntime().freeMemory() > lm)
> +                return false;
> +            
> +            // Only allow an low memory watermark to be valid
> +            // for five seconds after it was set. This stops
> +            // an incorrect limit being set for ever. This could
> +            // occur if other threads were freeing memory when
> +            // we called Runtime.getRuntime().freeMemory()
> +           
> +            long now = System.currentTimeMillis();
> +            if ((now - this.whenLowMemorySet) > 5000L) {
> +                lowMemory = 0L;
> +                whenLowMemorySet = 0L;
> +                return false;
> +            }
> +            return true;
> +        }
> +    }
> +}
> 
> Propchange: db/derby/code/trunk/java/engine/org/apache/derby/iapi/services/memory/LowMemory.java
> ------------------------------------------------------------------------------
>     svn:eol-style = native
> 
> Modified: db/derby/code/trunk/java/engine/org/apache/derby/impl/jdbc/EmbedConnection.java
> URL: http://svn.apache.org/viewcvs/db/derby/code/trunk/java/engine/org/apache/derby/impl/jdbc/EmbedConnection.java?rev=384605&r1=384604&r2=384605&view=diff
> ==============================================================================
> --- db/derby/code/trunk/java/engine/org/apache/derby/impl/jdbc/EmbedConnection.java (original)
> +++ db/derby/code/trunk/java/engine/org/apache/derby/impl/jdbc/EmbedConnection.java Thu
Mar  9 12:29:47 2006
> @@ -30,6 +30,7 @@
>  import org.apache.derby.iapi.reference.SQLState;
>  
>  import org.apache.derby.iapi.services.context.ContextManager;
> +import org.apache.derby.iapi.services.memory.LowMemory;
>  import org.apache.derby.iapi.services.monitor.Monitor;
>  import org.apache.derby.iapi.services.sanity.SanityManager;
>  
> @@ -91,6 +92,21 @@
>  {
>  
>  	private static final StandardException exceptionClose = StandardException.closeException();
> +    
> +    /**
> +     * Static exception to be thrown when a Connection request can not
> +     * be fulfilled due to lack of memory. A static exception as the lack
> +     * of memory would most likely cause another OutOfMemoryException and
> +     * if there is not enough memory to create the OOME exception then something
> +     * like the VM dying could occur. Simpler just to throw a static.
> +     */
> +    public static final SQLException NO_MEM =
> +        Util.generateCsSQLException(SQLState.LOGIN_FAILED, "java.lang.OutOfMemoryError");
> +    
> +    /**
> +     * Low memory state object for connection requests.
> +     */
> +    public static final LowMemory memoryState = new LowMemory();
>  
>  	//////////////////////////////////////////////////////////
>  	// OBJECTS SHARED ACROSS CONNECTION NESTING
> @@ -250,11 +266,26 @@
>  				throw tr.shutdownDatabaseException();
>  			}
>  
> -		} catch (Throwable t) {
> +		}
> +        catch (OutOfMemoryError noMemory)
> +		{
> +			//System.out.println("freeA");
> +			restoreContextStack();
> +			tr.lcc = null;
> +			tr.cm = null;
> +			
> +			//System.out.println("free");
> +			//System.out.println(Runtime.getRuntime().freeMemory());
> +            memoryState.setLowMemory();
> +			
> +			//noMemory.printStackTrace();
> +			// throw Util.generateCsSQLException(SQLState.LOGIN_FAILED, noMemory.getMessage(),
noMemory);
> +			throw NO_MEM;
> +		}
> +		catch (Throwable t) {
>  			throw handleException(t);
>  		} finally {
>  			restoreContextStack();
> -			info = null;
>  		}
>  	}
>  
> 
> Modified: db/derby/code/trunk/java/engine/org/apache/derby/jdbc/InternalDriver.java
> URL: http://svn.apache.org/viewcvs/db/derby/code/trunk/java/engine/org/apache/derby/jdbc/InternalDriver.java?rev=384605&r1=384604&r2=384605&view=diff
> ==============================================================================
> --- db/derby/code/trunk/java/engine/org/apache/derby/jdbc/InternalDriver.java (original)
> +++ db/derby/code/trunk/java/engine/org/apache/derby/jdbc/InternalDriver.java Thu Mar
 9 12:29:47 2006
> @@ -1,3 +1,4 @@
> +
>  /*
>  
>     Derby - Class org.apache.derby.jdbc.InternalDriver
> @@ -55,7 +56,7 @@
>  */
>  
>  public abstract class InternalDriver implements ModuleControl {
> -
> +    
>  	private static final Object syncMe = new Object();
>  	private static InternalDriver activeDriver;
>  
> @@ -113,8 +114,16 @@
>  		 throws SQLException 
>  	{
>  		if (!acceptsURL(url)) { return null; }
> -
> -			
> +		
> +        /**
> +         * If we are below the low memory watermark for obtaining
> +         * a connection, then don't even try. Just throw an exception.
> +         */
> +		if (EmbedConnection.memoryState.isLowMemory())
> +		{
> +			throw EmbedConnection.NO_MEM;
> +		}
> +        			
>  		/*
>  		** A url "jdbc:default:connection" means get the current
>  		** connection.  From within a method called from JSQL, the
> @@ -142,9 +151,12 @@
>  
>  		// convert the ;name=value attributes in the URL into
>  		// properties.
> -		FormatableProperties finfo = getAttributes(url, info);
> -		info = null; // ensure we don't use this reference directly again.
> +		FormatableProperties finfo = null;
> +        
>  		try {
> +            
> +            finfo = getAttributes(url, info);
> +            info = null; // ensure we don't use this reference directly again.
>  
>  			/*
>  			** A property "shutdown=true" means shut the system or database down
> @@ -194,9 +206,15 @@
>  
>  			return conn;
>  		}
> +		catch (OutOfMemoryError noMemory)
> +		{
> +			EmbedConnection.memoryState.setLowMemory();
> +			throw EmbedConnection.NO_MEM;
> +		}
>  		finally {
>  			// break any link with the user's Properties set.
> -			finfo.clearDefaults();
> +            if (finfo != null)
> +			    finfo.clearDefaults();
>  		}
>  	}
>  
> 
> Modified: db/derby/code/trunk/java/testing/org/apache/derbyTesting/functionTests/tests/memory/ConnectionHandling.java
> URL: http://svn.apache.org/viewcvs/db/derby/code/trunk/java/testing/org/apache/derbyTesting/functionTests/tests/memory/ConnectionHandling.java?rev=384605&r1=384604&r2=384605&view=diff
> ==============================================================================
> --- db/derby/code/trunk/java/testing/org/apache/derbyTesting/functionTests/tests/memory/ConnectionHandling.java
(original)
> +++ db/derby/code/trunk/java/testing/org/apache/derbyTesting/functionTests/tests/memory/ConnectionHandling.java
Thu Mar  9 12:29:47 2006
> @@ -88,6 +88,10 @@
>          int ok = 0;
>          for (int i = 0; i < 500; i++)
>          {
> +            // Sleep for 10 secs as we know the implementation
> +            // of the low meory watermark resets after 5 seconds.
> +            if (i == 300)
> +                Thread.sleep(10000L);
>              try {
>                    Connection c = DriverManager.getConnection("jdbc:derby:wombat", p);
>                    list.add(c);
> 
> 

Mime
View raw message