jakarta-jcs-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Smuts <asm...@yahoo.com>
Subject Re: 100% cpu issue with LateralTCPCache
Date Fri, 22 Sep 2006 17:32:45 GMT
I created a bug for this and am looking into it.

http://issues.apache.org/jira/browse/JCS-12


--- jklame <jlame@qovia.com> wrote:

> 
> Hi,
> 
> We encountered a problem with an infinite loop in
> java.io.ObjectOutputStream
> which we were able to fix by ensuring that access to
> it was synchronized
> inside LateralTCPSender.java.    I thought I should
> report exactly what we
> found and what we did to solve the problem so that
> perhaps similar changes
> might be made for the next release of jcs.  
> 
> We were using the LateralTCPCache in jcs.1.2.7.9 and
> were performing
> serveral thousand gets and puts per minute from
> multiple threads all using
> the same JCS object.   Very quickly, one of these
> threads would end up in an
> infinite loop in ObjectOutputStream.lookup()
> (offending loop in bold)
> 
> 	int lookup(Object obj) {
> 	    if (size == 0) {
> 		return -1;
> 	    }
> 	    int index = hash(obj) % spine.length;
> 	    for (int i = spine[index]; i >= 0; i = next[i])
> {
> 		if (objs[i] == obj) {
> 		    return i;
> 		}
> 	    }
> 	    return -1;
> 	}
>    
> Once one thread ended up in this loop, all other
> threads would end up
> blocking, waiting to obtain a lock on this.getLock
> in
> LateralTCPSender.sendAndReceive() (found by
> performing a thread dump after
> the application had stopped responding).    But
> since the thread in the
> inifinite loop already had a lock on this.getLock, 
> our application was
> effectively dead.
> 
> Now ObjectOutputStream is not thread-safe.  Indeed,
> if multiple threads
> attempt to write objects to the same
> ObjectOutputStream simultaneously, it
> is likely that the internal object cache will become
> corrupted and
> subsequent writes will end up locked in the
> aforementioned infinite loop.   
> 
> Looking at LateralTCPSender.java more carefully, we
> noticed that the  send(
> LateralElementDescriptor led ) method did perform
> non-thread-safe writes to
> the shared ObjectOutputStream.    Wrapping access to
> it with a
> synchronized(this.getLock) block (exactly as was
> done in the sendAndReceive
> method appears to have fixed out problem.    I've
> included the 1.2.7.9
> version of this method with the changes we made in
> bold.
> 
> 
>     /**
>      * Sends commands to the lateral cache listener.
>      * <p>
>      * @param led
>      * @throws IOException
>      */
>     public void send( LateralElementDescriptor led )
>         throws IOException
>     {
>         sendCnt++;
>         if ( log.isInfoEnabled() )
>         {
>             if ( sendCnt % 100 == 0 )
>             {
>                 log.info( "Send Count (port " + port
> + ") = " + sendCnt );
>             }
>         }
> 
>         if ( log.isDebugEnabled() )
>         {
>             log.debug( "sending
> LateralElementDescriptor" );
>         }
> 
>         if ( led == null )
>         {
>             return;
>         }
> 
>         if ( address == null )
>         {
>             throw new IOException( "No remote host
> is set for
> LateralTCPSender." );
>         }
> 
>         if ( oos != null )
>         {
>             synchronized(this.getLock)
>             {                try
>                 {
>                     oos.writeObject( led );
>                     oos.flush();
>                     if ( ++counter >=
> RESET_FREQUENCY )
>                     {
>                         counter = 0;
>                         // Failing to reset the
> object output stream every
> now and
>                         // then creates a serious
> memory leak.
>                         if ( log.isDebugEnabled() )
>                         {
>                             log.debug( "Doing
> oos.reset()" );
>                         }
>                         oos.reset();
>                     }
>                 }
>                 catch ( IOException e )
>                 {
>                     oos = null;
>                     log.error( "Detected problem
> with connection: " + e );
>                     throw e;
>                 }
>             }        }
>     }
> 
> Comments?
> John
> 
> 
> -- 
> View this message in context:
>
http://www.nabble.com/100--cpu-issue-with-LateralTCPCache-tf2318435.html#a6449346
> Sent from the JCS - Users mailing list archive at
> Nabble.com.
> 
> 
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> jcs-users-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
> jcs-users-help@jakarta.apache.org
> 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: jcs-users-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jcs-users-help@jakarta.apache.org


Mime
View raw message