geronimo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Jencks <david_jen...@yahoo.com>
Subject Re: [discuss] Release Geronimo 1.2
Date Fri, 06 Apr 2007 20:46:06 GMT
I don't think this is acceptable.  There should be only one thread  
working with the context at a time.  Either this exception is caused  
by modifying the collection in the same thread in which case we can  
fix it easily or it is caused by more than one thread having access  
to a context at once.  Since by my reading of the code this is a  
context attached to a stateless session bean instance, that would  
mean that more than one thread is using a stateless session bean  
instance at once, which is definitely cause to -1 the release.

I hope there's another possibility I haven't thought of..... but  
hiding the problem is not acceptable unless we really understand what  
is going on and are really convinced it's harmless.

thanks
david jencks

On Apr 6, 2007, at 1:40 PM, Jay D. McHugh wrote:

> Chris (do you go by Chris or Christopher?),
>
> Here is a patch that I just wrote that allows the exit routine of  
> ConnectionTrackingCoordinator to finish cleanly after a number (5)  
> of attempts at removing the resource.
>
> If it fails after five tries, then the routine exits and throws a  
> ResourceException (that will hopefully be caught further up the  
> stack).
>
> Would you like to try it to see if it solves your concurrency problem?
>
> Jay
>
> Christopher Blythe wrote:
>> Doubtful... everything tested fine under light browser based  
>> testing. As the exception suggests, this is a concurrency problem  
>> that you would only hit under load.
>>
>> On 4/6/07, * Jay D. McHugh* <jay@joyfulnoisewebdesign.com  
>> <mailto:jay@joyfulnoisewebdesign.com>> wrote:
>>
>>     If Matt had no problem deploying and testing DT, could it be a  
>> Java
>>     version or classpath issue?
>>
>>     That could explain the difference in the exception during  
>> deployment
>>     (and the problems during deployment could possibly explain the  
>> run
>>     time
>>     problems).
>>
>>     Jay
>>
>>     Christopher Blythe wrote:
>>     > I use a commercial load driving tool... FYI, I'm fairly  
>> certain that
>>     > G-2.0 has the same issue.
>>     >
>>     > On 4/6/07, *David Jencks* < david_jencks@yahoo.com
>>     <mailto:david_jencks@yahoo.com>
>>     > <mailto:david_jencks@yahoo.com  
>> <mailto:david_jencks@yahoo.com>>>
>>     wrote:
>>     >
>>     >     I think we need to figure out why the
>>     >     concurrentModificationException is happening before we
>>     release.  I
>>     >     think that one possible reason is that we are  
>> multithreading
>>     >     stateless session bean instances.  I hope this isn't the
>>     cause....
>>     >     but IMO we need to find out.
>>     >
>>     >     Chris, how do you run the several clients?  manually or  
>> with
>>     a tool?
>>     >
>>     >     thanks
>>     >     david jencks
>>     >
>>     >
>>     >     On Apr 6, 2007, at 11:09 AM, Christopher Blythe wrote:
>>     >
>>     >>     Gave it a shot... no luck. As soon as I started 2  
>> clients, the
>>     >>     same exceptions started to pile up. I have attached the
>>     >>     geronimo.log. Also, noticed the following exception during
>>     startup.
>>     >>
>>     >>     14:05:00,640 ERROR [TransportConnector] Could not accept
>>     >>     connection from /127.0.0.1:28428: java.io.IOException:  
>> Wire
>>     >>     format negociation timeout: peer did not send his wire  
>> format.
>>     >>     java.io.IOException: Wire format negociation timeout:  
>> peer did
>>     >>     not send his wire format.
>>     >>         at
>>     org.apache.activemq.transport.WireFormatNegotiator.oneway
>>     >>     (WireFormatNegotiator.java :88)
>>     >>         at
>>     >>        org.apache.activemq.transport.MutexTransport.oneway 
>> (MutexTransport.java:47)
>>     >>         at  
>> org.apache.activemq.broker.TransportConnection.dispatch
>>     >>     (TransportConnection.java :1138)
>>     >>         at
>>     >>         
>> org.apache.activemq.broker.TransportConnection.processDispatch 
>> (TransportConnection.java:805)
>>     >>         at
>>     >>     org.apache.activemq.broker.TransportConnection.start
>>     (TransportConnection.java
>>     >>     :885)
>>     >>         at org.apache.activemq.broker.TransportConnector 
>> $1.onAccept
>>     >>     (TransportConnector.java:148)
>>     >>         at
>>     >>     org.apache.activemq.transport.tcp.TcpTransportServer.run
>>     (TcpTransportServer.java:167)
>>     >>         at java.lang.Thread.run (Thread.java:797)
>>     >>
>>     >>
>>     >>
>>     >>     On 4/6/07, *Matt Hogstrom* < matt@hogstrom.org
>>     <mailto:matt@hogstrom.org>
>>     >>     <mailto:matt@hogstrom.org <mailto:matt@hogstrom.org>>>
 
>> wrote:
>>     >>
>>     >>         Only a very light load from a few browsers.  One thing
>>     to try
>>     >>         is to increase the number of SLSBs in the pool.
>>     >>
>>     >>         Can you add
>>     >>
>>     >>                         <session>
>>     >>                             <ejb-name>TradeJDBC</ejb-name>
>>     >>                             <jndi-name>ejb/TradeJDBC</jndi-

>> name>
>>     >>                             <cache-size>100</cache-size>
>>     >>                         </session>
>>     >>
>>     >>         to your plan and redeploy.  I added some support for
>>     multiple
>>     >>         SLSBs in a pool for 1.2 which we did not have  
>> before.  This
>>     >>         will hopefully make it better and not worse :)
>>     >>
>>     >>         On Apr 6, 2007, at 11:32 AM, Christopher Blythe wrote:
>>     >>
>>     >>>         Matt...
>>     >>>
>>     >>>         You mentioned that you deployed DayTrader 1.2...  
>> did you
>>     >>>         happen to run it under load? JDBC/Direct mode  
>> looks good;
>>     >>>         however, I am still seeing
>>     ConcurrentModificationExceptions
>>     >>>         while attempting to run more than 1 client in Session
>>     Direct
>>     >>>         mode (
>>     https://issues.apache.org/jira/browse/GERONIMO-2708).
>>     >>>         These exceptions are thrown throughout the  
>> duration of the
>>     >>>         run. FYI - I deployed the same ear on Geronimo  
>> 1.1.1 and
>>     >>>         didn't have a problem scaling up the users for  
>> Session
>>     >>>         Direct mode.
>>     >>>
>>     >>>         java.util.ConcurrentModificationException
>>     >>>             at
>>     java.util.HashMap$HashIterator.remove(HashMap.java:861)
>>     >>>             at
>>     >>>             
>> org.apache.geronimo.connector.outbound.connectiontracking.ConnectionT 
>> rackingCoordinator.exit
>>     >>>         ( ConnectionTrackingCoordinator.java :127)
>>     >>>             at
>>     >>>             
>> org.apache.geronimo.connector.outbound.connectiontracking.ConnectionT 
>> rackingCoordinator$$FastClassByCGLIB$$5d33aabf.invoke(<generated>)
>>
>>     >>>             at net.sf.cglib.reflect.FastMethod.invoke
>>     >>>         (FastMethod.java :53)
>>     >>>             at
>>     >>>             
>> org.apache.geronimo.gbean.runtime.FastMethodInvoker.invoke
>>     (FastMethodInvoker.java:38)
>>     >>>             at
>>     >>>             
>> org.apache.geronimo.gbean.runtime.GBeanOperation.invoke 
>> (GBeanOperation.java:122)
>>     >>>             at
>>     >>>          
>> org.apache.geronimo.gbean.runtime.GBeanInstance.invoke
>>     >>>         (GBeanInstance.java:820)
>>     >>>             at
>>     >>>             
>> org.apache.geronimo.gbean.runtime.RawInvoker.invoke 
>> (RawInvoker.java:57)
>>     >>>             at
>>     >>>             
>> org.apache.geronimo.kernel.basic.RawOperationInvoker.invoke 
>> (RawOperationInvoker.java:35)
>>     >>>             at
>>     >>>             
>> org.apache.geronimo.kernel.basic.ProxyMethodInterceptor.intercept
>>     >>>         ( ProxyMethodInterceptor.java:96)
>>     >>>             at
>>     >>>             
>> org.apache.geronimo.connector.outbound.connectiontracking.ConnectionT 
>> racker$$EnhancerByCGLIB$$b6b1324a.exit(<generated>)
>>     >>>             at
>>     >>>          
>> org.apache.openejb.NoConnectionEnlistingInterceptor.invoke
>>     >>>         (NoConnectionEnlistingInterceptor.java:70)
>>     >>>             at
>>     >>>         org.apache.openejb.SystemExceptionInterceptor.invoke
>>     (SystemExceptionInterceptor.java:35)
>>     >>>             at
>>     >>>             
>> org.apache.openejb.security.DefaultSubjectInterceptor.invoke 
>> (DefaultSubjectInterceptor.java
>>     >>>         :49)
>>     >>>             at
>>     >>>             
>> org.apache.openejb.slsb.DefaultStatelessEjbContainer.invoke 
>> (DefaultStatelessEjbContainer.java:178)
>>     >>>             at
>>     >>>             
>> org.apache.openejb.slsb.DefaultStatelessEjbContainer$ 
>> $FastClassByCGLIB$$7ad7a562.invoke
>>     (<generated>)
>>     >>>
>>     >>>             at
>>     >>>         net.sf.cglib.reflect.FastMethod.invoke 
>> (FastMethod.java:53)
>>     >>>             at
>>     >>>             
>> org.apache.geronimo.gbean.runtime.FastMethodInvoker.invoke
>>     (FastMethodInvoker.java:38)
>>     >>>             at
>>     >>>          
>> org.apache.geronimo.gbean.runtime.GBeanOperation.invoke
>>     >>>         (GBeanOperation.java:122)
>>     >>>             at
>>     >>>             
>> org.apache.geronimo.gbean.runtime.GBeanInstance.invoke 
>> (GBeanInstance.java:820)
>>     >>>             at
>>     >>>             
>> org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java
>>     :57)
>>     >>>             at
>>     >>>             
>> org.apache.geronimo.kernel.basic.RawOperationInvoker.invoke
>>     >>>         (RawOperationInvoker.java:35)
>>     >>>             at
>>     >>>             
>> org.apache.geronimo.kernel.basic.ProxyMethodInterceptor.intercept 
>> (ProxyMethodInterceptor.java:96)
>>     >>>             at
>>     >>>            org.apache.openejb.StatelessEjbContainer$ 
>> $EnhancerByCGLIB$$5c554f35.invoke
>>
>>     >>>         (<generated>)
>>     >>>             at
>>     >>>            org.apache.openejb.AbstractEjbDeployment.invoke 
>> (AbstractEjbDeployment.java:195)
>>     >>>             at
>>     >>>             
>> org.apache.openejb.proxy.EJBMethodInterceptor.intercept 
>> (EJBMethodInterceptor.java:145)
>>     >>>             at
>>     >>>            org.apache.openejb.proxy.SessionEJBObject$ 
>> $EnhancerByCGLIB$$f5a9c1b2.login
>>     >>>         (<generated>)
>>     >>>             at
>>     >>>             
>> org.apache.geronimo.samples.daytrader.TradeAction.login 
>> (TradeAction.java:449)
>>     >>>             at org.apache.geronimo.samples.daytrader.
>>     >>>         web.TradeServletAction.doLogin
>>     >>>            <http://web.TradeServletAction.doLogin> 
>> (TradeServletAction.java:364)
>>     >>>             at org.apache.geronimo.samples.daytrader .
>>     >>>         web.TradeAppServlet.performTask
>>     >>>            <http://web.TradeAppServlet.performTask> 
>> (TradeAppServlet.java:126)
>>     >>>             at org.apache.geronimo.samples.daytrader.
>>     >>>         web.TradeAppServlet.doPost
>>     >>>            <http://web.TradeAppServlet.doPost> 
>> (TradeAppServlet.java :91)
>>     >>>             at javax.servlet.http.HttpServlet.service
>>     >>>         (HttpServlet.java:617)
>>     >>>             at
>>     >>>            javax.servlet.http.HttpServlet.service 
>> (HttpServlet.java :690)
>>     >>>             at
>>     >>>             
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter
>>     >>>         (ApplicationFilterChain.java:252)
>>     >>>             at
>>     >>>          
>> org.apache.catalina.core.ApplicationFilterChain.doFilter
>>     >>>         (ApplicationFilterChain.java:173)
>>     >>>             at org.apache.geronimo.samples.daytrader.
>>     >>>         web.OrdersAlertFilter.doFilter
>>     >>>            <http://web.OrdersAlertFilter.doFilter> 
>> (OrdersAlertFilter.java:91)
>>     >>>             at
>>     >>>             
>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter
>>     (ApplicationFilterChain.java
>>     >>>         :202)
>>     >>>             at
>>     >>>          
>> org.apache.catalina.core.ApplicationFilterChain.doFilter
>>     >>>         (ApplicationFilterChain.java :173)
>>     >>>             at
>>     >>>             
>> org.apache.catalina.core.StandardWrapperValve.invoke 
>> (StandardWrapperValve.java:213)
>>     >>>             at
>>     org.apache.catalina.core.StandardContextValve.invoke
>>     >>>         (StandardContextValve.java:178)
>>     >>>             at
>>     >>>             
>> org.apache.geronimo.tomcat.valve.DefaultSubjectValve.invoke 
>> (DefaultSubjectValve.java:56)
>>     >>>             at
>>     >>>             
>> org.apache.geronimo.tomcat.GeronimoStandardContext 
>> $SystemMethodValve.invoke(GeronimoStandardContext.java
>>     >>>         :328)
>>     >>>             at
>>     >>>             
>> org.apache.geronimo.tomcat.valve.GeronimoBeforeAfterValve.invoke
>>     (GeronimoBeforeAfterValve.java:47)
>>     >>>             at
>>     >>>             
>> org.apache.catalina.core.StandardHostValve.invoke 
>> (StandardHostValve.java:126)
>>     >>>             at  
>> org.apache.catalina.valves.ErrorReportValve.invoke
>>     >>>         (ErrorReportValve.java:105)
>>     >>>             at
>>     >>>             
>> org.apache.catalina.core.StandardEngineValve.invoke 
>> (StandardEngineValve.java:107)
>>     >>>             at
>>     >>>            org.apache.catalina.valves.AccessLogValve.invoke 
>> (AccessLogValve.java:541)
>>     >>>             at  
>> org.apache.catalina.connector.CoyoteAdapter.service
>>     >>>         (CoyoteAdapter.java :148)
>>     >>>             at
>>     >>>            org.apache.coyote.http11.Http11Processor.process 
>> (Http11Processor.java:869)
>>     >>>             at
>>     >>>            org.apache.coyote.http11.Http11BaseProtocol 
>> $Http11ConnectionHandler.processConnection
>>     (Http11BaseProtocol.java
>>     >>>         :667)
>>     >>>             at
>>     >>>             
>> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket 
>> (PoolTcpEndpoint.java:527)
>>     >>>             at
>>     >>>             
>> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt 
>> (LeaderFollowerWorkerThread.java:80)
>>     >>>             at
>>     >>>            org.apache.tomcat.util.threads.ThreadPool 
>> $ControlRunnable.run
>>     >>>         (ThreadPool.java:684)
>>     >>>             at java.lang.Thread.run(Thread.java:797)
>>     >>>
>>     >>>         On 4/5/07, *Jason Dillon* < jason@planet57.com
>>     <mailto:jason@planet57.com>
>>     >>>         <mailto:jason@planet57.com
>>     <mailto:jason@planet57.com>>> wrote:
>>     >>>
>>     >>>             Aight, no worries.  I still don't fully  
>> understand
>>     all
>>     >>>             that plugin stuff... yet ;-)
>>     >>>
>>     >>>             --jason
>>     >>>
>>     >>>
>>     >>>             On Apr 5, 2007, at 3:38 PM, Paul McMahan wrote:
>>     >>>
>>     >>>>             The change I have cued up replaces "
>>     1.2-SNAPSHOT" with
>>     >>>>             " 1.2" for all the catalog entries.  So it would
>>     break
>>     >>>>             anyone using the Geronimo plugin repo from a
>>     >>>>             1.2-SNAPSHOT server (maybe not a huge deal).   
>> Also,
>>     >>>>             I've tested the catalog updates by looping http
>>     >>>>             requests to repo1.maven.org/maven2
>>     <http://repo1.maven.org/maven2>
>>     >>>>             <http://repo1.maven.org/maven2> back to my
local
>>     maven
>>     >>>>             repo.  So I've made some assumptions about  
>> the repo
>>     >>>>             layout that should probably be verified.
>>     >>>>
>>     >>>>             Best wishes,
>>     >>>>             Paul
>>     >>>>
>>     >>>>             On Apr 5, 2007, at 6:22 PM, Jason Dillon wrote:
>>     >>>>
>>     >>>>>             Will it hurt anything to commit it now?  Or
 
>> will it
>>     >>>>>             break things?
>>     >>>>>
>>     >>>>>             --jason
>>     >>>>>
>>     >>>>>
>>     >>>>>             On Apr 5, 2007, at 3:14 PM, Paul McMahan wrote:
>>     >>>>>
>>     >>>>>>
>>     >>>>>>             On Apr 5, 2007, at 2:11 PM, Joe Bohn wrote:
>>     >>>>>>
>>     >>>>>>>             I couldn't do much with the framework
 
>> assembly
>>     as it
>>     >>>>>>>             requires a plugin repository with 1.2
 
>> plugins and
>>     >>>>>>>             AFAIK there is no such plugin repository
 
>> available
>>     >>>>>>>             yet.  Will you be making the plugins
 
>> available for
>>     >>>>>>>             1.2 as you make the release available?
 If
>>     not, then
>>     >>>>>>>             perhaps we shouldn't include the framework
>>     assembly
>>     >>>>>>>             in the distribution.
>>     >>>>>>
>>     >>>>>>             I updated the plugin catalog stuff in
>>     >>>>>>             site/trunk/docs/plugins/geronimo- 1.2 locally
>>     and ran
>>     >>>>>>             some quick tests of plugin download &
 
>> install from
>>     >>>>>>             maven repo.  I'm ready to commit if/when
 
>> the 1.2
>>     >>>>>>             artifacts are published to central.
>>     >>>>>>
>>     >>>>>>             Best wishes,
>>     >>>>>>             Paul
>>     >>>>>>
>>     >>>>>
>>     >>>>
>>     >>>
>>     >>>
>>     >>>
>>     >>>
>>     >>>         --
>>     >>>         "I say never be complete, I say stop being  
>> perfect, I say
>>     >>>         let... lets evolve, let the chips fall where they  
>> may." -
>>     >>>         Tyler Durden
>>     >>
>>     >>
>>     >>
>>     >>
>>     >>     --
>>     >>     "I say never be complete, I say stop being perfect, I say
>>     let...
>>     >>     lets evolve, let the chips fall where they may." -  
>> Tyler Durden
>>     >>     <geronimo.log>
>>     >
>>     >
>>     >
>>     >
>>     > --
>>     > "I say never be complete, I say stop being perfect, I say  
>> let...
>>     lets
>>     > evolve, let the chips fall where they may." - Tyler Durden
>>
>>
>>
>>
>> -- 
>> "I say never be complete, I say stop being perfect, I say let...  
>> lets evolve, let the chips fall where they may." - Tyler Durden
> Index: /usr/src/geronimo-1.2/modules/geronimo-connector/src/main/ 
> java/org/apache/geronimo/connector/outbound/connectiontracking/ 
> ConnectionTrackingCoordinator.java
> ===================================================================
> --- /usr/src/geronimo-1.2/modules/geronimo-connector/src/main/java/ 
> org/apache/geronimo/connector/outbound/connectiontracking/ 
> ConnectionTrackingCoordinator.java	(revision 526213)
> +++ /usr/src/geronimo-1.2/modules/geronimo-connector/src/main/java/ 
> org/apache/geronimo/connector/outbound/connectiontracking/ 
> ConnectionTrackingCoordinator.java	(working copy)
> @@ -124,7 +124,23 @@
>
>                  // if no connection remain clear context... we  
> could support automatic commit, rollback or exception here
>                  if (connections.isEmpty()) {
> -                    i.remove();
> +                    boolean retry = false;
> +                    int numberOfRetries = 0;
> +                    do {
> +                        try {
> +                            i.remove();
> +                        } catch  
> (java.util.ConcurrentModificationException ex) {
> +                            if (numberOfRetries < 5) {
> +                                retry = true;
> +                            } else {
> +                                retry = false;
> +                            }
> +                            numberOfRetries += 1;
> +                        }
> +                    } while (retry);
> +                    if (numberOfRetries >= 5) {
> +                        throw new ResourceException 
> ("ConcurrentModificationException - Unable to remove resource");
> +                    }
>                  }
>              }
>          } finally {


Mime
View raw message