Return-Path: Delivered-To: apmail-geronimo-dev-archive@www.apache.org Received: (qmail 48835 invoked from network); 6 Apr 2007 20:46:47 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 6 Apr 2007 20:46:47 -0000 Received: (qmail 6540 invoked by uid 500); 6 Apr 2007 20:46:53 -0000 Delivered-To: apmail-geronimo-dev-archive@geronimo.apache.org Received: (qmail 5976 invoked by uid 500); 6 Apr 2007 20:46:52 -0000 Mailing-List: contact dev-help@geronimo.apache.org; run by ezmlm Precedence: bulk list-help: list-unsubscribe: List-Post: Reply-To: dev@geronimo.apache.org List-Id: Delivered-To: mailing list dev@geronimo.apache.org Received: (qmail 5964 invoked by uid 99); 6 Apr 2007 20:46:52 -0000 Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 Apr 2007 13:46:52 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received-SPF: pass (herse.apache.org: local policy) Received: from [68.142.206.241] (HELO smtp108.plus.mail.mud.yahoo.com) (68.142.206.241) by apache.org (qpsmtpd/0.29) with SMTP; Fri, 06 Apr 2007 13:46:44 -0700 Received: (qmail 24407 invoked from network); 6 Apr 2007 20:46:23 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Received:X-YMail-OSG:Mime-Version:In-Reply-To:References:Content-Type:Message-Id:Content-Transfer-Encoding:From:Subject:Date:To:X-Mailer; b=GT81ifThxTC463q4fPoonRsftxQmz//9zAosP4FGQloSTDMYUz8ST227BCAE9nglS5aoJGr286irc92bZV5mCmMgm/oZjl1JwYOD1QIEhK+OtJ8a4inbMiZUJXTGF4bGll3PCDhrAiCTSSTxKay0c8+kcM0hawPFoPElGFJ6o3M= ; Received: from unknown (HELO ?10.11.55.8?) (david_jencks@63.105.20.225 with plain) by smtp108.plus.mail.mud.yahoo.com with SMTP; 6 Apr 2007 20:46:22 -0000 X-YMail-OSG: lb5J4q0VM1ngrb4ILFrvco8Hbqd_HHWvv9UWGZKYIPiNZkhCyO6403dHDYf2LCTf8Kmj_QQpQfPNewr1kebMLp17_vAvKVMRM.0btero8kP96usg8bmIDKfU29RiOOHfKi3QsfCwaXmrw_zQb5reCfFLK6.Rag.n5Nx3l_c1mck- Mime-Version: 1.0 (Apple Message framework v752.3) In-Reply-To: <4616B025.20807@joyfulnoisewebdesign.com> References: <421012bd0704060832n47eca4b8odc33c821c940379f@mail.gmail.com> <421012bd0704061109l50cf840eocbabebfb5a72e467@mail.gmail.com> <421012bd0704061149k98d3955lf281b238130648d6@mail.gmail.com> <46169D44.1070203@joyfulnoisewebdesign.com> <421012bd0704061221r249af866o36f6022141bd1e2d@mail.gmail.com> <4616B025.20807@joyfulnoisewebdesign.com> Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <545F4E47-F522-4A70-9B32-EB4F632940E1@yahoo.com> Content-Transfer-Encoding: 7bit From: David Jencks Subject: Re: [discuss] Release Geronimo 1.2 Date: Fri, 6 Apr 2007 13:46:06 -0700 To: dev@geronimo.apache.org X-Mailer: Apple Mail (2.752.3) X-Virus-Checked: Checked by ClamAV on apache.org I don't think this is acceptable. There should be only one thread working with the context at a time. Either this exception is caused by modifying the collection in the same thread in which case we can fix it easily or it is caused by more than one thread having access to a context at once. Since by my reading of the code this is a context attached to a stateless session bean instance, that would mean that more than one thread is using a stateless session bean instance at once, which is definitely cause to -1 the release. I hope there's another possibility I haven't thought of..... but hiding the problem is not acceptable unless we really understand what is going on and are really convinced it's harmless. thanks david jencks On Apr 6, 2007, at 1:40 PM, Jay D. McHugh wrote: > Chris (do you go by Chris or Christopher?), > > Here is a patch that I just wrote that allows the exit routine of > ConnectionTrackingCoordinator to finish cleanly after a number (5) > of attempts at removing the resource. > > If it fails after five tries, then the routine exits and throws a > ResourceException (that will hopefully be caught further up the > stack). > > Would you like to try it to see if it solves your concurrency problem? > > Jay > > Christopher Blythe wrote: >> Doubtful... everything tested fine under light browser based >> testing. As the exception suggests, this is a concurrency problem >> that you would only hit under load. >> >> On 4/6/07, * Jay D. McHugh* > > wrote: >> >> If Matt had no problem deploying and testing DT, could it be a >> Java >> version or classpath issue? >> >> That could explain the difference in the exception during >> deployment >> (and the problems during deployment could possibly explain the >> run >> time >> problems). >> >> Jay >> >> Christopher Blythe wrote: >> > I use a commercial load driving tool... FYI, I'm fairly >> certain that >> > G-2.0 has the same issue. >> > >> > On 4/6/07, *David Jencks* < david_jencks@yahoo.com >> >> > > >> >> wrote: >> > >> > I think we need to figure out why the >> > concurrentModificationException is happening before we >> release. I >> > think that one possible reason is that we are >> multithreading >> > stateless session bean instances. I hope this isn't the >> cause.... >> > but IMO we need to find out. >> > >> > Chris, how do you run the several clients? manually or >> with >> a tool? >> > >> > thanks >> > david jencks >> > >> > >> > On Apr 6, 2007, at 11:09 AM, Christopher Blythe wrote: >> > >> >> Gave it a shot... no luck. As soon as I started 2 >> clients, the >> >> same exceptions started to pile up. I have attached the >> >> geronimo.log. Also, noticed the following exception during >> startup. >> >> >> >> 14:05:00,640 ERROR [TransportConnector] Could not accept >> >> connection from /127.0.0.1:28428: java.io.IOException: >> Wire >> >> format negociation timeout: peer did not send his wire >> format. >> >> java.io.IOException: Wire format negociation timeout: >> peer did >> >> not send his wire format. >> >> at >> org.apache.activemq.transport.WireFormatNegotiator.oneway >> >> (WireFormatNegotiator.java :88) >> >> at >> >> org.apache.activemq.transport.MutexTransport.oneway >> (MutexTransport.java:47) >> >> at >> org.apache.activemq.broker.TransportConnection.dispatch >> >> (TransportConnection.java :1138) >> >> at >> >> >> org.apache.activemq.broker.TransportConnection.processDispatch >> (TransportConnection.java:805) >> >> at >> >> org.apache.activemq.broker.TransportConnection.start >> (TransportConnection.java >> >> :885) >> >> at org.apache.activemq.broker.TransportConnector >> $1.onAccept >> >> (TransportConnector.java:148) >> >> at >> >> org.apache.activemq.transport.tcp.TcpTransportServer.run >> (TcpTransportServer.java:167) >> >> at java.lang.Thread.run (Thread.java:797) >> >> >> >> >> >> >> >> On 4/6/07, *Matt Hogstrom* < matt@hogstrom.org >> >> >> >> >> wrote: >> >> >> >> Only a very light load from a few browsers. One thing >> to try >> >> is to increase the number of SLSBs in the pool. >> >> >> >> Can you add >> >> >> >> >> >> TradeJDBC >> >> ejb/TradeJDBC> name> >> >> 100 >> >> >> >> >> >> to your plan and redeploy. I added some support for >> multiple >> >> SLSBs in a pool for 1.2 which we did not have >> before. This >> >> will hopefully make it better and not worse :) >> >> >> >> On Apr 6, 2007, at 11:32 AM, Christopher Blythe wrote: >> >> >> >>> Matt... >> >>> >> >>> You mentioned that you deployed DayTrader 1.2... >> did you >> >>> happen to run it under load? JDBC/Direct mode >> looks good; >> >>> however, I am still seeing >> ConcurrentModificationExceptions >> >>> while attempting to run more than 1 client in Session >> Direct >> >>> mode ( >> https://issues.apache.org/jira/browse/GERONIMO-2708). >> >>> These exceptions are thrown throughout the >> duration of the >> >>> run. FYI - I deployed the same ear on Geronimo >> 1.1.1 and >> >>> didn't have a problem scaling up the users for >> Session >> >>> Direct mode. >> >>> >> >>> java.util.ConcurrentModificationException >> >>> at >> java.util.HashMap$HashIterator.remove(HashMap.java:861) >> >>> at >> >>> >> org.apache.geronimo.connector.outbound.connectiontracking.ConnectionT >> rackingCoordinator.exit >> >>> ( ConnectionTrackingCoordinator.java :127) >> >>> at >> >>> >> org.apache.geronimo.connector.outbound.connectiontracking.ConnectionT >> rackingCoordinator$$FastClassByCGLIB$$5d33aabf.invoke() >> >> >>> at net.sf.cglib.reflect.FastMethod.invoke >> >>> (FastMethod.java :53) >> >>> at >> >>> >> org.apache.geronimo.gbean.runtime.FastMethodInvoker.invoke >> (FastMethodInvoker.java:38) >> >>> at >> >>> >> org.apache.geronimo.gbean.runtime.GBeanOperation.invoke >> (GBeanOperation.java:122) >> >>> at >> >>> >> org.apache.geronimo.gbean.runtime.GBeanInstance.invoke >> >>> (GBeanInstance.java:820) >> >>> at >> >>> >> org.apache.geronimo.gbean.runtime.RawInvoker.invoke >> (RawInvoker.java:57) >> >>> at >> >>> >> org.apache.geronimo.kernel.basic.RawOperationInvoker.invoke >> (RawOperationInvoker.java:35) >> >>> at >> >>> >> org.apache.geronimo.kernel.basic.ProxyMethodInterceptor.intercept >> >>> ( ProxyMethodInterceptor.java:96) >> >>> at >> >>> >> org.apache.geronimo.connector.outbound.connectiontracking.ConnectionT >> racker$$EnhancerByCGLIB$$b6b1324a.exit() >> >>> at >> >>> >> org.apache.openejb.NoConnectionEnlistingInterceptor.invoke >> >>> (NoConnectionEnlistingInterceptor.java:70) >> >>> at >> >>> org.apache.openejb.SystemExceptionInterceptor.invoke >> (SystemExceptionInterceptor.java:35) >> >>> at >> >>> >> org.apache.openejb.security.DefaultSubjectInterceptor.invoke >> (DefaultSubjectInterceptor.java >> >>> :49) >> >>> at >> >>> >> org.apache.openejb.slsb.DefaultStatelessEjbContainer.invoke >> (DefaultStatelessEjbContainer.java:178) >> >>> at >> >>> >> org.apache.openejb.slsb.DefaultStatelessEjbContainer$ >> $FastClassByCGLIB$$7ad7a562.invoke >> () >> >>> >> >>> at >> >>> net.sf.cglib.reflect.FastMethod.invoke >> (FastMethod.java:53) >> >>> at >> >>> >> org.apache.geronimo.gbean.runtime.FastMethodInvoker.invoke >> (FastMethodInvoker.java:38) >> >>> at >> >>> >> org.apache.geronimo.gbean.runtime.GBeanOperation.invoke >> >>> (GBeanOperation.java:122) >> >>> at >> >>> >> org.apache.geronimo.gbean.runtime.GBeanInstance.invoke >> (GBeanInstance.java:820) >> >>> at >> >>> >> org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java >> :57) >> >>> at >> >>> >> org.apache.geronimo.kernel.basic.RawOperationInvoker.invoke >> >>> (RawOperationInvoker.java:35) >> >>> at >> >>> >> org.apache.geronimo.kernel.basic.ProxyMethodInterceptor.intercept >> (ProxyMethodInterceptor.java:96) >> >>> at >> >>> org.apache.openejb.StatelessEjbContainer$ >> $EnhancerByCGLIB$$5c554f35.invoke >> >> >>> () >> >>> at >> >>> org.apache.openejb.AbstractEjbDeployment.invoke >> (AbstractEjbDeployment.java:195) >> >>> at >> >>> >> org.apache.openejb.proxy.EJBMethodInterceptor.intercept >> (EJBMethodInterceptor.java:145) >> >>> at >> >>> org.apache.openejb.proxy.SessionEJBObject$ >> $EnhancerByCGLIB$$f5a9c1b2.login >> >>> () >> >>> at >> >>> >> org.apache.geronimo.samples.daytrader.TradeAction.login >> (TradeAction.java:449) >> >>> at org.apache.geronimo.samples.daytrader. >> >>> web.TradeServletAction.doLogin >> >>> >> (TradeServletAction.java:364) >> >>> at org.apache.geronimo.samples.daytrader . >> >>> web.TradeAppServlet.performTask >> >>> >> (TradeAppServlet.java:126) >> >>> at org.apache.geronimo.samples.daytrader. >> >>> web.TradeAppServlet.doPost >> >>> >> (TradeAppServlet.java :91) >> >>> at javax.servlet.http.HttpServlet.service >> >>> (HttpServlet.java:617) >> >>> at >> >>> javax.servlet.http.HttpServlet.service >> (HttpServlet.java :690) >> >>> at >> >>> >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter >> >>> (ApplicationFilterChain.java:252) >> >>> at >> >>> >> org.apache.catalina.core.ApplicationFilterChain.doFilter >> >>> (ApplicationFilterChain.java:173) >> >>> at org.apache.geronimo.samples.daytrader. >> >>> web.OrdersAlertFilter.doFilter >> >>> >> (OrdersAlertFilter.java:91) >> >>> at >> >>> >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter >> (ApplicationFilterChain.java >> >>> :202) >> >>> at >> >>> >> org.apache.catalina.core.ApplicationFilterChain.doFilter >> >>> (ApplicationFilterChain.java :173) >> >>> at >> >>> >> org.apache.catalina.core.StandardWrapperValve.invoke >> (StandardWrapperValve.java:213) >> >>> at >> org.apache.catalina.core.StandardContextValve.invoke >> >>> (StandardContextValve.java:178) >> >>> at >> >>> >> org.apache.geronimo.tomcat.valve.DefaultSubjectValve.invoke >> (DefaultSubjectValve.java:56) >> >>> at >> >>> >> org.apache.geronimo.tomcat.GeronimoStandardContext >> $SystemMethodValve.invoke(GeronimoStandardContext.java >> >>> :328) >> >>> at >> >>> >> org.apache.geronimo.tomcat.valve.GeronimoBeforeAfterValve.invoke >> (GeronimoBeforeAfterValve.java:47) >> >>> at >> >>> >> org.apache.catalina.core.StandardHostValve.invoke >> (StandardHostValve.java:126) >> >>> at >> org.apache.catalina.valves.ErrorReportValve.invoke >> >>> (ErrorReportValve.java:105) >> >>> at >> >>> >> org.apache.catalina.core.StandardEngineValve.invoke >> (StandardEngineValve.java:107) >> >>> at >> >>> org.apache.catalina.valves.AccessLogValve.invoke >> (AccessLogValve.java:541) >> >>> at >> org.apache.catalina.connector.CoyoteAdapter.service >> >>> (CoyoteAdapter.java :148) >> >>> at >> >>> org.apache.coyote.http11.Http11Processor.process >> (Http11Processor.java:869) >> >>> at >> >>> org.apache.coyote.http11.Http11BaseProtocol >> $Http11ConnectionHandler.processConnection >> (Http11BaseProtocol.java >> >>> :667) >> >>> at >> >>> >> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket >> (PoolTcpEndpoint.java:527) >> >>> at >> >>> >> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt >> (LeaderFollowerWorkerThread.java:80) >> >>> at >> >>> org.apache.tomcat.util.threads.ThreadPool >> $ControlRunnable.run >> >>> (ThreadPool.java:684) >> >>> at java.lang.Thread.run(Thread.java:797) >> >>> >> >>> On 4/5/07, *Jason Dillon* < jason@planet57.com >> >> >>> > >> wrote: >> >>> >> >>> Aight, no worries. I still don't fully >> understand >> all >> >>> that plugin stuff... yet ;-) >> >>> >> >>> --jason >> >>> >> >>> >> >>> On Apr 5, 2007, at 3:38 PM, Paul McMahan wrote: >> >>> >> >>>> The change I have cued up replaces " >> 1.2-SNAPSHOT" with >> >>>> " 1.2" for all the catalog entries. So it would >> break >> >>>> anyone using the Geronimo plugin repo from a >> >>>> 1.2-SNAPSHOT server (maybe not a huge deal). >> Also, >> >>>> I've tested the catalog updates by looping http >> >>>> requests to repo1.maven.org/maven2 >> >> >>>> back to my local >> maven >> >>>> repo. So I've made some assumptions about >> the repo >> >>>> layout that should probably be verified. >> >>>> >> >>>> Best wishes, >> >>>> Paul >> >>>> >> >>>> On Apr 5, 2007, at 6:22 PM, Jason Dillon wrote: >> >>>> >> >>>>> Will it hurt anything to commit it now? Or >> will it >> >>>>> break things? >> >>>>> >> >>>>> --jason >> >>>>> >> >>>>> >> >>>>> On Apr 5, 2007, at 3:14 PM, Paul McMahan wrote: >> >>>>> >> >>>>>> >> >>>>>> On Apr 5, 2007, at 2:11 PM, Joe Bohn wrote: >> >>>>>> >> >>>>>>> I couldn't do much with the framework >> assembly >> as it >> >>>>>>> requires a plugin repository with 1.2 >> plugins and >> >>>>>>> AFAIK there is no such plugin repository >> available >> >>>>>>> yet. Will you be making the plugins >> available for >> >>>>>>> 1.2 as you make the release available? If >> not, then >> >>>>>>> perhaps we shouldn't include the framework >> assembly >> >>>>>>> in the distribution. >> >>>>>> >> >>>>>> I updated the plugin catalog stuff in >> >>>>>> site/trunk/docs/plugins/geronimo- 1.2 locally >> and ran >> >>>>>> some quick tests of plugin download & >> install from >> >>>>>> maven repo. I'm ready to commit if/when >> the 1.2 >> >>>>>> artifacts are published to central. >> >>>>>> >> >>>>>> Best wishes, >> >>>>>> Paul >> >>>>>> >> >>>>> >> >>>> >> >>> >> >>> >> >>> >> >>> >> >>> -- >> >>> "I say never be complete, I say stop being >> perfect, I say >> >>> let... lets evolve, let the chips fall where they >> may." - >> >>> Tyler Durden >> >> >> >> >> >> >> >> >> >> -- >> >> "I say never be complete, I say stop being perfect, I say >> let... >> >> lets evolve, let the chips fall where they may." - >> Tyler Durden >> >> >> > >> > >> > >> > >> > -- >> > "I say never be complete, I say stop being perfect, I say >> let... >> lets >> > evolve, let the chips fall where they may." - Tyler Durden >> >> >> >> >> -- >> "I say never be complete, I say stop being perfect, I say let... >> lets evolve, let the chips fall where they may." - Tyler Durden > Index: /usr/src/geronimo-1.2/modules/geronimo-connector/src/main/ > java/org/apache/geronimo/connector/outbound/connectiontracking/ > ConnectionTrackingCoordinator.java > =================================================================== > --- /usr/src/geronimo-1.2/modules/geronimo-connector/src/main/java/ > org/apache/geronimo/connector/outbound/connectiontracking/ > ConnectionTrackingCoordinator.java (revision 526213) > +++ /usr/src/geronimo-1.2/modules/geronimo-connector/src/main/java/ > org/apache/geronimo/connector/outbound/connectiontracking/ > ConnectionTrackingCoordinator.java (working copy) > @@ -124,7 +124,23 @@ > > // if no connection remain clear context... we > could support automatic commit, rollback or exception here > if (connections.isEmpty()) { > - i.remove(); > + boolean retry = false; > + int numberOfRetries = 0; > + do { > + try { > + i.remove(); > + } catch > (java.util.ConcurrentModificationException ex) { > + if (numberOfRetries < 5) { > + retry = true; > + } else { > + retry = false; > + } > + numberOfRetries += 1; > + } > + } while (retry); > + if (numberOfRetries >= 5) { > + throw new ResourceException > ("ConcurrentModificationException - Unable to remove resource"); > + } > } > } > } finally {