geronimo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kevan Miller <kevan.mil...@gmail.com>
Subject Re: HOWL exception
Date Tue, 18 Mar 2008 13:23:39 GMT

On Mar 14, 2008, at 6:25 AM, Tomasz Mazan wrote:

>
> Hi Guys
> I got very ugly (and blocking) issue with HOWL.
> After processing 20k request to my webservice whose are translated  
> to ~120k
> XA transactions (postgres  + jms) Geronimo hangs up and does not  
> respond on
> requests via HTTP, request to JMS engine (from HermesJMS) and  
> ignores tries
> to shutdown server.
>
> I stopped Geronimo with kill -9 and tried to start it again and got
> exception:
>
> Module 11/69 org.apache.geronimo.configs/activemq-ra/2.1-SNAPSHOT/car
> 10:22:15,325 ERROR [GBeanInstanceState] Error while starting; GBean  
> is now
> in the FAILED state:
> abstractName="org.apache.geronimo.configs/transaction/2.1-SNAPSHOT/ 
> car?ServiceModule=org.apache.geronimo.configs/transaction/2.1- 
> SNAPSHOT/car,j2eeType=TransactionLog,name=HOWLTransactionLog"
> java.lang.IllegalArgumentException: Negative position
>        at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:613)
>        at
> org.objectweb.howl.log.BlockLogBuffer.read(BlockLogBuffer.java:412)
>        at
> org.objectweb.howl.log.LogFileManager.read(LogFileManager.java:641)
>        at
> org.objectweb.howl.log.LogBufferManager.replay(LogBufferManager.java: 
> 869)
>        at org.objectweb.howl.log.Logger.replay(Logger.java:396)
>        at org.objectweb.howl.log.xa.XALogger.open(XALogger.java:897)
>        at
> org.apache.geronimo.transaction.log.HOWLLog.doStart(HOWLLog.java:224)
>        at
> org 
> .apache 
> .geronimo 
> .gbean.runtime.GBeanInstance.createInstance(GBeanInstance.java:996)
>        at
> org 
> .apache 
> .geronimo 
> .gbean 
> .runtime.GBeanInstanceState.attemptFullStart(GBeanInstanceState.java: 
> 268)
>        at
> org 
> .apache 
> .geronimo 
> .gbean.runtime.GBeanInstanceState.start(GBeanInstanceState.java:102)
>        at
> org 
> .apache 
> .geronimo.gbean.runtime.GBeanInstance.start(GBeanInstance.java:539)
>        at
> org 
> .apache 
> .geronimo 
> .gbean.runtime.GBeanDependency.attemptFullStart(GBeanDependency.java: 
> 111)
>        at
> org 
> .apache 
> .geronimo 
> .gbean.runtime.GBeanDependency.addTarget(GBeanDependency.java:146)
>        at
> org.apache.geronimo.gbean.runtime.GBeanDependency 
> $1.running(GBeanDependency.java:120)
>        at
> org 
> .apache 
> .geronimo 
> .kernel 
> .basic 
> .BasicLifecycleMonitor.fireRunningEvent(BasicLifecycleMonitor.java: 
> 176)
>        at
> org.apache.geronimo.kernel.basic.BasicLifecycleMonitor.access 
> $300(BasicLifecycleMonitor.java:44)
>        at
> org.apache.geronimo.kernel.basic.BasicLifecycleMonitor 
> $RawLifecycleBroadcaster.fireRunningEvent(BasicLifecycleMonitor.java: 
> 254)
>        at
> org 
> .apache 
> .geronimo 
> .gbean 
> .runtime.GBeanInstanceState.attemptFullStart(GBeanInstanceState.java: 
> 294)
>        at
> org 
> .apache 
> .geronimo 
> .gbean.runtime.GBeanInstanceState.start(GBeanInstanceState.java:102)
>        at
> org 
> .apache 
> .geronimo 
> .gbean 
> .runtime.GBeanInstanceState.startRecursive(GBeanInstanceState.java: 
> 124)
>        at
> org 
> .apache 
> .geronimo 
> .gbean.runtime.GBeanInstance.startRecursive(GBeanInstance.java:553)
>        at
> org 
> .apache 
> .geronimo 
> .kernel.basic.BasicKernel.startRecursiveGBean(BasicKernel.java:379)
>        at
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .ConfigurationUtil.startConfigurationGBeans(ConfigurationUtil.java: 
> 448)
>        at
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .KernelConfigurationManager.start(KernelConfigurationManager.java:187)
>        at
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .SimpleConfigurationManager 
> .startConfiguration(SimpleConfigurationManager.java:534)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun 
> .reflect 
> .NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun 
> .reflect 
> .DelegatingMethodAccessorImpl 
> .invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at
> org 
> .apache 
> .geronimo 
> .gbean 
> .runtime.ReflectionMethodInvoker.invoke(ReflectionMethodInvoker.java: 
> 34)
>        at
> org 
> .apache 
> .geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:124)
>        at
> org 
> .apache 
> .geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:830)
>        at
> org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java: 
> 57)
>        at
> org 
> .apache 
> .geronimo 
> .kernel.basic.RawOperationInvoker.invoke(RawOperationInvoker.java:35)
>        at
> org 
> .apache 
> .geronimo 
> .kernel 
> .basic.ProxyMethodInterceptor.intercept(ProxyMethodInterceptor.java: 
> 96)
>        at
> org.apache.geronimo.gbean.GBeanLifecycle$$EnhancerByCGLIB$ 
> $5ebee84a.startConfiguration(<generated>)
>        at
> org 
> .apache 
> .geronimo.system.main.EmbeddedDaemon.doStartup(EmbeddedDaemon.java: 
> 156)
>        at
> org 
> .apache 
> .geronimo.system.main.EmbeddedDaemon.execute(EmbeddedDaemon.java:78)
>        at
> org 
> .apache 
> .geronimo 
> .kernel 
> .util 
> .MainConfigurationBootstrapper 
> .main(MainConfigurationBootstrapper.java:45)
>        at
> org.apache.geronimo.cli.AbstractCLI.executeMain(AbstractCLI.java:67)
>        at  
> org.apache.geronimo.cli.daemon.DaemonCLI.main(DaemonCLI.java:30)
> Server Startup failed
>
> org.apache.geronimo.kernel.config.LifecycleException: start of
> org.apache.geronimo.configs/activemq-ra/2.1-SNAPSHOT/car failed
>        at
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .SimpleConfigurationManager 
> .startConfiguration(SimpleConfigurationManager.java:551)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at
> sun 
> .reflect 
> .NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>        at
> sun 
> .reflect 
> .DelegatingMethodAccessorImpl 
> .invoke(DelegatingMethodAccessorImpl.java:25)
>        at java.lang.reflect.Method.invoke(Method.java:597)
>        at
> org 
> .apache 
> .geronimo 
> .gbean 
> .runtime.ReflectionMethodInvoker.invoke(ReflectionMethodInvoker.java: 
> 34)
>        at
> org 
> .apache 
> .geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:124)
>        at
> org 
> .apache 
> .geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:830)
>        at
> org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java: 
> 57)
>        at
> org 
> .apache 
> .geronimo 
> .kernel.basic.RawOperationInvoker.invoke(RawOperationInvoker.java:35)
>        at
> org 
> .apache 
> .geronimo 
> .kernel 
> .basic.ProxyMethodInterceptor.intercept(ProxyMethodInterceptor.java: 
> 96)
>        at
> org.apache.geronimo.gbean.GBeanLifecycle$$EnhancerByCGLIB$ 
> $5ebee84a.startConfiguration(<generated>)
>        at
> org 
> .apache 
> .geronimo.system.main.EmbeddedDaemon.doStartup(EmbeddedDaemon.java: 
> 156)
>        at
> org 
> .apache 
> .geronimo.system.main.EmbeddedDaemon.execute(EmbeddedDaemon.java:78)
>        at
> org 
> .apache 
> .geronimo 
> .kernel 
> .util 
> .MainConfigurationBootstrapper 
> .main(MainConfigurationBootstrapper.java:45)
>        at
> org.apache.geronimo.cli.AbstractCLI.executeMain(AbstractCLI.java:67)
>        at  
> org.apache.geronimo.cli.daemon.DaemonCLI.main(DaemonCLI.java:30)
> Caused by: org.apache.geronimo.kernel.config.InvalidConfigException:  
> Unknown
> start exception
>        at
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .ConfigurationUtil.startConfigurationGBeans(ConfigurationUtil.java: 
> 514)
>        at
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .KernelConfigurationManager.start(KernelConfigurationManager.java:187)
>        at
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .SimpleConfigurationManager 
> .startConfiguration(SimpleConfigurationManager.java:534)
>        ... 16 more
> Caused by: org.apache.geronimo.gbean.InvalidConfigurationException:
> Configuration org.apache.geronimo.configs/transaction/2.1-SNAPSHOT/car
> failed to start due to the following reasons:
>  The service
> ServiceModule=org.apache.geronimo.configs/transaction/2.1-SNAPSHOT/ 
> car,j2eeType=JCAConnectionTracker,name=ConnectionTracker
> did not start because
> org.apache.geronimo.configs/transaction/2.1-SNAPSHOT/car? 
> ServiceModule=org.apache.geronimo.configs/transaction/2.1-SNAPSHOT/ 
> car,j2eeType=JTAResource,name=TransactionManager
> did not start.
>  The service
> ServiceModule=org.apache.geronimo.configs/transaction/2.1-SNAPSHOT/ 
> car,j2eeType=JCAWorkManager,name=DefaultWorkManager
> did not start because
> org.apache.geronimo.configs/transaction/2.1-SNAPSHOT/car? 
> ServiceModule=org.apache.geronimo.configs/transaction/2.1-SNAPSHOT/ 
> car,j2eeType=JTAResource,name=TransactionManager
> did not start.
>  The service
> ServiceModule=org.apache.geronimo.configs/transaction/2.1-SNAPSHOT/ 
> car,j2eeType=TransactionLog,name=HOWLTransactionLog
> did not start because Negative position
>  The service
> ServiceModule=org.apache.geronimo.configs/transaction/2.1-SNAPSHOT/ 
> car,j2eeType=JTAResource,name=TransactionManager
> did not start because
> org.apache.geronimo.configs/transaction/2.1-SNAPSHOT/car? 
> ServiceModule=org.apache.geronimo.configs/transaction/2.1-SNAPSHOT/ 
> car,j2eeType=TransactionLog,name=HOWLTransactionLog
> did not start.
>
>        at
> org 
> .apache 
> .geronimo 
> .kernel 
> .config 
> .ConfigurationUtil.startConfigurationGBeans(ConfigurationUtil.java: 
> 477)
>        ... 18 more
>
> I've found that HOWL's log-file reached 4G !

Hi Beniamin,
The size of the logfile is the reason for the IllegalArgumentException  
-- seems to be an integer overflow. I think the following bug report  
describes the situation -- http://forge.objectweb.org/tracker/index.php?func=detail&aid=306425&group_id=92&atid=100092

. HOWL 1.0.2 is supposed to detect the condition, but, IIUC, does not  
fix the underlying problem. Also, I'm not finding 1.0.2 on a maven  
repo...

Looks like we have some digging to do around our transaction logging.  
I'm not familiar with the specifics of the HOWL implementation... I'll  
see if I can devote some spare cycles to this, but would be great if  
anyone else could investigate.

The cause of the IllegalStateException is understood and should be  
fixed in http://issues.apache.org/jira/browse/OPENEJB-702

--kevan

>
>
> I have to add that each XA transaction with accesing activemq causes  
> log
> like below:
>
> 10:49:11,398 ERROR [Transaction] Please correct the integration and  
> supply a
> NamedXAResource
> java.lang.IllegalStateException: Cannot log transactions as
> org.apache.activemq.ra.LocalAndXATransaction@e363593 is not a
> NamedXAResource.
>        at
> org.apache.geronimo.transaction.manager.TransactionImpl 
> $TransactionBranch.getResourceName(TransactionImpl.java:697)
>        at
> org.apache.geronimo.transaction.log.HOWLLog.prepare(HOWLLog.java:254)
>        at
> org 
> .apache 
> .geronimo 
> .transaction 
> .manager.TransactionImpl.internalPrepare(TransactionImpl.java:444)
>        at
> org 
> .apache 
> .geronimo 
> .transaction.manager.TransactionImpl.commit(TransactionImpl.java:316)
>        at
> org 
> .apache 
> .geronimo 
> .transaction 
> .manager.TransactionManagerImpl.commit(TransactionManagerImpl.java: 
> 245)
>        at
> org 
> .apache 
> .openejb 
> .core 
> .transaction 
> .TransactionPolicy.commitTransaction(TransactionPolicy.java:140)
>        at
> org 
> .apache 
> .openejb.core.transaction.TxRequired.afterInvoke(TxRequired.java:75)
>        at
> org 
> .apache 
> .openejb.core.mdb.MdbContainer.afterDelivery(MdbContainer.java:388)
>        at
> org 
> .apache 
> .openejb.core.mdb.EndpointHandler.afterDelivery(EndpointHandler.java: 
> 274)
>        at
> org 
> .apache.openejb.core.mdb.EndpointHandler.invoke(EndpointHandler.java: 
> 164)
>        at $Proxy42.afterDelivery(Unknown Source)
>        at
> org.apache.activemq.ra.MessageEndpointProxy 
> $MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:126)
>        at
> org 
> .apache 
> .activemq 
> .ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:65)
>        at
> org 
> .apache 
> .activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java: 
> 216)
>        at  
> org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:751)
>        at
> org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java: 
> 165)
>        at
> org 
> .apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java: 
> 290)
>        at
> org 
> .apache 
> .geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
>        at org.apache.geronimo.pool.ThreadPool$1.run(ThreadPool.java: 
> 214)
>        at
> org.apache.geronimo.pool.ThreadPool 
> $ContextClassLoaderRunnable.run(ThreadPool.java:344)
>        at
> java.util.concurrent.ThreadPoolExecutor 
> $Worker.runTask(ThreadPoolExecutor.java:885)
>        at
> java.util.concurrent.ThreadPoolExecutor 
> $Worker.run(ThreadPoolExecutor.java:907)
>        at java.lang.Thread.run(Thread.java:619)
>
> I'm looking for solution for this issue.
>
> Thanks
> Beniamin
> -- 
> View this message in context: http://www.nabble.com/HOWL-exception-tp16047940s134p16047940.html
> Sent from the Apache Geronimo - Users mailing list archive at  
> Nabble.com.
>


Mime
View raw message