Mailing-List: contact dev-help@geronimo.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@geronimo.apache.org
Message-ID: <28029424.212841296041563349.JavaMail.jira@thor>
Date: Wed, 26 Jan 2011 06:32:43 -0500 (EST)
From: "Daniel Stoddart (JIRA)" <jira@apache.org>
To: dev@geronimo.apache.org
Subject: [jira] Created: (GERONIMO-5785) Geronimo Derby database corrupted
 after unix server restart (geronimo running at the time of restart)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Geronimo Derby database corrupted after unix server restart (geronimo running at the time of restart)
-----------------------------------------------------------------------------------------------------

                 Key: GERONIMO-5785
                 URL: https://issues.apache.org/jira/browse/GERONIMO-5785
             Project: Geronimo
          Issue Type: Bug
      Security Level: public (Regular issues)
          Components: startup/shutdown
    Affects Versions: 2.1.4
            Reporter: Daniel Stoddart


Hi,

We have several process running an a unix server and if we are not aware that it is going to be restarted and Geronimo is not shutdown first via the shutdown.sh script or through the Console we cannot restart geronimo and need to reinstall which can take some time.

This is what we see in /var/log/geronimo.log

2011-01-26 10:51:05,289 ERROR [GBeanInstanceState] Error while starting; GBean is now in the FAILED state: abstractName="org.apache.geronimo.configs/transaction/2.1.4/car?ServiceModule=org.apache.geronimo.configs/transaction/2.1.4/car,j2eeType=TransactionLog,name=HOWLTransactionLog"
org.objectweb.howl.log.LogConfigurationException: Unable to obtain lock on /home/ceh1/poseidon_dev/janap01/apps/uat/uat.geronimo-tomcat6-javaee5-2.1.4/./var/txlog/howl_1.log
        at org.objectweb.howl.log.LogFile.open(LogFile.java:191)
        at org.objectweb.howl.log.LogFileManager.open(LogFileManager.java:784)
        at org.objectweb.howl.log.Logger.open(Logger.java:303)
        at org.objectweb.howl.log.xa.XALogger.open(XALogger.java:893)
        at org.apache.geronimo.transaction.log.HOWLLog.doStart(HOWLLog.java:224)
        at org.apache.geronimo.gbean.runtime.GBeanInstance.createInstance(GBeanInstance.java:998)
        at org.apache.geronimo.gbean.runtime.GBeanInstanceState.attemptFullStart(GBeanInstanceState.java:268)
        at org.apache.geronimo.gbean.runtime.GBeanInstanceState.start(GBeanInstanceState.java:102)
        at org.apache.geronimo.gbean.runtime.GBeanInstance.start(GBeanInstance.java:541)
        at org.apache.geronimo.gbean.runtime.GBeanDependency.attemptFullStart(GBeanDependency.java:111)
        at org.apache.geronimo.gbean.runtime.GBeanDependency.addTarget(GBeanDependency.java:146)
        at org.apache.geronimo.gbean.runtime.GBeanDependency$1.running(GBeanDependency.java:120)
        at org.apache.geronimo.kernel.basic.BasicLifecycleMonitor.fireRunningEvent(BasicLifecycleMonitor.java:176)
        at org.apache.geronimo.kernel.basic.BasicLifecycleMonitor.access$300(BasicLifecycleMonitor.java:44)
        at org.apache.geronimo.kernel.basic.BasicLifecycleMonitor$RawLifecycleBroadcaster.fireRunningEvent(BasicLifecycleMonitor.java:254)
        at org.apache.geronimo.gbean.runtime.GBeanInstanceState.attemptFullStart(GBeanInstanceState.java:294)
        at org.apache.geronimo.gbean.runtime.GBeanInstanceState.start(GBeanInstanceState.java:102)
        at org.apache.geronimo.gbean.runtime.GBeanInstanceState.startRecursive(GBeanInstanceState.java:124)
        at org.apache.geronimo.gbean.runtime.GBeanInstance.startRecursive(GBeanInstance.java:555)
        at org.apache.geronimo.kernel.basic.BasicKernel.startRecursiveGBean(BasicKernel.java:379)
        at org.apache.geronimo.kernel.config.ConfigurationUtil.startConfigurationGBeans(ConfigurationUtil.java:456)
        at org.apache.geronimo.kernel.config.KernelConfigurationManager.start(KernelConfigurationManager.java:188)
        at org.apache.geronimo.kernel.config.SimpleConfigurationManager.startConfiguration(SimpleConfigurationManager.java:563)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.geronimo.gbean.runtime.ReflectionMethodInvoker.invoke(ReflectionMethodInvoker.java:34)
        at org.apache.geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:124)
        at org.apache.geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:832)
        at org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java:57)
        at org.apache.geronimo.kernel.basic.RawOperationInvoker.invoke(RawOperationInvoker.java:35)
        at org.apache.geronimo.kernel.basic.ProxyMethodInterceptor.intercept(ProxyMethodInterceptor.java:96)
        at org.apache.geronimo.gbean.GBeanLifecycle$$EnhancerByCGLIB$$dd83360e.startConfiguration(<generated>)
        at org.apache.geronimo.system.main.EmbeddedDaemon.doStartup(EmbeddedDaemon.java:162)
        at org.apache.geronimo.system.main.EmbeddedDaemon.execute(EmbeddedDaemon.java:79)
        at org.apache.geronimo.kernel.util.MainConfigurationBootstrapper.main(MainConfigurationBootstrapper.java:45)
        at org.apache.geronimo.cli.AbstractCLI.executeMain(AbstractCLI.java:67)
        at org.apache.geronimo.cli.daemon.DaemonCLI.main(DaemonCLI.java:30)


If we delete/reinstate the how1_1.log file we then get to another error which seems to be the real problem:


2011-01-26 11:13:29,843 ERROR [MCFConnectionInterceptor] Error occurred creating ManagedConnection for org.apache.geronimo.connector.outbound.ConnectionInfo@18dd7403
javax.resource.spi.ResourceAdapterInternalException: Unable to obtain physical connection to org.apache.derby.jdbc.EmbeddedXADataSource@27daf835
        at org.tranql.connector.jdbc.AbstractXADataSourceMCF.getPhysicalConnection(AbstractXADataSourceMCF.java:76)
        at org.tranql.connector.derby.EmbeddedXAMCF.createManagedConnection(EmbeddedXAMCF.java:53)
        at org.apache.geronimo.connector.outbound.MCFConnectionInterceptor.getConnection(MCFConnectionInterceptor.java:49)
        at org.apache.geronimo.connector.outbound.XAResourceInsertionInterceptor.getConnection(XAResourceInsertionInterceptor.java:41)
        at org.apache.geronimo.connector.outbound.SinglePoolConnectionInterceptor.internalGetConnection(SinglePoolConnectionInterceptor.java:71)
        at org.apache.geronimo.connector.outbound.AbstractSinglePoolConnectionInterceptor.getConnection(AbstractSinglePoolConnectionInterceptor.java:80)
        at org.apache.geronimo.connector.outbound.TransactionEnlistingInterceptor.getConnection(TransactionEnlistingInterceptor.java:46)
        at org.apache.geronimo.connector.outbound.TransactionCachingInterceptor.getConnection(TransactionCachingInterceptor.java:96)
        at org.apache.geronimo.connector.outbound.TCCLInterceptor.getConnection(TCCLInterceptor.java:39)
        at org.apache.geronimo.connector.outbound.AbstractConnectionManager.doRecovery(AbstractConnectionManager.java:65)
        at org.apache.geronimo.connector.outbound.ManagedConnectionFactoryWrapper.doStart(ManagedConnectionFactoryWrapper.java:166)
        at org.apache.geronimo.gbean.runtime.GBeanInstance.createInstance(GBeanInstance.java:998)
        at org.apache.geronimo.gbean.runtime.GBeanInstanceState.attemptFullStart(GBeanInstanceState.java:268)
        at org.apache.geronimo.gbean.runtime.GBeanInstanceState.start(GBeanInstanceState.java:102)
.......


Caused by: java.sql.SQLException: Failed to start database 'SystemDatabase', see the next exception for details.
        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(Unknown Source)
        at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(Unknown Source)
        at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Unknown Source)
        at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
        at org.apache.derby.jdbc.EmbeddedDataSource.getConnection(Unknown Source)
        at org.apache.derby.jdbc.EmbeddedDataSource.getConnection(Unknown Source)
        at org.apache.derby.jdbc.EmbeddedXADataSource.setupResourceAdapter(Unknown Source)
        at org.apache.derby.jdbc.EmbeddedXADataSource.getXAConnection(Unknown Source)
        at org.tranql.connector.jdbc.AbstractXADataSourceMCF.getPhysicalConnection(AbstractXADataSourceMCF.java:74)
        ... 44 more
Caused by: java.sql.SQLException: Failed to start database 'SystemDatabase', see the next exception for details.
        at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
        ... 58 more
Caused by: java.sql.SQLException: Another instance of Derby may have already booted the database /home/ceh1/poseidon_dev/janap01/apps/uat/uat.geronimo-tomcat6-javaee5-2.1.4/var/derby/SystemDatabase.
        at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source)
        at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
        at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
        ... 55 more
Caused by: ERROR XSDB6: Another instance of Derby may have already booted the database /home/ceh1/poseidon_dev/janap01/apps/uat/uat.geronimo-tomcat6-javaee5-2.1.4/var/derby/SystemDatabase.
        at org.apache.derby.iapi.error.StandardException.newException(Unknown Source)
        at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.privGetJBMSLockOnDB(Unknown Source)
        at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.getJBMSLockOnDB(Unknown Source)
        at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.boot(Unknown Source)
        at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
        at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
        at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
        at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
        at org.apache.derby.impl.store.raw.RawStore.boot(Unknown Source)
        at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source)
        at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source)
        at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
        at org.apache.derby.iapi.services.monitor.Moni......


Is there a clean up script available or instructions on how to reinstall Derby without re-installing the entire geronimo server? Any help would be appreciated as I spend most of a day sorting this out once a month to re-instate our dev and uat geronimo instances.

Thanks
Dan


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.