cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam <adam.scarce...@gmail.com>
Subject Re: Management Server Startup Issue
Date Wed, 16 Jul 2014 19:25:44 GMT
Here are the last few lines in my catalina.out:

INFO: Deploying module: addressing-1.5.6 -
file:/usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/lib/axis2-1.5.6.jar
Jul 16, 2014 9:30:08 AM org.apache.catalina.loader.WebappClassLoader
loadClass
INFO: Illegal access: this web application instance has been stopped
already.  Could not load
org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.  The
eventual following stack trace is caused by an error thrown for debugging
purposes as well as to attempt to terminate the thread which caused the
illegal access, and has no functional impact.
java.lang.IllegalStateException
        at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1369)
        at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
        at
org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)

Exception in thread "Timer-2" java.lang.NoClassDefFoundError:
org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
        at
org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
        at java.util.TimerThread.mainLoop(Timer.java:555)
        at java.util.TimerThread.run(Timer.java:505)
Caused by: java.lang.ClassNotFoundException:
org.apache.cloudstack.managed.context.ManagedContextTimerTask$1
        at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1483)
        at
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
        ... 3 more
Jul 16, 2014 9:30:14 AM org.apache.axis2.deployment.ServiceDeployer deploy
INFO: Deploying Web service: cloud-ec2.aar -
file:/usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/services/cloud-ec2.aar
Jul 16, 2014 9:30:14 AM org.apache.axis2.deployment.ServiceDeployer deploy
INFO: Deploying Web service: version-1.5.6.aar -
file:/usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/services/version-1.5.6.aar
Jul 16, 2014 9:30:14 AM org.apache.axis2.util.OnDemandLogger warn
WARNING: No transportReceiver for
org.apache.axis2.transport.http.AxisServletListener found. An instance for
HTTP will be configured automatically. Please update your axis2.xml file!
Jul 16, 2014 9:30:14 AM
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor
<init>
INFO: JSR-330 'javax.inject.Inject' annotation found and supported for
autowiring
Jul 16, 2014 9:30:14 AM org.apache.catalina.startup.HostConfig
deployDirectory
INFO: Deploying web application directory webapps
Jul 16, 2014 9:30:14 AM org.apache.coyote.http11.Http11NioProtocol start
INFO: Starting Coyote HTTP/1.1 on http-7080
Jul 16, 2014 9:30:14 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 50870 ms



Best Regards,



Adam Scarcella


On Tue, Jul 15, 2014 at 7:04 PM, Rafael Weingartner <
rafaelweingartner@gmail.com> wrote:

> The awsapi not getting up is not a problem; hence you are not using it. Why
> is not your CS app going up? Do you have any others log entries?
>
>
> On Tue, Jul 15, 2014 at 5:24 PM, Adam <adam.scarcella@gmail.com> wrote:
>
> > I do not use awsapi either, and have never turned it on. If I look in the
> > database I can see that those features are turned off too:
> >
> > SELECT * FROM cloud.configuration where name = 'enable.ec2.api' or name =
> > 'enable.s3.api';
> >
> > # category, instance, component, name, value, description, default_value,
> > updated, scope, is_dynamic
> > 'Advanced', 'DEFAULT', 'management-server', 'enable.ec2.api', 'false',
> > 'enable EC2 API on CloudStack', 'false', NULL, NULL, '0'
> > 'Advanced', 'DEFAULT', 'management-server', 'enable.s3.api', 'false',
> > 'enable Amazon S3 API on CloudStack', 'false', NULL, NULL, '0'
> >
> >
> > I found this though:
> https://issues.apache.org/jira/browse/CLOUDSTACK-2758
> >
> > Not sure if it's related.
> >
> > I tried deleted everything from both the work and temp dirs, but it had
> no
> > affect.
> >
> > :(
> >
> > Best Regards,
> >
> >
> >
> > Adam Scarcella
> >
> >
> > On Tue, Jul 15, 2014 at 12:22 PM, Rafael Weingartner <
> > rafaelweingartner@gmail.com> wrote:
> >
> > > Have you tried to clean the directories work and temp of the tomcat
> > > installation that the CS is using?
> > >
> > > rm -rf /usr/share/cloudstack-management/temp/*
> > >
> > > rm -rf /usr/share/cloudstack-management/work/*
> > >
> > >
> > > In my CS installation I do not even have the awsapi files.
> > >
> > > I have just noticed that now, it has a soft link pointing to
> > > ../../cloudstack-bridge/webapps/awsapi, However, that folder does not
> > > exist.
> > >
> > > Are you using awsapi?
> > >
> > > I do not use it, so that is not a problem.
> > >
> > >
> > > On Tue, Jul 15, 2014 at 1:05 PM, Adam <adam.scarcella@gmail.com>
> wrote:
> > >
> > > > OK, so I have this issue HTTP Status 404
> > > >
> > > > [image: Inline image 1]
> > > >
> > > >
> > > > ...except it must be different in some way because I've gone through
> > this
> > > > entire thread and even though I have the same original error
> mentioned
> > > and
> > > > the same behavior, I do not have the same "smoking gun" so to speak
> in
> > > the
> > > > localhost.log where my logging type was STATEMENT instead of  MIXED.
> > > >
> > > > [root@cs-east-prod1 logs]# cat /etc/my.cnf
> > > > [mysqld]
> > > > datadir=/var/lib/mysql
> > > > sockeet=/var/lib/mysql/mysql.sock
> > > > user=mysql
> > > > # Disabling symbolic-links is recommended to prevent assorted
> security
> > > > risks
> > > > symbolic-links=0
> > > >
> > > >
> > > > #
> > > > # BEGIN CloudStack 4.3 Requirements
> > > > #
> > > > innodb_rollback_on_timeout=1
> > > > innodb_lock_wait_timeout=600
> > > > max_connections=350
> > > > log-bin=mysql-bin
> > > > binlog-format = 'ROW'
> > > > #
> > > > # END CloudStack 4.3 Requirements
> > > > #
> > > >
> > > > [mysqld_safe]
> > > > log-error=/var/log/mysqld.log
> > > > pid-file=/var/run/mysqld/mysqld.pid
> > > > [root@cs-east-prod1 logs]#
> > > >
> > > >
> > > >
> > > > I upgraded from 4.2.1 to 4.3 (CentOS 6.5 x86_64, KVM) and ever since
> > I've
> > > > not been able to start the cloudstack-management ui. I see these
> errors
> > > in
> > > > the catalina.out log:
> > > >
> > > > Jul 15, 2014 10:13:57 AM org.apache.axis2.deployment.ModuleDeployer
> > > deploy
> > > > SEVERE: The addressing-1.5.4.mar module, which is not valid, caused
> The
> > > > module.xml file cannot be found for the module:
> > > >
> > >
> >
> /usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules/addressing-1.5.4.mar
> > > > org.apache.axis2.deployment.DeploymentException: The module.xml file
> > > > cannot be found for the module:
> > > >
> > >
> >
> /usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules/addressing-1.5.4.mar
> > > >         at
> > > >
> > >
> >
> org.apache.axis2.deployment.repository.util.ArchiveReader.readModuleArchive(ArchiveReader.java:529)
> > > >         at
> > > >
> > org.apache.axis2.deployment.ModuleDeployer.deploy(ModuleDeployer.java:75)
> > > >         at
> > > >
> > >
> >
> org.apache.axis2.deployment.repository.util.DeploymentFileData.deploy(DeploymentFileData.java:136)
> > > >         at
> > > >
> > >
> >
> org.apache.axis2.deployment.DeploymentEngine.doDeploy(DeploymentEngine.java:661)
> > > >         at
> > > >
> > >
> >
> org.apache.axis2.deployment.RepositoryListener.init(RepositoryListener.java:251)
> > > >         at
> > > >
> > >
> >
> org.apache.axis2.deployment.RepositoryListener.init2(RepositoryListener.java:68)
> > > >         at
> > > >
> > >
> >
> org.apache.axis2.deployment.RepositoryListener.<init>(RepositoryListener.java:63)
> > > >         at
> > > >
> > >
> >
> org.apache.axis2.deployment.DeploymentEngine.loadRepository(DeploymentEngine.java:147)
> > > >         at
> > > >
> > >
> >
> org.apache.axis2.deployment.WarBasedAxisConfigurator.getAxisConfiguration(WarBasedAxisConfigurator.java:229)
> > > >         at
> > > >
> > >
> >
> org.apache.axis2.context.ConfigurationContextFactory.createConfigurationContext(ConfigurationContextFactory.java:64)
> > > >         at
> > > >
> > >
> >
> org.apache.axis2.transport.http.AxisServlet.initConfigContext(AxisServlet.java:567)
> > > >         at
> > > >
> org.apache.axis2.transport.http.AxisServlet.init(AxisServlet.java:452)
> > > >         at
> > > >
> > >
> >
> org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1173)
> > > >         at
> > > >
> org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:993)
> > > >         at
> > > >
> > >
> >
> org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4187)
> > > >         at
> > > >
> > org.apache.catalina.core.StandardContext.start(StandardContext.java:4496)
> > > >         at
> > > >
> > >
> >
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
> > > >         at
> > > >
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
> > > >         at
> > > > org.apache.catalina.core.StandardHost.addChild(StandardHost.java:526)
> > > >         at
> > > >
> > >
> >
> org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1041)
> > > >         at
> > > >
> > >
> >
> org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:964)
> > > >         at
> > > >
> org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
> > > >         at
> > > > org.apache.catalina.startup.HostConfig.start(HostConfig.java:1277)
> > > >         at
> > > >
> > >
> >
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:321)
> > > >         at
> > > >
> > >
> >
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
> > > >         at
> > > > org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
> > > >         at
> > > > org.apache.catalina.core.StandardHost.start(StandardHost.java:722)
> > > >         at
> > > > org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
> > > >         at
> > > >
> org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
> > > >         at
> > > >
> > org.apache.catalina.core.StandardService.start(StandardService.java:516)
> > > >         at
> > > >
> org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
> > > >         at
> > org.apache.catalina.startup.Catalina.start(Catalina.java:593)
> > > >         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > >         at
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > > >         at
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > >         at java.lang.reflect.Method.invoke(Method.java:606)
> > > >         at
> > > org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
> > > >         at
> > org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
> > > > Caused by: org.apache.axis2.deployment.DeploymentException: The
> > > module.xml
> > > > file cannot be found for the module:
> > > >
> > >
> >
> /usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules/addressing-1.5.4.mar
> > > >         at
> > > >
> > >
> >
> org.apache.axis2.deployment.repository.util.ArchiveReader.readModuleArchive(ArchiveReader.java:523)
> > > >         ... 37 more
> > > >
> > > > Jul 15, 2014 10:13:57 AM org.apache.axis2.deployment.ModuleDeployer
> > > deploy
> > > > INFO: Deploying module: addressing-1.5.6 -
> > > >
> > >
> >
> file:/usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules/addressing-1.5.6.mar
> > > > Jul 15, 2014 10:13:57 AM org.apache.axis2.deployment.ModuleDeployer
> > > deploy
> > > > INFO: Deploying module: jaxws-1.5.6 -
> > > >
> > >
> >
> file:/usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules/axis2-jaxws-mar-1.5.6.mar
> > > > Jul 15, 2014 10:13:57 AM org.apache.axis2.deployment.ModuleDeployer
> > > deploy
> > > > INFO: Deploying module: mtompolicy-1.5.6 -
> > > >
> > >
> >
> file:/usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules/mtompolicy-1.5.6.mar
> > > > Jul 15, 2014 10:13:57 AM org.apache.axis2.deployment.ModuleDeployer
> > > deploy
> > > > INFO: Deploying module: addressing-1.5.6 -
> > > >
> > >
> >
> file:/usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/lib/axis2-1.5.6.jar
> > > > Jul 15, 2014 10:13:57 AM org.apache.catalina.loader.WebappClassLoader
> > > > loadClass
> > > > INFO: Illegal access: this web application instance has been stopped
> > > > already.  Could not load
> > > > org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.  The
> > > > eventual following stack trace is caused by an error thrown for
> > debugging
> > > > purposes as well as to attempt to terminate the thread which caused
> the
> > > > illegal access, and has no functional impact.
> > > > java.lang.IllegalStateException
> > > >         at
> > > >
> > >
> >
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1369)
> > > >         at
> > > >
> > >
> >
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
> > > >         at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
> > > >         at java.util.TimerThread.mainLoop(Timer.java:555)
> > > >         at java.util.TimerThread.run(Timer.java:505)
> > > >
> > > > Exception in thread "Timer-2" java.lang.NoClassDefFoundError:
> > > > org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
> > > >         at
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
> > > >         at java.util.TimerThread.mainLoop(Timer.java:555)
> > > >         at java.util.TimerThread.run(Timer.java:505)
> > > > Caused by: java.lang.ClassNotFoundException:
> > > > org.apache.cloudstack.managed.context.ManagedContextTimerTask$1
> > > >         at
> > > >
> > >
> >
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1483)
> > > >         at
> > > >
> > >
> >
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
> > > >         ... 3 more
> > > >
> > > >
> > > > I have absolutely no errors of any kind in the management-server.log.
> > > >
> > > > [root@cs-east-prod1 logs]# grep -i -E
> > > > 'exception|unable|fail|invalid|leak|warn|error'
> ./management-server.log
> > > > [root@cs-east-prod1 logs]#
> > > >
> > > >
> > > > The only thing I see in the access.log is:
> > > >
> > > > [root@cs-east-prod1 logs]# cat access_log.2014-07-15.txt
> > > > 10.97.34.115 - - [15/Jul/2014:10:27:15 -0500] "GET /client/ HTTP/1.1"
> > 404
> > > > 952
> > > > [root@cs-east-prod1 logs]#
> > > >
> > > >
> > > > ... and my localhost.log is entirely empty.
> > > >
> > > >
> > > > [root@cs-east-prod1 logs]# du -sh localhost.2014-07-15.log
> > > > 0       localhost.2014-07-15.log
> > > >
> > > >
> > > > And even more bizzare is the fact that I have another CloudStack 4.3
> > > > server that is running on XenServer and it's perfectly fine. It was
> not
> > > an
> > > > upgrade though, it was a clean install of 4.3. I've compared all the
> > > files
> > > > associated with the addressing-1.5.4.mar but could not find anything
> > > > significant.
> > > >
> > > > Here is the output from my upgraded server (cs-east-prod1: broken
> with
> > > 404)
> > > >
> > > > [root@cs-east-prod1 modules]# pwd
> > > > /usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules
> > > > [root@cs-east-prod1 modules]# md5sum addressing-1.5.4.mar
> > > > 1ef211c1a042e7d4b9b7e5f914222929  addressing-1.5.4.mar
> > > > [root@cs-east-prod1 modules]# cat modules.list
> > > > soapmonitor-1.5.6.mar
> > > > addressing-1.5.6.mar
> > > > mtompolicy-1.5.6.mar
> > > > mex-1.5.6.mar
> > > > ping-1.5.6.mar
> > > > axis2-jaxws-mar-1.5.6.mar
> > > > scripting-1.5.6.mar
> > > > [root@cs-east-prod1 modules]# ls -l
> > > > total 340
> > > > -rwxrwxrwx 1 cloud cloud 126981 Mar 25 11:01 addressing-1.5.4.mar
> > > > -rwxrwxrwx 1 cloud cloud  34518 Mar 25 11:01 addressing-1.5.6.mar
> > > > -rwxrwxrwx 1 cloud cloud  14363 Mar 25 11:01
> axis2-jaxws-mar-1.5.6.mar
> > > > -rwxrwxrwx 1 cloud cloud  38620 Mar 25 11:01 mex-1.5.6.mar
> > > > -rwxrwxrwx 1 cloud cloud    139 Mar 25 11:01 modules.list
> > > > -rwxrwxrwx 1 cloud cloud  18682 Mar 25 11:01 mtompolicy-1.5.6.mar
> > > > -rwxrwxrwx 1 cloud cloud  17647 Mar 25 11:01 ping-1.5.6.mar
> > > > -rwxrwxrwx 1 cloud cloud   9721 Mar 25 11:01 rahas-1.5.1.mar
> > > > -rwxrwxrwx 1 cloud cloud   9612 Mar 25 11:01 rampart-1.5.1.mar
> > > > -rwxrwxrwx 1 cloud cloud  33779 Mar 25 11:01 scripting-1.5.6.mar
> > > > -rwxrwxrwx 1 cloud cloud  15650 Mar 25 11:01 soapmonitor-1.5.6.mar
> > > > [root@cs-east-prod1 modules]# cd /
> > > > [root@cs-east-prod1 /]# find . -name "module.xml"
> > > >
> > > >
> > >
> >
> ./usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/classes/com/cloud/bridge/auth/s3/module.xml
> > > >
> > > >
> > >
> >
> ./usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/classes/com/cloud/bridge/auth/ec2/module.xml
> > > > [root@cs-east-prod1 /]# md5sum
> > > >
> > >
> >
> ./usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/classes/com/cloud/bridge/auth/s3/module.xml
> > > > baf5f63f5dd57484e9a88be6689939fc
> > > >
> > >
> >
>  ./usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/classes/com/cloud/bridge/auth/s3/module.xml
> > > > [root@cs-east-prod1 /]# md5sum
> > > >
> > >
> >
> ./usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/classes/com/cloud/bridge/auth/ec2/module.xml
> > > > 913b4561258980f7b1b4c7cb3de261ba
> > > >
> > >
> >
>  ./usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/classes/com/cloud/bridge/auth/ec2/module.xml
> > > > [root@cs-east-prod1 /]#
> > > >
> > > >
> > > > And here is the output from the exact same commands on my 4.3 server
> > > > that's working fine (na-nyc-bos-cs-poc2):
> > > >
> > > > [root@na-nyc-bos-cs-poc2 modules]# pwd
> > > > /usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules
> > > > [root@na-nyc-bos-cs-poc2 modules]# md5sum addressing-1.5.4.mar
> > > > 1ef211c1a042e7d4b9b7e5f914222929  addressing-1.5.4.mar
> > > > [root@na-nyc-bos-cs-poc2 modules]# cat modules.list
> > > > soapmonitor-1.5.6.mar
> > > > addressing-1.5.6.mar
> > > > mtompolicy-1.5.6.mar
> > > > mex-1.5.6.mar
> > > > ping-1.5.6.mar
> > > > axis2-jaxws-mar-1.5.6.mar
> > > > scripting-1.5.6.mar
> > > > [root@na-nyc-bos-cs-poc2 modules]# ls -l
> > > > total 340
> > > > -rw-r--r--. 1 cloud cloud 126981 Mar 25 11:01 addressing-1.5.4.mar
> > > > -rw-r--r--. 1 cloud cloud  34518 Mar 25 11:01 addressing-1.5.6.mar
> > > > -rw-r--r--. 1 cloud cloud  14363 Mar 25 11:01
> axis2-jaxws-mar-1.5.6.mar
> > > > -rw-r--r--. 1 cloud cloud  38620 Mar 25 11:01 mex-1.5.6.mar
> > > > -rw-r--r--. 1 cloud cloud    139 Mar 25 11:01 modules.list
> > > > -rw-r--r--. 1 cloud cloud  18682 Mar 25 11:01 mtompolicy-1.5.6.mar
> > > > -rw-r--r--. 1 cloud cloud  17647 Mar 25 11:01 ping-1.5.6.mar
> > > > -rw-r--r--. 1 cloud cloud   9721 Mar 25 11:01 rahas-1.5.1.mar
> > > > -rw-r--r--. 1 cloud cloud   9612 Mar 25 11:01 rampart-1.5.1.mar
> > > > -rw-r--r--. 1 cloud cloud  33779 Mar 25 11:01 scripting-1.5.6.mar
> > > > -rw-r--r--. 1 cloud cloud  15650 Mar 25 11:01 soapmonitor-1.5.6.mar
> > > > [root@na-nyc-bos-cs-poc2 modules]# cd /
> > > > [root@na-nyc-bos-cs-poc2 /]# find . -name "module.xml"
> > > >
> > > >
> > >
> >
> ./usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/classes/com/cloud/bridge/auth/s3/module.xml
> > > >
> > > >
> > >
> >
> ./usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/classes/com/cloud/bridge/auth/ec2/module.xml
> > > > [root@na-nyc-bos-cs-poc2 /]# md5sum
> > > >
> > >
> >
> ./usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/classes/com/cloud/bridge/auth/s3/module.xml
> > > > baf5f63f5dd57484e9a88be6689939fc
> > > >
> > >
> >
>  ./usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/classes/com/cloud/bridge/auth/s3/module.xml
> > > > [root@na-nyc-bos-cs-poc2 /]# md5sum
> > > >
> > >
> >
> ./usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/classes/com/cloud/bridge/auth/ec2/module.xml
> > > > 913b4561258980f7b1b4c7cb3de261ba
> > > >
> > >
> >
>  ./usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/classes/com/cloud/bridge/auth/ec2/module.xml
> > > > [root@na-nyc-bos-cs-poc2 /]#
> > > >
> > > >
> > > >
> > > > *HELP!!!*
> > > >
> > > > I need to get my cloud back up and running.
> > > >
> > > >
> > > >
> > > >
> > > > Best Regards,
> > > >
> > > >
> > > >
> > > > Adam Scarcella
> > > >
> > > >
> > > > On Tue, Jun 10, 2014 at 8:41 PM, Rafael Weingartner <
> > > > rafaelweingartner@gmail.com> wrote:
> > > >
> > > >> Nice, no doubts that others will run into this.
> > > >> But when they run into that problem, it is going to be just a query
> at
> > > >> Google to uncover this thread ;)
> > > >>
> > > >>
> > > >> On Tue, Jun 10, 2014 at 8:39 PM, Steve Searles <ssearles@zimcom.net
> >
> > > >> wrote:
> > > >>
> > > >> >  Rafael,
> > > >> >
> > > >> >  Ok, I seem to have found the smoking gun in the localhost.log
> > > >> >
> > > >> >  Caused by: java.sql.SQLException: Binary logging not possible.
> > > Message:
> > > >> > Transaction level 'READ-COMMITTED' in InnoDB is not safe for
> binlog
> > > mode
> > > >> > 'STATEMENT'
> > > >> > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
> > > >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
> > > >> > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
> > > >> > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
> > > >> > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
> > > >> > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2333)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2318)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
> > > >> > at
> > > >> >
> > > >>
> > >
> >
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
> > > >> > at
> com.cloud.utils.db.GenericDaoBase.update(GenericDaoBase.java:823)
> > > >> > ... 198 more
> > > >> > Jun 10, 2014 7:02:25 PM
> org.apache.catalina.core.ApplicationContext
> > > log
> > > >> > INFO: Shutting down log4j
> > > >> >
> > > >> >  Oddly enough I thought the default logging type was MIXED not
> > > >> STATEMENT.
> > > >> >  I am sure others will run into this. I am running (5.1.73) This
> is
> > > what
> > > >> > the MySQL website had to say.
> > > >> >
> > > >> >           The default binary logging format depends on the version
> > of
> > > >> > MySQL you are using:
> > > >> >
> > > >> >    -
> > > >> >
> > > >> >    For MySQL 5.1.11 and earlier, and for MySQL 5.1.29 and later,
> > > >> >    statement-based logging is used by default.
> > > >> >    -
> > > >> >
> > > >> >    For MySQL 5.1.12 through MySQL 5.1.28, mixed logging is used by
> > > >> default
> > > >> >
> > > >> >             If you are using InnoDB
> > > >> >    <
> > http://dev.mysql.com/doc/refman/5.1/en/innodb-storage-engine.html
> > > >
> > > >>
> > > >> >    tables and the transaction isolation level is READ COMMITTED
> > > >> >    <
> > > >>
> > >
> >
> http://dev.mysql.com/doc/refman/5.1/en/set-transaction.html#isolevel_read-committed
> > > >> >
> > > >> >    or READ UNCOMMITTED
> > > >> >    <
> > > >>
> > >
> >
> http://dev.mysql.com/doc/refman/5.1/en/set-transaction.html#isolevel_read-uncommitted
> > > >> >,
> > > >> >    only row-based logging can be used. It is *possible* to change
> > the
> > > >>
> > > >> >    logging format to STATEMENT, but doing so at runtime leads very
> > > >> >    rapidly to errors because InnoDB can no longer perform inserts.
> > > >> >
> > > >> >    Switching the replication format at runtime is not recommended
> > when
> > > >> >    any temporary tables exist, because temporary tables are logged
> > > only
> > > >> when
> > > >> >    using statement-based replication, whereas with row-based
> > > >> replication they
> > > >> >    are not logged. With mixed replication, temporary tables are
> > > usually
> > > >> >    logged; exceptions happen with user-defined functions (UDFs)
> and
> > > >> with the
> > > >> >    UUID()
> > > >> >    <
> > > >>
> > >
> >
> http://dev.mysql.com/doc/refman/5.1/en/miscellaneous-functions.html#function_uuid
> > > >> >
> > > >>
> > > >> >    function.
> > > >> >
> > > >> >    With the binary log format set to ROW, many changes are written
> > to
> > > >> the
> > > >> >    binary log using the row-based format. Some changes, however,
> > still
> > > >> use the
> > > >> >    statement-based format. Examples include all DDL (data
> definition
> > > >> language)
> > > >> >    statements such as CREATE TABLE
> > > >> >    <http://dev.mysql.com/doc/refman/5.1/en/create-table.html>,
> > ALTER
> > > >> TABLE
> > > >> >    <http://dev.mysql.com/doc/refman/5.1/en/alter-table.html>, or
> > DROP
> > > >> >    TABLE <http://dev.mysql.com/doc/refman/5.1/en/drop-table.html
> >.
> > > >>
> > > >> >
> > > >> >
> > > >> >
> > > >> >  Add
> > > >> >  binlog-format=ROW
> > > >> > to /etc/my.cnf on the master DB server and you are good to go.
> > > >> >
> > > >> >
> > > >> >
> > > >> >    Steven Searles, CTO | ssearles@zimcom.net
> > > >> >  Zimcom Internet Solutions  | www.zimcom.net
> > > >> >  O: 513.231.9500  |  D: 513.233.4130
> > > >> >
> > > >> >
> > > >> >  On Jun 10, 2014, at 6:18 PM, Rafael Weingartner <
> > > >> > rafaelweingartner@gmail.com> wrote:
> > > >> >
> > > >> > maybe, but, I actually do not care that much about the app that is
> > > used
> > > >> to
> > > >> > integrate CS with AWS, hence they are two separated apps, a
> problem
> > > with
> > > >> > AWS integration app should not affect the CS normal app.
> > > >> >
> > > >> > Could you please provide me again with the log file from
> > > >> > /var/log/cloudstack/management/management-server.log? There must
> be
> > > >> > something here.
> > > >> >
> > > >> >
> > > >> > On Tue, Jun 10, 2014 at 6:34 PM, Steve Searles <
> ssearles@zimcom.net
> > >
> > > >> > wrote:
> > > >> >
> > > >> > Is
> > > >> >
> > > >> >
> > > >>
> > >
> >
> http://mail-archives.apache.org/mod_mbox/cloudstack-issues/201401.mbox/%3CJIRA.12687852.1389186771873.61435.1389186832715@arcas%3E
> > > >> >  relevant here?
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Jun 10, 2014, at 5:31 PM, Steve Searles <ssearles@zimcom.net
> > > <mailto:
> > > >> > ssearles@zimcom.net>> wrote:
> > > >> >
> > > >> > Ok I was wrong, the addressing-1.5.4.mar is present.
> > > >> >
> > > >> > [root@csmgmt01 modules]# pwd
> > > >> >
> /usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules
> > > >> > [root@csmgmt01 modules]# ls -lash
> > > >> > total 344K
> > > >> > 4.0K drwxr-xr-x. 2 cloud cloud 4.0K Mar 31 18:21 .
> > > >> > 4.0K drwxr-xr-x. 7 cloud cloud 4.0K Mar 31 18:21 ..
> > > >> > 124K -rw-r--r--. 1 cloud cloud 124K Mar 27 01:06
> > addressing-1.5.4.mar
> > > >> > 36K -rw-r--r--. 1 cloud cloud  34K Mar 27 01:06
> addressing-1.5.6.mar
> > > >> > 16K -rw-r--r--. 1 cloud cloud  15K Mar 27 01:06
> > > >> axis2-jaxws-mar-1.5.6.mar
> > > >> > 40K -rw-r--r--. 1 cloud cloud  38K Mar 27 01:06 mex-1.5.6.mar
> > > >> > 4.0K -rw-r--r--. 1 cloud cloud  139 Mar 27 01:06 modules.list
> > > >> > 20K -rw-r--r--. 1 cloud cloud  19K Mar 27 01:06
> mtompolicy-1.5.6.mar
> > > >> > 20K -rw-r--r--. 1 cloud cloud  18K Mar 27 01:06 ping-1.5.6.mar
> > > >> > 12K -rw-r--r--. 1 cloud cloud 9.5K Mar 27 01:06 rahas-1.5.1.mar
> > > >> > 12K -rw-r--r--. 1 cloud cloud 9.4K Mar 27 01:06 rampart-1.5.1.mar
> > > >> > 36K -rw-r--r--. 1 cloud cloud  33K Mar 27 01:06
> scripting-1.5.6.mar
> > > >> > 16K -rw-r--r--. 1 cloud cloud  16K Mar 27 01:06
> > soapmonitor-1.5.6.mar
> > > >> > [root@csmgmt01 modules]#
> > > >> >
> > > >> >
> > > >> >
> > > >> > The error is complaining about the module.xml file missing.
> > > >> >
> > > >> > Steve Searles
> > > >> >
> > > >> > On Jun 10, 2014, at 5:21 PM, Steve Searles <ssearles@zimcom.net
> > > <mailto:
> > > >> > ssearles@zimcom.net><mailto:ssearles@zimcom.net>> wrote:
> > > >> >
> > > >> > I agree with what you are saying.  Yes. addressing-1.5.4.mar
> errors
> > I
> > > >> have
> > > >> > seen before.  I have no webapps7080 directory so that is why that
> > > error
> > > >> is
> > > >> > being thrown.  I still am wondering even if a dependency is
> missing
> > > why
> > > >> is
> > > >> > it only a dependency when sql replication(bin log) is on? Can you
> > tell
> > > >> me
> > > >> > how to get a stack trace from CS startup?
> > > >> >
> > > >> > That file is present in these locations on the management node.
> > > >> >
> > > >> > [root@csmgmt01 /]# find / -name addressing-1.5.4.mar
> > > >> >
> > > >>
> > >
> >
> /root/.m2/repository/org/apache/axis2/addressing/1.5.4/addressing-1.5.4.mar
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > >
> >
> /root/apache-cloudstack-4.3.0-src/dist/rpmbuild/BUILD/cloudstack-4.3.0/awsapi/target/cloud-awsapi-4.3.0/WEB-INF/modules/addressing-1.5.4.mar
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > >
> >
> /usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/modules/addressing-1.5.4.mar
> > > >> > [root@csmgmt01 /]#
> > > >> >
> > > >> >
> > > >> >
> > > >> > is it just a symlink that is missing?
> > > >> >
> > > >> >
> > > >> >
> > > >> > [root@csmgmt01 webapps]# ls -las
> > > >> > total 12
> > > >> > 4 drwxr-xr-x.  3 root root 4096 Mar 31 18:21 .
> > > >> > 4 drwxr-xr-x.  4 root root 4096 Mar 31 18:21 ..
> > > >> > 4 drwxr-xr-x. 11 root root 4096 Mar 31 18:21 client
> > > >> > [root@csmgmt01 webapps]# cd client/
> > > >> > [root@csmgmt01 client]# ls
> > > >> > css  dictionary.jsp  images  index.jsp  lib  META-INF  modules
> > >  plugins
> > > >> > scripts  tests  WEB-INF
> > > >> > [root@csmgmt01 client]# ls -lash
> > > >> > total 264K
> > > >> > 4.0K drwxr-xr-x. 11 root root 4.0K Mar 31 18:21 .
> > > >> > 4.0K drwxr-xr-x.  3 root root 4.0K Mar 31 18:21 ..
> > > >> > 4.0K drwxr-xr-x.  2 root root 4.0K Mar 31 18:21 css
> > > >> > 116K -rw-r--r--.  1 root root 115K Mar 27 01:06 dictionary.jsp
> > > >> > 4.0K drwxr-xr-x.  2 root root 4.0K Mar 31 18:21 images
> > > >> > 104K -rw-r--r--.  1 root root 103K Mar 27 01:06 index.jsp
> > > >> > 4.0K drwxr-xr-x.  5 root root 4.0K Mar 31 18:21 lib
> > > >> > 4.0K drwxr-xr-x.  2 root root 4.0K Mar 27 01:06 META-INF
> > > >> > 4.0K drwxr-xr-x.  6 root root 4.0K Mar 31 18:21 modules
> > > >> > 4.0K drwxr-xr-x.  3 root root 4.0K Mar 31 18:21 plugins
> > > >> > 4.0K drwxr-xr-x.  4 root root 4.0K Mar 31 18:21 scripts
> > > >> > 4.0K drwxr-xr-x.  2 root root 4.0K Mar 31 18:21 tests
> > > >> > 4.0K drwxr-xr-x.  4 root root 4.0K Mar 31 18:21 WEB-INF
> > > >> > [root@csmgmt01 client]# pwd
> > > >> > /usr/share/cloudstack-management/webapps/client
> > > >> > [root@csmgmt01 client]#
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > Steve Searles
> > > >> >
> > > >> >
> > > >> > On Jun 10, 2014, at 3:04 PM, Rafael Weingartner <
> > > >> > rafaelweingartner@gmail.com<mailto:rafaelweingartner@gmail.com
> > > ><mailto:
> > > >> > rafaelweingartner@gmail.com><mailto:rafaelweingartner@gmail.com>>
> > > >> wrote:
> > > >> >
> > > >> > Look at this point:
> > > >> > "SEVERE: Error listenerStart
> > > >> > Jun 10, 2014 2:12:48 PM org.apache.catalina.core.StandardContext
> > start
> > > >> > SEVERE: Context [/client] startup failed due to previous errors"
> > > >> > I could not find the stacktrace into the logs you provided. But,
> it
> > > has
> > > >> to
> > > >> > have some stack which would help to understand why the CS is not
> > > getting
> > > >> > up.
> > > >> >
> > > >> > This is a good point to look at:
> > > >> > "SEVERE: The addressing-1.5.4.mar module, which is not valid,
> caused
> > > The
> > > >> > module.xml file cannot be found for the module:
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > >
> >
> /usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules/addressing-1.5.4.mar
> > > >> > org.apache.axis2.deployment.DeploymentException: The module.xml
> file
> > > >> cannot
> > > >> > be found for the module:
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > >
> >
> /usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules/addressing-1.5.4.mar
> > > >> > at
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.axis2.deployment.repository.util.ArchiveReader.readModuleArchive(ArchiveReader.java:529)
> > > >> > at
> > > >> >
> > > >>
> > >
> org.apache.axis2.deployment.ModuleDeployer.deploy(ModuleDeployer.java:75)
> > > >> > at
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.axis2.deployment.repository.util.DeploymentFileData.deploy(DeploymentFileData.java:136)
> > > >> > at
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.axis2.deployment.DeploymentEngine.doDeploy(DeploymentEngine.java:661)"
> > > >> >
> > > >> >
> > > >> >
> > > >> > It seems there is a .xml file being missing.
> > > >> > But, still even though the awsapi is not working, the CS app
> should
> > > run
> > > >> > just fine, unless it depends on that service.
> > > >> >
> > > >> > This is also something interesting:
> > > >> > "Jun 10, 2014 2:12:58 PM
> > org.apache.catalina.loader.WebappClassLoader
> > > >> > loadClass
> > > >> > INFO: Illegal access: this web application instance has been
> stopped
> > > >> > already.  Could not load
> > > >> > org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.
> >  The
> > > >> > eventual following stack trace is caused by an error thrown for
> > > >> debugging
> > > >> > purposes as well as to attempt to terminate the thread which
> caused
> > > the
> > > >> > illegal access, and has no functional impact.
> > > >> > java.lang.IllegalStateException
> > > >> > at
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > >
> >
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1369)"
> > > >> >
> > > >> > it seems there is a missing dependency.
> > > >> >
> > > >> > BTW: out of curiosity, what do tou have in folder
> > > >> > "/usr/share/cloudstack-management/webapps"
> > > >> >
> > > >> >
> > > >> > On Tue, Jun 10, 2014 at 3:53 PM, Steve Searles <
> ssearles@zimcom.net
> > > >> > <mailto:ssearles@zimcom.net><mailto:ssearles@zimcom.net><mailto:
> > > >> > ssearles@zimcom.net>> wrote:
> > > >> >
> > > >> > Nothing interesting in the management server logs, you saw the
> > > complete
> > > >> > ones I sent yesterday.  I have attached the complete catalina.out
> > > below.
> > > >> > This is starting to feel like a problem with the DB locking the
> > tables
> > > >> > during the startup integrity check.  I am able to lock the tables
> > via
> > > >> the
> > > >> > command line with the bin log enabled without issue though.
> > > >> >
> > > >> > https://dl.dropboxusercontent.com/u/51895629/catalina.binlog.gz
> > > >> >
> > > >> > Steve Searles
> > > >> >
> > > >> >
> > > >> > On Jun 10, 2014, at 2:29 PM, Rafael Weingartner <
> > > >> > rafaelweingartner@gmail.com<mailto:rafaelweingartner@gmail.com>>
> > > wrote:
> > > >> >
> > > >> > this is interesting:
> > > >> > "
> > > >> > Jun 10, 2014 2:12:48 PM org.apache.catalina.core.StandardContext
> > start
> > > >> > SEVERE: Context [/client] startup failed due to previous error"
> > > >> >
> > > >> > does it have any extra log before that message?
> > > >> > Did you find anything at
> /var/log/cloudstack/management-server.log?
> > > >> >
> > > >> >
> > > >> > On Tue, Jun 10, 2014 at 3:24 PM, Steve Searles <
> ssearles@zimcom.net
> > > >> > <mailto:ssearles@zimcom.net>>
> > > >> > wrote:
> > > >> >
> > > >> > I found the following differences in the catalina.out file with
> bin
> > > log
> > > >> > enabled and disabled.  Can you point me in the right direction on
> > > >> debugging
> > > >> > the actual /client application further.  Obviously the problem is
> > > >> >
> > > >> > Jun 10, 2014 2:12:48 PM org.apache.catalina.core.StandardContext
> > start
> > > >> > SEVERE: Error listenerStart
> > > >> >
> > > >> > I could use a point in the right direction to get further
> debugging
> > > >> > info.
> > > >> >
> > > >> >
> > > >> > WITH BINLOG
> > > >> >
> > > >> > INFO  [c.c.u.c.ComponentContext] (main:null) Setup Spring
> > Application
> > > >> > context
> > > >> > INFO  [c.c.s.GsonHelper] (main:null) Default Builder inited.
> > > >> > Jun 10, 2014 2:12:48 PM org.apache.catalina.core.StandardContext
> > start
> > > >> > SEVERE: Error listenerStart
> > > >> > Jun 10, 2014 2:12:48 PM org.apache.catalina.core.StandardContext
> > start
> > > >> > SEVERE: Context [/client] startup failed due to previous errors
> > > >> > Jun 10, 2014 2:12:48 PM
> org.apache.catalina.loader.WebappClassLoader
> > > >> > clearReferencesThreads
> > > >> > SEVERE: A web application appears to have started a thread named
> > > >> > [FileWatchdog] but has failed to stop it. This is very likely to
> > > create
> > > >> a
> > > >> > memory leak.
> > > >> > Jun 10, 2014 2:12:48 PM
> org.apache.catalina.loader.WebappClassLoader
> > > >> > clearReferencesThreads
> > > >> > SEVERE: A web application appears to have started a thread named
> > > >> [Timer-0]
> > > >> > but has failed to stop it. This is very likely to create a memory
> > > leak.
> > > >> > Jun 10, 2014 2:12:48 PM
> org.apache.catalina.loader.WebappClassLoader
> > > >> > clearReferencesThreads
> > > >> > SEVERE: A web application appears to have started a thread named
> > > >> > [ConnectionConcierge-1] but has failed to stop it. This is very
> > likely
> > > >> to
> > > >> > create a memory leak.
> > > >> > Jun 10, 2014 2:12:48 PM
> org.apache.catalina.loader.WebappClassLoader
> > > >> > clearReferencesThreads
> > > >> > SEVERE: A web application appears to have started a thread named
> > > >> > [ClusteredAgentManager Timer] but has failed to stop it. This is
> > very
> > > >> > likely to create a memory leak.
> > > >> > Jun 10, 2014 2:12:48 PM
> org.apache.catalina.loader.WebappClassLoader
> > > >> > clearReferencesThreads
> > > >> > SEVERE: A web application appears to have started a thread named
> > > >> [Timer-1]
> > > >> > but has failed to stop it. This is very likely to create a memory
> > > leak.
> > > >> > Jun 10, 2014 2:12:48 PM
> org.apache.catalina.loader.WebappClassLoader
> > > >> > clearThreadLocalMap
> > > >> > SEVERE: A web application created a ThreadLocal with key of type
> > > >> > [org.springframework.core.NamedThreadLocal] (value [Prototype
> beans
> > > >> > currently in creation]) and a value of type [null] (value [null])
> > but
> > > >> > failed to remove it when the web application was stopped. To
> > prevent a
> > > >> > memory leak, the ThreadLocal has be
> > > >> > en forcibly removed.
> > > >> >
> > > >> >
> > > >> > WITHOUT BINLOG
> > > >> >
> > > >> >
> > > >> > INFO  [c.c.u.c.ComponentContext] (main:null) Setup Spring
> > Application
> > > >> > context
> > > >> > INFO  [c.c.s.GsonHelper] (main:null) Default Builder inited.
> > > >> > INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:null) Running
> > > system
> > > >> > integrity checker
> > com.cloud.upgrade.DatabaseIntegrityChecker@7f2af4a3
> > > >> > INFO  [c.c.u.DatabaseIntegrityChecker] (main:null) Grabbing lock
> to
> > > >> check
> > > >> > for database integrity.
> > > >> > INFO  [c.c.u.DatabaseIntegrityChecker] (main:null) Performing
> > database
> > > >> > integrity check
> > > >> > INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:null) Running
> > > system
> > > >> > integrity checker
> > > >> > org.apache.cloudstack.utils.identity.ManagementServerNode@7c73b9ad
> > > >> > INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:null)
> > Configuring
> > > >> > CloudStack Components
> > > >> > INFO  [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name =
> > > >> > UserDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk =
> > > false
> > > >> > maxEntriesLocalHeap = 5000 maxEntriesLocalDisk = 0
> > > >> > memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 300
> > > >> timeToIdleSeconds =
> > > >> > 300 persistence = none diskExpiryThreadInt
> > > >> > ervalSeconds = 120 cacheEventListeners:
> > > >> > net.sf.ehcache.statistics.LiveCacheStatisticsWrapper  hitCount = 0
> > > >> > memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound =
> 0
> > > >> > missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap =
> false
> > > >> > maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
> > > >> > INFO  [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name =
> > > >> > VMTemplateDaoImpl status = STATUS_ALIVE eternal = false
> > > overflowToDisk =
> > > >> > false maxEntriesLocalHeap = 100 maxEntriesLocalDisk = 0
> > > >> > memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600
> > > >> timeToIdleSeconds =
> > > >> > 300 persistence = none diskExpiryThre
> > > >> > adIntervalSeconds = 120 cacheEventListeners:
> > > >> > net.sf.ehcache.statistics.LiveCacheStatisticsWrapper  hitCount = 0
> > > >> > memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound =
> 0
> > > >> > missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap =
> false
> > > >> > maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = fals
> > > >> > e ]
> > > >> > INFO  [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name =
> > > >> > ServiceOfferingDaoImpl status = STATUS_ALIVE eternal = false
> > > >> overflowToDisk
> > > >> > = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0
> > > >> > memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600
> > > >> timeToIdleSeconds =
> > > >> > 300 persistence = none diskExpiry
> > > >> > ThreadIntervalSeconds = 120 cacheEventListeners:
> > > >> > net.sf.ehcache.statistics.LiveCacheStatisticsWrapper  hitCount = 0
> > > >> > memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound =
> 0
> > > >> > missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap =
> false
> > > >> > maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned =
> > > >> > false ]
> > > >> > INFO  [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name =
> > > >> > DiskOfferingDaoImpl status = STATUS_ALIVE eternal = false
> > > >> overflowToDisk =
> > > >> > false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0
> > > >> > memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600
> > > >> timeToIdleSeconds =
> > > >> > 300 persistence = none diskExpiryThr
> > > >> > eadIntervalSeconds = 120 cacheEventListeners:
> > > >> > net.sf.ehcache.statistics.LiveCacheStatisticsWrapper  hitCount = 0
> > > >> > memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound =
> 0
> > > >> > missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap =
> false
> > > >> > maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = fal
> > > >> > se ]
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Jun 10, 2014, at 10:30 AM, Rafael Weingartner <
> > > >> > rafaelweingartner@gmail.com<mailto:rafaelweingartner@gmail.com>>
> > > wrote:
> > > >> >
> > > >> > you do not need to debug into the mysql, you just need to find the
> > > stack
> > > >> > trace in the tomcat that is causing the app to shutdown during the
> > > >> start up
> > > >> > process
> > > >> >
> > > >> >
> > > >> > On Tue, Jun 10, 2014 at 11:26 AM, Steve Searles <
> > ssearles@zimcom.net
> > > >> > <mailto:ssearles@zimcom.net>>
> > > >> > wrote:
> > > >> >
> > > >> > Yes, I will see if I can get any more debugging info out of it.
>  Yes
> > > >> that
> > > >> > is the URL we are using to access.  In the condition where the bin
> > log
> > > >> is
> > > >> > on I get a 404 from tomcat meaning that the application has not
> > > >> started.  I
> > > >> > will try some further debugging on the mysql side as well.  This
> is
> > > not
> > > >> an
> > > >> > upgrade but rather a fresh install of 4.3.  That is the problem I
> am
> > > >> seeing
> > > >> > as well, no useful debug information.
> > > >> >
> > > >> > Steve Searles
> > > >> >
> > > >> > On Jun 10, 2014, at 8:21 AM, Rafael Weingartner <
> > > >> > rafaelweingartner@gmail.com<mailto:rafaelweingartner@gmail.com>>
> > > wrote:
> > > >> >
> > > >> > I thought that it could be a missing dependency problem, because
> > with
> > > CS
> > > >> > 4.3 upgrade from 4.2.1 we had one. The mysql jdbc .jar was not in
> > the
> > > >> > classpath. So, we had to manually add it.
> > > >> >
> > > >> > I did not see any expressive tty in your log files.
> > > >> > Could you look for some more meaningful java stack traces, then we
> > > might
> > > >> >
> > > >> > be
> > > >> >
> > > >> > able to help you.
> > > >> > BTW, how are you trying to access the CS?
> > http://serverIp:8080/client
> > > ?
> > > >> >
> > > >> >
> > > >> > On Mon, Jun 9, 2014 at 11:31 PM, Steve Searles <
> ssearles@zimcom.net
> > > >> > <mailto:ssearles@zimcom.net>>
> > > >> >
> > > >> > wrote:
> > > >> >
> > > >> >
> > > >> > It is also worth noting that these are the last two DB queries
> that
> > > >> >
> > > >> > happen
> > > >> >
> > > >> > before the stop. So it may have something to do with cloudbridge?
> > > >> >
> > > >> >
> > > >> >
> > > >> > 140609 22:28:00     8 Connect   cloud@db01.cvg.cs.zimcom.net
> > <mailto:
> > > >> > cloud@db01.cvg.cs.zimcom.net> on
> > > >> > cloudbridge
> > > >> >              8 Query     /* mysql-connector-java-5.1.17-SNAPSHOT (
> > > >> > Revision: ${bzr.revision-id} ) */SHOW VARIABLES WHERE
> Variable_name
> > > >> > ='language' OR Variable_name = 'net_write_timeout' OR
> Variable_name
> > =
> > > >> > 'interactive_timeout' OR Variable_name = 'wait_timeout' OR
> > > >> >
> > > >> > Variable_name =
> > > >> >
> > > >> > 'character_set_client' OR Variable_name =
> 'character_set_connection'
> > > OR
> > > >> > Variable_name = 'character_set' OR Variable_name =
> > > >> >
> > > >> > 'character_set_server'
> > > >> >
> > > >> > OR Variable_name = 'tx_isolation' OR Variable_name =
> > > >> > 'transaction_isolation' OR Variable_name = 'character_set_results'
> > OR
> > > >> > Variable_name = 'timezone' OR Variable_name = 'time_zone' OR
> > > >> >
> > > >> > Variable_name
> > > >> >
> > > >> > = 'system_time_zone' OR Variable_name = 'lower_case_table_names'
> OR
> > > >> > Variable_name = 'max_allowed_packet' OR Variable_name =
> > > >> >
> > > >> > 'net_buffer_length'
> > > >> >
> > > >> > OR Variable_name = 'sql_mode' OR Variable_name =
> 'query_cache_type'
> > OR
> > > >> > Variable_name = 'query_cache_size' OR Variable_name =
> 'init_connect'
> > > >> >              8 Query     /* mysql-connector-java-5.1.17-SNAPSHOT (
> > > >> > Revision: ${bzr.revision-id} ) */SELECT
> > > >> >
> > > >> > @@session.auto_increment_increment
> > > >> >
> > > >> >              8 Query     SHOW COLLATION
> > > >> >              8 Query     SET NAMES latin1
> > > >> >              8 Query     SET character_set_results = NULL
> > > >> >              8 Query     SET autocommit=1
> > > >> >              8 Query     SET sql_mode='STRICT_TRANS_TABLES'
> > > >> >              8 Query     SET autocommit=0
> > > >> >              8 Query     SET autocommit=0
> > > >> >              8 Query     SELECT offering_bundle.ID,
> > > >> > offering_bundle.AmazonEC2Offering,
> > offering_bundle.CloudStackOffering
> > > >> >
> > > >> > FROM
> > > >> >
> > > >> > offering_bundle
> > > >> >              8 Query     commit
> > > >> >              8 Query     rollback
> > > >> >              8 Query     SET autocommit=1
> > > >> >              8 Query     SET autocommit=0
> > > >> >              8 Query     SET autocommit=0
> > > >> >              8 Query     SELECT mhost.ID, mhost.MHostKey,
> > > >> > mhost.Host, mhost.Version, mhost.LastHeartbeatTime FROM mhost
> WHERE
> > > >> > mhost.MHostKey = _binary'00:50:56:92:04:18'  ORDER BY RAND()
> LIMIT 1
> > > >> >              8 Query     commit
> > > >> >              8 Query     rollback
> > > >> >              8 Query     SET autocommit=1
> > > >> >              8 Query     SET autocommit=0
> > > >> >              8 Query     SET autocommit=0
> > > >> >              8 Query     UPDATE mhost SET mhost.Host=_binary'
> > > >> > http://localhost:7080/awsapi' WHERE mhost.ID = 1
> > > >> >              8 Query     commit
> > > >> >              8 Query     rollback
> > > >> >              8 Query     SET autocommit=1
> > > >> >              8 Query     SET autocommit=0
> > > >> >              8 Query     SET autocommit=0
> > > >> >              8 Query     SELECT shost.ID, shost.Host,
> > > >> > shost.HostType, shost.MHostID, shost.ExportRoot, shost.UserOnHost,
> > > >> > shost.UserPassword FROM shost WHERE shost.MHostID = 1  AND
> > > >> >
> > > >> > shost.ExportRoot
> > > >> >
> > > >> > = _binary'/Users/john1/S3-Mount'  ORDER BY RAND() LIMIT 1
> > > >> >              8 Query     commit
> > > >> >              8 Query     rollback
> > > >> >              8 Query     SET autocommit=1
> > > >> >              9 Connect   cloud@db01.cvg.cs.zimcom.net<mailto:
> > > >> > cloud@db01.cvg.cs.zimcom.net> on cloud
> > > >> >              9 Query     /* mysql-connector-java-5.1.17-SNAPSHOT (
> > > >> > Revision: ${bzr.revision-id} ) */SHOW VARIABLES WHERE
> Variable_name
> > > >> > ='language' OR Variable_name = 'net_write_timeout' OR
> Variable_name
> > =
> > > >> > 'interactive_timeout' OR Variable_name = 'wait_timeout' OR
> > > >> >
> > > >> > Variable_name =
> > > >> >
> > > >> > 'character_set_client' OR Variable_name =
> 'character_set_connection'
> > > OR
> > > >> > Variable_name = 'character_set' OR Variable_name =
> > > >> >
> > > >> > 'character_set_server'
> > > >> >
> > > >> > OR Variable_name = 'tx_isolation' OR Variable_name =
> > > >> > 'transaction_isolation' OR Variable_name = 'character_set_results'
> > OR
> > > >> > Variable_name = 'timezone' OR Variable_name = 'time_zone' OR
> > > >> >
> > > >> > Variable_name
> > > >> >
> > > >> > = 'system_time_zone' OR Variable_name = 'lower_case_table_names'
> OR
> > > >> > Variable_name = 'max_allowed_packet' OR Variable_name =
> > > >> >
> > > >> > 'net_buffer_length'
> > > >> >
> > > >> > OR Variable_name = 'sql_mode' OR Variable_name =
> 'query_cache_type'
> > OR
> > > >> > Variable_name = 'query_cache_size' OR Variable_name =
> 'init_connect'
> > > >> >              9 Query     /* mysql-connector-java-5.1.17-SNAPSHOT (
> > > >> > Revision: ${bzr.revision-id} ) */SELECT
> > > >> >
> > > >> > @@session.auto_increment_increment
> > > >> >
> > > >> >              9 Query     SHOW COLLATION
> > > >> >              9 Query     SET NAMES latin1
> > > >> >              9 Query     SET character_set_results = NULL
> > > >> >              9 Query     SET autocommit=1
> > > >> >              9 Query     SET sql_mode='STRICT_TRANS_TABLES'
> > > >> >              9 Query     SET autocommit=0
> > > >> >              9 Query     SELECT @@session.tx_isolation
> > > >> >              9 Query     SET SESSION TRANSACTION ISOLATION LEVEL
> > > >> > READ COMMITTED
> > > >> >              9 Query     SELECT 1
> > > >> >              9 Query     SET autocommit=0
> > > >> >              9 Query     SELECT configuration.name,
> > > >> > configuration.value FROM configuration WHERE configuration.name =
> > > >> > _binary'enable.ec2.api'  ORDER BY RAND() LIMIT 1
> > > >> >              9 Query     commit
> > > >> >              9 Query     rollback
> > > >> >              9 Query     SET autocommit=1
> > > >> >
> > > >> >
> > > >> >
> > > >> > Steve Searles
> > > >> >
> > > >> >
> > > >> >
> > > >> > -----Original Message-----
> > > >> > From: Steve Searles [mailto:ssearles@zimcom.net <
> > ssearles@zimcom.net
> > > >> > <mailto:ssearles@zimcom.net>> <
> > > >> > ssearles@zimcom.net<mailto:ssearles@zimcom.net>>]
> > > >> > Sent: Monday, June 9, 2014 10:19 PM
> > > >> > To: users@cloudstack.apache.org<mailto:
> users@cloudstack.apache.org>
> > > >> > Subject: RE: Management Server Startup Issue
> > > >> >
> > > >> > Rafael, Thank you for responding so quickly.  Yes 4.3.0. I am
> using
> > a
> > > >> > dedicated DB servers and dedicated management servers.  I will
> > provide
> > > >> > whatever log files you would find useful. It would be odd if it
> was
> > a
> > > >> > dependancy since CS really has no visibility to the sql server
> other
> > > >> >
> > > >> > than a
> > > >> >
> > > >> > connect string? Or so I thought; and the system is running fine
> with
> > > the
> > > >> > binlog off.  With the bin log on I can run queries from the
> command
> > > line
> > > >> > and everything seems normal from the sql end,  but the management
> > > server
> > > >> > will not run.  Toggle binlog off CS works fine.  The replication
> > slave
> > > >> >
> > > >> > has
> > > >> >
> > > >> > not even been started yet.  Let me know what other info I can send
> > > over
> > > >> > that would be of help.  There is nothing in the catalina logs of
> > > >> >
> > > >> > interest
> > > >> >
> > > >> > and my logging level is currently set to trace for the
> > > >> > management-server.log.  I will turn on the sql query log and
> compare
> > > >> >
> > > >> > them
> > > >> >
> > > >> > between startups of the management nodes perhaps that will yeild
> > > >> >
> > > >> > something.
> > > >> >
> > > >> > I have included a dropbox link with the management server log
> from a
> > > >> > normal startup of CS and one with mysql binlog enabled.  Any help
> > > would
> > > >> >
> > > >> > be
> > > >> >
> > > >> > greatly appriciated.
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > >
> >
> https://dl.dropboxusercontent.com/u/51895629/management-server.logs.tar.gz
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > Steve Searles
> > > >> >
> > > >> >
> > > >> > -----Original Message-----
> > > >> > From: Rafael Weingartner [mailto:rafaelweingartner@gmail.com]
> > > >> > Sent: Monday, June 9, 2014 6:57 PM
> > > >> > To: users@cloudstack.apache.org
> > > >> > Subject: Re: Management Server Startup Issue
> > > >> >
> > > >> > please give more logs.
> > > >> > It is the 4.3.0 right? I bet there is a problem with a missing
> > > >> >
> > > >> > dependency.
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Mon, Jun 9, 2014 at 7:32 PM, Steve Searles <
> ssearles@zimcom.net>
> > > >> >
> > > >> > wrote:
> > > >> >
> > > >> >
> > > >> > Anyone have any idea why the management service would refuse to
> > start
> > > >> > if the DB server is configured as a master for replication?
> > > >> >
> > > >> >
> > > >> >
> > > >> > log_bin=mysql-bin
> > > >> > server_id=1
> > > >> >
> > > >> > Adding the above the my.cnf file of my DB server, restarting mysql
> > > >> > then restarting cloudstack-managment will cause cloudstack to get
> > hung
> > > >> > on startup here.
> > > >> >
> > > >> > 2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext]
> (main:null)
> > > >> > Starting
> > > >> >
> > com.cloud.bridge.persist.dao.SHostDaoImpl_EnhancerByCloudStack_9da5499
> > > >> > b
> > > >> > 2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext]
> (main:null)
> > > >> > Starting
> > > >> >
> > com.cloud.bridge.persist.dao.SObjectDaoImpl_EnhancerByCloudStack_847fb
> > > >> > 98c
> > > >> > 2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext]
> (main:null)
> > > >> > Starting
> > > >> >
> > com.cloud.bridge.persist.dao.CloudStackUserDaoImpl_EnhancerByCloudStac
> > > >> > k_7fce3ee6
> > > >> > 2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext]
> (main:null)
> > > >> > Starting
> > > >> >
> > com.cloud.bridge.service.core.ec2.EC2Engine_EnhancerByCloudStack_d70c9
> > > >> > e3c
> > > >> > 2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext]
> (main:null)
> > > >> > Starting
> > > >> >
> > com.cloud.bridge.service.controller.s3.ServiceProvider_EnhancerByCloud
> > > >> > Stack_23d38b1
> > > >> >
> > > >> > That is the last few lines of the management-server.log and it
> will
> > > >> > hang there indefinitely.
> > > >> >
> > > >> > I noticed this on startup as well.
> > > >> >
> > > >> > 2014-06-09 18:25:16,841 INFO  [c.c.u.d.T.Transaction] (main:null)
> Is
> > > >> > Data Base High Availiability enabled? Ans : false
> > > >> > 2014-06-09 18:25:17,164 DEBUG [c.c.u.d.ConnectionConcierge]
> > > >> > (main:null) Registering a database connection for LockMaster1
> > > >> > 2014-06-09 18:25:17,164 INFO  [c.c.u.d.Merovingian2] (main:null)
> > > >> > Cleaning up locks for 345049793560
> > > >> > 2014-06-09 18:25:17,175 INFO  [c.c.u.d.Merovingian2] (main:null)
> > > >> > Released
> > > >> > 0 locks for 345049793560
> > > >> > 2014-06-09 18:25:17,206 INFO
> >  [o.a.c.s.l.CloudStackExtendedLifeCycle]
> > > >> > (main:null) Running system integrity checker
> > > >> > com.cloud.upgrade.DatabaseUpgradeChecker@63c8e4ae
> > > >> > 2014-06-09 18:25:17,207 INFO  [c.c.u.DatabaseUpgradeChecker]
> > > >> > (main:null) Grabbing lock to check for database upgrade.
> > > >> > 2014-06-09 18:25:17,254 DEBUG [c.c.u.d.VersionDaoImpl] (main:null)
> > > >> > Checking to see if the database is at a version before it was the
> > > >> > version table is created
> > > >> > 2014-06-09 18:25:17,295 INFO  [c.c.u.DatabaseUpgradeChecker]
> > > >> > (main:null) DB version = 4.3.0 Code Version = 4.3.0
> > > >> > 2014-06-09 18:25:17,296 INFO  [c.c.u.DatabaseUpgradeChecker]
> > > >> > (main:null) DB version and code version matches so no upgrade
> > needed.
> > > >> >
> > > >> > The management server talking to the DB server at least enough to
> > > >> > check the version; and mysql from the command line on the
> management
> > > >> > server works fine as well.
> > > >> >
> > > >> >
> > > >> > This line caught my attention.
> > > >> >
> > > >> >
> > > >> > 2014-06-09 18:25:16,841 INFO  [c.c.u.d.T.Transaction] (main:null)
> Is
> > > >> > Data Base High Availiability enabled? Ans : false
> > > >> >
> > > >> > Is there something I need to set when using mysql replication
> > specific
> > > >> > to cloudstack? I do not see anything in the documentation on this
> > > >> > other than setting up basic replication.
> > > >> >
> > > >> >
> > > >> > if I comment out the above two lines in my.cnf and restart the
> > > >> > management server it fires up just fine.  I don’t get how enabling
> > bin
> > > >> > log would make a difference at all.
> > > >> >
> > > >> > Any help or explication for this strange issue would be
> appreciated.
> > > >> >
> > > >> >
> > > >> > - Steve Searles
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Rafael Weingärtner
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Rafael Weingärtner
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Rafael Weingärtner
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Rafael Weingärtner
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Rafael Weingärtner
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Rafael Weingärtner
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >> --
> > > >> Rafael Weingärtner
> > > >>
> > > >
> > > >
> > >
> > >
> > > --
> > > Rafael Weingärtner
> > >
> >
>
>
>
> --
> Rafael Weingärtner
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message