cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam <adam.scarce...@gmail.com>
Subject Re: Management Server Startup Issue
Date Thu, 17 Jul 2014 12:48:38 GMT
OK, so this morning I did a complete re-install again (clean wipe and clean
install of both mysqld and cloudstack) and this time I did not restore my
db backup and cloudstack started up no problem.

I then shut cloudstack down and restored my mysql db for cloudstack and
tried starting cloudstack-management again, but I got a 404. So it's
something in the database that's actually causing the management UI not to
load. I'll need to keep restoring back in time to see if this goes away.

I'll keep you all posted...

Best Regards,



Adam Scarcella


On Wed, Jul 16, 2014 at 7:40 PM, Rafael Weingartner <
rafaelweingartner@gmail.com> wrote:

> could you please send us the tomcat's log files?
>
>
> On Wed, Jul 16, 2014 at 7:16 PM, Adam <adam.scarcella@gmail.com> wrote:
>
> > I have done a complete wipe & reinstall with full db restore from 3 days
> > ago (last known good state) and still have exactly the same issue so it's
> > environmental. I'm on a train now so will have to post the new logs
> > tomorrow.
> >
> > Needless to say this issue is extremely frustrating.
> >
> > Best Regards,
> >
> > Adam Scarcella
> > 44 Avery Street
> > Ipswich, MA 01938
> > H: (978) 471-4000
> > M: (781) 367-8015
> > On Jul 16, 2014 3:44 PM, "Rafael Weingartner" <
> rafaelweingartner@gmail.com
> > >
> > wrote:
> >
> > > I did not find any errors there.
> > > Isn't the http://server:port/client working?
> > > Could you please provide us with more logs entries?
> > >
> > >
> > >
> > > On Wed, Jul 16, 2014 at 4:25 PM, Adam <adam.scarcella@gmail.com>
> wrote:
> > >
> > > > Here are the last few lines in my catalina.out:
> > > >
> > > > INFO: Deploying module: addressing-1.5.6 -
> > > >
> > > >
> > >
> >
> file:/usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/lib/axis2-1.5.6.jar
> > > > Jul 16, 2014 9:30:08 AM org.apache.catalina.loader.WebappClassLoader
> > > > loadClass
> > > > INFO: Illegal access: this web application instance has been stopped
> > > > already.  Could not load
> > > > org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.  The
> > > > eventual following stack trace is caused by an error thrown for
> > debugging
> > > > purposes as well as to attempt to terminate the thread which caused
> the
> > > > illegal access, and has no functional impact.
> > > > java.lang.IllegalStateException
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1369)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
> > > >         at java.util.TimerThread.mainLoop(Timer.java:555)
> > > >         at java.util.TimerThread.run(Timer.java:505)
> > > >
> > > > Exception in thread "Timer-2" java.lang.NoClassDefFoundError:
> > > > org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
> > > >         at java.util.TimerThread.mainLoop(Timer.java:555)
> > > >         at java.util.TimerThread.run(Timer.java:505)
> > > > Caused by: java.lang.ClassNotFoundException:
> > > > org.apache.cloudstack.managed.context.ManagedContextTimerTask$1
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1483)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
> > > >         ... 3 more
> > > > Jul 16, 2014 9:30:14 AM org.apache.axis2.deployment.ServiceDeployer
> > > deploy
> > > > INFO: Deploying Web service: cloud-ec2.aar -
> > > >
> > > >
> > >
> >
> file:/usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/services/cloud-ec2.aar
> > > > Jul 16, 2014 9:30:14 AM org.apache.axis2.deployment.ServiceDeployer
> > > deploy
> > > > INFO: Deploying Web service: version-1.5.6.aar -
> > > >
> > > >
> > >
> >
> file:/usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/services/version-1.5.6.aar
> > > > Jul 16, 2014 9:30:14 AM org.apache.axis2.util.OnDemandLogger warn
> > > > WARNING: No transportReceiver for
> > > > org.apache.axis2.transport.http.AxisServletListener found. An
> instance
> > > for
> > > > HTTP will be configured automatically. Please update your axis2.xml
> > file!
> > > > Jul 16, 2014 9:30:14 AM
> > > >
> > > >
> > >
> >
> org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor
> > > > <init>
> > > > INFO: JSR-330 'javax.inject.Inject' annotation found and supported
> for
> > > > autowiring
> > > > Jul 16, 2014 9:30:14 AM org.apache.catalina.startup.HostConfig
> > > > deployDirectory
> > > > INFO: Deploying web application directory webapps
> > > > Jul 16, 2014 9:30:14 AM org.apache.coyote.http11.Http11NioProtocol
> > start
> > > > INFO: Starting Coyote HTTP/1.1 on http-7080
> > > > Jul 16, 2014 9:30:14 AM org.apache.catalina.startup.Catalina start
> > > > INFO: Server startup in 50870 ms
> > > >
> > > >
> > > >
> > > > Best Regards,
> > > >
> > > >
> > > >
> > > > Adam Scarcella
> > > >
> > > >
> > > > On Tue, Jul 15, 2014 at 7:04 PM, Rafael Weingartner <
> > > > rafaelweingartner@gmail.com> wrote:
> > > >
> > > > > The awsapi not getting up is not a problem; hence you are not using
> > it.
> > > > Why
> > > > > is not your CS app going up? Do you have any others log entries?
> > > > >
> > > > >
> > > > > On Tue, Jul 15, 2014 at 5:24 PM, Adam <adam.scarcella@gmail.com>
> > > wrote:
> > > > >
> > > > > > I do not use awsapi either, and have never turned it on. If I
> look
> > in
> > > > the
> > > > > > database I can see that those features are turned off too:
> > > > > >
> > > > > > SELECT * FROM cloud.configuration where name = 'enable.ec2.api'
> or
> > > > name =
> > > > > > 'enable.s3.api';
> > > > > >
> > > > > > # category, instance, component, name, value, description,
> > > > default_value,
> > > > > > updated, scope, is_dynamic
> > > > > > 'Advanced', 'DEFAULT', 'management-server', 'enable.ec2.api',
> > > 'false',
> > > > > > 'enable EC2 API on CloudStack', 'false', NULL, NULL, '0'
> > > > > > 'Advanced', 'DEFAULT', 'management-server', 'enable.s3.api',
> > 'false',
> > > > > > 'enable Amazon S3 API on CloudStack', 'false', NULL, NULL, '0'
> > > > > >
> > > > > >
> > > > > > I found this though:
> > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-2758
> > > > > >
> > > > > > Not sure if it's related.
> > > > > >
> > > > > > I tried deleted everything from both the work and temp dirs, but
> it
> > > had
> > > > > no
> > > > > > affect.
> > > > > >
> > > > > > :(
> > > > > >
> > > > > > Best Regards,
> > > > > >
> > > > > >
> > > > > >
> > > > > > Adam Scarcella
> > > > > >
> > > > > >
> > > > > > On Tue, Jul 15, 2014 at 12:22 PM, Rafael Weingartner <
> > > > > > rafaelweingartner@gmail.com> wrote:
> > > > > >
> > > > > > > Have you tried to clean the directories work and temp of the
> > tomcat
> > > > > > > installation that the CS is using?
> > > > > > >
> > > > > > > rm -rf /usr/share/cloudstack-management/temp/*
> > > > > > >
> > > > > > > rm -rf /usr/share/cloudstack-management/work/*
> > > > > > >
> > > > > > >
> > > > > > > In my CS installation I do not even have the awsapi files.
> > > > > > >
> > > > > > > I have just noticed that now, it has a soft link pointing to
> > > > > > > ../../cloudstack-bridge/webapps/awsapi, However, that folder
> does
> > > not
> > > > > > > exist.
> > > > > > >
> > > > > > > Are you using awsapi?
> > > > > > >
> > > > > > > I do not use it, so that is not a problem.
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jul 15, 2014 at 1:05 PM, Adam <
> adam.scarcella@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > OK, so I have this issue HTTP Status 404
> > > > > > > >
> > > > > > > > [image: Inline image 1]
> > > > > > > >
> > > > > > > >
> > > > > > > > ...except it must be different in some way because I've gone
> > > > through
> > > > > > this
> > > > > > > > entire thread and even though I have the same original error
> > > > > mentioned
> > > > > > > and
> > > > > > > > the same behavior, I do not have the same "smoking gun" so to
> > > speak
> > > > > in
> > > > > > > the
> > > > > > > > localhost.log where my logging type was STATEMENT instead of
> > > >  MIXED.
> > > > > > > >
> > > > > > > > [root@cs-east-prod1 logs]# cat /etc/my.cnf
> > > > > > > > [mysqld]
> > > > > > > > datadir=/var/lib/mysql
> > > > > > > > sockeet=/var/lib/mysql/mysql.sock
> > > > > > > > user=mysql
> > > > > > > > # Disabling symbolic-links is recommended to prevent assorted
> > > > > security
> > > > > > > > risks
> > > > > > > > symbolic-links=0
> > > > > > > >
> > > > > > > >
> > > > > > > > #
> > > > > > > > # BEGIN CloudStack 4.3 Requirements
> > > > > > > > #
> > > > > > > > innodb_rollback_on_timeout=1
> > > > > > > > innodb_lock_wait_timeout=600
> > > > > > > > max_connections=350
> > > > > > > > log-bin=mysql-bin
> > > > > > > > binlog-format = 'ROW'
> > > > > > > > #
> > > > > > > > # END CloudStack 4.3 Requirements
> > > > > > > > #
> > > > > > > >
> > > > > > > > [mysqld_safe]
> > > > > > > > log-error=/var/log/mysqld.log
> > > > > > > > pid-file=/var/run/mysqld/mysqld.pid
> > > > > > > > [root@cs-east-prod1 logs]#
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > I upgraded from 4.2.1 to 4.3 (CentOS 6.5 x86_64, KVM) and
> ever
> > > > since
> > > > > > I've
> > > > > > > > not been able to start the cloudstack-management ui. I see
> > these
> > > > > errors
> > > > > > > in
> > > > > > > > the catalina.out log:
> > > > > > > >
> > > > > > > > Jul 15, 2014 10:13:57 AM
> > > org.apache.axis2.deployment.ModuleDeployer
> > > > > > > deploy
> > > > > > > > SEVERE: The addressing-1.5.4.mar module, which is not valid,
> > > caused
> > > > > The
> > > > > > > > module.xml file cannot be found for the module:
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules/addressing-1.5.4.mar
> > > > > > > > org.apache.axis2.deployment.DeploymentException: The
> module.xml
> > > > file
> > > > > > > > cannot be found for the module:
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules/addressing-1.5.4.mar
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.axis2.deployment.repository.util.ArchiveReader.readModuleArchive(ArchiveReader.java:529)
> > > > > > > >         at
> > > > > > > >
> > > > > >
> > > >
> > org.apache.axis2.deployment.ModuleDeployer.deploy(ModuleDeployer.java:75)
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.axis2.deployment.repository.util.DeploymentFileData.deploy(DeploymentFileData.java:136)
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.axis2.deployment.DeploymentEngine.doDeploy(DeploymentEngine.java:661)
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.axis2.deployment.RepositoryListener.init(RepositoryListener.java:251)
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.axis2.deployment.RepositoryListener.init2(RepositoryListener.java:68)
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.axis2.deployment.RepositoryListener.<init>(RepositoryListener.java:63)
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.axis2.deployment.DeploymentEngine.loadRepository(DeploymentEngine.java:147)
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.axis2.deployment.WarBasedAxisConfigurator.getAxisConfiguration(WarBasedAxisConfigurator.java:229)
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.axis2.context.ConfigurationContextFactory.createConfigurationContext(ConfigurationContextFactory.java:64)
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.axis2.transport.http.AxisServlet.initConfigContext(AxisServlet.java:567)
> > > > > > > >         at
> > > > > > > >
> > > > >
> > org.apache.axis2.transport.http.AxisServlet.init(AxisServlet.java:452)
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.catalina.core.StandardWrapper.loadServlet(StandardWrapper.java:1173)
> > > > > > > >         at
> > > > > > > >
> > > > >
> > org.apache.catalina.core.StandardWrapper.load(StandardWrapper.java:993)
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.catalina.core.StandardContext.loadOnStartup(StandardContext.java:4187)
> > > > > > > >         at
> > > > > > > >
> > > > > >
> > > >
> > org.apache.catalina.core.StandardContext.start(StandardContext.java:4496)
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:791)
> > > > > > > >         at
> > > > > > > >
> > > > >
> > org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:771)
> > > > > > > >         at
> > > > > > > >
> > > > org.apache.catalina.core.StandardHost.addChild(StandardHost.java:526)
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1041)
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:964)
> > > > > > > >         at
> > > > > > > >
> > > > >
> > org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:502)
> > > > > > > >         at
> > > > > > > >
> > > org.apache.catalina.startup.HostConfig.start(HostConfig.java:1277)
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:321)
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:119)
> > > > > > > >         at
> > > > > > > >
> > > > org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1053)
> > > > > > > >         at
> > > > > > > >
> > > org.apache.catalina.core.StandardHost.start(StandardHost.java:722)
> > > > > > > >         at
> > > > > > > >
> > > > org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1045)
> > > > > > > >         at
> > > > > > > >
> > > > >
> > org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
> > > > > > > >         at
> > > > > > > >
> > > > > >
> > > >
> > org.apache.catalina.core.StandardService.start(StandardService.java:516)
> > > > > > > >         at
> > > > > > > >
> > > > >
> > org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
> > > > > > > >         at
> > > > > > org.apache.catalina.startup.Catalina.start(Catalina.java:593)
> > > > > > > >         at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > > > Method)
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > > > > >         at java.lang.reflect.Method.invoke(Method.java:606)
> > > > > > > >         at
> > > > > > > org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
> > > > > > > >         at
> > > > > > org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
> > > > > > > > Caused by: org.apache.axis2.deployment.DeploymentException:
> The
> > > > > > > module.xml
> > > > > > > > file cannot be found for the module:
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules/addressing-1.5.4.mar
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.axis2.deployment.repository.util.ArchiveReader.readModuleArchive(ArchiveReader.java:523)
> > > > > > > >         ... 37 more
> > > > > > > >
> > > > > > > > Jul 15, 2014 10:13:57 AM
> > > org.apache.axis2.deployment.ModuleDeployer
> > > > > > > deploy
> > > > > > > > INFO: Deploying module: addressing-1.5.6 -
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> file:/usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules/addressing-1.5.6.mar
> > > > > > > > Jul 15, 2014 10:13:57 AM
> > > org.apache.axis2.deployment.ModuleDeployer
> > > > > > > deploy
> > > > > > > > INFO: Deploying module: jaxws-1.5.6 -
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> file:/usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules/axis2-jaxws-mar-1.5.6.mar
> > > > > > > > Jul 15, 2014 10:13:57 AM
> > > org.apache.axis2.deployment.ModuleDeployer
> > > > > > > deploy
> > > > > > > > INFO: Deploying module: mtompolicy-1.5.6 -
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> file:/usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules/mtompolicy-1.5.6.mar
> > > > > > > > Jul 15, 2014 10:13:57 AM
> > > org.apache.axis2.deployment.ModuleDeployer
> > > > > > > deploy
> > > > > > > > INFO: Deploying module: addressing-1.5.6 -
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> file:/usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/lib/axis2-1.5.6.jar
> > > > > > > > Jul 15, 2014 10:13:57 AM
> > > > org.apache.catalina.loader.WebappClassLoader
> > > > > > > > loadClass
> > > > > > > > INFO: Illegal access: this web application instance has been
> > > > stopped
> > > > > > > > already.  Could not load
> > > > > > > >
> > org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.
> > > >  The
> > > > > > > > eventual following stack trace is caused by an error thrown
> for
> > > > > > debugging
> > > > > > > > purposes as well as to attempt to terminate the thread which
> > > caused
> > > > > the
> > > > > > > > illegal access, and has no functional impact.
> > > > > > > > java.lang.IllegalStateException
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1369)
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
> > > > > > > >         at java.util.TimerThread.mainLoop(Timer.java:555)
> > > > > > > >         at java.util.TimerThread.run(Timer.java:505)
> > > > > > > >
> > > > > > > > Exception in thread "Timer-2" java.lang.NoClassDefFoundError:
> > > > > > > >
> org/apache/cloudstack/managed/context/ManagedContextTimerTask$1
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
> > > > > > > >         at java.util.TimerThread.mainLoop(Timer.java:555)
> > > > > > > >         at java.util.TimerThread.run(Timer.java:505)
> > > > > > > > Caused by: java.lang.ClassNotFoundException:
> > > > > > > >
> org.apache.cloudstack.managed.context.ManagedContextTimerTask$1
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1483)
> > > > > > > >         at
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1329)
> > > > > > > >         ... 3 more
> > > > > > > >
> > > > > > > >
> > > > > > > > I have absolutely no errors of any kind in the
> > > > management-server.log.
> > > > > > > >
> > > > > > > > [root@cs-east-prod1 logs]# grep -i -E
> > > > > > > > 'exception|unable|fail|invalid|leak|warn|error'
> > > > > ./management-server.log
> > > > > > > > [root@cs-east-prod1 logs]#
> > > > > > > >
> > > > > > > >
> > > > > > > > The only thing I see in the access.log is:
> > > > > > > >
> > > > > > > > [root@cs-east-prod1 logs]# cat access_log.2014-07-15.txt
> > > > > > > > 10.97.34.115 - - [15/Jul/2014:10:27:15 -0500] "GET /client/
> > > > HTTP/1.1"
> > > > > > 404
> > > > > > > > 952
> > > > > > > > [root@cs-east-prod1 logs]#
> > > > > > > >
> > > > > > > >
> > > > > > > > ... and my localhost.log is entirely empty.
> > > > > > > >
> > > > > > > >
> > > > > > > > [root@cs-east-prod1 logs]# du -sh localhost.2014-07-15.log
> > > > > > > > 0       localhost.2014-07-15.log
> > > > > > > >
> > > > > > > >
> > > > > > > > And even more bizzare is the fact that I have another
> > CloudStack
> > > > 4.3
> > > > > > > > server that is running on XenServer and it's perfectly fine.
> It
> > > was
> > > > > not
> > > > > > > an
> > > > > > > > upgrade though, it was a clean install of 4.3. I've compared
> > all
> > > > the
> > > > > > > files
> > > > > > > > associated with the addressing-1.5.4.mar but could not find
> > > > anything
> > > > > > > > significant.
> > > > > > > >
> > > > > > > > Here is the output from my upgraded server (cs-east-prod1:
> > broken
> > > > > with
> > > > > > > 404)
> > > > > > > >
> > > > > > > > [root@cs-east-prod1 modules]# pwd
> > > > > > > >
> > > /usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules
> > > > > > > > [root@cs-east-prod1 modules]# md5sum addressing-1.5.4.mar
> > > > > > > > 1ef211c1a042e7d4b9b7e5f914222929  addressing-1.5.4.mar
> > > > > > > > [root@cs-east-prod1 modules]# cat modules.list
> > > > > > > > soapmonitor-1.5.6.mar
> > > > > > > > addressing-1.5.6.mar
> > > > > > > > mtompolicy-1.5.6.mar
> > > > > > > > mex-1.5.6.mar
> > > > > > > > ping-1.5.6.mar
> > > > > > > > axis2-jaxws-mar-1.5.6.mar
> > > > > > > > scripting-1.5.6.mar
> > > > > > > > [root@cs-east-prod1 modules]# ls -l
> > > > > > > > total 340
> > > > > > > > -rwxrwxrwx 1 cloud cloud 126981 Mar 25 11:01
> > addressing-1.5.4.mar
> > > > > > > > -rwxrwxrwx 1 cloud cloud  34518 Mar 25 11:01
> > addressing-1.5.6.mar
> > > > > > > > -rwxrwxrwx 1 cloud cloud  14363 Mar 25 11:01
> > > > > axis2-jaxws-mar-1.5.6.mar
> > > > > > > > -rwxrwxrwx 1 cloud cloud  38620 Mar 25 11:01 mex-1.5.6.mar
> > > > > > > > -rwxrwxrwx 1 cloud cloud    139 Mar 25 11:01 modules.list
> > > > > > > > -rwxrwxrwx 1 cloud cloud  18682 Mar 25 11:01
> > mtompolicy-1.5.6.mar
> > > > > > > > -rwxrwxrwx 1 cloud cloud  17647 Mar 25 11:01 ping-1.5.6.mar
> > > > > > > > -rwxrwxrwx 1 cloud cloud   9721 Mar 25 11:01 rahas-1.5.1.mar
> > > > > > > > -rwxrwxrwx 1 cloud cloud   9612 Mar 25 11:01
> rampart-1.5.1.mar
> > > > > > > > -rwxrwxrwx 1 cloud cloud  33779 Mar 25 11:01
> > scripting-1.5.6.mar
> > > > > > > > -rwxrwxrwx 1 cloud cloud  15650 Mar 25 11:01
> > > soapmonitor-1.5.6.mar
> > > > > > > > [root@cs-east-prod1 modules]# cd /
> > > > > > > > [root@cs-east-prod1 /]# find . -name "module.xml"
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ./usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/classes/com/cloud/bridge/auth/s3/module.xml
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ./usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/classes/com/cloud/bridge/auth/ec2/module.xml
> > > > > > > > [root@cs-east-prod1 /]# md5sum
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ./usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/classes/com/cloud/bridge/auth/s3/module.xml
> > > > > > > > baf5f63f5dd57484e9a88be6689939fc
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>  ./usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/classes/com/cloud/bridge/auth/s3/module.xml
> > > > > > > > [root@cs-east-prod1 /]# md5sum
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ./usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/classes/com/cloud/bridge/auth/ec2/module.xml
> > > > > > > > 913b4561258980f7b1b4c7cb3de261ba
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>  ./usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/classes/com/cloud/bridge/auth/ec2/module.xml
> > > > > > > > [root@cs-east-prod1 /]#
> > > > > > > >
> > > > > > > >
> > > > > > > > And here is the output from the exact same commands on my 4.3
> > > > server
> > > > > > > > that's working fine (na-nyc-bos-cs-poc2):
> > > > > > > >
> > > > > > > > [root@na-nyc-bos-cs-poc2 modules]# pwd
> > > > > > > >
> > > /usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules
> > > > > > > > [root@na-nyc-bos-cs-poc2 modules]# md5sum
> addressing-1.5.4.mar
> > > > > > > > 1ef211c1a042e7d4b9b7e5f914222929  addressing-1.5.4.mar
> > > > > > > > [root@na-nyc-bos-cs-poc2 modules]# cat modules.list
> > > > > > > > soapmonitor-1.5.6.mar
> > > > > > > > addressing-1.5.6.mar
> > > > > > > > mtompolicy-1.5.6.mar
> > > > > > > > mex-1.5.6.mar
> > > > > > > > ping-1.5.6.mar
> > > > > > > > axis2-jaxws-mar-1.5.6.mar
> > > > > > > > scripting-1.5.6.mar
> > > > > > > > [root@na-nyc-bos-cs-poc2 modules]# ls -l
> > > > > > > > total 340
> > > > > > > > -rw-r--r--. 1 cloud cloud 126981 Mar 25 11:01
> > > addressing-1.5.4.mar
> > > > > > > > -rw-r--r--. 1 cloud cloud  34518 Mar 25 11:01
> > > addressing-1.5.6.mar
> > > > > > > > -rw-r--r--. 1 cloud cloud  14363 Mar 25 11:01
> > > > > axis2-jaxws-mar-1.5.6.mar
> > > > > > > > -rw-r--r--. 1 cloud cloud  38620 Mar 25 11:01 mex-1.5.6.mar
> > > > > > > > -rw-r--r--. 1 cloud cloud    139 Mar 25 11:01 modules.list
> > > > > > > > -rw-r--r--. 1 cloud cloud  18682 Mar 25 11:01
> > > mtompolicy-1.5.6.mar
> > > > > > > > -rw-r--r--. 1 cloud cloud  17647 Mar 25 11:01 ping-1.5.6.mar
> > > > > > > > -rw-r--r--. 1 cloud cloud   9721 Mar 25 11:01 rahas-1.5.1.mar
> > > > > > > > -rw-r--r--. 1 cloud cloud   9612 Mar 25 11:01
> rampart-1.5.1.mar
> > > > > > > > -rw-r--r--. 1 cloud cloud  33779 Mar 25 11:01
> > scripting-1.5.6.mar
> > > > > > > > -rw-r--r--. 1 cloud cloud  15650 Mar 25 11:01
> > > soapmonitor-1.5.6.mar
> > > > > > > > [root@na-nyc-bos-cs-poc2 modules]# cd /
> > > > > > > > [root@na-nyc-bos-cs-poc2 /]# find . -name "module.xml"
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ./usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/classes/com/cloud/bridge/auth/s3/module.xml
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ./usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/classes/com/cloud/bridge/auth/ec2/module.xml
> > > > > > > > [root@na-nyc-bos-cs-poc2 /]# md5sum
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ./usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/classes/com/cloud/bridge/auth/s3/module.xml
> > > > > > > > baf5f63f5dd57484e9a88be6689939fc
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>  ./usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/classes/com/cloud/bridge/auth/s3/module.xml
> > > > > > > > [root@na-nyc-bos-cs-poc2 /]# md5sum
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ./usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/classes/com/cloud/bridge/auth/ec2/module.xml
> > > > > > > > 913b4561258980f7b1b4c7cb3de261ba
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>  ./usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/classes/com/cloud/bridge/auth/ec2/module.xml
> > > > > > > > [root@na-nyc-bos-cs-poc2 /]#
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > *HELP!!!*
> > > > > > > >
> > > > > > > > I need to get my cloud back up and running.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Best Regards,
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Adam Scarcella
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Jun 10, 2014 at 8:41 PM, Rafael Weingartner <
> > > > > > > > rafaelweingartner@gmail.com> wrote:
> > > > > > > >
> > > > > > > >> Nice, no doubts that others will run into this.
> > > > > > > >> But when they run into that problem, it is going to be just
> a
> > > > query
> > > > > at
> > > > > > > >> Google to uncover this thread ;)
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Tue, Jun 10, 2014 at 8:39 PM, Steve Searles <
> > > > ssearles@zimcom.net
> > > > > >
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >> >  Rafael,
> > > > > > > >> >
> > > > > > > >> >  Ok, I seem to have found the smoking gun in the
> > localhost.log
> > > > > > > >> >
> > > > > > > >> >  Caused by: java.sql.SQLException: Binary logging not
> > > possible.
> > > > > > > Message:
> > > > > > > >> > Transaction level 'READ-COMMITTED' in InnoDB is not safe
> for
> > > > > binlog
> > > > > > > mode
> > > > > > > >> > 'STATEMENT'
> > > > > > > >> > at
> > > > com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
> > > > > > > >> > at
> > com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
> > > > > > > >> > at
> > com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
> > > > > > > >> > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
> > > > > > > >> > at
> com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
> > > > > > > >> > at
> > > > com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
> > > > > > > >> > at
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2119)
> > > > > > > >> > at
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2415)
> > > > > > > >> > at
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2333)
> > > > > > > >> > at
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2318)
> > > > > > > >> > at
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
> > > > > > > >> > at
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeUpdate(DelegatingPreparedStatement.java:105)
> > > > > > > >> > at
> > > > > com.cloud.utils.db.GenericDaoBase.update(GenericDaoBase.java:823)
> > > > > > > >> > ... 198 more
> > > > > > > >> > Jun 10, 2014 7:02:25 PM
> > > > > org.apache.catalina.core.ApplicationContext
> > > > > > > log
> > > > > > > >> > INFO: Shutting down log4j
> > > > > > > >> >
> > > > > > > >> >  Oddly enough I thought the default logging type was MIXED
> > not
> > > > > > > >> STATEMENT.
> > > > > > > >> >  I am sure others will run into this. I am running
> (5.1.73)
> > > This
> > > > > is
> > > > > > > what
> > > > > > > >> > the MySQL website had to say.
> > > > > > > >> >
> > > > > > > >> >           The default binary logging format depends on the
> > > > version
> > > > > > of
> > > > > > > >> > MySQL you are using:
> > > > > > > >> >
> > > > > > > >> >    -
> > > > > > > >> >
> > > > > > > >> >    For MySQL 5.1.11 and earlier, and for MySQL 5.1.29 and
> > > later,
> > > > > > > >> >    statement-based logging is used by default.
> > > > > > > >> >    -
> > > > > > > >> >
> > > > > > > >> >    For MySQL 5.1.12 through MySQL 5.1.28, mixed logging is
> > > used
> > > > by
> > > > > > > >> default
> > > > > > > >> >
> > > > > > > >> >             If you are using InnoDB
> > > > > > > >> >    <
> > > > > >
> http://dev.mysql.com/doc/refman/5.1/en/innodb-storage-engine.html
> > > > > > > >
> > > > > > > >>
> > > > > > > >> >    tables and the transaction isolation level is READ
> > > COMMITTED
> > > > > > > >> >    <
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://dev.mysql.com/doc/refman/5.1/en/set-transaction.html#isolevel_read-committed
> > > > > > > >> >
> > > > > > > >> >    or READ UNCOMMITTED
> > > > > > > >> >    <
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://dev.mysql.com/doc/refman/5.1/en/set-transaction.html#isolevel_read-uncommitted
> > > > > > > >> >,
> > > > > > > >> >    only row-based logging can be used. It is *possible* to
> > > > change
> > > > > > the
> > > > > > > >>
> > > > > > > >> >    logging format to STATEMENT, but doing so at runtime
> > leads
> > > > very
> > > > > > > >> >    rapidly to errors because InnoDB can no longer perform
> > > > inserts.
> > > > > > > >> >
> > > > > > > >> >    Switching the replication format at runtime is not
> > > > recommended
> > > > > > when
> > > > > > > >> >    any temporary tables exist, because temporary tables
> are
> > > > logged
> > > > > > > only
> > > > > > > >> when
> > > > > > > >> >    using statement-based replication, whereas with
> row-based
> > > > > > > >> replication they
> > > > > > > >> >    are not logged. With mixed replication, temporary
> tables
> > > are
> > > > > > > usually
> > > > > > > >> >    logged; exceptions happen with user-defined functions
> > > (UDFs)
> > > > > and
> > > > > > > >> with the
> > > > > > > >> >    UUID()
> > > > > > > >> >    <
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://dev.mysql.com/doc/refman/5.1/en/miscellaneous-functions.html#function_uuid
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >> >    function.
> > > > > > > >> >
> > > > > > > >> >    With the binary log format set to ROW, many changes are
> > > > written
> > > > > > to
> > > > > > > >> the
> > > > > > > >> >    binary log using the row-based format. Some changes,
> > > however,
> > > > > > still
> > > > > > > >> use the
> > > > > > > >> >    statement-based format. Examples include all DDL (data
> > > > > definition
> > > > > > > >> language)
> > > > > > > >> >    statements such as CREATE TABLE
> > > > > > > >> >    <
> > http://dev.mysql.com/doc/refman/5.1/en/create-table.html
> > > >,
> > > > > > ALTER
> > > > > > > >> TABLE
> > > > > > > >> >    <
> http://dev.mysql.com/doc/refman/5.1/en/alter-table.html
> > >,
> > > > or
> > > > > > DROP
> > > > > > > >> >    TABLE <
> > > > http://dev.mysql.com/doc/refman/5.1/en/drop-table.html
> > > > > >.
> > > > > > > >>
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >  Add
> > > > > > > >> >  binlog-format=ROW
> > > > > > > >> > to /etc/my.cnf on the master DB server and you are good to
> > go.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >    Steven Searles, CTO | ssearles@zimcom.net
> > > > > > > >> >  Zimcom Internet Solutions  | www.zimcom.net
> > > > > > > >> >  O: 513.231.9500  |  D: 513.233.4130
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >  On Jun 10, 2014, at 6:18 PM, Rafael Weingartner <
> > > > > > > >> > rafaelweingartner@gmail.com> wrote:
> > > > > > > >> >
> > > > > > > >> > maybe, but, I actually do not care that much about the app
> > > that
> > > > is
> > > > > > > used
> > > > > > > >> to
> > > > > > > >> > integrate CS with AWS, hence they are two separated apps,
> a
> > > > > problem
> > > > > > > with
> > > > > > > >> > AWS integration app should not affect the CS normal app.
> > > > > > > >> >
> > > > > > > >> > Could you please provide me again with the log file from
> > > > > > > >> > /var/log/cloudstack/management/management-server.log?
> There
> > > must
> > > > > be
> > > > > > > >> > something here.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Tue, Jun 10, 2014 at 6:34 PM, Steve Searles <
> > > > > ssearles@zimcom.net
> > > > > > >
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> > Is
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/cloudstack-issues/201401.mbox/%3CJIRA.12687852.1389186771873.61435.1389186832715@arcas%3E
> > > > > > > >> >  relevant here?
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Jun 10, 2014, at 5:31 PM, Steve Searles <
> > > ssearles@zimcom.net
> > > > > > > <mailto:
> > > > > > > >> > ssearles@zimcom.net>> wrote:
> > > > > > > >> >
> > > > > > > >> > Ok I was wrong, the addressing-1.5.4.mar is present.
> > > > > > > >> >
> > > > > > > >> > [root@csmgmt01 modules]# pwd
> > > > > > > >> >
> > > > > /usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules
> > > > > > > >> > [root@csmgmt01 modules]# ls -lash
> > > > > > > >> > total 344K
> > > > > > > >> > 4.0K drwxr-xr-x. 2 cloud cloud 4.0K Mar 31 18:21 .
> > > > > > > >> > 4.0K drwxr-xr-x. 7 cloud cloud 4.0K Mar 31 18:21 ..
> > > > > > > >> > 124K -rw-r--r--. 1 cloud cloud 124K Mar 27 01:06
> > > > > > addressing-1.5.4.mar
> > > > > > > >> > 36K -rw-r--r--. 1 cloud cloud  34K Mar 27 01:06
> > > > > addressing-1.5.6.mar
> > > > > > > >> > 16K -rw-r--r--. 1 cloud cloud  15K Mar 27 01:06
> > > > > > > >> axis2-jaxws-mar-1.5.6.mar
> > > > > > > >> > 40K -rw-r--r--. 1 cloud cloud  38K Mar 27 01:06
> > mex-1.5.6.mar
> > > > > > > >> > 4.0K -rw-r--r--. 1 cloud cloud  139 Mar 27 01:06
> > modules.list
> > > > > > > >> > 20K -rw-r--r--. 1 cloud cloud  19K Mar 27 01:06
> > > > > mtompolicy-1.5.6.mar
> > > > > > > >> > 20K -rw-r--r--. 1 cloud cloud  18K Mar 27 01:06
> > ping-1.5.6.mar
> > > > > > > >> > 12K -rw-r--r--. 1 cloud cloud 9.5K Mar 27 01:06
> > > rahas-1.5.1.mar
> > > > > > > >> > 12K -rw-r--r--. 1 cloud cloud 9.4K Mar 27 01:06
> > > > rampart-1.5.1.mar
> > > > > > > >> > 36K -rw-r--r--. 1 cloud cloud  33K Mar 27 01:06
> > > > > scripting-1.5.6.mar
> > > > > > > >> > 16K -rw-r--r--. 1 cloud cloud  16K Mar 27 01:06
> > > > > > soapmonitor-1.5.6.mar
> > > > > > > >> > [root@csmgmt01 modules]#
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > The error is complaining about the module.xml file
> missing.
> > > > > > > >> >
> > > > > > > >> > Steve Searles
> > > > > > > >> >
> > > > > > > >> > On Jun 10, 2014, at 5:21 PM, Steve Searles <
> > > ssearles@zimcom.net
> > > > > > > <mailto:
> > > > > > > >> > ssearles@zimcom.net><mailto:ssearles@zimcom.net>> wrote:
> > > > > > > >> >
> > > > > > > >> > I agree with what you are saying.  Yes.
> addressing-1.5.4.mar
> > > > > errors
> > > > > > I
> > > > > > > >> have
> > > > > > > >> > seen before.  I have no webapps7080 directory so that is
> why
> > > > that
> > > > > > > error
> > > > > > > >> is
> > > > > > > >> > being thrown.  I still am wondering even if a dependency
> is
> > > > > missing
> > > > > > > why
> > > > > > > >> is
> > > > > > > >> > it only a dependency when sql replication(bin log) is on?
> > Can
> > > > you
> > > > > > tell
> > > > > > > >> me
> > > > > > > >> > how to get a stack trace from CS startup?
> > > > > > > >> >
> > > > > > > >> > That file is present in these locations on the management
> > > node.
> > > > > > > >> >
> > > > > > > >> > [root@csmgmt01 /]# find / -name addressing-1.5.4.mar
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /root/.m2/repository/org/apache/axis2/addressing/1.5.4/addressing-1.5.4.mar
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /root/apache-cloudstack-4.3.0-src/dist/rpmbuild/BUILD/cloudstack-4.3.0/awsapi/target/cloud-awsapi-4.3.0/WEB-INF/modules/addressing-1.5.4.mar
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /usr/share/cloudstack-bridge/webapps/awsapi/WEB-INF/modules/addressing-1.5.4.mar
> > > > > > > >> > [root@csmgmt01 /]#
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > is it just a symlink that is missing?
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > [root@csmgmt01 webapps]# ls -las
> > > > > > > >> > total 12
> > > > > > > >> > 4 drwxr-xr-x.  3 root root 4096 Mar 31 18:21 .
> > > > > > > >> > 4 drwxr-xr-x.  4 root root 4096 Mar 31 18:21 ..
> > > > > > > >> > 4 drwxr-xr-x. 11 root root 4096 Mar 31 18:21 client
> > > > > > > >> > [root@csmgmt01 webapps]# cd client/
> > > > > > > >> > [root@csmgmt01 client]# ls
> > > > > > > >> > css  dictionary.jsp  images  index.jsp  lib  META-INF
> >  modules
> > > > > > >  plugins
> > > > > > > >> > scripts  tests  WEB-INF
> > > > > > > >> > [root@csmgmt01 client]# ls -lash
> > > > > > > >> > total 264K
> > > > > > > >> > 4.0K drwxr-xr-x. 11 root root 4.0K Mar 31 18:21 .
> > > > > > > >> > 4.0K drwxr-xr-x.  3 root root 4.0K Mar 31 18:21 ..
> > > > > > > >> > 4.0K drwxr-xr-x.  2 root root 4.0K Mar 31 18:21 css
> > > > > > > >> > 116K -rw-r--r--.  1 root root 115K Mar 27 01:06
> > dictionary.jsp
> > > > > > > >> > 4.0K drwxr-xr-x.  2 root root 4.0K Mar 31 18:21 images
> > > > > > > >> > 104K -rw-r--r--.  1 root root 103K Mar 27 01:06 index.jsp
> > > > > > > >> > 4.0K drwxr-xr-x.  5 root root 4.0K Mar 31 18:21 lib
> > > > > > > >> > 4.0K drwxr-xr-x.  2 root root 4.0K Mar 27 01:06 META-INF
> > > > > > > >> > 4.0K drwxr-xr-x.  6 root root 4.0K Mar 31 18:21 modules
> > > > > > > >> > 4.0K drwxr-xr-x.  3 root root 4.0K Mar 31 18:21 plugins
> > > > > > > >> > 4.0K drwxr-xr-x.  4 root root 4.0K Mar 31 18:21 scripts
> > > > > > > >> > 4.0K drwxr-xr-x.  2 root root 4.0K Mar 31 18:21 tests
> > > > > > > >> > 4.0K drwxr-xr-x.  4 root root 4.0K Mar 31 18:21 WEB-INF
> > > > > > > >> > [root@csmgmt01 client]# pwd
> > > > > > > >> > /usr/share/cloudstack-management/webapps/client
> > > > > > > >> > [root@csmgmt01 client]#
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > Steve Searles
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Jun 10, 2014, at 3:04 PM, Rafael Weingartner <
> > > > > > > >> > rafaelweingartner@gmail.com<mailto:
> > > rafaelweingartner@gmail.com
> > > > > > > ><mailto:
> > > > > > > >> > rafaelweingartner@gmail.com><mailto:
> > > rafaelweingartner@gmail.com
> > > > >>
> > > > > > > >> wrote:
> > > > > > > >> >
> > > > > > > >> > Look at this point:
> > > > > > > >> > "SEVERE: Error listenerStart
> > > > > > > >> > Jun 10, 2014 2:12:48 PM
> > > org.apache.catalina.core.StandardContext
> > > > > > start
> > > > > > > >> > SEVERE: Context [/client] startup failed due to previous
> > > errors"
> > > > > > > >> > I could not find the stacktrace into the logs you
> provided.
> > > But,
> > > > > it
> > > > > > > has
> > > > > > > >> to
> > > > > > > >> > have some stack which would help to understand why the CS
> is
> > > not
> > > > > > > getting
> > > > > > > >> > up.
> > > > > > > >> >
> > > > > > > >> > This is a good point to look at:
> > > > > > > >> > "SEVERE: The addressing-1.5.4.mar module, which is not
> > valid,
> > > > > caused
> > > > > > > The
> > > > > > > >> > module.xml file cannot be found for the module:
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules/addressing-1.5.4.mar
> > > > > > > >> > org.apache.axis2.deployment.DeploymentException: The
> > > module.xml
> > > > > file
> > > > > > > >> cannot
> > > > > > > >> > be found for the module:
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> /usr/share/cloudstack-management/webapps7080/awsapi/WEB-INF/modules/addressing-1.5.4.mar
> > > > > > > >> > at
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.axis2.deployment.repository.util.ArchiveReader.readModuleArchive(ArchiveReader.java:529)
> > > > > > > >> > at
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > >
> > >
> org.apache.axis2.deployment.ModuleDeployer.deploy(ModuleDeployer.java:75)
> > > > > > > >> > at
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.axis2.deployment.repository.util.DeploymentFileData.deploy(DeploymentFileData.java:136)
> > > > > > > >> > at
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.axis2.deployment.DeploymentEngine.doDeploy(DeploymentEngine.java:661)"
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > It seems there is a .xml file being missing.
> > > > > > > >> > But, still even though the awsapi is not working, the CS
> app
> > > > > should
> > > > > > > run
> > > > > > > >> > just fine, unless it depends on that service.
> > > > > > > >> >
> > > > > > > >> > This is also something interesting:
> > > > > > > >> > "Jun 10, 2014 2:12:58 PM
> > > > > > org.apache.catalina.loader.WebappClassLoader
> > > > > > > >> > loadClass
> > > > > > > >> > INFO: Illegal access: this web application instance has
> been
> > > > > stopped
> > > > > > > >> > already.  Could not load
> > > > > > > >> >
> > > org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.
> > > > > >  The
> > > > > > > >> > eventual following stack trace is caused by an error
> thrown
> > > for
> > > > > > > >> debugging
> > > > > > > >> > purposes as well as to attempt to terminate the thread
> which
> > > > > caused
> > > > > > > the
> > > > > > > >> > illegal access, and has no functional impact.
> > > > > > > >> > java.lang.IllegalStateException
> > > > > > > >> > at
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1369)"
> > > > > > > >> >
> > > > > > > >> > it seems there is a missing dependency.
> > > > > > > >> >
> > > > > > > >> > BTW: out of curiosity, what do tou have in folder
> > > > > > > >> > "/usr/share/cloudstack-management/webapps"
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Tue, Jun 10, 2014 at 3:53 PM, Steve Searles <
> > > > > ssearles@zimcom.net
> > > > > > > >> > <mailto:ssearles@zimcom.net><mailto:ssearles@zimcom.net
> > > > ><mailto:
> > > > > > > >> > ssearles@zimcom.net>> wrote:
> > > > > > > >> >
> > > > > > > >> > Nothing interesting in the management server logs, you saw
> > the
> > > > > > > complete
> > > > > > > >> > ones I sent yesterday.  I have attached the complete
> > > > catalina.out
> > > > > > > below.
> > > > > > > >> > This is starting to feel like a problem with the DB
> locking
> > > the
> > > > > > tables
> > > > > > > >> > during the startup integrity check.  I am able to lock the
> > > > tables
> > > > > > via
> > > > > > > >> the
> > > > > > > >> > command line with the bin log enabled without issue
> though.
> > > > > > > >> >
> > > > > > > >> >
> > > https://dl.dropboxusercontent.com/u/51895629/catalina.binlog.gz
> > > > > > > >> >
> > > > > > > >> > Steve Searles
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Jun 10, 2014, at 2:29 PM, Rafael Weingartner <
> > > > > > > >> > rafaelweingartner@gmail.com<mailto:
> > > rafaelweingartner@gmail.com
> > > > >>
> > > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > this is interesting:
> > > > > > > >> > "
> > > > > > > >> > Jun 10, 2014 2:12:48 PM
> > > org.apache.catalina.core.StandardContext
> > > > > > start
> > > > > > > >> > SEVERE: Context [/client] startup failed due to previous
> > > error"
> > > > > > > >> >
> > > > > > > >> > does it have any extra log before that message?
> > > > > > > >> > Did you find anything at
> > > > > /var/log/cloudstack/management-server.log?
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Tue, Jun 10, 2014 at 3:24 PM, Steve Searles <
> > > > > ssearles@zimcom.net
> > > > > > > >> > <mailto:ssearles@zimcom.net>>
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> > I found the following differences in the catalina.out file
> > > with
> > > > > bin
> > > > > > > log
> > > > > > > >> > enabled and disabled.  Can you point me in the right
> > direction
> > > > on
> > > > > > > >> debugging
> > > > > > > >> > the actual /client application further.  Obviously the
> > problem
> > > > is
> > > > > > > >> >
> > > > > > > >> > Jun 10, 2014 2:12:48 PM
> > > org.apache.catalina.core.StandardContext
> > > > > > start
> > > > > > > >> > SEVERE: Error listenerStart
> > > > > > > >> >
> > > > > > > >> > I could use a point in the right direction to get further
> > > > > debugging
> > > > > > > >> > info.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > WITH BINLOG
> > > > > > > >> >
> > > > > > > >> > INFO  [c.c.u.c.ComponentContext] (main:null) Setup Spring
> > > > > > Application
> > > > > > > >> > context
> > > > > > > >> > INFO  [c.c.s.GsonHelper] (main:null) Default Builder
> inited.
> > > > > > > >> > Jun 10, 2014 2:12:48 PM
> > > org.apache.catalina.core.StandardContext
> > > > > > start
> > > > > > > >> > SEVERE: Error listenerStart
> > > > > > > >> > Jun 10, 2014 2:12:48 PM
> > > org.apache.catalina.core.StandardContext
> > > > > > start
> > > > > > > >> > SEVERE: Context [/client] startup failed due to previous
> > > errors
> > > > > > > >> > Jun 10, 2014 2:12:48 PM
> > > > > org.apache.catalina.loader.WebappClassLoader
> > > > > > > >> > clearReferencesThreads
> > > > > > > >> > SEVERE: A web application appears to have started a thread
> > > named
> > > > > > > >> > [FileWatchdog] but has failed to stop it. This is very
> > likely
> > > to
> > > > > > > create
> > > > > > > >> a
> > > > > > > >> > memory leak.
> > > > > > > >> > Jun 10, 2014 2:12:48 PM
> > > > > org.apache.catalina.loader.WebappClassLoader
> > > > > > > >> > clearReferencesThreads
> > > > > > > >> > SEVERE: A web application appears to have started a thread
> > > named
> > > > > > > >> [Timer-0]
> > > > > > > >> > but has failed to stop it. This is very likely to create a
> > > > memory
> > > > > > > leak.
> > > > > > > >> > Jun 10, 2014 2:12:48 PM
> > > > > org.apache.catalina.loader.WebappClassLoader
> > > > > > > >> > clearReferencesThreads
> > > > > > > >> > SEVERE: A web application appears to have started a thread
> > > named
> > > > > > > >> > [ConnectionConcierge-1] but has failed to stop it. This is
> > > very
> > > > > > likely
> > > > > > > >> to
> > > > > > > >> > create a memory leak.
> > > > > > > >> > Jun 10, 2014 2:12:48 PM
> > > > > org.apache.catalina.loader.WebappClassLoader
> > > > > > > >> > clearReferencesThreads
> > > > > > > >> > SEVERE: A web application appears to have started a thread
> > > named
> > > > > > > >> > [ClusteredAgentManager Timer] but has failed to stop it.
> > This
> > > is
> > > > > > very
> > > > > > > >> > likely to create a memory leak.
> > > > > > > >> > Jun 10, 2014 2:12:48 PM
> > > > > org.apache.catalina.loader.WebappClassLoader
> > > > > > > >> > clearReferencesThreads
> > > > > > > >> > SEVERE: A web application appears to have started a thread
> > > named
> > > > > > > >> [Timer-1]
> > > > > > > >> > but has failed to stop it. This is very likely to create a
> > > > memory
> > > > > > > leak.
> > > > > > > >> > Jun 10, 2014 2:12:48 PM
> > > > > org.apache.catalina.loader.WebappClassLoader
> > > > > > > >> > clearThreadLocalMap
> > > > > > > >> > SEVERE: A web application created a ThreadLocal with key
> of
> > > type
> > > > > > > >> > [org.springframework.core.NamedThreadLocal] (value
> > [Prototype
> > > > > beans
> > > > > > > >> > currently in creation]) and a value of type [null] (value
> > > > [null])
> > > > > > but
> > > > > > > >> > failed to remove it when the web application was stopped.
> To
> > > > > > prevent a
> > > > > > > >> > memory leak, the ThreadLocal has be
> > > > > > > >> > en forcibly removed.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > WITHOUT BINLOG
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > INFO  [c.c.u.c.ComponentContext] (main:null) Setup Spring
> > > > > > Application
> > > > > > > >> > context
> > > > > > > >> > INFO  [c.c.s.GsonHelper] (main:null) Default Builder
> inited.
> > > > > > > >> > INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:null)
> > > > Running
> > > > > > > system
> > > > > > > >> > integrity checker
> > > > > > com.cloud.upgrade.DatabaseIntegrityChecker@7f2af4a3
> > > > > > > >> > INFO  [c.c.u.DatabaseIntegrityChecker] (main:null)
> Grabbing
> > > lock
> > > > > to
> > > > > > > >> check
> > > > > > > >> > for database integrity.
> > > > > > > >> > INFO  [c.c.u.DatabaseIntegrityChecker] (main:null)
> > Performing
> > > > > > database
> > > > > > > >> > integrity check
> > > > > > > >> > INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:null)
> > > > Running
> > > > > > > system
> > > > > > > >> > integrity checker
> > > > > > > >> >
> > > > org.apache.cloudstack.utils.identity.ManagementServerNode@7c73b9ad
> > > > > > > >> > INFO  [o.a.c.s.l.CloudStackExtendedLifeCycle] (main:null)
> > > > > > Configuring
> > > > > > > >> > CloudStack Components
> > > > > > > >> > INFO  [c.c.u.d.GenericDaoBase] (main:null) Cache created:
> [
> > > > name =
> > > > > > > >> > UserDaoImpl status = STATUS_ALIVE eternal = false
> > > > overflowToDisk =
> > > > > > > false
> > > > > > > >> > maxEntriesLocalHeap = 5000 maxEntriesLocalDisk = 0
> > > > > > > >> > memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 300
> > > > > > > >> timeToIdleSeconds =
> > > > > > > >> > 300 persistence = none diskExpiryThreadInt
> > > > > > > >> > ervalSeconds = 120 cacheEventListeners:
> > > > > > > >> > net.sf.ehcache.statistics.LiveCacheStatisticsWrapper
> >  hitCount
> > > > = 0
> > > > > > > >> > memoryStoreHitCount = 0 diskStoreHitCount = 0
> > > missCountNotFound
> > > > =
> > > > > 0
> > > > > > > >> > missCountExpired = 0 maxBytesLocalHeap = 0
> > overflowToOffHeap =
> > > > > false
> > > > > > > >> > maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned =
> > false
> > > ]
> > > > > > > >> > INFO  [c.c.u.d.GenericDaoBase] (main:null) Cache created:
> [
> > > > name =
> > > > > > > >> > VMTemplateDaoImpl status = STATUS_ALIVE eternal = false
> > > > > > > overflowToDisk =
> > > > > > > >> > false maxEntriesLocalHeap = 100 maxEntriesLocalDisk = 0
> > > > > > > >> > memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600
> > > > > > > >> timeToIdleSeconds =
> > > > > > > >> > 300 persistence = none diskExpiryThre
> > > > > > > >> > adIntervalSeconds = 120 cacheEventListeners:
> > > > > > > >> > net.sf.ehcache.statistics.LiveCacheStatisticsWrapper
> >  hitCount
> > > > = 0
> > > > > > > >> > memoryStoreHitCount = 0 diskStoreHitCount = 0
> > > missCountNotFound
> > > > =
> > > > > 0
> > > > > > > >> > missCountExpired = 0 maxBytesLocalHeap = 0
> > overflowToOffHeap =
> > > > > false
> > > > > > > >> > maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned =
> fals
> > > > > > > >> > e ]
> > > > > > > >> > INFO  [c.c.u.d.GenericDaoBase] (main:null) Cache created:
> [
> > > > name =
> > > > > > > >> > ServiceOfferingDaoImpl status = STATUS_ALIVE eternal =
> false
> > > > > > > >> overflowToDisk
> > > > > > > >> > = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0
> > > > > > > >> > memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600
> > > > > > > >> timeToIdleSeconds =
> > > > > > > >> > 300 persistence = none diskExpiry
> > > > > > > >> > ThreadIntervalSeconds = 120 cacheEventListeners:
> > > > > > > >> > net.sf.ehcache.statistics.LiveCacheStatisticsWrapper
> >  hitCount
> > > > = 0
> > > > > > > >> > memoryStoreHitCount = 0 diskStoreHitCount = 0
> > > missCountNotFound
> > > > =
> > > > > 0
> > > > > > > >> > missCountExpired = 0 maxBytesLocalHeap = 0
> > overflowToOffHeap =
> > > > > false
> > > > > > > >> > maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned =
> > > > > > > >> > false ]
> > > > > > > >> > INFO  [c.c.u.d.GenericDaoBase] (main:null) Cache created:
> [
> > > > name =
> > > > > > > >> > DiskOfferingDaoImpl status = STATUS_ALIVE eternal = false
> > > > > > > >> overflowToDisk =
> > > > > > > >> > false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0
> > > > > > > >> > memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600
> > > > > > > >> timeToIdleSeconds =
> > > > > > > >> > 300 persistence = none diskExpiryThr
> > > > > > > >> > eadIntervalSeconds = 120 cacheEventListeners:
> > > > > > > >> > net.sf.ehcache.statistics.LiveCacheStatisticsWrapper
> >  hitCount
> > > > = 0
> > > > > > > >> > memoryStoreHitCount = 0 diskStoreHitCount = 0
> > > missCountNotFound
> > > > =
> > > > > 0
> > > > > > > >> > missCountExpired = 0 maxBytesLocalHeap = 0
> > overflowToOffHeap =
> > > > > false
> > > > > > > >> > maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned =
> fal
> > > > > > > >> > se ]
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Jun 10, 2014, at 10:30 AM, Rafael Weingartner <
> > > > > > > >> > rafaelweingartner@gmail.com<mailto:
> > > rafaelweingartner@gmail.com
> > > > >>
> > > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > you do not need to debug into the mysql, you just need to
> > find
> > > > the
> > > > > > > stack
> > > > > > > >> > trace in the tomcat that is causing the app to shutdown
> > during
> > > > the
> > > > > > > >> start up
> > > > > > > >> > process
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Tue, Jun 10, 2014 at 11:26 AM, Steve Searles <
> > > > > > ssearles@zimcom.net
> > > > > > > >> > <mailto:ssearles@zimcom.net>>
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> > Yes, I will see if I can get any more debugging info out
> of
> > > it.
> > > > >  Yes
> > > > > > > >> that
> > > > > > > >> > is the URL we are using to access.  In the condition where
> > the
> > > > bin
> > > > > > log
> > > > > > > >> is
> > > > > > > >> > on I get a 404 from tomcat meaning that the application
> has
> > > not
> > > > > > > >> started.  I
> > > > > > > >> > will try some further debugging on the mysql side as well.
> > >  This
> > > > > is
> > > > > > > not
> > > > > > > >> an
> > > > > > > >> > upgrade but rather a fresh install of 4.3.  That is the
> > > problem
> > > > I
> > > > > am
> > > > > > > >> seeing
> > > > > > > >> > as well, no useful debug information.
> > > > > > > >> >
> > > > > > > >> > Steve Searles
> > > > > > > >> >
> > > > > > > >> > On Jun 10, 2014, at 8:21 AM, Rafael Weingartner <
> > > > > > > >> > rafaelweingartner@gmail.com<mailto:
> > > rafaelweingartner@gmail.com
> > > > >>
> > > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > I thought that it could be a missing dependency problem,
> > > because
> > > > > > with
> > > > > > > CS
> > > > > > > >> > 4.3 upgrade from 4.2.1 we had one. The mysql jdbc .jar was
> > not
> > > > in
> > > > > > the
> > > > > > > >> > classpath. So, we had to manually add it.
> > > > > > > >> >
> > > > > > > >> > I did not see any expressive tty in your log files.
> > > > > > > >> > Could you look for some more meaningful java stack traces,
> > > then
> > > > we
> > > > > > > might
> > > > > > > >> >
> > > > > > > >> > be
> > > > > > > >> >
> > > > > > > >> > able to help you.
> > > > > > > >> > BTW, how are you trying to access the CS?
> > > > > > http://serverIp:8080/client
> > > > > > > ?
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Mon, Jun 9, 2014 at 11:31 PM, Steve Searles <
> > > > > ssearles@zimcom.net
> > > > > > > >> > <mailto:ssearles@zimcom.net>>
> > > > > > > >> >
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > It is also worth noting that these are the last two DB
> > queries
> > > > > that
> > > > > > > >> >
> > > > > > > >> > happen
> > > > > > > >> >
> > > > > > > >> > before the stop. So it may have something to do with
> > > > cloudbridge?
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > 140609 22:28:00     8 Connect
> > cloud@db01.cvg.cs.zimcom.net
> > > > > > <mailto:
> > > > > > > >> > cloud@db01.cvg.cs.zimcom.net> on
> > > > > > > >> > cloudbridge
> > > > > > > >> >              8 Query     /*
> > > > mysql-connector-java-5.1.17-SNAPSHOT (
> > > > > > > >> > Revision: ${bzr.revision-id} ) */SHOW VARIABLES WHERE
> > > > > Variable_name
> > > > > > > >> > ='language' OR Variable_name = 'net_write_timeout' OR
> > > > > Variable_name
> > > > > > =
> > > > > > > >> > 'interactive_timeout' OR Variable_name = 'wait_timeout' OR
> > > > > > > >> >
> > > > > > > >> > Variable_name =
> > > > > > > >> >
> > > > > > > >> > 'character_set_client' OR Variable_name =
> > > > > 'character_set_connection'
> > > > > > > OR
> > > > > > > >> > Variable_name = 'character_set' OR Variable_name =
> > > > > > > >> >
> > > > > > > >> > 'character_set_server'
> > > > > > > >> >
> > > > > > > >> > OR Variable_name = 'tx_isolation' OR Variable_name =
> > > > > > > >> > 'transaction_isolation' OR Variable_name =
> > > > 'character_set_results'
> > > > > > OR
> > > > > > > >> > Variable_name = 'timezone' OR Variable_name = 'time_zone'
> OR
> > > > > > > >> >
> > > > > > > >> > Variable_name
> > > > > > > >> >
> > > > > > > >> > = 'system_time_zone' OR Variable_name =
> > > 'lower_case_table_names'
> > > > > OR
> > > > > > > >> > Variable_name = 'max_allowed_packet' OR Variable_name =
> > > > > > > >> >
> > > > > > > >> > 'net_buffer_length'
> > > > > > > >> >
> > > > > > > >> > OR Variable_name = 'sql_mode' OR Variable_name =
> > > > > 'query_cache_type'
> > > > > > OR
> > > > > > > >> > Variable_name = 'query_cache_size' OR Variable_name =
> > > > > 'init_connect'
> > > > > > > >> >              8 Query     /*
> > > > mysql-connector-java-5.1.17-SNAPSHOT (
> > > > > > > >> > Revision: ${bzr.revision-id} ) */SELECT
> > > > > > > >> >
> > > > > > > >> > @@session.auto_increment_increment
> > > > > > > >> >
> > > > > > > >> >              8 Query     SHOW COLLATION
> > > > > > > >> >              8 Query     SET NAMES latin1
> > > > > > > >> >              8 Query     SET character_set_results = NULL
> > > > > > > >> >              8 Query     SET autocommit=1
> > > > > > > >> >              8 Query     SET
> sql_mode='STRICT_TRANS_TABLES'
> > > > > > > >> >              8 Query     SET autocommit=0
> > > > > > > >> >              8 Query     SET autocommit=0
> > > > > > > >> >              8 Query     SELECT offering_bundle.ID,
> > > > > > > >> > offering_bundle.AmazonEC2Offering,
> > > > > > offering_bundle.CloudStackOffering
> > > > > > > >> >
> > > > > > > >> > FROM
> > > > > > > >> >
> > > > > > > >> > offering_bundle
> > > > > > > >> >              8 Query     commit
> > > > > > > >> >              8 Query     rollback
> > > > > > > >> >              8 Query     SET autocommit=1
> > > > > > > >> >              8 Query     SET autocommit=0
> > > > > > > >> >              8 Query     SET autocommit=0
> > > > > > > >> >              8 Query     SELECT mhost.ID, mhost.MHostKey,
> > > > > > > >> > mhost.Host, mhost.Version, mhost.LastHeartbeatTime FROM
> > mhost
> > > > > WHERE
> > > > > > > >> > mhost.MHostKey = _binary'00:50:56:92:04:18'  ORDER BY
> RAND()
> > > > > LIMIT 1
> > > > > > > >> >              8 Query     commit
> > > > > > > >> >              8 Query     rollback
> > > > > > > >> >              8 Query     SET autocommit=1
> > > > > > > >> >              8 Query     SET autocommit=0
> > > > > > > >> >              8 Query     SET autocommit=0
> > > > > > > >> >              8 Query     UPDATE mhost SET
> > mhost.Host=_binary'
> > > > > > > >> > http://localhost:7080/awsapi' WHERE mhost.ID = 1
> > > > > > > >> >              8 Query     commit
> > > > > > > >> >              8 Query     rollback
> > > > > > > >> >              8 Query     SET autocommit=1
> > > > > > > >> >              8 Query     SET autocommit=0
> > > > > > > >> >              8 Query     SET autocommit=0
> > > > > > > >> >              8 Query     SELECT shost.ID, shost.Host,
> > > > > > > >> > shost.HostType, shost.MHostID, shost.ExportRoot,
> > > > shost.UserOnHost,
> > > > > > > >> > shost.UserPassword FROM shost WHERE shost.MHostID = 1  AND
> > > > > > > >> >
> > > > > > > >> > shost.ExportRoot
> > > > > > > >> >
> > > > > > > >> > = _binary'/Users/john1/S3-Mount'  ORDER BY RAND() LIMIT 1
> > > > > > > >> >              8 Query     commit
> > > > > > > >> >              8 Query     rollback
> > > > > > > >> >              8 Query     SET autocommit=1
> > > > > > > >> >              9 Connect   cloud@db01.cvg.cs.zimcom.net
> > <mailto:
> > > > > > > >> > cloud@db01.cvg.cs.zimcom.net> on cloud
> > > > > > > >> >              9 Query     /*
> > > > mysql-connector-java-5.1.17-SNAPSHOT (
> > > > > > > >> > Revision: ${bzr.revision-id} ) */SHOW VARIABLES WHERE
> > > > > Variable_name
> > > > > > > >> > ='language' OR Variable_name = 'net_write_timeout' OR
> > > > > Variable_name
> > > > > > =
> > > > > > > >> > 'interactive_timeout' OR Variable_name = 'wait_timeout' OR
> > > > > > > >> >
> > > > > > > >> > Variable_name =
> > > > > > > >> >
> > > > > > > >> > 'character_set_client' OR Variable_name =
> > > > > 'character_set_connection'
> > > > > > > OR
> > > > > > > >> > Variable_name = 'character_set' OR Variable_name =
> > > > > > > >> >
> > > > > > > >> > 'character_set_server'
> > > > > > > >> >
> > > > > > > >> > OR Variable_name = 'tx_isolation' OR Variable_name =
> > > > > > > >> > 'transaction_isolation' OR Variable_name =
> > > > 'character_set_results'
> > > > > > OR
> > > > > > > >> > Variable_name = 'timezone' OR Variable_name = 'time_zone'
> OR
> > > > > > > >> >
> > > > > > > >> > Variable_name
> > > > > > > >> >
> > > > > > > >> > = 'system_time_zone' OR Variable_name =
> > > 'lower_case_table_names'
> > > > > OR
> > > > > > > >> > Variable_name = 'max_allowed_packet' OR Variable_name =
> > > > > > > >> >
> > > > > > > >> > 'net_buffer_length'
> > > > > > > >> >
> > > > > > > >> > OR Variable_name = 'sql_mode' OR Variable_name =
> > > > > 'query_cache_type'
> > > > > > OR
> > > > > > > >> > Variable_name = 'query_cache_size' OR Variable_name =
> > > > > 'init_connect'
> > > > > > > >> >              9 Query     /*
> > > > mysql-connector-java-5.1.17-SNAPSHOT (
> > > > > > > >> > Revision: ${bzr.revision-id} ) */SELECT
> > > > > > > >> >
> > > > > > > >> > @@session.auto_increment_increment
> > > > > > > >> >
> > > > > > > >> >              9 Query     SHOW COLLATION
> > > > > > > >> >              9 Query     SET NAMES latin1
> > > > > > > >> >              9 Query     SET character_set_results = NULL
> > > > > > > >> >              9 Query     SET autocommit=1
> > > > > > > >> >              9 Query     SET
> sql_mode='STRICT_TRANS_TABLES'
> > > > > > > >> >              9 Query     SET autocommit=0
> > > > > > > >> >              9 Query     SELECT @@session.tx_isolation
> > > > > > > >> >              9 Query     SET SESSION TRANSACTION ISOLATION
> > > LEVEL
> > > > > > > >> > READ COMMITTED
> > > > > > > >> >              9 Query     SELECT 1
> > > > > > > >> >              9 Query     SET autocommit=0
> > > > > > > >> >              9 Query     SELECT configuration.name,
> > > > > > > >> > configuration.value FROM configuration WHERE
> > > configuration.name
> > > > =
> > > > > > > >> > _binary'enable.ec2.api'  ORDER BY RAND() LIMIT 1
> > > > > > > >> >              9 Query     commit
> > > > > > > >> >              9 Query     rollback
> > > > > > > >> >              9 Query     SET autocommit=1
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > Steve Searles
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > -----Original Message-----
> > > > > > > >> > From: Steve Searles [mailto:ssearles@zimcom.net <
> > > > > > ssearles@zimcom.net
> > > > > > > >> > <mailto:ssearles@zimcom.net>> <
> > > > > > > >> > ssearles@zimcom.net<mailto:ssearles@zimcom.net>>]
> > > > > > > >> > Sent: Monday, June 9, 2014 10:19 PM
> > > > > > > >> > To: users@cloudstack.apache.org<mailto:
> > > > > users@cloudstack.apache.org>
> > > > > > > >> > Subject: RE: Management Server Startup Issue
> > > > > > > >> >
> > > > > > > >> > Rafael, Thank you for responding so quickly.  Yes 4.3.0. I
> > am
> > > > > using
> > > > > > a
> > > > > > > >> > dedicated DB servers and dedicated management servers.  I
> > will
> > > > > > provide
> > > > > > > >> > whatever log files you would find useful. It would be odd
> if
> > > it
> > > > > was
> > > > > > a
> > > > > > > >> > dependancy since CS really has no visibility to the sql
> > server
> > > > > other
> > > > > > > >> >
> > > > > > > >> > than a
> > > > > > > >> >
> > > > > > > >> > connect string? Or so I thought; and the system is running
> > > fine
> > > > > with
> > > > > > > the
> > > > > > > >> > binlog off.  With the bin log on I can run queries from
> the
> > > > > command
> > > > > > > line
> > > > > > > >> > and everything seems normal from the sql end,  but the
> > > > management
> > > > > > > server
> > > > > > > >> > will not run.  Toggle binlog off CS works fine.  The
> > > replication
> > > > > > slave
> > > > > > > >> >
> > > > > > > >> > has
> > > > > > > >> >
> > > > > > > >> > not even been started yet.  Let me know what other info I
> > can
> > > > send
> > > > > > > over
> > > > > > > >> > that would be of help.  There is nothing in the catalina
> > logs
> > > of
> > > > > > > >> >
> > > > > > > >> > interest
> > > > > > > >> >
> > > > > > > >> > and my logging level is currently set to trace for the
> > > > > > > >> > management-server.log.  I will turn on the sql query log
> and
> > > > > compare
> > > > > > > >> >
> > > > > > > >> > them
> > > > > > > >> >
> > > > > > > >> > between startups of the management nodes perhaps that will
> > > yeild
> > > > > > > >> >
> > > > > > > >> > something.
> > > > > > > >> >
> > > > > > > >> > I have included a dropbox link with the management server
> > log
> > > > > from a
> > > > > > > >> > normal startup of CS and one with mysql binlog enabled.
>  Any
> > > > help
> > > > > > > would
> > > > > > > >> >
> > > > > > > >> > be
> > > > > > > >> >
> > > > > > > >> > greatly appriciated.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://dl.dropboxusercontent.com/u/51895629/management-server.logs.tar.gz
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > Steve Searles
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > -----Original Message-----
> > > > > > > >> > From: Rafael Weingartner [mailto:
> > rafaelweingartner@gmail.com]
> > > > > > > >> > Sent: Monday, June 9, 2014 6:57 PM
> > > > > > > >> > To: users@cloudstack.apache.org
> > > > > > > >> > Subject: Re: Management Server Startup Issue
> > > > > > > >> >
> > > > > > > >> > please give more logs.
> > > > > > > >> > It is the 4.3.0 right? I bet there is a problem with a
> > missing
> > > > > > > >> >
> > > > > > > >> > dependency.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Mon, Jun 9, 2014 at 7:32 PM, Steve Searles <
> > > > > ssearles@zimcom.net>
> > > > > > > >> >
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > Anyone have any idea why the management service would
> refuse
> > > to
> > > > > > start
> > > > > > > >> > if the DB server is configured as a master for
> replication?
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > log_bin=mysql-bin
> > > > > > > >> > server_id=1
> > > > > > > >> >
> > > > > > > >> > Adding the above the my.cnf file of my DB server,
> restarting
> > > > mysql
> > > > > > > >> > then restarting cloudstack-managment will cause cloudstack
> > to
> > > > get
> > > > > > hung
> > > > > > > >> > on startup here.
> > > > > > > >> >
> > > > > > > >> > 2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext]
> > > > > (main:null)
> > > > > > > >> > Starting
> > > > > > > >> >
> > > > > >
> > > com.cloud.bridge.persist.dao.SHostDaoImpl_EnhancerByCloudStack_9da5499
> > > > > > > >> > b
> > > > > > > >> > 2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext]
> > > > > (main:null)
> > > > > > > >> > Starting
> > > > > > > >> >
> > > > > >
> > > com.cloud.bridge.persist.dao.SObjectDaoImpl_EnhancerByCloudStack_847fb
> > > > > > > >> > 98c
> > > > > > > >> > 2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext]
> > > > > (main:null)
> > > > > > > >> > Starting
> > > > > > > >> >
> > > > > >
> > > com.cloud.bridge.persist.dao.CloudStackUserDaoImpl_EnhancerByCloudStac
> > > > > > > >> > k_7fce3ee6
> > > > > > > >> > 2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext]
> > > > > (main:null)
> > > > > > > >> > Starting
> > > > > > > >> >
> > > > > >
> > > com.cloud.bridge.service.core.ec2.EC2Engine_EnhancerByCloudStack_d70c9
> > > > > > > >> > e3c
> > > > > > > >> > 2014-06-09 18:25:33,769 INFO  [c.c.u.c.ComponentContext]
> > > > > (main:null)
> > > > > > > >> > Starting
> > > > > > > >> >
> > > > > >
> > > com.cloud.bridge.service.controller.s3.ServiceProvider_EnhancerByCloud
> > > > > > > >> > Stack_23d38b1
> > > > > > > >> >
> > > > > > > >> > That is the last few lines of the management-server.log
> and
> > it
> > > > > will
> > > > > > > >> > hang there indefinitely.
> > > > > > > >> >
> > > > > > > >> > I noticed this on startup as well.
> > > > > > > >> >
> > > > > > > >> > 2014-06-09 18:25:16,841 INFO  [c.c.u.d.T.Transaction]
> > > > (main:null)
> > > > > Is
> > > > > > > >> > Data Base High Availiability enabled? Ans : false
> > > > > > > >> > 2014-06-09 18:25:17,164 DEBUG
> [c.c.u.d.ConnectionConcierge]
> > > > > > > >> > (main:null) Registering a database connection for
> > LockMaster1
> > > > > > > >> > 2014-06-09 18:25:17,164 INFO  [c.c.u.d.Merovingian2]
> > > (main:null)
> > > > > > > >> > Cleaning up locks for 345049793560
> > > > > > > >> > 2014-06-09 18:25:17,175 INFO  [c.c.u.d.Merovingian2]
> > > (main:null)
> > > > > > > >> > Released
> > > > > > > >> > 0 locks for 345049793560
> > > > > > > >> > 2014-06-09 18:25:17,206 INFO
> > > > > >  [o.a.c.s.l.CloudStackExtendedLifeCycle]
> > > > > > > >> > (main:null) Running system integrity checker
> > > > > > > >> > com.cloud.upgrade.DatabaseUpgradeChecker@63c8e4ae
> > > > > > > >> > 2014-06-09 18:25:17,207 INFO
>  [c.c.u.DatabaseUpgradeChecker]
> > > > > > > >> > (main:null) Grabbing lock to check for database upgrade.
> > > > > > > >> > 2014-06-09 18:25:17,254 DEBUG [c.c.u.d.VersionDaoImpl]
> > > > (main:null)
> > > > > > > >> > Checking to see if the database is at a version before it
> > was
> > > > the
> > > > > > > >> > version table is created
> > > > > > > >> > 2014-06-09 18:25:17,295 INFO
>  [c.c.u.DatabaseUpgradeChecker]
> > > > > > > >> > (main:null) DB version = 4.3.0 Code Version = 4.3.0
> > > > > > > >> > 2014-06-09 18:25:17,296 INFO
>  [c.c.u.DatabaseUpgradeChecker]
> > > > > > > >> > (main:null) DB version and code version matches so no
> > upgrade
> > > > > > needed.
> > > > > > > >> >
> > > > > > > >> > The management server talking to the DB server at least
> > enough
> > > > to
> > > > > > > >> > check the version; and mysql from the command line on the
> > > > > management
> > > > > > > >> > server works fine as well.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > This line caught my attention.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > 2014-06-09 18:25:16,841 INFO  [c.c.u.d.T.Transaction]
> > > > (main:null)
> > > > > Is
> > > > > > > >> > Data Base High Availiability enabled? Ans : false
> > > > > > > >> >
> > > > > > > >> > Is there something I need to set when using mysql
> > replication
> > > > > > specific
> > > > > > > >> > to cloudstack? I do not see anything in the documentation
> on
> > > > this
> > > > > > > >> > other than setting up basic replication.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > if I comment out the above two lines in my.cnf and restart
> > the
> > > > > > > >> > management server it fires up just fine.  I don’t get how
> > > > enabling
> > > > > > bin
> > > > > > > >> > log would make a difference at all.
> > > > > > > >> >
> > > > > > > >> > Any help or explication for this strange issue would be
> > > > > appreciated.
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > - Steve Searles
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > --
> > > > > > > >> > Rafael Weingärtner
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > --
> > > > > > > >> > Rafael Weingärtner
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > --
> > > > > > > >> > Rafael Weingärtner
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > --
> > > > > > > >> > Rafael Weingärtner
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > --
> > > > > > > >> > Rafael Weingärtner
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > --
> > > > > > > >> > Rafael Weingärtner
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> --
> > > > > > > >> Rafael Weingärtner
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Rafael Weingärtner
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Rafael Weingärtner
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Rafael Weingärtner
> > >
> >
>
>
>
> --
> Rafael Weingärtner
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message