incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohit Yadav (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CLOUDSTACK-100) capacityChecker throws exception in specific scenario
Date Tue, 18 Sep 2012 07:13:07 GMT

    [ https://issues.apache.org/jira/browse/CLOUDSTACK-100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13457655#comment-13457655
] 

Rohit Yadav commented on CLOUDSTACK-100:
----------------------------------------

@Sadhu: The bugs (you mentioned) suggest that the MySQL server is unable to serve under such
load (capacity checking queries are intensive)
http://bugs.cloudstack.org/browse/CS-16309
http://bugs.cloudstack.org/browse/CS-10501

Due to inadequate memory, my hypothesis is that the sql server process has many page error,
swap ins/outs, hence slow response from the mysql-connector. Same goes for mgmt server. If
the minimum requirement are not met for a process, it is bound to fail. (for example, trying
to run Battlefield3 on 256MB RAM and saying that it does not work hence it's a bug is ridiculous
:)

So, this bug ticket is invalid. I'm closing the ticket.

But just to be sure, let's create a new test environment and reopen or file a new bug if this
fails;
1. Find out the minimum memory requirement for MySQL server and create a VM with that memory
and host mgmt server and mysql on different VMs.
2. Perform the operations as in this bug.
3. If the bug is caught, perform it at least couple of times. If the bug is found consistently,
then we've a problem.
                
> capacityChecker throws exception in specific scenario
> -----------------------------------------------------
>
>                 Key: CLOUDSTACK-100
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-100
>             Project: CloudStack
>          Issue Type: Bug
>          Components: Management Server
>    Affects Versions: pre-4.0.0
>         Environment: Git Revision: 03df2fa9dd45c938f72cd1866044b09d1b0cc978
> Git URL: https://git-wip-us.apache.org/repos/asf/incubator-cloudstack.gi
>            Reporter: sadhu suresh
>            Assignee: Rohit Yadav
>             Fix For: pre-4.0.0
>
>         Attachments: management-server.log.2012-09-12.gz
>
>
> Runtime exception noticed during zone creation on a setup where  Management server installed
on a VM with less memory resources
> Steps:
> 1.Install new build on rhel6.3 OS
> 2.make sure your Ms has less memory capacity(i.e In the current environment Management
server heartbeat takes too long to finish)
> 3.try to create advance zone with xen as hypervior and observe the behavior
> Actual result:
> Noticed exception in the log while its adding primary storage during zone creation operation
and it took long time to finish addZone operation due to  communication took long time to
 finish between host and Management server.
> Capacity Checker throws runtime exception caused by some sql exception...and exception
says "you have error in SQL syntax"
> *********************
> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error
in your SQL syntax; check the manual that corresponds to your MySQL server version for the
right syntax to use near 'B' at line 1
> **********************
> Content of management log:
> 2012-09-12 16:34:44,074 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null)
Management server heartbeat takes too long to finish. profiler: Done. Duration: 8837ms, profilerHeartbeatUpdate:
Done. Duration: 3455ms, profilerPeerScan: Done. Duration: 5382ms, profilerAgentLB: Done. Duration:
0ms
> 2012-09-12 16:34:49,824 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null)
Peer scan takes too long to finish. profiler: Done. Duration: 4631ms, profilerQueryActiveList:
Done. Duration: 2637ms, profilerSyncClusterInfo: Done. Duration: 1994ms, profilerInvalidatedNodeList:
Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms
> 2012-09-12 16:34:52,543 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null)
Management server heartbeat takes too long to finish. profiler: Done. Duration: 4699ms, profilerHeartbeatUpdate:
Done. Duration: 68ms, profilerPeerScan: Done. Duration: 4631ms, profilerAgentLB: Done. Duration:
0ms
> 2012-09-12 16:34:54,854 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null)
Found 0 routers.
> 2012-09-12 16:34:56,316 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null)
Management server heartbeat takes too long to finish. profiler: Done. Duration: 2112ms, profilerHeartbeatUpdate:
Done. Duration: 704ms, profilerPeerScan: Done. Duration: 1408ms, profilerAgentLB: Done. Duration:
0ms
> 2012-09-12 16:35:01,183 INFO  [agent.manager.AgentMonitor] (AgentMonitor:null) Found
the following agents behind on ping: [1]
> 2012-09-12 16:35:01,276 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null)
Peer scan takes too long to finish. profiler: Done. Duration: 4690ms, profilerQueryActiveList:
Done. Duration: 4394ms, profilerSyncClusterInfo: Done. Duration: 296ms, profilerInvalidatedNodeList:
Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms
> 2012-09-12 16:35:01,634 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null)
Management server heartbeat takes too long to finish. profiler: Done. Duration: 5318ms, profilerHeartbeatUpdate:
Done. Duration: 265ms, profilerPeerScan: Done. Duration: 5053ms, profilerAgentLB: Done. Duration:
0ms
> 2012-09-12 16:35:01,874 ERROR [cloud.alert.AlertManagerImpl] (CapacityChecker:null) Exception
in CapacityChecker
> com.cloud.utils.exception.CloudRuntimeException: DB Exception on: org.apache.commons.dbcp.DelegatingPreparedStatement@a9ee006
>         at com.cloud.utils.db.GenericDaoBase.customSearchIncludingRemoved(GenericDaoBase.java:479)
>         at com.cloud.utils.db.DatabaseCallback.intercept(DatabaseCallback.java:34)
>         at com.cloud.capacity.dao.CapacityDaoImpl.findCapacityBy(CapacityDaoImpl.java:355)
>         at com.cloud.utils.db.DatabaseCallback.intercept(DatabaseCallback.java:34)
>         at com.cloud.alert.AlertManagerImpl.checkForAlerts(AlertManagerImpl.java:473)
>         at com.cloud.alert.AlertManagerImpl$CapacityChecker.run(AlertManagerImpl.java:442)
>         at java.util.TimerThread.mainLoop(Timer.java:534)
>         at java.util.TimerThread.run(Timer.java:484)
> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error
in your SQL syntax; check the manual that corresponds to your MySQL server version for the
right syntax to use near 'B' at line 1
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>         at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>         at com.mysql.jdbc.Util.getInstance(Util.java:386)
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1053)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
>         at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
>         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
>         at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
>         at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2155)
>         at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2318)
>         at org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:97)
>         at com.cloud.utils.db.GenericDaoBase.customSearchIncludingRemoved(GenericDaoBase.java:459)
>         ... 15 more
> 2012-09-12 16:35:33,894 DEBUG [cloud.alert.AlertManagerImpl] (CapacityChecker:null) Running
Capacity Checker ...
> 2012-09-12 16:35:31,364 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null)
Found 0 routers.
> 2012-09-12 16:35:31,191 DEBUG [cloud.server.StatsCollector] (StatsCollector-3:null) StorageCollector
is running...
> 2012-09-12 16:35:31,191 DEBUG [cloud.server.StatsCollector] (StatsCollector-2:null) VmStatsCollector
is running...
> 2012-09-12 16:35:15,394 DEBUG [agent.manager.AgentAttache] (AgentTaskPool-1:null) Seq
1-1550843917: Waiting some more time because this is the current command
> 2012-09-12 16:35:13,612 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-9:null)
Ping from 1
> 2012-09-12 16:35:08,496 DEBUG [cloud.host.Status] (AgentMonitor:null) Ping timeout for
host 1, do invstigation
> 2012-09-12 16:35:05,456 DEBUG [cloud.cluster.ClusterManagerImpl] (Cluster-Heartbeat-1:null)
Peer scan takes too long to finish. profiler: Done. Duration: 3620ms, profilerQueryActiveList:
Done. Duration: 188ms, profilerSyncClusterInfo: Done. Duration: 3432ms, profilerInvalidatedNodeList:
Done. Duration: 0ms, profilerRemovedList: Done. Duration: 0ms
> 2012-09-12 16:35:01,875 ERROR [cloud.api.ApiDispatcher] (catalina-exec-13:null) Exception
while executing ListCapacityCmd:
> java.lang.NullPointerException
>         at com.cloud.capacity.dao.CapacityDaoImpl.findCapacityBy(CapacityDaoImpl.java:319)
>         at com.cloud.utils.db.DatabaseCallback.intercept(DatabaseCallback.java:34)
>         at com.cloud.server.ManagementServerImpl.listCapacities(ManagementServerImpl.java:2216)
>         at com.cloud.api.commands.ListCapacityCmd.execute(ListCapacityCmd.java:129)
>         at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138)
>         at com.cloud.api.ApiServer.queueCommand(ApiServer.java:543)
>         at com.cloud.api.ApiServer.handleRequest(ApiServer.java:422)
>         at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:304)
>         at com.cloud.api.ApiServlet.doGet(ApiServlet.java:63)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
>         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
>         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
>         at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
>         at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
>         at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
>         at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>         at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
>         at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
>         at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
>         at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
>         at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:679)
> Expected result:
>  AddZone should be completed with out any errors and exception should be caught and alert
the user with proper information.
> should not get any syntax error exception like  MySQLSyntaxErrorExceptions seen above
logs.
> Note:Not seen the problem when increase the memory on VM  from 512 to 2GB.also did xe-toolstack-restart
... on xen host.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message