cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Animesh Chaturvedi (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CLOUDSTACK-4371) [Performance Testing] Basic zone with 20K Hosts, management server restart leaves the hosts in disconnected state for very long time
Date Thu, 07 Nov 2013 22:28:03 GMT

     [ https://issues.apache.org/jira/browse/CLOUDSTACK-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Animesh Chaturvedi updated CLOUDSTACK-4371:
-------------------------------------------


BULK EDIT> These issues are open to be picked up. Help in resolution is appreciated.

> [Performance Testing] Basic zone with 20K Hosts, management server restart leaves the
hosts in disconnected state for very long time
> ------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: CLOUDSTACK-4371
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4371
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: Management Server
>    Affects Versions: 4.2.0
>         Environment: Basic zone, with over 20K simulator hosts
>            Reporter: Sowmya Krishnan
>              Labels: performance
>             Fix For: 4.3.0
>
>         Attachments: agenttaskpool_334.log, ms1_restartfail.log.gz, ms2_restartfail.log.gz,
ms3_restartfail.log.gz
>
>
> Basic zone performance test bed:
> 20K simulator hosts,
> 3 Management servers
> 1 host/cluster
> Local storage
> Java heap size: 12GB
> db.cloud.maxActive=2000
> direct.agent.load.size=1000
> agent.lb.enabled=true
> Deploy around 20K simulator hosts with 3 Management servers clustered
> (Not deployed any VMs yet)
> After all hosts are deployed, stop all 3 Management servers and then start all 3 one
after another
> Result
> =====
> Hosts don't get to connected state at all even after 10 minutes. While around 2K of them
go into alert state while rest are in disconnected state.
> mysql> select count(*), status, resource_state, type, mgmt_server_id from host group
by mgmt_server_id, status, type, resource_state;
> +----------+--------------+----------------+--------------------+----------------+
> | count(*) | status       | resource_state | type               | mgmt_server_id |
> +----------+--------------+----------------+--------------------+----------------+
> |     1946 | Alert        | Enabled        | Routing            |           NULL |
> |    18054 | Disconnected | Enabled        | Routing            |           NULL |
> |        1 | Disconnected | Enabled        | SecondaryStorageVM |           NULL |
> +----------+--------------+----------------+--------------------+----------------+
> 3 rows in set (0.11 sec)
> MS Logs show lot of storage pool exceptions while hosts try to get connected:
> 2013-08-16 05:49:25,592 DEBUG [agent.transport.Request] (AgentTaskPool-12:null) Seq 13-32440322:
Sending  { Cmd , MgmtId: 206915885094132, via: 13, Ver: v1, Flags: 100011, [{"com.cloud.agen
> t.api.CleanupNetworkRulesCmd":{"interval":2028,"wait":0}}] }
> 2013-08-16 05:49:25,592 DEBUG [agent.transport.Request] (AgentTaskPool-12:null) Seq 13-32440322:
Executing:  { Cmd , MgmtId: 206915885094132, via: 13, Ver: v1, Flags: 100011, [{"com.cloud.a
> gent.api.CleanupNetworkRulesCmd":{"interval":2028,"wait":0}}] }
> 2013-08-16 05:49:25,592 DEBUG [xen.discoverer.XcpServerDiscoverer] (AgentTaskPool-14:null)
Not XenServer so moving on.
> 2013-08-16 05:49:25,592 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-14:null)
Sending Connect to listener: DeploymentPlanningManagerImpl_EnhancerByCloudStack_76f3d8e4
> 2013-08-16 05:49:25,591 DEBUG [cloud.resource.AgentResourceBase] (ClusteredAgentManager
Timer:null) Deserializing simulated agent on reconnect
> 2013-08-16 05:49:25,594 INFO  [network.security.SecurityGroupListener] (AgentTaskPool-12:null)
Scheduled network rules cleanup, interval=2028
> 2013-08-16 05:49:25,594 INFO  [network.security.SecurityGroupListener] (AgentTaskPool-12:null)
Received a host startup notification
> 2013-08-16 05:49:25,595 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-12:null)
Sending Connect to listener: StoragePoolMonitor
> ...
> ...
> 2013-08-16 05:49:25,761 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-12:null)
Sending Connect to listener: ClusteredVirtualMachineManagerImpl_EnhancerByCloudStack_b5459b7b
> 2013-08-16 05:49:25,764 DEBUG [cloud.vm.VirtualMachineManagerImpl] (AgentTaskPool-12:null)
Found 0 VMs for host 13
> 2013-08-16 05:49:25,765 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-12:null)
Sending Connect to listener: LocalStoragePoolListener
> 2013-08-16 05:49:25,768 DEBUG [datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl]
(AgentTaskPool-12:null) createPool Params @ scheme - Filesystem storageHost - 172.1.3.131
hostPath - /mnt/2a2463b4-4fd2-4ac7-ad3f-040a3046e478 port - -1
> 2013-08-16 05:49:25,771 DEBUG [datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl]
(AgentTaskPool-12:null) Another active pool with the same uuid already exists
> 2013-08-16 05:49:25,772 WARN  [cloud.storage.StorageManagerImpl] (AgentTaskPool-12:null)
Unable to setup the local storage pool for Host[-13-Routing]
> com.cloud.utils.exception.CloudRuntimeException: Another active pool with the same uuid
already exists
>         at org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.initialize(CloudStackPrimaryDataStoreLifeCycleImpl.java:319)
>         at com.cloud.storage.StorageManagerImpl.createLocalStorage(StorageManagerImpl.java:647)
>         at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
>         at com.cloud.storage.LocalStoragePoolListener.processConnect(LocalStoragePoolListener.java:86)
>         at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:587)
>         at com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1479)
>         at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1739)
>         at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1901)
>         at com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.run(AgentManagerImpl.java:1130)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:679)
> 2013-08-16 05:49:25,773 INFO  [utils.exception.CSExceptionErrorCode] (AgentTaskPool-12:null)
Could not find exception: com.cloud.exception.ConnectionException in error code list for exceptions
> 2013-08-16 05:49:25,773 WARN  [agent.manager.AgentManagerImpl] (AgentTaskPool-12:null)
Monitor LocalStoragePoolListener says there is an error in the connect process for 13 due
to Unable to setup the local storage pool for Host[-13-Routing]
> 2013-08-16 05:49:25,773 INFO  [agent.manager.AgentManagerImpl] (AgentTaskPool-12:null)
Host 13 is disconnecting with event AgentDisconnected



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message