incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hieu Le <hieul...@gmail.com>
Subject Unable to add more XCP host in cluster
Date Tue, 09 Oct 2012 01:58:19 GMT
Hi everyone,

I have some troubles while adding more XCP host in cluster. I have 2
running hosts which are homogeneous and another host that is inhomogeneous.
Now I want to add this inhomogeneous host to CS in the same cluster with 2
running hosts but it is always fail. The step I follow is:

   - Force the inhomogeneous to join existing pool with xe command (xe
   pool-join --force=true --master-address=....)
   - Add new host in the same cluster with 2 existing hosts via CS UI.
   - Getting the error.

The error log I got here is:

2012-10-09 09:19:04,334 DEBUG [cloud.storage.StorageManagerImpl]
(catalina-exec-2:null) Adding pool XenPrimary to  host 14
2012-10-09 09:19:04,336 DEBUG [agent.transport.Request]
(catalina-exec-2:null) Seq 14-976879620: Sending  { Cmd , MgmtId:
148045887322015, via: 14, Ver: v1, Flags: 100011, [{"ModifyStoragePoolC$
2012-10-09 09:19:04,337 DEBUG [agent.transport.Request]
(catalina-exec-2:null) Seq 14-976879620: Executing:  { Cmd , MgmtId:
148045887322015, via: 14, Ver: v1, Flags: 100011, [{"ModifyStoragePo$
2012-10-09 09:19:04,337 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-20:null) Seq 14-976879620: Executing request
2012-10-09 09:19:04,381 WARN  [xen.resource.CitrixResourceBase]
(DirectAgent-20:null) ModifyStoragePoolCommand add XenAPIException:Can not
see storage pool: b43e6f96-d047-3f77-ab5c-5a2abb13a732$
com.cloud.utils.exception.CloudRuntimeException: Can not see storage pool:
b43e6f96-d047-3f77-ab5c-5a2abb13a732 from on
host:1949af01-e15e-467c-8d27-42678a5f826c
        at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.getStorageRepository(CitrixResourceBase.java:6217)
        at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:4617)
        at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:426)
        at
com.cloud.hypervisor.xen.resource.XcpServerResource.executeRequest(XcpServerResource.java:48)
        at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:187)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)
2012-10-09 09:19:04,382 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-20:null) Seq 14-976879620: Response Received:
2012-10-09 09:19:04,382 DEBUG [agent.transport.Request]
(DirectAgent-20:null) Seq 14-976879620: Processing:  { Ans: , MgmtId:
148045887322015, via: 14, Ver: v1, Flags: 10, [{"Answer":{"result":$
2012-10-09 09:19:04,382 WARN  [cloud.vm.VirtualMachineManagerImpl]
(DirectAgent-20:null) Cleanup failed due to ModifyStoragePoolCommand add
XenAPIException:Can not see storage pool: b43e6f96-d0$
2012-10-09 09:19:04,382 DEBUG [agent.transport.Request]
(catalina-exec-2:null) Seq 14-976879620: Received:  { Ans: , MgmtId:
148045887322015, via: 14, Ver: v1, Flags: 10, { Answer } }
2012-10-09 09:19:04,383 WARN  [cloud.vm.VirtualMachineManagerImpl]
(catalina-exec-2:null) Cleanup failed due to ModifyStoragePoolCommand add
XenAPIException:Can not see storage pool: b43e6f96-d$
2012-10-09 09:19:04,383 DEBUG [agent.manager.AgentManagerImpl]
(catalina-exec-2:null) Details from executing class
com.cloud.agent.api.ModifyStoragePoolCommand: ModifyStoragePoolCommand add
Xen$
2012-10-09 09:19:04,461 WARN  [agent.manager.AgentManagerImpl]
(catalina-exec-2:null) Monitor StoragePoolMonitor says there is an error in
the connect process for 14 due to Unable to connect to$
2012-10-09 09:19:04,461 INFO  [agent.manager.AgentManagerImpl]
(catalina-exec-2:null) Host 14 is disconnecting with event AgentDisconnected
2012-10-09 09:19:04,462 DEBUG [agent.manager.AgentManagerImpl]
(catalina-exec-2:null) The next status of agent 14is Alert, current status
is Connecting
2012-10-09 09:19:04,463 DEBUG [agent.manager.AgentManagerImpl]
(catalina-exec-2:null) Deregistering link for 14 with state Alert
2012-10-09 09:19:04,463 DEBUG [agent.manager.AgentManagerImpl]
(catalina-exec-2:null) Remove Agent : 14
2012-10-09 09:19:04,463 DEBUG [agent.manager.DirectAgentAttache]
(catalina-exec-2:null) Processing disconnect 14
2012-10-09 09:19:04,463 DEBUG [agent.manager.AgentManagerImpl]
(catalina-exec-2:null) Sending Disconnect to listener:
com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer$$EnhancerByCGLIB$$4$
2012-10-09 09:19:04,463 DEBUG [agent.manager.AgentManagerImpl]
(catalina-exec-2:null) Sending Disconnect to listener:
com.cloud.hypervisor.vmware.VmwareManagerImpl$$EnhancerByCGLIB$$8d05fd47
2012-10-09 09:19:04,463 DEBUG [agent.manager.AgentManagerImpl]
(catalina-exec-2:null) Sending Disconnect to listener:
com.cloud.vm.ClusteredVirtualMachineManagerImpl$$EnhancerByCGLIB$$4b7c62a0
2012-10-09 09:19:04,463 DEBUG [agent.manager.AgentManagerImpl]
(catalina-exec-2:null) Sending Disconnect to listener:
com.cloud.network.security.SecurityGroupListener
2012-10-09 09:19:04,463 DEBUG [agent.manager.AgentManagerImpl]
(catalina-exec-2:null) Sending Disconnect to listener:
com.cloud.storage.listener.StoragePoolMonitor
2012-10-09 09:19:04,463 DEBUG [agent.manager.AgentManagerImpl]
(catalina-exec-2:null) Sending Disconnect to listener:
com.cloud.storage.secondary.SecondaryStorageListener
2012-10-09 09:19:04,463 DEBUG [agent.manager.AgentManagerImpl]
(catalina-exec-2:null) Sending Disconnect to listener:
com.cloud.network.NetworkManagerImpl$$EnhancerByCGLIB$$f53c41bc
2012-10-09 09:19:04,463 DEBUG [agent.manager.AgentManagerImpl]
(catalina-exec-2:null) Sending Disconnect to listener:
com.cloud.agent.manager.AgentMonitor$$EnhancerByCGLIB$$65dcc41c
2012-10-09 09:19:04,463 DEBUG [agent.manager.AgentManagerImpl]
(catalina-exec-2:null) Sending Disconnect to listener:
com.cloud.storage.download.DownloadListener
2012-10-09 09:19:04,463 DEBUG [agent.manager.AgentManagerImpl]
(catalina-exec-2:null) Sending Disconnect to listener:
com.cloud.storage.upload.UploadListener
2012-10-09 09:19:04,463 DEBUG [agent.manager.AgentManagerImpl]
(catalina-exec-2:null) Sending Disconnect to listener:
com.cloud.network.SshKeysDistriMonitor
2012-10-09 09:19:04,463 DEBUG [agent.manager.AgentManagerImpl]
(catalina-exec-2:null) Sending Disconnect to listener:
com.cloud.network.router.VirtualNetworkApplianceManagerImpl$$EnhancerByCGLI$
2012-10-09 09:19:04,463 DEBUG [agent.manager.AgentManagerImpl]
(catalina-exec-2:null) Sending Disconnect to listener:
com.cloud.capacity.StorageCapacityListener
2012-10-09 09:19:04,463 DEBUG [agent.manager.AgentManagerImpl]
(catalina-exec-2:null) Sending Disconnect to listener:
com.cloud.capacity.ComputeCapacityListener
2012-10-09 09:19:04,463 DEBUG [agent.manager.AgentManagerImpl]
(catalina-exec-2:null) Sending Disconnect to listener:
com.cloud.consoleproxy.ConsoleProxyListener
2012-10-09 09:19:04,464 DEBUG [agent.manager.AgentManagerImpl]
(catalina-exec-2:null) Sending Disconnect to listener:
com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
2012-10-09 09:19:04,464 DEBUG [cloud.network.NetworkUsageManagerImpl]
(catalina-exec-2:null) Disconnected called on 14 with status Alert
2012-10-09 09:19:04,464 DEBUG [cloud.host.Status] (catalina-exec-2:null)
Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host
id = 14, name = node1.viettelrd.vn]
2012-10-09 09:19:04,532 DEBUG [cloud.host.Status] (catalina-exec-2:null)
Agent status update: [id = 14; name = node1.viettelrd.vn; old status =
Connecting; event = AgentDisconnected; new status$
2012-10-09 09:19:04,532 DEBUG [agent.manager.ClusteredAgentManagerImpl]
(catalina-exec-2:null) Notifying other nodes of to disconnect
2012-10-09 09:19:04,534 WARN  [cloud.resource.ResourceManagerImpl]
(catalina-exec-2:null) Unable to connect due to
com.cloud.exception.ConnectionException: Unable to connect to pool
Pool[200|NetworkFilesystem]
        at
com.cloud.storage.listener.StoragePoolMonitor.processConnect(StoragePoolMonitor.java:91)
        at
com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:601)
        at
com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1496)
        at
com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1574)
        at
com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:677)
        at
com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:514)
        at com.cloud.api.commands.AddHostCmd.execute(AddHostCmd.java:136)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:132)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:509)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:416)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:300)
        at com.cloud.api.ApiServlet.doGet(ApiServlet.java:59)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at
org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
        at
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
        at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)
Caused by: com.cloud.exception.StorageUnavailableException: Resource
[StoragePool:200] is unreachable: Unable establish connection from storage
head to storage pool 200 due to ModifyStoragePool$
        at
com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(StorageManagerImpl.java:1567)
        at
com.cloud.storage.listener.StoragePoolMonitor.processConnect(StoragePoolMonitor.java:88)
        ... 28 more
2012-10-09 09:19:04,535 DEBUG [cloud.host.Status] (catalina-exec-2:null)
Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host
id = 14, name = node1.viettelrd.vn]
2012-10-09 09:19:04,535 DEBUG [cloud.host.Status] (catalina-exec-2:null)
Cannot transit agent status with event AgentDisconnected for host 14, name=
node1.viettelrd.vn, mangement server id is 14$
2012-10-09 09:19:04,535 ERROR [cloud.api.ApiDispatcher]
(catalina-exec-2:null) Exception while executing AddHostCmd:
com.cloud.utils.exception.CloudRuntimeException: Cannot transit agent
status with event AgentDisconnected for host 14, mangement server id is
148045887322015,Unable to transition to a new state$
        at
com.cloud.agent.manager.AgentManagerImpl.agentStatusTransitTo(AgentManagerImpl.java:1428)
        at
com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1588)
        at
com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:677)
        at
com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:514)
        at com.cloud.api.commands.AddHostCmd.execute(AddHostCmd.java:136)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:132)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:509)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:416)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:300)
        at com.cloud.api.ApiServlet.doGet(ApiServlet.java:59)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at
org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
        at
org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
        at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)
2012-10-09 09:19:08,143 WARN  [xen.resource.CitrixResourceBase]
(DirectAgent-9:null) ModifyStoragePoolCommand add XenAPIException:Can not
see storage pool: b43e6f96-d047-3f77-ab5c-5a2abb13a732 $
com.cloud.utils.exception.CloudRuntimeException: Can not see storage pool:
b43e6f96-d047-3f77-ab5c-5a2abb13a732 from on
host:1949af01-e15e-467c-8d27-42678a5f826c
        at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.getStorageRepository(CitrixResourceBase.java:6217)
        at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:4617)
        at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:426)
        at
com.cloud.hypervisor.xen.resource.XcpServerResource.executeRequest(XcpServerResource.java:48)
        at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:187)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)

- When force joining the inhomogeneous host, Xen Center throws an error
that it can not see the SR (primary storage) and can not connect to host
via PBD.

Please support !

Best regards !

-- 

------------------------------------

Le Quang Hieu

Specialist – Core Cloud Computing  Dept

Cloud Computing Research Center

Viettel Research and Development Institute

No. 380 Lac Long Quan Str, Tay Ho Dist, Hanoi, Vietnam

Mobile: (84) 974616850

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message