cloudstack-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Animesh Chaturvedi (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CLOUDSTACK-1967) Maintenance mode not fully working with VMWare
Date Tue, 04 Jun 2013 22:33:20 GMT

     [ https://issues.apache.org/jira/browse/CLOUDSTACK-1967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Animesh Chaturvedi updated CLOUDSTACK-1967:
-------------------------------------------

    Affects Version/s: 4.2.0
    
> Maintenance mode not fully working with VMWare
> ----------------------------------------------
>
>                 Key: CLOUDSTACK-1967
>                 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-1967
>             Project: CloudStack
>          Issue Type: Bug
>      Security Level: Public(Anyone can view this level - this is the default.) 
>          Components: VMware
>    Affects Versions: 4.1.0, 4.2.0
>            Reporter: Francois Gaudreault
>         Attachments: managementlog_maint.log
>
>
> When we turn on maintenance mode on one node of the cluster, the VMs are migrating fine,
but the host is never fully sent to maintenance mode on the VMWare side.  We do see bunch
of stack traces related to the management server loosing connectivity, but machine will be
available to ICMP for the entire process. The host will appear as Disconnected. This is not
a critical issue (major tho) since the VMs are migrated properly, but the error should be
investigated/reproduced.
> See the attachments for logs.
> Catalina.out StackTrace:
> INFO  [vmware.resource.VmwareResource] (ClusteredAgentManager Timer:) VmwareResource
network configuration info. private vSwitch: vSwitch0, public vSwitch: vSwitch0, guest network:
vSwitch0
> INFO  [vmware.resource.VmwareResource] (AgentTaskPool-6:) Startup report on host hardware
info. {"memoryBytes":68708646912,"cpuCount":8,"cpuSpeed":2800}
> INFO  [vmware.resource.VmwareResource] (AgentTaskPool-6:) Startup report on host network
info. {"hostIp":"10.0.22.65","hostNetmask":"255.255.255.0","hostMacAddress":"00:26:55:2b:c0:8a"}
> INFO  [vmware.resource.VmwareResource] (AgentTaskPool-6:) Add local storage startup command:
{"totalSize":0,"poolInfo":{"uuid":"4e656219-c3f9-4a7d-93e3-5c4d09e8c59c","host":"yul01vi13.ops.dot","localPath":"","hostPath":"datastore-9206","poolType":"LVM","capacityBytes":67914170368,"availableBytes":66896003072},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"3","pod":"3","cluster":"12","guid":"4e656219-c3f9-4a7d-93e3-5c4d09e8c59c","name":"4e656219-c3f9-4a7d-93e3-5c4d09e8c59c","wait":0}
> INFO  [vmware.resource.VmwareResource] (AgentTaskPool-6:) Add local storage startup command:
{"totalSize":0,"poolInfo":{"uuid":"ddd756cb-8592-4473-9482-5a4b68085a05","host":"yul01vi13.ops.dot","localPath":"","hostPath":"datastore-9221","poolType":"LVM","capacityBytes":733634101248,"availableBytes":732612788224},"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"3","pod":"3","cluster":"12","guid":"ddd756cb-8592-4473-9482-5a4b68085a05","name":"ddd756cb-8592-4473-9482-5a4b68085a05","wait":0}
> INFO  [agent.manager.DirectAgentAttache] (AgentTaskPool-6:) StartupAnswer received 4
Interval = 60
> INFO  [vmware.resource.VmwareResource] (DirectAgent-494:yul01vi13.ops.dot) Executing
resource CheckNetworkCommand {"networkInfoList":[{"physicalNetworkId":206,"privateNetworkName":"vSwitch0"},{"physicalNetworkId":207,"storageNetworkName":"vSwitch1"},{"physicalNetworkId":208,"publicNetworkName":"vSwitch2","guestNetworkName":"vSwitch2"}],"wait":0}
> INFO  [network.security.SecurityGroupListener] (AgentTaskPool-6:) Received a host startup
notification
> INFO  [network.security.SecurityGroupListener] (AgentTaskPool-6:) Scheduled network rules
cleanup, interval=1961
> INFO  [network.security.SecurityGroupListener] (AgentTaskPool-6:) Received a host startup
notification
> INFO  [network.security.SecurityGroupListener] (AgentTaskPool-6:) Received a host startup
notification
> WARN  [agent.manager.AgentManagerImpl] (AgentTaskPool-6:) Resource [Host:4] is unreachable:
Host 4: Unable to send class com.cloud.agent.api.ModifyStoragePoolCommand because agent is
in maintenance mode
> WARN  [storage.listener.StoragePoolMonitor] (AgentTaskPool-6:) Unable to connect host
4 to pool Pool[205|VMFS] due to com.cloud.exception.StorageUnavailableException: Resource
[StoragePool:205] is unreachable: Unable to get an answer to the modify storage pool command
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:205] is unreachable:
Unable to get an answer to the modify storage pool command
>         at com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(StorageManagerImpl.java:1615)
>         at com.cloud.storage.listener.StoragePoolMonitor.processConnect(StoragePoolMonitor.java:94)
>         at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:611)
>         at com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1504)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
>         at com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:45)
>         at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
>         at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
>         at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
>         at sun.proxy.$Proxy109.handleDirectConnectAgent(Unknown Source)
>         at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1937)
>         at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:2074)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
>         at com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:45)
>         at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
>         at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
>         at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
>         at sun.proxy.$Proxy119.createHostAndAgent(Unknown Source)
>         at com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.run(AgentManagerImpl.java:1156)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:679)
> WARN  [agent.manager.AgentManagerImpl] (AgentTaskPool-6:) Resource [Host:4] is unreachable:
Host 4: Unable to send class com.cloud.agent.api.ModifyStoragePoolCommand because agent is
in maintenance mode
> WARN  [storage.listener.StoragePoolMonitor] (AgentTaskPool-6:) Unable to connect host
4 to pool Pool[206|VMFS] due to com.cloud.exception.StorageUnavailableException: Resource
[StoragePool:206] is unreachable: Unable to get an answer to the modify storage pool command
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:206] is unreachable:
Unable to get an answer to the modify storage pool command
>         at com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(StorageManagerImpl.java:1615)
>         at com.cloud.storage.listener.StoragePoolMonitor.processConnect(StoragePoolMonitor.java:94)
>         at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:611)
>         at com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1504)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
>         at com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:45)
>         at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
>         at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
>         at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
>         at sun.proxy.$Proxy109.handleDirectConnectAgent(Unknown Source)
>         at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1937)
>         at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:2074)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>         at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
>         at com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:45)
>         at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
>         at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
>         at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
>         at sun.proxy.$Proxy119.createHostAndAgent(Unknown Source)
>         at com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.run(AgentManagerImpl.java:1156)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         at java.lang.Thread.run(Thread.java:679)
> INFO  [vmware.resource.VmwareResource] (DirectAgent-403:yul01vi13.ops.dot) Executing
resource ReadyCommand: {"dcId":3,"hostId":4,"wait":0}
> INFO  [agent.manager.AgentMonitor] (Thread-5:) Found the following agents behind on ping:
[8, 4]
> INFO  [agent.manager.AgentManagerImpl] (AgentTaskPool-7:) Host 4 is disconnecting with
event ShutdownRequested
> INFO  [agent.manager.AgentManagerImpl] (AgentTaskPool-10:) Investigating why host 8 has
disconnected with event PingTimeout
> INFO  [agent.manager.AgentManagerImpl] (AgentTaskPool-10:) The state determined is Up
> INFO  [agent.manager.AgentManagerImpl] (AgentTaskPool-10:) Agent is determined to be
up and running

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message