cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastien Goasguen <run...@gmail.com>
Subject Re: how to get hosts out of 'avoid' state
Date Fri, 22 Feb 2013 10:53:36 GMT
Just to close this thread, I think I ran into:

https://issues.apache.org/jira/browse/CLOUDSTACK-587

Which was not applied to 4.0 or 4.0.1.

I will check with 4.0.2 and 4.1 when they come out.

On Feb 21, 2013, at 9:10 PM, Sebastien Goasguen <runseb@gmail.com> wrote:

> I am getting to types of warning:
> 
> 2013-02-21 20:58:37,544 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-395:null)
1. The VM s-342-VM is in Starting state.
> 2013-02-21 20:58:37,612 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-395:null)
Created VM 2b3bfdc6-cc92-e618-ee18-29c1ad1e3f59 for s-342-VM
> 2013-02-21 20:58:37,835 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-395:null)
PV args are -- quiet console=hvc0%template=domP%type=secstorage%host=10.0.0.100%port=8250%name=s-342-VM%zone=1%pod=1%guid=s-342-VM%resource=com.cloud.storage.resource.NfsSecondaryStorageResource%instance=SecStorage%sslcopy=true%role=templateProcessor%mtu=1500%eth2ip=10.0.0.111%eth2mask=255.255.255.0%gateway=10.0.0.1%eth0ip=169.254.0.91%eth0mask=255.255.0.0%eth1ip=10.0.0.97%eth1mask=255.255.255.0%mgmtcidr=10.0.0.0/24%localgw=10.0.0.1%private.network.device=eth1%eth3ip=10.0.0.92%eth3mask=255.255.255.0%storageip=10.0.0.92%storagenetmask=255.255.255.0%storagegateway=10.0.0.1%internaldns1=8.8.8.8%dns1=8.8.8.8
> 2013-02-21 20:58:37,864 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-395:null)
VBD aef11fc5-e2af-a414-f3d9-de743890a784 created for Vol[342|ROOT|ac2fee6f-3a9e-4edc-9a13-9833a895b54e|2147483648]
> 2013-02-21 20:58:37,864 WARN  [xen.resource.CitrixResourceBase] (DirectAgent-395:null)
Catch Exception: class com.cloud.utils.exception.CloudRuntimeException due to com.cloud.utils.exception.CloudRuntimeException:
can not find local sr
> com.cloud.utils.exception.CloudRuntimeException: can not find local sr
> 	at com.cloud.hypervisor.xen.resource.XcpOssResource.createPatchVbd(XcpOssResource.java:109)
> 	at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1285)
> 	at com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:142)
> 	at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:497)
> 	at com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:136)
> 	at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:191)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:679)
> 2013-02-21 20:58:37,866 WARN  [xen.resource.CitrixResourceBase] (DirectAgent-395:null)
Unable to start s-342-VM due to 
> com.cloud.utils.exception.CloudRuntimeException: can not find local sr
> 	at com.cloud.hypervisor.xen.resource.XcpOssResource.createPatchVbd(XcpOssResource.java:109)
> 	at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1285)
> 	at com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:142)
> 	at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:497)
> 	at com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:136)
> 	at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:191)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:679)
> 2013-02-21 20:58:37,900 WARN  [xen.resource.CitrixResourceBase] (DirectAgent-395:null)
Unable to clean up VBD due to 
> You gave an invalid object reference.  The object may have recently been deleted.  The
class parameter gives the type of reference given, and the handle parameter echoes the bad
value given.
> 	at com.xensource.xenapi.Types.checkResponse(Types.java:207)
> 	at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
> 	at com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909)
> 	at com.xensource.xenapi.VBD.unplug(VBD.java:1054)
> 	at com.cloud.hypervisor.xen.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:1092)
> 	at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1353)
> 	at com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:142)
> 	at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:497)
> 	at com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:136)
> 	at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:191)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:679)
> 
> Silly question, but do you need to actually mount the nfs share or is just creating the
sr with xe enough ?
> 
> And this :
> 
> 2013-02-21 20:58:41,989 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-181:null)
1. The VM v-2-VM is in Starting state.
> 2013-02-21 20:58:42,344 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-181:null)
Created VM 93db21ef-604d-990c-27e1-712b39d97a18 for v-2-VM
> 2013-02-21 20:58:42,395 WARN  [xen.resource.CitrixResourceBase] (DirectAgent-181:null)
Catch Exception: class com.xensource.xenapi.Types$XenAPIException due to MEMORY_CONSTRAINT_VIOLATIONMemory
limits must satisfy: static_min ≤ dynamic_min ≤ dynamic_max ≤ static_max
> MEMORY_CONSTRAINT_VIOLATIONMemory limits must satisfy: static_min ≤ dynamic_min ≤
dynamic_max ≤ static_max
> 	at com.xensource.xenapi.Types.checkResponse(Types.java:1732)
> 	at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
> 	at com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909)
> 	at com.xensource.xenapi.VM.setMemoryStaticMin(VM.java:3450)
> 	at com.cloud.hypervisor.xen.resource.CitrixResourceBase.setMemory(CitrixResourceBase.java:3116)
> 	at com.cloud.hypervisor.xen.resource.CitrixResourceBase.createVmFromTemplate(CitrixResourceBase.java:985)
> 	at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1278)
> 	at com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:142)
> 	at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:497)
> 	at com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:136)
> 	at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:191)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:679)
> 2013-02-21 20:58:42,397 WARN  [xen.resource.CitrixResourceBase] (DirectAgent-181:null)
Unable to start v-2-VM due to 
> MEMORY_CONSTRAINT_VIOLATIONMemory limits must satisfy: static_min ≤ dynamic_min ≤
dynamic_max ≤ static_max
> 	at com.xensource.xenapi.Types.checkResponse(Types.java:1732)
> 	at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
> 	at com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909)
> 	at com.xensource.xenapi.VM.setMemoryStaticMin(VM.java:3450)
> 	at com.cloud.hypervisor.xen.resource.CitrixResourceBase.setMemory(CitrixResourceBase.java:3116)
> 	at com.cloud.hypervisor.xen.resource.CitrixResourceBase.createVmFromTemplate(CitrixResourceBase.java:985)
> 	at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1278)
> 	at com.cloud.hypervisor.xen.resource.XcpOssResource.execute(XcpOssResource.java:142)
> 	at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:497)
> 	at com.cloud.hypervisor.xen.resource.XcpOssResource.executeRequest(XcpOssResource.java:136)
> 	at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:191)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> 	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:679)
> 2013-02-21 20:58:42,398 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-181:null)
The VM is in stopped state, detected problem during startup : v-2-VM
> 2013-02-21 20:58:42,625 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-181:null)
Seq 1-603786545: Cancelling because one of the answers is false and it is stop on error.
> 2013-02-21 20:58:42,625 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-181:null)
Seq 1-603786545: Response Received: 
> 
> I don't know how to affect the memory with xen,
> 
> thoughts ?
> 
> -sebastien
> 
> On Feb 21, 2013, at 7:14 PM, Clayton Weise <cweise@iswest.net> wrote:
> 
>> Also, the avoid set is flushed when you restart the management service.  So it might
be easier to start tailing a log, restart the management service, and watch it fail and stick
the host in the avoid set.  Then you can more easily see how it got there in the first place.
>> 
>> -----Original Message-----
>> From: Nitin Mehta [mailto:Nitin.Mehta@citrix.com] 
>> Sent: Thursday, February 21, 2013 7:31 AM
>> To: cloudstack-users@incubator.apache.org
>> Subject: Re: how to get hosts out of 'avoid' state
>> 
>> I think the logs aren't complete. There must be logs indicating that the
>> host is put in avoid set. Can you paste logs before this as well
>> Also the avoid set gets reset after the vm deployment operation (which
>> consists of certain tries.)
>> 
>> On 21/02/13 7:55 PM, "Sebastien Goasguen" <runseb@gmail.com> wrote:
>> 
>>> Hi,
>>> 
>>> I got a basic zone with xen hosts. running 4.0.1.
>>> 
>>> My hosts are stuck in 'avoid' state according to the logs at:
>>> http://pastebin.com/Vb1tg6Gq
>>> 
>>> And therefore the systemvms don't start.
>>> 
>>> I tried to restart the mgt server but that did not do anything.
>>> 
>>> -sebastien
>> 
> 


Mime
View raw message