Mailing-List: contact cloudstack-users-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cloudstack-users@incubator.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Message-ID: <50F5EF42.501@nfinausa.com>
Date: Tue, 15 Jan 2013 18:07:30 -0600
From: Nik Martin <nik.martin@nfinausa.com>
Organization: nfina Technologies, Inc.
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:16.0) Gecko/20121028 Thunderbird/16.0.2
MIME-Version: 1.0
To: cloudstack-users@incubator.apache.org
Subject: Re: Console Proxy VM unavailable
References: <50F5CAD0.9020303@nfinausa.com> <50F5CCE2.2090800@nfinausa.com>
 <B1DF26ECC0458748AC97CECE2DA98D41012FA33B21D3@SJCPMAILBOX01.citrite.net>
 <50F5D566.6050504@nfinausa.com>
 <B1DF26ECC0458748AC97CECE2DA98D41012FA33B21D5@SJCPMAILBOX01.citrite.net>
 <50F5DAD2.6020202@nfinausa.com>
 <B1DF26ECC0458748AC97CECE2DA98D41012FA33B21D6@SJCPMAILBOX01.citrite.net>
In-Reply-To: 
 <B1DF26ECC0458748AC97CECE2DA98D41012FA33B21D6@SJCPMAILBOX01.citrite.net>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 01/15/2013 04:47 PM, Anthony Xu wrote:
> destroy_domain is a very low level command, it talks to Xen hypervisor directly. The command hangs might mean it is waiting for a lock in hypervisor which is held by other command/xapi.
>
> Try restaringing xapi and destroy_domain.
> xe-toolstack-restart
>
> if it doesn't work, the only solution I know is to reboot the host, you can migrate VMs to other hosts before you reboot this host.
>
>
> Anthony
>
I restarted the toolstack, then tried to migrate a VM poff of it, and 
here is what I got in the management log:

2013-01-15 18:05:12,394 DEBUG [cloud.capacity.CapacityManagerImpl] 
(Job-Executor-12:job-715) CPU STATS after allocation: for host: 6, old 
used: 0, old reserved: 0, actual total: 57456, total with 
overprovisioning: 71820; new used:500, reserved:0; requested 
cpu:500,alloc_from_last:false
2013-01-15 18:05:12,394 DEBUG [cloud.capacity.CapacityManagerImpl] 
(Job-Executor-12:job-715) RAM STATS after allocation: for host: 6, old 
used: 0, old reserved: 0, total: 98962234560; new used: 536870912, 
reserved: 0; requested mem: 536870912,alloc_from_last:false
2013-01-15 18:05:12,399 DEBUG [agent.transport.Request] 
(Job-Executor-12:job-715) Seq 1-1375404813: Sending  { Cmd , MgmtId: 
130577622632, via: 1, Ver: v1, Flags: 100111, 
[{"MigrateCommand":{"vmName":"i-17-67-VM","destIp":"172.16.5.5","hostGuid":"e1deec38-20f5-46e1-aa11-7d18ce7bd97c","isWindows":false,"wait":0}}] 
}
2013-01-15 18:05:12,399 DEBUG [agent.transport.Request] 
(Job-Executor-12:job-715) Seq 1-1375404813: Executing:  { Cmd , MgmtId: 
130577622632, via: 1, Ver: v1, Flags: 100111, 
[{"MigrateCommand":{"vmName":"i-17-67-VM","destIp":"172.16.5.5","hostGuid":"e1deec38-20f5-46e1-aa11-7d18ce7bd97c","isWindows":false,"wait":0}}] 
}
2013-01-15 18:05:12,399 DEBUG [agent.manager.DirectAgentAttache] 
(DirectAgent-483:null) Seq 1-1375404813: Executing request
2013-01-15 18:05:12,567 DEBUG [xen.resource.CitrixResourceBase] 
(DirectAgent-483:null) 5. The VM i-17-67-VM is in Stopping state
2013-01-15 18:05:14,020 DEBUG [agent.manager.DirectAgentAttache] 
(DirectAgent-111:null) Ping from 6
2013-01-15 18:05:14,361 DEBUG [agent.manager.DirectAgentAttache] 
(DirectAgent-145:null) Ping from 3
2013-01-15 18:05:15,923 DEBUG [agent.manager.DirectAgentAttache] 
(DirectAgent-337:null) Ping from 1
2013-01-15 18:05:19,650 DEBUG [agent.manager.DirectAgentAttache] 
(DirectAgent-337:null) Ping from 1
2013-01-15 18:05:23,217 DEBUG [agent.manager.DirectAgentAttache] 
(DirectAgent-337:null) Ping from 1
2013-01-15 18:05:35,161 DEBUG 
[storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 
1 is ready to launch secondary storage VM
2013-01-15 18:05:35,268 DEBUG 
[cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 
1 is ready to launch console proxy
2013-01-15 18:05:35,483 DEBUG 
[network.router.VirtualNetworkApplianceManagerImpl] 
(RouterStatusMonitor-1:null) Found 8 routers.
2013-01-15 18:05:37,448 WARN  [agent.manager.DirectAgentAttache] 
(DirectAgent-97:null) Seq 1-1375404807: Exception Caught while executing 
command
com.cloud.utils.exception.CloudRuntimeException: Unable to reset master 
of slave 172.16.5.3 to 172.16.5.5 due to 
com.cloud.utils.exception.CloudRuntimeException: Unable to reset master 
of slave 172.16.5.3 to 172.16.5.5after 30 retry
	at 
com.cloud.hypervisor.xen.resource.XenServerConnectionPool.PoolEmergencyResetMaster(XenServerConnectionPool.java:439)
	at 
com.cloud.hypervisor.xen.resource.XenServerConnectionPool.connect(XenServerConnectionPool.java:651)
	at 
com.cloud.hypervisor.xen.resource.CitrixResourceBase.getConnection(CitrixResourceBase.java:4872)
	at 
com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1749)
	at 
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:404)
	at 
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:69)
	at 
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:187)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
	at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
	at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:679)
2013-01-15 18:05:37,449 DEBUG [agent.manager.DirectAgentAttache] 
(DirectAgent-97:null) Seq 1-1375404807: Response Received:
2013-01-15 18:05:37,449 DEBUG [agent.transport.Request] 
(DirectAgent-97:null) Seq 1-1375404807: Processing:  { Ans: , MgmtId: 
130577622632, via: 1, Ver: v1, Flags: 10, 
[{"Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException: 
Unable to reset master of slave 172.16.5.3 to 172.16.5.5 due to 
com.cloud.utils.exception.CloudRuntimeException: Unable to reset master 
of slave 172.16.5.3 to 172.16.5.5after 30 retry","wait":0}}] }
2013-01-15 18:05:37,449 WARN  [cloud.vm.VirtualMachineManagerImpl] 
(DirectAgent-97:null) Cleanup failed due to 
com.cloud.utils.exception.CloudRuntimeException: Unable to reset master 
of slave 172.16.5.3 to 172.16.5.5 due to 
com.cloud.utils.exception.CloudRuntimeException: Unable to reset master 
of slave 172.16.5.3 to 172.16.5.5after 30 retry
2013-01-15 18:05:37,449 DEBUG [agent.transport.Request] 
(StatsCollector-3:null) Seq 1-1375404807: Received:  { Ans: , MgmtId: 
130577622632, via: 1, Ver: v1, Flags: 10, { Answer } }
2013-01-15 18:05:37,449 WARN  [cloud.vm.VirtualMachineManagerImpl] 
(StatsCollector-3:null) Cleanup failed due to 
com.cloud.utils.exception.CloudRuntimeException: Unable to reset master 
of slave 172.16.5.3 to 172.16.5.5 due to 
com.cloud.utils.exception.CloudRuntimeException: Unable to reset master 
of slave 172.16.5.3 to 172.16.5.5after 30 retry
2013-01-15 18:05:37,449 DEBUG [agent.manager.AgentManagerImpl] 
(StatsCollector-3:null) Details from executing class 
com.cloud.agent.api.GetHostStatsCommand: 
com.cloud.utils.exception.CloudRuntimeException: Unable to reset master 
of slave 172.16.5.3 to 172.16.5.5 due to 
com.cloud.utils.exception.CloudRuntimeException: Unable to reset master 
of slave 172.16.5.3 to 172.16.5.5after 30 retry


>
>
>> -----Original Message-----
>> From: Nik Martin [mailto:nik.martin@nfinausa.com]
>> Sent: Tuesday, January 15, 2013 2:40 PM
>> To: cloudstack-users@incubator.apache.org
>> Subject: Re: Console Proxy VM unavailable
>>
>> On 01/15/2013 04:28 PM, Anthony Xu wrote:
>>> Hi Nik,
>>>
>>> List_domains and destroy_domain are working on host not pool.
>>>
>>> So if you have multiple hosts in a XenServer pool , you might need to
>> execute above command on that host,
>>>
>>> You can try following to find the host
>>>
>>> Xe vm-param-list uuid="vm uuid" | grep resident-on
>>> You will get the host uuid
>>> Xe host-param-list uuid="above host uuid"
>>>
>>> Then you get the host information.
>>>
>>> Log into the host and retry the procedure to destroy VM.
>> I ran that command on the VM in question, and it hangs, on any host in
>> the pool.  The command works fine for any other VM.  This VM must be in
>> bad shape.
>>
>>>
>>>
>>> Anthony
>>>
>>>> -----Original Message-----
>>>> From: Nik Martin [mailto:nik.martin@nfinausa.com]
>>>> Sent: Tuesday, January 15, 2013 2:17 PM
>>>> To: cloudstack-users@incubator.apache.org
>>>> Subject: Re: Console Proxy VM unavailable
>>>>
>>>> On 01/15/2013 03:55 PM, Anthony Xu wrote:
>>>>> Hi Nik,
>>>>>
>>>>> Try this
>>>>> http://support.citrix.com/article/CTX131421
>>>>>
>>>>> Anthony
>>>>
>>>> Anthony,
>>>> Running list_domains with the VMs UUID does not return any results,
>> but
>>>> if I just run list_domains, I get all the control domains, and one
>> odd
>>>> one:
>>>>
>>>> 117 | deadbeef-dead-beef-dead-beef00000075 | DS
>>>>
>>>> could this be the control domain for the hung VM?
>>>>
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Nik Martin [mailto:nik.martin@nfinausa.com]
>>>>>> Sent: Tuesday, January 15, 2013 1:41 PM
>>>>>> To: cloudstack-users@incubator.apache.org
>>>>>> Subject: Re: Console Proxy VM unavailable
>>>>>>
>>>>>> A bit more information, I logged in to Xen center and the VM is
>> hung
>>>>>> with a yellow arrow.  There are no options to shutdown, reboot, or
>>>>>> anything.  How do I force this VM off?
>>>>>>
>>>>>> On 01/15/2013 03:32 PM, Nik Martin wrote:
>>>>>>> I have a CS 3.02 cloud in production, and it has been running
>> fine
>>>>>> since
>>>>>>> December 1.  Today, I tried to use the web console, and it times
>>>> out.
>>>>>> I
>>>>>>> turned debug up on the management server, and see the command to
>>>>>> launch
>>>>>>> the proxy go by, but no errors.  I tried to migrate it, and now
>>>> it's
>>>>>>> hung in a migrating state.  What is the best way to reset this VM
>>>> and
>>>>>>> get back on track, and also diagnose why this happened?
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Nik
>>>>>>
>>>>>> Nik Martin
>>>>>> nfina Technologies, Inc.
>>>>>> +1.251.243.0043 x1003
>>>>>> http://nfinausa.com
>>>>>> Relentless Reliability
>>>>
>>>>
>>>> --
>>>>
>>>> Regards,
>>>>
>>>> Nik
>>>>
>>>> Nik Martin
>>>> nfina Technologies, Inc.
>>>> +1.251.243.0043 x1003
>>>> http://nfinausa.com
>>>> Relentless Reliability
>>
>>
>> --
>>
>> Regards,
>>
>> Nik
>>
>> Nik Martin
>> nfina Technologies, Inc.
>> +1.251.243.0043 x1003
>> http://nfinausa.com
>> Relentless Reliability


-- 

Regards,

Nik

Nik Martin
nfina Technologies, Inc.
+1.251.243.0043 x1003
http://nfinausa.com
Relentless Reliability