cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alena Prokharchyk <Alena.Prokharc...@citrix.com>
Subject Re: Problem: Unable to create a deployment for VM[User|i-2-8-VM]
Date Tue, 17 Jul 2012 18:43:44 GMT
On 7/17/12 11:20 AM, "Evan Miller" <Evan.Miller@citrix.com> wrote:

>host id=1 was always removed using the API.
>
>I don't believe any non-system instances were ever running when
>deleteHost was executed.
>
>No error was ever observed after running deleteHost API.
>
>Regards,
>Evan



But in your previous email you've mentioned that there were vms running on
the host id=1:

mysql> Select * from vm_instance where state in ('Running', 'Starting',
'Stopping') AND host_id in (select id from host where removed is not
null);        
+----+--------+--------------------------------------+---------------+-----
----+----------------+-------------+---------------------+-----------------
---+--------+----------------+---------+--------------+----------+---------
----------+----------------------------------------------+------------+----
-----------+--------------+---------------------+---------------------+----
-----+--------------------+--------------------+------------+-----------+--
-------------------+--------------------------------------+----------------
-+
| id | name   | uuid                                 | instance_name |
state   | vm_template_id | guest_os_id | private_mac_address |
private_ip_address | pod_id | data_center_id | host_id | last_host_id |
proxy_id | proxy_assign_time | vnc_password
 | ha_enabled | limit_cpu_use | update_count | update_time         |
created             | removed | type               | vm_type            |
account_id | domain_id | service_offering_id | reservation_id
         | hypervisor_type |
+----+--------+--------------------------------------+---------------+-----
----+----------------+-------------+---------------------+-----------------
---+--------+----------------+---------+--------------+----------+---------
----------+----------------------------------------------+------------+----
-----------+--------------+---------------------+---------------------+----
-----+--------------------+--------------------+------------+-----------+--
-------------------+--------------------------------------+----------------
-+
|  1 | s-1-VM | 4780c7ba-dd55-45a2-a2e5-ed2974c070b5 | s-1-VM        |
Running |              1 |          15 | 06:9e:ec:00:00:07   |
10.217.5.199       |      1 |              1 |       1 |            1 |
 NULL | NULL              | caqFANzAxnvRMnVMJeajvIY4x3W8NMjFUkj/RkQmhSU= |
         0 |             0 |            3 | 2012-07-14 00:02:27 |
2012-07-13 23:59:42 | NULL    | SecondaryStorageVm | SecondaryStorageVm |
        1 |         1 |                   6 |
e62a93bd-9bb0-4640-8a89-dc1049f300fa | XenServer       |
|  2 | v-2-VM | 79924f6e-66e2-429f-b38a-9d3bd3f6a575 | v-2-VM        |
Running |              1 |          15 | 06:31:80:00:00:06   |
10.217.5.198       |      1 |              1 |       1 |            1 |
 NULL | NULL              | cF8VnXqcjTMS8dUfURYlShHJhV55iUFKMyyEKA/SNhw= |
         0 |             0 |            3 | 2012-07-14 00:04:19 |
2012-07-13 23:59:42 | NULL    | ConsoleProxy       | ConsoleProxy       |
        1 |         1 |                   8 |
c1e4d053-95c6-4804-b73f-c38e7e4e887a | XenServer       |
|  4 | r-4-VM | 593d5b00-9c73-485f-b091-121b69171e65 | r-4-VM        |
Running |              1 |          15 | 0e:00:a9:fe:00:88   |
169.254.0.136      |      1 |              1 |       1 |            1 |
 NULL | NULL              | AZ69gd1Mp6IO4gt0AWAeAQawEXZtjh9mA7cQ6ZCDcJg= |
         1 |             0 |            3 | 2012-07-14 01:53:39 |
2012-07-14 01:52:48 | NULL    | DomainRouter       | DomainRouter       |
        1 |         1 |                   7 |
4f69471a-c0f0-48d2-a39e-3dda20b20c56 | XenServer       |
+----+--------+--------------------------------------+---------------+-----
----+----------------+-------------+---------------------+-----------------
---+--------+----------------+---------+--------------+----------+---------
----------+----------------------------------------------+------------+----
-----------+--------------+---------------------+---------------------+----
-----+--------------------+--------------------+------------+-----------+--
-------------------+--------------------------------------+----------------
-+
3 rows in set (0.00 sec)




I would advise to file a bug; and attach management server log file from
the day where host was removed + the day vm r-4-VM was started.

-Alena.

>
>
>-----Original Message-----
>From: Alena Prokharchyk [mailto:Alena.Prokharchyk@citrix.com]
>Sent: Monday, July 16, 2012 7:25 PM
>To: cloudstack-users@incubator.apache.org
>Subject: Re: Problem: Unable to create a deployment for VM[User|i-2-8-VM]
>
>In your case problematic host is not id=5, but id=1. How that host was
>removed?
>
>cloudStack should have been resulted with error when you attempt host
>id=1 deletion while vms are running on it. Right now the system is in bad
>state, and you can't send Stop command for the router as it will fail
>because router is shown as running on removed host, and there is no place
>to send the command to. The only one workaround for situation like that -
>DB modification.
>
>-Alena.
>
>
>
>On 7/16/12 5:38 PM, "Evan Miller" <Evan.Miller@citrix.com> wrote:
>
>>Hi Alena:
>>
>>For example, I removed the host id=5 through the API.
>>
>>Here is the url:
>>http://10.217.5.192:8080/client/api?apikey=iFl88lw1Pk6gKqUIFPN8vzZbJNsU
>>VdY 
>>GIJKBTEXtrymcIH5UWp9VHjgnpP_zCmaucmi8XmwK75TR70z-2ayjGA&command=deleteH
>>ost 
>>&id=dff5e16e-7d8b-47d1-a9a3-8c8e18ca8d23&response=json&signature=CM007f
>>HHr
>>Lsi70cXdMAUIXa6Jbw%3D
>>
>>The JSON output I received was this:
>>
>>{ "deletehostresponse" : { "success" : "true"}  }
>>
>>So, I thought I was good to go.
>>
>>I didn't know to stop a virtual router first. I can do that.
>>Like so?
>>  1) listRouters setting listall=true
>>  2) get the id of the router associated with the host to be deleted.
>>  3) stopRouter id of router from step 2.
>>I need to understand all of the API commands to properly delete a host.
>>
>>Regards,
>>Evan
>>
>>
>>
>>-----Original Message-----
>>From: Alena Prokharchyk [mailto:Alena.Prokharchyk@citrix.com]
>>Sent: Monday, July 16, 2012 5:26 PM
>>To: cloudstack-users@incubator.apache.org
>>Subject: Re: Problem: Unable to create a deployment for
>>VM[User|i-2-8-VM]
>>
>>How did you remove the host id=1, through the API or UI? It should have
>>been resulted with error because it has vms running on it - your
>>Virtual router r-4-VM.
>>
>>Looks like the system is in a pretty bad state. I can think of one way
>>to recover the vms:
>>
>>1) stop the Management server, save the cloud DB dump.
>>2) In the DB find all Running vms referring the Removed hosts:
>>
>>Select * from vm_instance where state in (Running,Starting, Stopping)
>>and host_id in (select id from host where removed is not null).
>>
>>3) update all these vms with the following info:
>>
>>Update vm_instance set state='Stopped', host_id=null where id in (<ids
>>found on prev step>)
>>
>>4) start the management server
>>5) start the vms updated on step 3).
>>
>>
>>-Alena.
>>
>>
>>
>>On 7/16/12 5:16 PM, "Evan Miller" <Evan.Miller@citrix.com> wrote:
>>
>>>Hi Alena:
>>>
>>>It looks like the host table in the database needs to be cleaned up:
>>>
>>>mysql> select id, name, uuid, status from host;
>>>+----+-----------------------------------------------------+----------
>>>+----+-----------------------------------------------------+-
>>>+----+-----------------------------------------------------+---
>>>------------------------+--------------+
>>>| id | name                                                | uuid
>>>                        | status       |
>>>+----+-----------------------------------------------------+----------
>>>+----+-----------------------------------------------------+-
>>>+----+-----------------------------------------------------+---
>>>------------------------+--------------+
>>>|  1 | xenserver1-cs                                       |
>>>5da2e723-c41a-45bb-9f87-6048c5487ea2 | Removed      |
>>>|  2 | nfs://cumulus.eng.citrite.net/home/export/secondary |
>>>7bf58f58-0cf5-4487-9346-9481fa28ad7a | Alert        |
>>>|  3 | s-1-VM                                              |
>>>780d68d6-f6d7-45de-bbd6-766d8e17ee48 | Disconnected |
>>>|  4 | v-2-VM                                              |
>>>ca98385f-90f5-49ba-8d88-50af5587e7a9 | Disconnected |
>>>|  5 | xenserver1-cs                                       |
>>>dff5e16e-7d8b-47d1-a9a3-8c8e18ca8d23 | Removed      |
>>>|  6 | xenserver1-cs                                       |
>>>7b966f69-e792-4fb5-a5d7-2c3acb294f79 | Up           |
>>>+----+-----------------------------------------------------+----------
>>>+----+-----------------------------------------------------+-
>>>+----+-----------------------------------------------------+---
>>>------------------------+--------------+
>>>6 rows in set (0.00 sec)
>>>
>>>mysql>
>>>
>>>id 1 and id 5 should be deleted.
>>>id 6 has the correct state and uuid.
>>>id 2, 3 and 4 have the wrong state.
>>>
>>>I can fix directly with SQL.
>>>
>>>Any other tables that need fixing besides host?
>>>
>>>Regards,
>>>Evan
>>>
>>>
>>>
>>>-----Original Message-----
>>>From: Alena Prokharchyk [mailto:Alena.Prokharchyk@citrix.com]
>>>Sent: Monday, July 16, 2012 4:55 PM
>>>To: cloudstack-users@incubator.apache.org
>>>Subject: Re: Problem: Unable to create a deployment for
>>>VM[User|i-2-8-VM]
>>>
>>>The log states that the host doesn't exist:
>>>
>>>com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
>>>unreachable: Host 1: Can't find the host
>>>
>>>
>>>
>>>Can you check the database for host id=1? Make sure its not removed.
>>>
>>>
>>>On 7/16/12 4:45 PM, "Evan Miller" <Evan.Miller@citrix.com> wrote:
>>>
>>>>From CSMS GUI, the State is Up for the host, xenserver1-cs.
>>>>
>>>>Regards,
>>>>Evan
>>>>
>>>>
>>>>-----Original Message-----
>>>>From: Alena Prokharchyk [mailto:Alena.Prokharchyk@citrix.com]
>>>>Sent: Monday, July 16, 2012 4:45 PM
>>>>To: cloudstack-users@incubator.apache.org
>>>>Subject: Re: Problem: Unable to create a deployment for
>>>>VM[User|i-2-8-VM]
>>>>
>>>>Looks like the Xen host your Virtual Router r-4-VM is running on, is
>>>>in not in UP state. Can you please confirm. If this is the case, then
>>>>the failure is expected. The Xen host should be in UP state.
>>>>
>>>>
>>>>On 7/16/12 4:16 PM, "Evan Miller" <Evan.Miller@citrix.com> wrote:
>>>>
>>>>>Running CloudStack Management Server:
>>>>>  v3.0.2.1
>>>>>On:
>>>>>  [root@cumulus management]# uname -a
>>>>>  Linux cumulus.eng.citrite.net 2.6.32-220.el6.x86_64 #1 SMP Tue Dec
>>>>>6
>>>>>19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux
>>>>>  [root@cumulus management]#
>>>>>Hypervisor :
>>>>>  XenServer v6.02
>>>>> 
>>>>>Hi:
>>>>> 
>>>>>Can¹t add VM instance.
>>>>>Getting this error message from the GUI:
>>>>>  Unable to create a deployment for VM[User|i-2-8-VM]
>>>>> 
>>>>>In the attached log file, there are a few ³apparent² issues when
>>>>>trying to create the instance :
>>>>> 
>>>>>1)
>>>>>2012-07-16 15:56:19,928 DEBUG
>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>(Job-Executor-3:job-33) Applying dhcp entry in network
>>>>>Ntwk[204|Guest|5]
>>>>>2012-07-16 15:56:19,943 WARN
>>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>>(Job-Executor-3:job-33) Unable to apply dhcp entry on disconnected
>>>>>router r-4-VM
>>>>>com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
>>>>>unreachable: Host 1: Can't find the host
>>>>> 
>>>>>This one occurred at the beginning of the async job. Not sure if it
>>>>>is an error or not.
>>>>> 
>>>>>2)
>>>>>2012-07-16 15:56:20,521 DEBUG [cloud.capacity.CapacityManagerImpl]
>>>>>(Job-Executor-3:job-33) VM state transitted from :Stopped to Error
>>>>>with
>>>>>event: OperationFailedToErrorvm's original host id: null new host id:
>>>>>null host id before state transition:
>>>>> null
>>>>>2012-07-16 15:56:21,192 INFO  [api.commands.DeployVMCmd]
>>>>>(Job-Executor-3:job-33)
>>>>>com.cloud.exception.InsufficientServerCapacityException: Unable to
>>>>>create a deployment for VM[User|i-2-8-VM]Scope=interface
>>>>>com.cloud.dc.DataCenter; id=1
>>>>>2012-07-16 15:56:21,192 WARN  [cloud.api.ApiDispatcher]
>>>>>(Job-Executor-3:job-33) class com.cloud.api.ServerApiException :
>>>>>Unable to create a deployment for VM[User|i-2-8-VM]
>>>>> 
>>>>>That one doesn¹t look good.
>>>>> 
>>>>>3)
>>>>>2012-07-16 15:56:30,263 DEBUG [cloud.server.StatsCollector]
>>>>>(StatsCollector-1:null) There is no secondary storage VM for
>>>>>secondary storage host
>>>>>nfs://cumulus.eng.citrite.net/home/export/secondary
>>>>>
>>>>>
>>>>>From CloudStack GUI, secondary storage is defined and enabled.
>>>>> 
>>>>>What¹s important, which needs to be fixed and how to fix?
>>>>> 
>>>>>Thanks.
>>>>> 
>>>>>Regards,
>>>>> 
>>>>>Evan Miller
>>>>> 
>>>>>Citrix Systems. Inc.
>>>>>Desktop and Cloud Engineering Infrastructure
>>>>>4988 Great America Parkway
>>>>>Santa Clara, CA 95054
>>>>> 
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>
>
>


Mime
View raw message