cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Evan Miller <Evan.Mil...@citrix.com>
Subject RE: Problem: Unable to create a deployment for VM[User|i-2-8-VM]
Date Tue, 17 Jul 2012 03:09:07 GMT
FYI: From the GUI, I noticed that the cluster for the only host has an undefined State.
FYI: From the GUI, the only host has a State of Up.

First ...

I stopped the cloud-management service.
I dumped the cloud DB:

[root@cumulus management]# mysqldump -u root -p cloud > /tmp/cloud.sql
Enter password:
[root@cumulus management]# ls -alg /tmp/cloud.sql
-rw-r--r--. 1 root root 401072 Jul 16 19:35 /tmp/cloud.sql
[root@cumulus management]#

Got the ids for the VMs ...

mysql> Select * from vm_instance where state in ('Running', 'Starting', 'Stopping') AND
host_id in (select id from host where removed is not null);        +----+--------+--------------------------------------+---------------+---------+----------------+-------------+---------------------+--------------------+--------+----------------+---------+--------------+----------+-------------------+----------------------------------------------+------------+---------------+--------------+---------------------+---------------------+---------+--------------------+--------------------+------------+-----------+---------------------+--------------------------------------+-----------------+
| id | name   | uuid                                 | instance_name | state   | vm_template_id
| guest_os_id | private_mac_address | private_ip_address | pod_id | data_center_id | host_id
| last_host_id | proxy_id | proxy_assign_time | vnc_password                             
   | ha_enabled | limit_cpu_use | update_count | update_time         | created           
 | removed | type               | vm_type            | account_id | domain_id | service_offering_id
| reservation_id                       | hypervisor_type |
+----+--------+--------------------------------------+---------------+---------+----------------+-------------+---------------------+--------------------+--------+----------------+---------+--------------+----------+-------------------+----------------------------------------------+------------+---------------+--------------+---------------------+---------------------+---------+--------------------+--------------------+------------+-----------+---------------------+--------------------------------------+-----------------+
|  1 | s-1-VM | 4780c7ba-dd55-45a2-a2e5-ed2974c070b5 | s-1-VM        | Running |         
    1 |          15 | 06:9e:ec:00:00:07   | 10.217.5.199       |      1 |              1 |
      1 |            1 |     NULL | NULL              | caqFANzAxnvRMnVMJeajvIY4x3W8NMjFUkj/RkQmhSU=
|          0 |             0 |            3 | 2012-07-14 00:02:27 | 2012-07-13 23:59:42 |
NULL    | SecondaryStorageVm | SecondaryStorageVm |          1 |         1 |             
     6 | e62a93bd-9bb0-4640-8a89-dc1049f300fa | XenServer       |
|  2 | v-2-VM | 79924f6e-66e2-429f-b38a-9d3bd3f6a575 | v-2-VM        | Running |         
    1 |          15 | 06:31:80:00:00:06   | 10.217.5.198       |      1 |              1 |
      1 |            1 |     NULL | NULL              | cF8VnXqcjTMS8dUfURYlShHJhV55iUFKMyyEKA/SNhw=
|          0 |             0 |            3 | 2012-07-14 00:04:19 | 2012-07-13 23:59:42 |
NULL    | ConsoleProxy       | ConsoleProxy       |          1 |         1 |             
     8 | c1e4d053-95c6-4804-b73f-c38e7e4e887a | XenServer       |
|  4 | r-4-VM | 593d5b00-9c73-485f-b091-121b69171e65 | r-4-VM        | Running |         
    1 |          15 | 0e:00:a9:fe:00:88   | 169.254.0.136      |      1 |              1 |
      1 |            1 |     NULL | NULL              | AZ69gd1Mp6IO4gt0AWAeAQawEXZtjh9mA7cQ6ZCDcJg=
|          1 |             0 |            3 | 2012-07-14 01:53:39 | 2012-07-14 01:52:48 |
NULL    | DomainRouter       | DomainRouter       |          1 |         1 |             
     7 | 4f69471a-c0f0-48d2-a39e-3dda20b20c56 | XenServer       |
+----+--------+--------------------------------------+---------------+---------+----------------+-------------+---------------------+--------------------+--------+----------------+---------+--------------+----------+-------------------+----------------------------------------------+------------+---------------+--------------+---------------------+---------------------+---------+--------------------+--------------------+------------+-----------+---------------------+--------------------------------------+-----------------+
3 rows in set (0.00 sec)

mysql> Select id, name, state from vm_instance where state in ('Running', 'Starting', 'Stopping')
AND host_id in (select id from host where removed is not null);
+----+--------+---------+
| id | name   | state   |
+----+--------+---------+
|  1 | s-1-VM | Running |
|  2 | v-2-VM | Running |
|  4 | r-4-VM | Running |
+----+--------+---------+
3 rows in set (0.00 sec)

mysql>

Next ...

Update vm_instance set state='Stopped', host_id=null where id in('1', '2', '4');

mysql> Update vm_instance set state='Stopped', host_id=null where id in('1', '2', '4');
Query OK, 3 rows affected (0.06 sec)
Rows matched: 3  Changed: 3  Warnings: 0

mysql>

Restarted CSMS ...

[root@cumulus management]# service cloud-management start
Starting cloud-management:                                 [  OK  ]
[root@cumulus management]#

>From the GUI, started the two system VMs ...

>From the API (no other way), started the Router ...

http://10.217.5.192:8080/client/api?apikey=iFl88lw1Pk6gKqUIFPN8vzZbJNsUVdYGIJKBTEXtrymcIH5UWp9VHjgnpP_zCmaucmi8XmwK75TR70z-2ayjGA&command=startR
outer&id=593d5b00-9c73-485f-b091-121b69171e65&response=json&signature=sx8etg9c1W5o%2BV7L2JUr1Bn%2BfEk%3D

{ "listroutersresponse" : { "count":1 ,"router" : [  {"id":"593d5b00-9c73-485f-b091-121b69171e65","zoneid":"9bf663f5-167a-4152-b80c-a1fb905e01ef
... etc} ] } }

>From the API, verified the Router had started ...

http://10.217.5.192:8080/client/api?apikey=iFl88lw1Pk6gKqUIFPN8vzZbJNsUVdYGIJKBTEXtrymcIH5UWp9VHjgnpP_zCmaucmi8XmwK75TR70z-2ayjGA&command=listRouters&listall=true&response=json&signature=aQWLi1pJXK2wKF5D2ljFR6rbsDA%3D

{ "listroutersresponse" : { "count":1 ,"router" : [  {"id":"593d5b00-9c73-485f-b091-121b69171e65","zoneid":"9bf663f5-167a-4152-b80c-a1fb905e01ef","zonename":"LS_ZONE1","dns1":"10.217.120.2","dns2":"10.217.120.3","networkdomain":"cs1cloud.internal","name":"r-4-VM","podid":"3d6a0b64-94c7-40e1-865b-c62469ca2bda","hostid":"7b966f69-e792-4fb5-a5d7-2c3acb294f79","hostname":"xenserver1-cs","linklocalip":"169.254.2.164","linklocalmacaddress":"0e:00:a9:fe:02:a4","linklocalnetmask":"255.255.0.0","linklocalnetworkid":"83214711-3a39-46ac-a482-95368cbd6950","guestipaddress":"10.217.5.214","guestmacaddress":"06:0e:d6:00:00:0d","guestnetmask":"255.255.255.0","guestnetworkid":"003cef18-fb0c-4b5b-8f4d-1b89eda55580","templateid":"d6260123-7192-4122-beda-d810a7c596b0","created":"2012-07-13T18:52:48-0700","state":"Running","account":"system","domainid":"b8d53f75-99be-432f-9a49-853509c461a3","domain":"ROOT","serviceofferingid":"43713ed5-61fe-4dbc-b676-f21f0d0ebbb0","serviceofferingname":"System
Offering For Software Router","isredundantrouter":false,"redundantstate":"UNKNOWN"} ] } }

The three VMs are now Running ...

>From the GUI, successfully added a VM to the only host.

Thanks.

Could you please give the sequence of API steps to deleting a host?
deleteHost reports success when devices other than the host have been adversely affected.

Regards,
Evan



-----Original Message-----
From: Alena Prokharchyk [mailto:Alena.Prokharchyk@citrix.com] 
Sent: Monday, July 16, 2012 5:26 PM
To: cloudstack-users@incubator.apache.org
Subject: Re: Problem: Unable to create a deployment for VM[User|i-2-8-VM]

How did you remove the host id=1, through the API or UI? It should have been resulted with
error because it has vms running on it - your Virtual router r-4-VM.

Looks like the system is in a pretty bad state. I can think of one way to recover the vms:

1) stop the Management server, save the cloud DB dump.
2) In the DB find all Running vms referring the Removed hosts:

Select * from vm_instance where state in (Running,Starting, Stopping) and host_id in (select
id from host where removed is not null).

3) update all these vms with the following info:

Update vm_instance set state='Stopped', host_id=null where id in (<ids found on prev step>)

4) start the management server
5) start the vms updated on step 3).


-Alena.



On 7/16/12 5:16 PM, "Evan Miller" <Evan.Miller@citrix.com> wrote:

>Hi Alena:
>
>It looks like the host table in the database needs to be cleaned up:
>
>mysql> select id, name, uuid, status from host;
>+----+-----------------------------------------------------+-----------
>+----+-----------------------------------------------------+---
>------------------------+--------------+
>| id | name                                                | uuid
>                        | status       |
>+----+-----------------------------------------------------+-----------
>+----+-----------------------------------------------------+---
>------------------------+--------------+
>|  1 | xenserver1-cs                                       |
>5da2e723-c41a-45bb-9f87-6048c5487ea2 | Removed      |
>|  2 | nfs://cumulus.eng.citrite.net/home/export/secondary |
>7bf58f58-0cf5-4487-9346-9481fa28ad7a | Alert        |
>|  3 | s-1-VM                                              |
>780d68d6-f6d7-45de-bbd6-766d8e17ee48 | Disconnected |
>|  4 | v-2-VM                                              |
>ca98385f-90f5-49ba-8d88-50af5587e7a9 | Disconnected |
>|  5 | xenserver1-cs                                       |
>dff5e16e-7d8b-47d1-a9a3-8c8e18ca8d23 | Removed      |
>|  6 | xenserver1-cs                                       |
>7b966f69-e792-4fb5-a5d7-2c3acb294f79 | Up           |
>+----+-----------------------------------------------------+-----------
>+----+-----------------------------------------------------+---
>------------------------+--------------+
>6 rows in set (0.00 sec)
>
>mysql>
>
>id 1 and id 5 should be deleted.
>id 6 has the correct state and uuid.
>id 2, 3 and 4 have the wrong state.
>
>I can fix directly with SQL.
>
>Any other tables that need fixing besides host?
>
>Regards,
>Evan
>
>
>
>-----Original Message-----
>From: Alena Prokharchyk [mailto:Alena.Prokharchyk@citrix.com]
>Sent: Monday, July 16, 2012 4:55 PM
>To: cloudstack-users@incubator.apache.org
>Subject: Re: Problem: Unable to create a deployment for 
>VM[User|i-2-8-VM]
>
>The log states that the host doesn't exist:
>
>com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
>unreachable: Host 1: Can't find the host
>
>
>
>Can you check the database for host id=1? Make sure its not removed.
>
>
>On 7/16/12 4:45 PM, "Evan Miller" <Evan.Miller@citrix.com> wrote:
>
>>From CSMS GUI, the State is Up for the host, xenserver1-cs.
>>
>>Regards,
>>Evan
>>
>>
>>-----Original Message-----
>>From: Alena Prokharchyk [mailto:Alena.Prokharchyk@citrix.com]
>>Sent: Monday, July 16, 2012 4:45 PM
>>To: cloudstack-users@incubator.apache.org
>>Subject: Re: Problem: Unable to create a deployment for 
>>VM[User|i-2-8-VM]
>>
>>Looks like the Xen host your Virtual Router r-4-VM is running on, is 
>>in not in UP state. Can you please confirm. If this is the case, then 
>>the failure is expected. The Xen host should be in UP state.
>>
>>
>>On 7/16/12 4:16 PM, "Evan Miller" <Evan.Miller@citrix.com> wrote:
>>
>>>Running CloudStack Management Server:
>>>  v3.0.2.1
>>>On:
>>>  [root@cumulus management]# uname -a
>>>  Linux cumulus.eng.citrite.net 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 
>>>6
>>>19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux
>>>  [root@cumulus management]#
>>>Hypervisor :
>>>  XenServer v6.02
>>> 
>>>Hi:
>>> 
>>>Can¹t add VM instance.
>>>Getting this error message from the GUI:
>>>  Unable to create a deployment for VM[User|i-2-8-VM]
>>> 
>>>In the attached log file, there are a few ³apparent² issues when 
>>>trying to create the instance :
>>> 
>>>1)
>>>2012-07-16 15:56:19,928 DEBUG
>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>(Job-Executor-3:job-33) Applying dhcp entry in network 
>>>Ntwk[204|Guest|5]
>>>2012-07-16 15:56:19,943 WARN
>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>(Job-Executor-3:job-33) Unable to apply dhcp entry on disconnected 
>>>router r-4-VM
>>>com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
>>>unreachable: Host 1: Can't find the host
>>> 
>>>This one occurred at the beginning of the async job. Not sure if it 
>>>is an error or not.
>>> 
>>>2)
>>>2012-07-16 15:56:20,521 DEBUG [cloud.capacity.CapacityManagerImpl]
>>>(Job-Executor-3:job-33) VM state transitted from :Stopped to Error 
>>>with
>>>event: OperationFailedToErrorvm's original host id: null new host id:
>>>null host id before state transition:
>>> null
>>>2012-07-16 15:56:21,192 INFO  [api.commands.DeployVMCmd]
>>>(Job-Executor-3:job-33)
>>>com.cloud.exception.InsufficientServerCapacityException: Unable to 
>>>create a deployment for VM[User|i-2-8-VM]Scope=interface 
>>>com.cloud.dc.DataCenter; id=1
>>>2012-07-16 15:56:21,192 WARN  [cloud.api.ApiDispatcher]
>>>(Job-Executor-3:job-33) class com.cloud.api.ServerApiException :
>>>Unable to create a deployment for VM[User|i-2-8-VM]
>>> 
>>>That one doesn¹t look good.
>>> 
>>>3)
>>>2012-07-16 15:56:30,263 DEBUG [cloud.server.StatsCollector]
>>>(StatsCollector-1:null) There is no secondary storage VM for 
>>>secondary storage host 
>>>nfs://cumulus.eng.citrite.net/home/export/secondary
>>>
>>>
>>>From CloudStack GUI, secondary storage is defined and enabled.
>>> 
>>>What¹s important, which needs to be fixed and how to fix?
>>> 
>>>Thanks.
>>> 
>>>Regards,
>>> 
>>>Evan Miller
>>> 
>>>Citrix Systems. Inc.
>>>Desktop and Cloud Engineering Infrastructure
>>>4988 Great America Parkway
>>>Santa Clara, CA 95054
>>> 
>>>
>>>
>>
>>
>>
>
>
>



Mime
View raw message