cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alena Prokharchyk <Alena.Prokharc...@citrix.com>
Subject Re: Problem: Unable to create a deployment for VM[User|i-2-8-VM]
Date Tue, 17 Jul 2012 04:47:05 GMT
Steps to remove the host:

* If this is the only one host in the cluster, add one more host so the
existing vms can migrate there
* put the host you want to remove into Maintenance.
* Once maintenance is completed, delete the host.

-Alena.


On 7/16/12 8:09 PM, "Evan Miller" <Evan.Miller@citrix.com> wrote:

>FYI: From the GUI, I noticed that the cluster for the only host has an
>undefined State.
>FYI: From the GUI, the only host has a State of Up.
>
>First ...
>
>I stopped the cloud-management service.
>I dumped the cloud DB:
>
>[root@cumulus management]# mysqldump -u root -p cloud > /tmp/cloud.sql
>Enter password:
>[root@cumulus management]# ls -alg /tmp/cloud.sql
>-rw-r--r--. 1 root root 401072 Jul 16 19:35 /tmp/cloud.sql
>[root@cumulus management]#
>
>Got the ids for the VMs ...
>
>mysql> Select * from vm_instance where state in ('Running', 'Starting',
>'Stopping') AND host_id in (select id from host where removed is not
>null);
>+----+--------+--------------------------------------+---------------+----
>-----+----------------+-------------+---------------------+---------------
>-----+--------+----------------+---------+--------------+----------+------
>-------------+----------------------------------------------+------------+
>---------------+--------------+---------------------+---------------------
>+---------+--------------------+--------------------+------------+--------
>---+---------------------+--------------------------------------+---------
>--------+
>| id | name   | uuid                                 | instance_name |
>state   | vm_template_id | guest_os_id | private_mac_address |
>private_ip_address | pod_id | data_center_id | host_id | last_host_id |
>proxy_id | proxy_assign_time | vnc_password
>  | ha_enabled | limit_cpu_use | update_count | update_time         |
>created             | removed | type               | vm_type            |
>account_id | domain_id | service_offering_id | reservation_id
>          | hypervisor_type |
>+----+--------+--------------------------------------+---------------+----
>-----+----------------+-------------+---------------------+---------------
>-----+--------+----------------+---------+--------------+----------+------
>-------------+----------------------------------------------+------------+
>---------------+--------------+---------------------+---------------------
>+---------+--------------------+--------------------+------------+--------
>---+---------------------+--------------------------------------+---------
>--------+
>|  1 | s-1-VM | 4780c7ba-dd55-45a2-a2e5-ed2974c070b5 | s-1-VM        |
>Running |              1 |          15 | 06:9e:ec:00:00:07   |
>10.217.5.199       |      1 |              1 |       1 |            1 |
>  NULL | NULL              | caqFANzAxnvRMnVMJeajvIY4x3W8NMjFUkj/RkQmhSU=
>|          0 |             0 |            3 | 2012-07-14 00:02:27 |
>2012-07-13 23:59:42 | NULL    | SecondaryStorageVm | SecondaryStorageVm |
>         1 |         1 |                   6 |
>e62a93bd-9bb0-4640-8a89-dc1049f300fa | XenServer       |
>|  2 | v-2-VM | 79924f6e-66e2-429f-b38a-9d3bd3f6a575 | v-2-VM        |
>Running |              1 |          15 | 06:31:80:00:00:06   |
>10.217.5.198       |      1 |              1 |       1 |            1 |
>  NULL | NULL              | cF8VnXqcjTMS8dUfURYlShHJhV55iUFKMyyEKA/SNhw=
>|          0 |             0 |            3 | 2012-07-14 00:04:19 |
>2012-07-13 23:59:42 | NULL    | ConsoleProxy       | ConsoleProxy       |
>         1 |         1 |                   8 |
>c1e4d053-95c6-4804-b73f-c38e7e4e887a | XenServer       |
>|  4 | r-4-VM | 593d5b00-9c73-485f-b091-121b69171e65 | r-4-VM        |
>Running |              1 |          15 | 0e:00:a9:fe:00:88   |
>169.254.0.136      |      1 |              1 |       1 |            1 |
>  NULL | NULL              | AZ69gd1Mp6IO4gt0AWAeAQawEXZtjh9mA7cQ6ZCDcJg=
>|          1 |             0 |            3 | 2012-07-14 01:53:39 |
>2012-07-14 01:52:48 | NULL    | DomainRouter       | DomainRouter       |
>         1 |         1 |                   7 |
>4f69471a-c0f0-48d2-a39e-3dda20b20c56 | XenServer       |
>+----+--------+--------------------------------------+---------------+----
>-----+----------------+-------------+---------------------+---------------
>-----+--------+----------------+---------+--------------+----------+------
>-------------+----------------------------------------------+------------+
>---------------+--------------+---------------------+---------------------
>+---------+--------------------+--------------------+------------+--------
>---+---------------------+--------------------------------------+---------
>--------+
>3 rows in set (0.00 sec)
>
>mysql> Select id, name, state from vm_instance where state in ('Running',
>'Starting', 'Stopping') AND host_id in (select id from host where removed
>is not null);
>+----+--------+---------+
>| id | name   | state   |
>+----+--------+---------+
>|  1 | s-1-VM | Running |
>|  2 | v-2-VM | Running |
>|  4 | r-4-VM | Running |
>+----+--------+---------+
>3 rows in set (0.00 sec)
>
>mysql>
>
>Next ...
>
>Update vm_instance set state='Stopped', host_id=null where id in('1',
>'2', '4');
>
>mysql> Update vm_instance set state='Stopped', host_id=null where id
>in('1', '2', '4');
>Query OK, 3 rows affected (0.06 sec)
>Rows matched: 3  Changed: 3  Warnings: 0
>
>mysql>
>
>Restarted CSMS ...
>
>[root@cumulus management]# service cloud-management start
>Starting cloud-management:                                 [  OK  ]
>[root@cumulus management]#
>
>From the GUI, started the two system VMs ...
>
>From the API (no other way), started the Router ...
>
>http://10.217.5.192:8080/client/api?apikey=iFl88lw1Pk6gKqUIFPN8vzZbJNsUVdY
>GIJKBTEXtrymcIH5UWp9VHjgnpP_zCmaucmi8XmwK75TR70z-2ayjGA&command=startR
>outer&id=593d5b00-9c73-485f-b091-121b69171e65&response=json&signature=sx8e
>tg9c1W5o%2BV7L2JUr1Bn%2BfEk%3D
>
>{ "listroutersresponse" : { "count":1 ,"router" : [
>{"id":"593d5b00-9c73-485f-b091-121b69171e65","zoneid":"9bf663f5-167a-4152-
>b80c-a1fb905e01ef ... etc} ] } }
>
>From the API, verified the Router had started ...
>
>http://10.217.5.192:8080/client/api?apikey=iFl88lw1Pk6gKqUIFPN8vzZbJNsUVdY
>GIJKBTEXtrymcIH5UWp9VHjgnpP_zCmaucmi8XmwK75TR70z-2ayjGA&command=listRouter
>s&listall=true&response=json&signature=aQWLi1pJXK2wKF5D2ljFR6rbsDA%3D
>
>{ "listroutersresponse" : { "count":1 ,"router" : [
>{"id":"593d5b00-9c73-485f-b091-121b69171e65","zoneid":"9bf663f5-167a-4152-
>b80c-a1fb905e01ef","zonename":"LS_ZONE1","dns1":"10.217.120.2","dns2":"10.
>217.120.3","networkdomain":"cs1cloud.internal","name":"r-4-VM","podid":"3d
>6a0b64-94c7-40e1-865b-c62469ca2bda","hostid":"7b966f69-e792-4fb5-a5d7-2c3a
>cb294f79","hostname":"xenserver1-cs","linklocalip":"169.254.2.164","linklo
>calmacaddress":"0e:00:a9:fe:02:a4","linklocalnetmask":"255.255.0.0","linkl
>ocalnetworkid":"83214711-3a39-46ac-a482-95368cbd6950","guestipaddress":"10
>.217.5.214","guestmacaddress":"06:0e:d6:00:00:0d","guestnetmask":"255.255.
>255.0","guestnetworkid":"003cef18-fb0c-4b5b-8f4d-1b89eda55580","templateid
>":"d6260123-7192-4122-beda-d810a7c596b0","created":"2012-07-13T18:52:48-07
>00","state":"Running","account":"system","domainid":"b8d53f75-99be-432f-9a
>49-853509c461a3","domain":"ROOT","serviceofferingid":"43713ed5-61fe-4dbc-b
>676-f21f0d0ebbb0","serviceofferingname":"System Offering For Software
>Router","isredundantrouter":false,"redundantstate":"UNKNOWN"} ] } }
>
>The three VMs are now Running ...
>
>From the GUI, successfully added a VM to the only host.
>
>Thanks.
>
>Could you please give the sequence of API steps to deleting a host?
>deleteHost reports success when devices other than the host have been
>adversely affected.
>
>Regards,
>Evan
>
>
>
>-----Original Message-----
>From: Alena Prokharchyk [mailto:Alena.Prokharchyk@citrix.com]
>Sent: Monday, July 16, 2012 5:26 PM
>To: cloudstack-users@incubator.apache.org
>Subject: Re: Problem: Unable to create a deployment for VM[User|i-2-8-VM]
>
>How did you remove the host id=1, through the API or UI? It should have
>been resulted with error because it has vms running on it - your Virtual
>router r-4-VM.
>
>Looks like the system is in a pretty bad state. I can think of one way to
>recover the vms:
>
>1) stop the Management server, save the cloud DB dump.
>2) In the DB find all Running vms referring the Removed hosts:
>
>Select * from vm_instance where state in (Running,Starting, Stopping) and
>host_id in (select id from host where removed is not null).
>
>3) update all these vms with the following info:
>
>Update vm_instance set state='Stopped', host_id=null where id in (<ids
>found on prev step>)
>
>4) start the management server
>5) start the vms updated on step 3).
>
>
>-Alena.
>
>
>
>On 7/16/12 5:16 PM, "Evan Miller" <Evan.Miller@citrix.com> wrote:
>
>>Hi Alena:
>>
>>It looks like the host table in the database needs to be cleaned up:
>>
>>mysql> select id, name, uuid, status from host;
>>+----+-----------------------------------------------------+-----------
>>+----+-----------------------------------------------------+---
>>------------------------+--------------+
>>| id | name                                                | uuid
>>                        | status       |
>>+----+-----------------------------------------------------+-----------
>>+----+-----------------------------------------------------+---
>>------------------------+--------------+
>>|  1 | xenserver1-cs                                       |
>>5da2e723-c41a-45bb-9f87-6048c5487ea2 | Removed      |
>>|  2 | nfs://cumulus.eng.citrite.net/home/export/secondary |
>>7bf58f58-0cf5-4487-9346-9481fa28ad7a | Alert        |
>>|  3 | s-1-VM                                              |
>>780d68d6-f6d7-45de-bbd6-766d8e17ee48 | Disconnected |
>>|  4 | v-2-VM                                              |
>>ca98385f-90f5-49ba-8d88-50af5587e7a9 | Disconnected |
>>|  5 | xenserver1-cs                                       |
>>dff5e16e-7d8b-47d1-a9a3-8c8e18ca8d23 | Removed      |
>>|  6 | xenserver1-cs                                       |
>>7b966f69-e792-4fb5-a5d7-2c3acb294f79 | Up           |
>>+----+-----------------------------------------------------+-----------
>>+----+-----------------------------------------------------+---
>>------------------------+--------------+
>>6 rows in set (0.00 sec)
>>
>>mysql>
>>
>>id 1 and id 5 should be deleted.
>>id 6 has the correct state and uuid.
>>id 2, 3 and 4 have the wrong state.
>>
>>I can fix directly with SQL.
>>
>>Any other tables that need fixing besides host?
>>
>>Regards,
>>Evan
>>
>>
>>
>>-----Original Message-----
>>From: Alena Prokharchyk [mailto:Alena.Prokharchyk@citrix.com]
>>Sent: Monday, July 16, 2012 4:55 PM
>>To: cloudstack-users@incubator.apache.org
>>Subject: Re: Problem: Unable to create a deployment for
>>VM[User|i-2-8-VM]
>>
>>The log states that the host doesn't exist:
>>
>>com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
>>unreachable: Host 1: Can't find the host
>>
>>
>>
>>Can you check the database for host id=1? Make sure its not removed.
>>
>>
>>On 7/16/12 4:45 PM, "Evan Miller" <Evan.Miller@citrix.com> wrote:
>>
>>>From CSMS GUI, the State is Up for the host, xenserver1-cs.
>>>
>>>Regards,
>>>Evan
>>>
>>>
>>>-----Original Message-----
>>>From: Alena Prokharchyk [mailto:Alena.Prokharchyk@citrix.com]
>>>Sent: Monday, July 16, 2012 4:45 PM
>>>To: cloudstack-users@incubator.apache.org
>>>Subject: Re: Problem: Unable to create a deployment for
>>>VM[User|i-2-8-VM]
>>>
>>>Looks like the Xen host your Virtual Router r-4-VM is running on, is
>>>in not in UP state. Can you please confirm. If this is the case, then
>>>the failure is expected. The Xen host should be in UP state.
>>>
>>>
>>>On 7/16/12 4:16 PM, "Evan Miller" <Evan.Miller@citrix.com> wrote:
>>>
>>>>Running CloudStack Management Server:
>>>>  v3.0.2.1
>>>>On:
>>>>  [root@cumulus management]# uname -a
>>>>  Linux cumulus.eng.citrite.net 2.6.32-220.el6.x86_64 #1 SMP Tue Dec
>>>>6
>>>>19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux
>>>>  [root@cumulus management]#
>>>>Hypervisor :
>>>>  XenServer v6.02
>>>>
>>>>Hi:
>>>>
>>>>Can¹t add VM instance.
>>>>Getting this error message from the GUI:
>>>>  Unable to create a deployment for VM[User|i-2-8-VM]
>>>>
>>>>In the attached log file, there are a few ³apparent² issues when
>>>>trying to create the instance :
>>>>
>>>>1)
>>>>2012-07-16 15:56:19,928 DEBUG
>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>(Job-Executor-3:job-33) Applying dhcp entry in network
>>>>Ntwk[204|Guest|5]
>>>>2012-07-16 15:56:19,943 WARN
>>>>[network.router.VirtualNetworkApplianceManagerImpl]
>>>>(Job-Executor-3:job-33) Unable to apply dhcp entry on disconnected
>>>>router r-4-VM
>>>>com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
>>>>unreachable: Host 1: Can't find the host
>>>>
>>>>This one occurred at the beginning of the async job. Not sure if it
>>>>is an error or not.
>>>>
>>>>2)
>>>>2012-07-16 15:56:20,521 DEBUG [cloud.capacity.CapacityManagerImpl]
>>>>(Job-Executor-3:job-33) VM state transitted from :Stopped to Error
>>>>with
>>>>event: OperationFailedToErrorvm's original host id: null new host id:
>>>>null host id before state transition:
>>>> null
>>>>2012-07-16 15:56:21,192 INFO  [api.commands.DeployVMCmd]
>>>>(Job-Executor-3:job-33)
>>>>com.cloud.exception.InsufficientServerCapacityException: Unable to
>>>>create a deployment for VM[User|i-2-8-VM]Scope=interface
>>>>com.cloud.dc.DataCenter; id=1
>>>>2012-07-16 15:56:21,192 WARN  [cloud.api.ApiDispatcher]
>>>>(Job-Executor-3:job-33) class com.cloud.api.ServerApiException :
>>>>Unable to create a deployment for VM[User|i-2-8-VM]
>>>>
>>>>That one doesn¹t look good.
>>>>
>>>>3)
>>>>2012-07-16 15:56:30,263 DEBUG [cloud.server.StatsCollector]
>>>>(StatsCollector-1:null) There is no secondary storage VM for
>>>>secondary storage host
>>>>nfs://cumulus.eng.citrite.net/home/export/secondary
>>>>
>>>>
>>>>From CloudStack GUI, secondary storage is defined and enabled.
>>>>
>>>>What¹s important, which needs to be fixed and how to fix?
>>>>
>>>>Thanks.
>>>>
>>>>Regards,
>>>>
>>>>Evan Miller
>>>>
>>>>Citrix Systems. Inc.
>>>>Desktop and Cloud Engineering Infrastructure
>>>>4988 Great America Parkway
>>>>Santa Clara, CA 95054
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>
>
>


Mime
View raw message