cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Hertanu <d...@unixmob.com>
Subject Re: Management IP address pool exhausted
Date Tue, 24 Sep 2013 13:07:58 GMT
Hello Ilya

I repeated the issue by rebooting the Xen host without having it in
maintenance mode in CS. At this moment I have the SSVM up and running and a
virtual router (not related to the management IPs, I know). Console Proxy
VM can't be started because:

2013-09-24 14:56:24,410 INFO  [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-22:job-72) Insufficient capacity
com.cloud.exception.InsufficientAddressCapacityException: Unable to get a
management ip addressScope=interface com.cloud.dc.Pod; id=1


The queries you were asking about:

mysql> SELECT * FROM cloud.nics where state="Reserved" and
(vm_type="ConsoleProxy" or vm_type="SecondaryStorageVM");
+----+--------------------------------------+-------------+-------------------+----------------+---------------+---------------+---------+-----------------+------------+--------+----------+----------+---------------------------+--------------------------------------+-----------+---------------------+----------------+-------------+-------------+--------------------+---------------------+---------+-------------+----------+
| id | uuid                                 | instance_id | mac_address
  | ip4_address    | netmask       | gateway       | ip_type |
broadcast_uri   | network_id | mode   | state    | strategy | reserver_name
            | reservation_id                       | device_id |
update_time         | isolation_uri  | ip6_address | default_nic | vm_type
           | created             | removed | ip6_gateway | ip6_cidr |
+----+--------------------------------------+-------------+-------------------+----------------+---------------+---------------+---------+-----------------+------------+--------+----------+----------+---------------------------+--------------------------------------+-----------+---------------------+----------------+-------------+-------------+--------------------+---------------------+---------+-------------+----------+
|  1 | c3c2d45d-94f9-4172-9eee-7829106380de |           1 |
06:e9:ea:00:00:0d | 195.238.173.10 | 255.255.255.0 | 195.238.173.1 | Ip4
  | vlan://untagged |        204 | Dhcp   | Reserved | Start    |
DirectPodBasedNetworkGuru | 6591a371-15e3-4396-b14e-d23de671bdaa |
2 | 2013-09-24 14:54:33 | ec2://untagged | NULL        |           1 |
SecondaryStorageVm | 2013-09-23 12:06:25 | NULL    | NULL        | NULL
|
|  2 | d2c11105-22fd-4c94-b54b-4039347e3ec2 |           1 |
0e:00:a9:fe:02:ea | 169.254.2.234  | 255.255.0.0   | 169.254.0.1   | Ip4
  | NULL            |        202 | Static | Reserved | Start    |
ControlNetworkGuru        | 6591a371-15e3-4396-b14e-d23de671bdaa |
0 | 2013-09-24 14:54:33 | NULL           | NULL        |           0 |
SecondaryStorageVm | 2013-09-23 12:06:25 | NULL    | NULL        | NULL
|
|  3 | 8c1c909c-82fb-4786-b1e6-efc9e3098d51 |           1 |
06:17:8c:00:00:04 | 192.168.0.11   | 255.255.255.0 | 192.168.0.1   | Ip4
  | NULL            |        201 | Static | Reserved | Start    |
PodBasedNetworkGuru       | 6591a371-15e3-4396-b14e-d23de671bdaa |
1 | 2013-09-24 14:54:34 | NULL           | NULL        |           0 |
SecondaryStorageVm | 2013-09-23 12:06:25 | NULL    | NULL        | NULL
|
|  4 | 0f440319-18b5-4129-9ff9-37088f794107 |           1 |
06:61:fc:00:00:08 | 192.168.0.15   | 255.255.255.0 | 192.168.0.1   | Ip4
  | NULL            |        203 | Static | Reserved | Start    |
StorageNetworkGuru        | 6591a371-15e3-4396-b14e-d23de671bdaa |
3 | 2013-09-24 14:54:34 | NULL           | NULL        |           0 |
SecondaryStorageVm | 2013-09-23 12:06:25 | NULL    | NULL        | NULL
|
+----+--------------------------------------+-------------+-------------------+----------------+---------------+---------------+---------+-----------------+------------+--------+----------+----------+---------------------------+--------------------------------------+-----------+---------------------+----------------+-------------+-------------+--------------------+---------------------+---------+-------------+----------+

mysql> SELECT * FROM cloud.vm_instance where (name like "s-%" or name like
"v-%");
+----+--------+--------------------------------------+---------------+---------+----------------+-------------+---------------------+--------------------+--------+----------------+---------+--------------+----------+-------------------+----------------------------------------------+------------+---------------+--------------+---------------------+---------------------+---------+--------------------+--------------------+------------+-----------+---------------------+--------------------------------------+-----------------+------------------+------+------+-------+-------+-----------+--------------+---------------+
| id | name   | uuid                                 | instance_name |
state   | vm_template_id | guest_os_id | private_mac_address |
private_ip_address | pod_id | data_center_id | host_id | last_host_id |
proxy_id | proxy_assign_time | vnc_password
| ha_enabled | limit_cpu_use | update_count | update_time         | created
            | removed | type               | vm_type            |
account_id | domain_id | service_offering_id | reservation_id
        | hypervisor_type | disk_offering_id | cpu  | ram  | owner | speed
| host_name | display_name | desired_state |
+----+--------+--------------------------------------+---------------+---------+----------------+-------------+---------------------+--------------------+--------+----------------+---------+--------------+----------+-------------------+----------------------------------------------+------------+---------------+--------------+---------------------+---------------------+---------+--------------------+--------------------+------------+-----------+---------------------+--------------------------------------+-----------------+------------------+------+------+-------+-------+-----------+--------------+---------------+
|  1 | s-1-VM | 2cb5f8db-91eb-4f16-a6fa-4bfed79c51b6 | s-1-VM        |
Running |              1 |         133 | 06:17:8c:00:00:04   | 192.168.0.11
      |      1 |              1 |       1 |            1 |     NULL | NULL
             | gGI1kYniavKLLwWWHO1/8oL62JDMUdEbsSpj48XfVCo= |          0 |
            0 |           13 | 2013-09-24 13:55:14 | 2013-09-23 12:06:25 |
NULL    | SecondaryStorageVm | SecondaryStorageVm |          1 |         1
|                   9 | 6591a371-15e3-4396-b14e-d23de671bdaa | XenServer
    |             NULL | NULL | NULL | NULL  |  NULL | NULL      | NULL
    | NULL          |
|  2 | v-2-VM | c1975f1c-300f-4a6d-879f-2b70b4842a8f | v-2-VM        |
Stopped |              1 |         133 | 06:c4:62:00:00:03   | 192.168.0.10
      |      1 |              1 |    NULL |            1 |     NULL | NULL
             | wRV0PYKhMMKR7ENEvJ212Zi3FP//FV1eGmTfIkrkLX4= |          0 |
            0 |          371 | 2013-09-24 14:03:35 | 2013-09-23 12:06:26 |
NULL    | ConsoleProxy       | ConsoleProxy       |          1 |         1
|                  10 | 1dea74ec-ee0c-4b6c-bc62-43093572f0f1 | XenServer
    |             NULL | NULL | NULL | NULL  |  NULL | NULL      | NULL
    | NULL          |
+----+--------+--------------------------------------+---------------+---------+----------------+-------------+---------------------+--------------------+--------+----------------+---------+--------------+----------+-------------------+----------------------------------------------+------------+---------------+--------------+---------------------+---------------------+---------+--------------------+--------------------+------------+-----------+---------------------+--------------------------------------+-----------------+------------------+------+------+-------+-------+-----------+--------------+---------------+

The expunge interval and delay is setup to 60 seconds.

Thanks,

Daniel




On Tue, Sep 24, 2013 at 12:34 AM, Musayev, Ilya <imusayev@webmd.net> wrote:

> I suspect the issue with IPs not being released right away.
>
> What do you get for running this query
>
> SELECT * FROM cloud.nics where state="Reserved" and
> (vm_type="ConsoleProxy" or vm_type="SecondaryStorageVM");
>
> SELECT * FROM cloud.vm_instance where (name like "s-%" or name like "v-%");
>
> If its set to expunging, what is the expunge interval set to in global
> settings?
>
>
> > -----Original Message-----
> > From: Daniel Hertanu [mailto:dart@unixmob.com]
> > Sent: Monday, September 23, 2013 10:08 AM
> > To: users@cloudstack.apache.org
> > Subject: Management IP address pool exhausted
> >
> > Hi
> >
> > I'm running CS 4.1.1 on CentOS 6.4 64 bit with one host Citrix Xenserver
> 6.1.
> >
> > Few days ago I ran into a problem while I was configuring CS. The zone
> was
> > active, Xen host added, SSVM and CPVM up & running  when someone from
> > our team rebooted the Xen host for whatever reason. Even though
> > CloudStack was complaining about the missing host in the logs, it
> deleted the
> > old SSVM and tried to deploy another one, it failed, then another one,
> and so
> > on, until the management IP addresses pool was exhausted, moment when
> > CloudStack reported the new error - that there are not enough management
> > IPs. The host came back online later but the situation didn't changed. I
> > lowered the expunge.delay and expunge.interval to 5 minutes, still no
> > change.
> > I wiped out CS installation and I start all over again. I was curious
> about the
> > previous issue, so I manually rebooted the Xen host (without putting it
> in
> > Maintenance mode in CS) and the issue showed up again.
> > While I admit that having one host is not ideal, I have two questions:
> > - why would CS try to deploy any VM when it actually knows there are no
> > valid hosts
> > - how do I make those management IPs available again after CS tried to
> > deploy 10-20 SSVMs and it exhausted the management IPs pool?
> >
> > Thanks,
> >
> > Daniel
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message