cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Özhan Rüzgar Karaman <oruzgarkara...@gmail.com>
Subject Re: [4.11] VR Problem on Releasing Expunged Instance IP from dnsmasq.leases file
Date Tue, 23 Jan 2018 12:01:16 GMT
Hi;
We made some more tests to find the root cause of the problem. The problem
occur because we crashed(power off) VR for a 4.11 HA test. After rebooting
VR, active VM's dhcp lease datas are stucked
in /var/lib/misc/dnsmasq.leases file and this creates problems.

Our solution is to clear/flush dnsmasq file using dhcp_release command.
After dhcp_release command problem disappears for new VMs with same ip
address.

So i think we need to add some code on VR startup to flush dnsmasq.leases
file before starting dnsmasq.

Thanks
Özhan

On Mon, Jan 22, 2018 at 1:13 PM, Özhan Rüzgar Karaman <
oruzgarkaraman@gmail.com> wrote:

> Hi Ivan;
> I am not sure PR 2393 directly points to my findings, i only tested this
> scenario on 4.11rc1.
>
> I am not a developer so i will not submit a fix, i am only testing 4.11rc
> because its a LTS release and its quality is very important.
>
> Please check the issue on your environment, all details and issue
> reproducing steps are written on my first email, but if you want i will
> create a PR only to report & record the situation, just send me message if
> you want.
>
> Thanks
> Özhan
>
> On Mon, Jan 22, 2018 at 1:01 PM, Ivan Kudryavtsev <
> kudryavtsev_ia@bw-sw.com> wrote:
>
>> Reopen issue, do a PR with fix if you can, could it be that VR doesn't
>> have
>> patched code? Also, describe testing scenario, I'll try to look at it in
>> my
>> patched 4.10.
>>
>> 22 янв. 2018 г. 16:52 пользователь "Özhan Rüzgar Karaman" <
>> oruzgarkaraman@gmail.com> написал:
>>
>> > Hi Ivan;
>> > I checked 2 PR's and they are exist on 4.11rc1 but issue still exists
>> on my
>> > environment. When a new vm uses IP from old expunged vm then leases file
>> > creates problem. Please check the logs that i submitted on first email,
>> > issue is clear there and in my opinion it still exists on 4.11rc1.
>> >
>> > By the way 2393 is about VM's IP Changing progress, maybe it does not
>> cover
>> > my scenario.
>> >
>> > Thanks
>> > Özhan
>> >
>> > On Mon, Jan 22, 2018 at 12:40 PM, Özhan Rüzgar Karaman <
>> > oruzgarkaraman@gmail.com> wrote:
>> >
>> > > Hi Ivan;
>> > > I made several tests with same scenario on 4.11rc1 and got same
>> results,
>> > > did your 2 PR's currently exists on 4.11 rc1 in which i am testing or
>> it
>> > > will exist on future rc2? If they exists on 4.11rc1 then we have a
>> > problem
>> > >
>> > > Thanks
>> > > Özhan
>> > >
>> > > On Mon, Jan 22, 2018 at 12:32 PM, Ivan Kudryavtsev <
>> > > kudryavtsev_ia@bw-sw.com> wrote:
>> > >
>> > >> Hi, Ozhan. MACs are not removed upon vm removal, but they are
>> overriden
>> > >> upon vm creation with same ip (or same hostname). It should work
>> fine,
>> > >> 4.10, 4.11 received 2 PRs to fix several possible bugs. I tested the
>> > case
>> > >> when IP is reused.
>> > >>
>> > >> 22 янв. 2018 г. 16:07 пользователь "Özhan Rüzgar
Karaman" <
>> > >> oruzgarkaraman@gmail.com> написал:
>> > >>
>> > >> Hi;
>> > >> Today we noticed that one of our new provisioned instance did not
>> get IP
>> > >> from VR. When we dig into the issue we find that one different mac
is
>> > >> written in dnsmasq.leases file holds new instances IP address.
>> > >>
>> > >> We checked this mac address from db and we noticed that this mac is
>> used
>> > >> for old expunged instance.
>> > >>
>> > >> So from this point we realised that when we destroy an instance its
>> mac
>> > >> did
>> > >> not removed from dnsmasq.leases file so if we use this ip for a new
>> > >> instance then we have a problem, our instance could not get IP from
>> VR.
>> > >>
>> > >> We have one host on our lab environment and its Ubuntu 16.04.3 KVM.
>> > Today
>> > >> we made a HA test and we crashed the host so VR and SystemVM's are
>> > >> rebooted
>> > >> after we boot host back. I do not think this issue is related to VR
>> > reboot
>> > >> but i like to give information about our environment.
>> > >>
>> > >> We need to manage dnsmasq.leases file when we expunge an instance.
>> > >>
>> > >> Thanks
>> > >> Özhan
>> > >>
>> > >> Logs are below:
>> > >>
>> > >> root@r-4-VM:/var/lib/misc# tail -4 /var/log/dnsmasq.log
>> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: not using configured address
>> > >> 192.168.18.186 because it is leased to 1e:00:25:00:00:b9
>> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0)
>> 1e:00:00:00:00:b9
>> > no
>> > >> address available
>> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: not using configured address
>> > >> 192.168.18.187 because it is leased to 1e:00:80:00:00:ba
>> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0)
>> 1e:00:9f:00:00:ba
>> > no
>> > >> address available
>> > >>
>> > >> root@r-4-VM:/var/lib/misc# cat /etc/dhcphosts.txt
>> > >> 1e:00:9f:00:00:ba,192.168.18.187,test411rc1mac,736h
>> > >> 1e:00:00:00:00:b9,192.168.18.186,sil3sameip,733h
>> > >> 1e:00:96:00:00:bf,192.168.18.192,TolgaTest02,707h
>> > >> 1e:00:90:00:00:bc,192.168.18.189,TolgaTest,758h
>> > >> 1e:00:40:00:00:bb,192.168.18.188,test411rc1,750h
>> > >> root@r-4-VM:/var/lib/misc# cat /var/lib/misc/dnsmasq.leases
>> > >> 1519339727 1e:00:25:00:00:b9 192.168.18.186 sil3sameip *
>> > >> 1519331409 1e:00:40:00:00:bb 192.168.18.188 test411rc1 *
>> > >> 1518921177 1e:00:80:00:00:ba 192.168.18.187 test411rc1mac *
>> > >> 1518904275 1e:00:90:00:00:bc 192.168.18.189 TolgaTest *
>> > >> 1519023297 1e:00:96:00:00:bf 192.168.18.192 TolgaTest02 *
>> > >>
>> > >> mysql> select name,state,private_mac_address,private_ip_address
from
>> > >> vm_instance;
>> > >> +---------------+-----------+---------------------+---------
>> > -----------+
>> > >> | name          | state     | private_mac_address |
>> private_ip_address |
>> > >> +---------------+-----------+---------------------+---------
>> > -----------+
>> > >> | s-1-VM        | Running   | 1e:00:34:00:01:00   | 172.16.50.143
>>   |
>> > >> | v-2-VM        | Running   | 1e:00:81:00:01:03   | 172.16.50.146
>>   |
>> > >> | Tolga         | Expunging | 1e:00:50:00:00:bc   | 192.168.18.189
>>    |
>> > >> | r-4-VM        | Running   | 0e:00:a9:fe:03:0e   | 169.254.3.14
>>    |
>> > >> | Tolga02       | Expunging | 1e:00:b4:00:00:bf   | 192.168.18.192
>>    |
>> > >> | Tolga03       | Expunging | 1e:00:99:00:00:bb   | 192.168.18.188
>>    |
>> > >> | deneme        | Expunging | 1e:00:80:00:00:ba   | 192.168.18.187
>>    |
>> > >> | snpvmtolga02  | Expunging | 1e:00:69:00:00:b9   | 192.168.18.186
>>    |
>> > >> | TolgaTest     | Stopped   | 1e:00:90:00:00:bc   | 192.168.18.189
>>    |
>> > >> | TolgaTest02   | Stopped   | 1e:00:96:00:00:bf   | 192.168.18.192
>>    |
>> > >> | test411rc1    | Running   | 1e:00:40:00:00:bb   | 192.168.18.188
>>    |
>> > >> | test411rc1mac | Running   | 1e:00:9f:00:00:ba   | 192.168.18.187
>>    |
>> > >> | sil1          | Expunging | 1e:00:25:00:00:b9   | 192.168.18.186
>>    |
>> > >> | sil2sameip    | Expunging | 1e:00:14:00:00:b9   | 192.168.18.186
>>    |
>> > >> | sil3sameip    | Running   | 1e:00:00:00:00:b9   | 192.168.18.186
>>    |
>> > >> +---------------+-----------+---------------------+---------
>> > -----------+
>> > >> 15 rows in set (0.00 sec)
>> > >>
>> > >
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message