cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Özhan Rüzgar Karaman <oruzgarkara...@gmail.com>
Subject Re: [4.11] VR Problem on Releasing Expunged Instance IP from dnsmasq.leases file
Date Wed, 24 Jan 2018 06:40:18 GMT
Hi Rohit;
Today i am short in time but tomorrow i will create a new network and test
your fix over this fresh VR.

I have one more question, with your current code fix do we still continue
reloading dnsmasq on normal operations and we only flush leases on
start/restart operations or after this fix we start to use restart the
dnsmasq instead of reloading it on all our operations? Thanks for all your
help.

Özhan

On Tue, Jan 23, 2018 at 11:45 PM, Rohit Yadav <rohit.yadav@shapeblue.com>
wrote:

> Hi Ozhan,
>
>
> During the 4.11-systemvmtemplate migration work (to debian9 based
> template), I refactored the code to reload dnsmasq instead of restart it.
> Based on your feedback, I've created a fix that will remove the leases file
> everytime dnsmasq needs to be restarted.
>
>
> Can you help test/verify it:
>
> https://github.com/apache/cloudstack/pull/2427/files
>
>
>
> - Rohit
>
> <https://cloudstack.apache.org>
>
>
>
> ________________________________
> From: Özhan Rüzgar Karaman <oruzgarkaraman@gmail.com>
> Sent: Tuesday, January 23, 2018 1:01:16 PM
> To: dev@cloudstack.apache.org
> Subject: Re: [4.11] VR Problem on Releasing Expunged Instance IP from
> dnsmasq.leases file
>
> Hi;
> We made some more tests to find the root cause of the problem. The problem
> occur because we crashed(power off) VR for a 4.11 HA test. After rebooting
> VR, active VM's dhcp lease datas are stucked
> in /var/lib/misc/dnsmasq.leases file and this creates problems.
>
> Our solution is to clear/flush dnsmasq file using dhcp_release command.
> After dhcp_release command problem disappears for new VMs with same ip
> address.
>
> So i think we need to add some code on VR startup to flush dnsmasq.leases
> file before starting dnsmasq.
>
> Thanks
> Özhan
>
> On Mon, Jan 22, 2018 at 1:13 PM, Özhan Rüzgar Karaman <
> oruzgarkaraman@gmail.com> wrote:
>
> > Hi Ivan;
> > I am not sure PR 2393 directly points to my findings, i only tested this
> > scenario on 4.11rc1.
> >
> > I am not a developer so i will not submit a fix, i am only testing 4.11rc
> > because its a LTS release and its quality is very important.
> >
> > Please check the issue on your environment, all details and issue
> > reproducing steps are written on my first email, but if you want i will
> > create a PR only to report & record the situation, just send me message
> if
> > you want.
> >
> > Thanks
> > Özhan
> >
> > On Mon, Jan 22, 2018 at 1:01 PM, Ivan Kudryavtsev <
> > kudryavtsev_ia@bw-sw.com> wrote:
> >
> >> Reopen issue, do a PR with fix if you can, could it be that VR doesn't
> >> have
> >> patched code? Also, describe testing scenario, I'll try to look at it in
> >> my
> >> patched 4.10.
> >>
> >> 22 янв. 2018 г. 16:52 пользователь "Özhan Rüzgar Karaman"
<
> >> oruzgarkaraman@gmail.com> написал:
> >>
> >> > Hi Ivan;
> >> > I checked 2 PR's and they are exist on 4.11rc1 but issue still exists
> >> on my
> >> > environment. When a new vm uses IP from old expunged vm then leases
> file
> >> > creates problem. Please check the logs that i submitted on first
> email,
> >> > issue is clear there and in my opinion it still exists on 4.11rc1.
> >> >
> >> > By the way 2393 is about VM's IP Changing progress, maybe it does not
> >> cover
> >> > my scenario.
> >> >
> >> > Thanks
> >> > Özhan
> >> >
> >> > On Mon, Jan 22, 2018 at 12:40 PM, Özhan Rüzgar Karaman <
> >> > oruzgarkaraman@gmail.com> wrote:
> >> >
> >> > > Hi Ivan;
> >> > > I made several tests with same scenario on 4.11rc1 and got same
> >> results,
> >> > > did your 2 PR's currently exists on 4.11 rc1 in which i am testing
> or
> >> it
> >> > > will exist on future rc2? If they exists on 4.11rc1 then we have a
> >> > problem
> >> > >
> >> > > Thanks
> >> > > Özhan
> >> > >
> >> > > On Mon, Jan 22, 2018 at 12:32 PM, Ivan Kudryavtsev <
> >> > > kudryavtsev_ia@bw-sw.com> wrote:
> >> > >
> >> > >> Hi, Ozhan. MACs are not removed upon vm removal, but they are
> >> overriden
> >> > >> upon vm creation with same ip (or same hostname). It should work
> >> fine,
> >> > >> 4.10, 4.11 received 2 PRs to fix several possible bugs. I tested
> the
> >> > case
> >> > >> when IP is reused.
> >> > >>
> >> > >> 22 янв. 2018 г. 16:07 пользователь "Özhan Rüzgar
Karaman" <
> >> > >> oruzgarkaraman@gmail.com> написал:
> >> > >>
> >> > >> Hi;
> >> > >> Today we noticed that one of our new provisioned instance did
not
> >> get IP
> >> > >> from VR. When we dig into the issue we find that one different
mac
> is
> >> > >> written in dnsmasq.leases file holds new instances IP address.
> >> > >>
> >> > >> We checked this mac address from db and we noticed that this mac
is
> >> used
> >> > >> for old expunged instance.
> >> > >>
> >> > >> So from this point we realised that when we destroy an instance
its
> >> mac
> >> > >> did
> >> > >> not removed from dnsmasq.leases file so if we use this ip for
a new
> >> > >> instance then we have a problem, our instance could not get IP
from
> >> VR.
> >> > >>
> >> > >> We have one host on our lab environment and its Ubuntu 16.04.3
KVM.
> >> > Today
> >> > >> we made a HA test and we crashed the host so VR and SystemVM's
are
> >> > >> rebooted
> >> > >> after we boot host back. I do not think this issue is related
to VR
> >> > reboot
> >> > >> but i like to give information about our environment.
> >> > >>
> >> > >> We need to manage dnsmasq.leases file when we expunge an instance.
> >> > >>
> >> > >> Thanks
> >> > >> Özhan
> >> > >>
> >> > >> Logs are below:
> >> > >>
> >> > >> root@r-4-VM:/var/lib/misc# tail -4 /var/log/dnsmasq.log
> >> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: not using configured address
> >> > >> 192.168.18.186 because it is leased to 1e:00:25:00:00:b9
> >> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0)
> >> 1e:00:00:00:00:b9
> >> > no
> >> > >> address available
> >> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: not using configured address
> >> > >> 192.168.18.187 because it is leased to 1e:00:80:00:00:ba
> >> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0)
> >> 1e:00:9f:00:00:ba
> >> > no
> >> > >> address available
> >> > >>
> >> > >> root@r-4-VM:/var/lib/misc# cat /etc/dhcphosts.txt
> >> > >> 1e:00:9f:00:00:ba,192.168.18.187,test411rc1mac,736h
> >> > >> 1e:00:00:00:00:b9,192.168.18.186,sil3sameip,733h
> >> > >> 1e:00:96:00:00:bf,192.168.18.192,TolgaTest02,707h
> >> > >> 1e:00:90:00:00:bc,192.168.18.189,TolgaTest,758h
> >> > >> 1e:00:40:00:00:bb,192.168.18.188,test411rc1,750h
> >> > >> root@r-4-VM:/var/lib/misc# cat /var/lib/misc/dnsmasq.leases
> >> > >> 1519339727 1e:00:25:00:00:b9 192.168.18.186 sil3sameip *
> >> > >> 1519331409 1e:00:40:00:00:bb 192.168.18.188 test411rc1 *
> >> > >> 1518921177 1e:00:80:00:00:ba 192.168.18.187 test411rc1mac *
> >> > >> 1518904275 1e:00:90:00:00:bc 192.168.18.189 TolgaTest *
> >> > >> 1519023297 1e:00:96:00:00:bf 192.168.18.192 TolgaTest02 *
> >> > >>
> >> > >> mysql> select name,state,private_mac_address,private_ip_address
> from
> >> > >> vm_instance;
> >> > >> +---------------+-----------+---------------------+---------
> >> > -----------+
> >> > >> | name          | state     | private_mac_address |
> >> private_ip_address |
> >> > >> +---------------+-----------+---------------------+---------
> >> > -----------+
> >> > >> | s-1-VM        | Running   | 1e:00:34:00:01:00   | 172.16.50.143
> >>   |
> >> > >> | v-2-VM        | Running   | 1e:00:81:00:01:03   | 172.16.50.146
> >>   |
> >> > >> | Tolga         | Expunging | 1e:00:50:00:00:bc   | 192.168.18.189
> >>    |
> >> > >> | r-4-VM        | Running   | 0e:00:a9:fe:03:0e   | 169.254.3.14
> >>    |
> >> > >> | Tolga02       | Expunging | 1e:00:b4:00:00:bf   | 192.168.18.192
> >>    |
> >> > >> | Tolga03       | Expunging | 1e:00:99:00:00:bb   | 192.168.18.188
> >>    |
> >> > >> | deneme        | Expunging | 1e:00:80:00:00:ba   | 192.168.18.187
> >>    |
> >> > >> | snpvmtolga02  | Expunging | 1e:00:69:00:00:b9   | 192.168.18.186
> >>    |
> >> > >> | TolgaTest     | Stopped   | 1e:00:90:00:00:bc   | 192.168.18.189
> >>    |
> >> > >> | TolgaTest02   | Stopped   | 1e:00:96:00:00:bf   | 192.168.18.192
> >>    |
> >> > >> | test411rc1    | Running   | 1e:00:40:00:00:bb   | 192.168.18.188
> >>    |
> >> > >> | test411rc1mac | Running   | 1e:00:9f:00:00:ba   | 192.168.18.187
> >>    |
> >> > >> | sil1          | Expunging | 1e:00:25:00:00:b9   | 192.168.18.186
> >>    |
> >> > >> | sil2sameip    | Expunging | 1e:00:14:00:00:b9   | 192.168.18.186
> >>    |
> >> > >> | sil3sameip    | Running   | 1e:00:00:00:00:b9   | 192.168.18.186
> >>    |
> >> > >> +---------------+-----------+---------------------+---------
> >> > -----------+
> >> > >> 15 rows in set (0.00 sec)
> >> > >>
> >> > >
> >> > >
> >> >
> >>
> >
> >
>
> rohit.yadav@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message