cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rohit Yadav <rohit.ya...@shapeblue.com>
Subject Re: [4.11] VR Problem on Releasing Expunged Instance IP from dnsmasq.leases file
Date Thu, 25 Jan 2018 14:21:08 GMT
Thanks Ozhan for confirming, the fix will make its way to RC2, which we'll hopefully cut by
EOD tomorrow.


- Rohit

<https://cloudstack.apache.org>



________________________________
From: Özhan Rüzgar Karaman <oruzgarkaraman@gmail.com>
Sent: Thursday, January 25, 2018 1:12:07 PM
To: dev@cloudstack.apache.org
Subject: Re: [4.11] VR Problem on Releasing Expunged Instance IP from dnsmasq.leases file

Hi Rohit;
I made the test again on a fresh VR and your solution fixed the issue.

Thanks
Özhan

On Wed, Jan 24, 2018 at 11:26 AM, Rohit Yadav <rohit.yadav@shapeblue.com>
wrote:

> Hi Ozhan,
>
>
> With my fix, whenever dnsmasq needs to be restarted the leases file will
> be removed, and dnsmasq will be restarted whenever /etc/dhcphosts.txt or
> /etc/dnsmasq.d/cloud.conf change, otherwise it will be reloaded.
>
>
> - Rohit
>
> <https://cloudstack.apache.org>
>
>
>
> ________________________________
> From: Özhan Rüzgar Karaman <oruzgarkaraman@gmail.com>
> Sent: Wednesday, January 24, 2018 7:40:18 AM
> To: dev@cloudstack.apache.org
> Subject: Re: [4.11] VR Problem on Releasing Expunged Instance IP from
> dnsmasq.leases file
>
> Hi Rohit;
> Today i am short in time but tomorrow i will create a new network and test
> your fix over this fresh VR.
>
> I have one more question, with your current code fix do we still continue
> reloading dnsmasq on normal operations and we only flush leases on
> start/restart operations or after this fix we start to use restart the
> dnsmasq instead of reloading it on all our operations? Thanks for all your
> help.
>
> Özhan
>
> On Tue, Jan 23, 2018 at 11:45 PM, Rohit Yadav <rohit.yadav@shapeblue.com>
> wrote:
>
> > Hi Ozhan,
> >
> >
> > During the 4.11-systemvmtemplate migration work (to debian9 based
> > template), I refactored the code to reload dnsmasq instead of restart it.
> > Based on your feedback, I've created a fix that will remove the leases
> file
> > everytime dnsmasq needs to be restarted.
> >
> >
> > Can you help test/verify it:
> >
> > https://github.com/apache/cloudstack/pull/2427/files
> >
> >
> >
> > - Rohit
> >
> > <https://cloudstack.apache.org>
> >
> >
> >
> > ________________________________
> > From: Özhan Rüzgar Karaman <oruzgarkaraman@gmail.com>
> > Sent: Tuesday, January 23, 2018 1:01:16 PM
> > To: dev@cloudstack.apache.org
> > Subject: Re: [4.11] VR Problem on Releasing Expunged Instance IP from
> > dnsmasq.leases file
> >
> > Hi;
> > We made some more tests to find the root cause of the problem. The
> problem
> > occur because we crashed(power off) VR for a 4.11 HA test. After
> rebooting
> > VR, active VM's dhcp lease datas are stucked
> > in /var/lib/misc/dnsmasq.leases file and this creates problems.
> >
> > Our solution is to clear/flush dnsmasq file using dhcp_release command.
> > After dhcp_release command problem disappears for new VMs with same ip
> > address.
> >
> > So i think we need to add some code on VR startup to flush dnsmasq.leases
> > file before starting dnsmasq.
> >
> > Thanks
> > Özhan
> >
> > On Mon, Jan 22, 2018 at 1:13 PM, Özhan Rüzgar Karaman <
> > oruzgarkaraman@gmail.com> wrote:
> >
> > > Hi Ivan;
> > > I am not sure PR 2393 directly points to my findings, i only tested
> this
> > > scenario on 4.11rc1.
> > >
> > > I am not a developer so i will not submit a fix, i am only testing
> 4.11rc
> > > because its a LTS release and its quality is very important.
> > >
> > > Please check the issue on your environment, all details and issue
> > > reproducing steps are written on my first email, but if you want i will
> > > create a PR only to report & record the situation, just send me message
> > if
> > > you want.
> > >
> > > Thanks
> > > Özhan
> > >
> > > On Mon, Jan 22, 2018 at 1:01 PM, Ivan Kudryavtsev <
> > > kudryavtsev_ia@bw-sw.com> wrote:
> > >
> > >> Reopen issue, do a PR with fix if you can, could it be that VR doesn't
> > >> have
> > >> patched code? Also, describe testing scenario, I'll try to look at it
> in
> > >> my
> > >> patched 4.10.
> > >>
> > >> 22 янв. 2018 г. 16:52 пользователь "Özhan Rüzgar Karaman"
<
> > >> oruzgarkaraman@gmail.com> написал:
> > >>
> > >> > Hi Ivan;
> > >> > I checked 2 PR's and they are exist on 4.11rc1 but issue still
> exists
> > >> on my
> > >> > environment. When a new vm uses IP from old expunged vm then leases
> > file
> > >> > creates problem. Please check the logs that i submitted on first
> > email,
> > >> > issue is clear there and in my opinion it still exists on 4.11rc1.
> > >> >
> > >> > By the way 2393 is about VM's IP Changing progress, maybe it does
> not
> > >> cover
> > >> > my scenario.
> > >> >
> > >> > Thanks
> > >> > Özhan
> > >> >
> > >> > On Mon, Jan 22, 2018 at 12:40 PM, Özhan Rüzgar Karaman <
> > >> > oruzgarkaraman@gmail.com> wrote:
> > >> >
> > >> > > Hi Ivan;
> > >> > > I made several tests with same scenario on 4.11rc1 and got same
> > >> results,
> > >> > > did your 2 PR's currently exists on 4.11 rc1 in which i am testing
> > or
> > >> it
> > >> > > will exist on future rc2? If they exists on 4.11rc1 then we have
a
> > >> > problem
> > >> > >
> > >> > > Thanks
> > >> > > Özhan
> > >> > >
> > >> > > On Mon, Jan 22, 2018 at 12:32 PM, Ivan Kudryavtsev <
> > >> > > kudryavtsev_ia@bw-sw.com> wrote:
> > >> > >
> > >> > >> Hi, Ozhan. MACs are not removed upon vm removal, but they
are
> > >> overriden
> > >> > >> upon vm creation with same ip (or same hostname). It should
work
> > >> fine,
> > >> > >> 4.10, 4.11 received 2 PRs to fix several possible bugs. I
tested
> > the
> > >> > case
> > >> > >> when IP is reused.
> > >> > >>
> > >> > >> 22 янв. 2018 г. 16:07 пользователь "Özhan
Rüzgar Karaman" <
> > >> > >> oruzgarkaraman@gmail.com> написал:
> > >> > >>
> > >> > >> Hi;
> > >> > >> Today we noticed that one of our new provisioned instance
did not
> > >> get IP
> > >> > >> from VR. When we dig into the issue we find that one different
> mac
> > is
> > >> > >> written in dnsmasq.leases file holds new instances IP address.
> > >> > >>
> > >> > >> We checked this mac address from db and we noticed that this
mac
> is
> > >> used
> > >> > >> for old expunged instance.
> > >> > >>
> > >> > >> So from this point we realised that when we destroy an instance
> its
> > >> mac
> > >> > >> did
> > >> > >> not removed from dnsmasq.leases file so if we use this ip
for a
> new
> > >> > >> instance then we have a problem, our instance could not get
IP
> from
> > >> VR.
> > >> > >>
> > >> > >> We have one host on our lab environment and its Ubuntu 16.04.3
> KVM.
> > >> > Today
> > >> > >> we made a HA test and we crashed the host so VR and SystemVM's
> are
> > >> > >> rebooted
> > >> > >> after we boot host back. I do not think this issue is related
to
> VR
> > >> > reboot
> > >> > >> but i like to give information about our environment.
> > >> > >>
> > >> > >> We need to manage dnsmasq.leases file when we expunge an
> instance.
> > >> > >>
> > >> > >> Thanks
> > >> > >> Özhan
> > >> > >>
> > >> > >> Logs are below:
> > >> > >>
> > >> > >> root@r-4-VM:/var/lib/misc# tail -4 /var/log/dnsmasq.log
> > >> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: not using configured address
> > >> > >> 192.168.18.186 because it is leased to 1e:00:25:00:00:b9
> > >> > >> Jan 22 08:57:27 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0)
> > >> 1e:00:00:00:00:b9
> > >> > no
> > >> > >> address available
> > >> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: not using configured address
> > >> > >> 192.168.18.187 because it is leased to 1e:00:80:00:00:ba
> > >> > >> Jan 22 08:57:29 dnsmasq-dhcp[850]: DHCPDISCOVER(eth0)
> > >> 1e:00:9f:00:00:ba
> > >> > no
> > >> > >> address available
> > >> > >>
> > >> > >> root@r-4-VM:/var/lib/misc# cat /etc/dhcphosts.txt
> > >> > >> 1e:00:9f:00:00:ba,192.168.18.187,test411rc1mac,736h
> > >> > >> 1e:00:00:00:00:b9,192.168.18.186,sil3sameip,733h
> > >> > >> 1e:00:96:00:00:bf,192.168.18.192,TolgaTest02,707h
> > >> > >> 1e:00:90:00:00:bc,192.168.18.189,TolgaTest,758h
> > >> > >> 1e:00:40:00:00:bb,192.168.18.188,test411rc1,750h
> > >> > >> root@r-4-VM:/var/lib/misc# cat /var/lib/misc/dnsmasq.leases
> > >> > >> 1519339727 1e:00:25:00:00:b9 192.168.18.186 sil3sameip *
> > >> > >> 1519331409 1e:00:40:00:00:bb 192.168.18.188 test411rc1 *
> > >> > >> 1518921177 1e:00:80:00:00:ba 192.168.18.187 test411rc1mac
*
> > >> > >> 1518904275 1e:00:90:00:00:bc 192.168.18.189 TolgaTest *
> > >> > >> 1519023297 1e:00:96:00:00:bf 192.168.18.192 TolgaTest02 *
> > >> > >>
> > >> > >> mysql> select name,state,private_mac_address,private_ip_address
> > from
> > >> > >> vm_instance;
> > >> > >> +---------------+-----------+---------------------+---------
> > >> > -----------+
> > >> > >> | name          | state     | private_mac_address |
> > >> private_ip_address |
> > >> > >> +---------------+-----------+---------------------+---------
> > >> > -----------+
> > >> > >> | s-1-VM        | Running   | 1e:00:34:00:01:00   | 172.16.50.143
> > >>   |
> > >> > >> | v-2-VM        | Running   | 1e:00:81:00:01:03   | 172.16.50.146
> > >>   |
> > >> > >> | Tolga         | Expunging | 1e:00:50:00:00:bc   |
> 192.168.18.189
> > >>    |
> > >> > >> | r-4-VM        | Running   | 0e:00:a9:fe:03:0e   | 169.254.3.14
> > >>    |
> > >> > >> | Tolga02       | Expunging | 1e:00:b4:00:00:bf   |
> 192.168.18.192
> > >>    |
> > >> > >> | Tolga03       | Expunging | 1e:00:99:00:00:bb   |
> 192.168.18.188
> > >>    |
> > >> > >> | deneme        | Expunging | 1e:00:80:00:00:ba   |
> 192.168.18.187
> > >>    |
> > >> > >> | snpvmtolga02  | Expunging | 1e:00:69:00:00:b9   |
> 192.168.18.186
> > >>    |
> > >> > >> | TolgaTest     | Stopped   | 1e:00:90:00:00:bc   |
> 192.168.18.189
> > >>    |
> > >> > >> | TolgaTest02   | Stopped   | 1e:00:96:00:00:bf   |
> 192.168.18.192
> > >>    |
> > >> > >> | test411rc1    | Running   | 1e:00:40:00:00:bb   |
> 192.168.18.188
> > >>    |
> > >> > >> | test411rc1mac | Running   | 1e:00:9f:00:00:ba   |
> 192.168.18.187
> > >>    |
> > >> > >> | sil1          | Expunging | 1e:00:25:00:00:b9   |
> 192.168.18.186
> > >>    |
> > >> > >> | sil2sameip    | Expunging | 1e:00:14:00:00:b9   |
> 192.168.18.186
> > >>    |
> > >> > >> | sil3sameip    | Running   | 1e:00:00:00:00:b9   |
> 192.168.18.186
> > >>    |
> > >> > >> +---------------+-----------+---------------------+---------
> > >> > -----------+
> > >> > >> 15 rows in set (0.00 sec)
> > >> > >>
> > >> > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
> > rohit.yadav@shapeblue.com
> > www.shapeblue.com<http://www.shapeblue.com>
> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
>
> rohit.yadav@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>

rohit.yadav@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message