cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From John Skinner <john.skin...@appcore.com>
Subject Re: CS 4.3.2 virtual router issue
Date Fri, 27 Feb 2015 18:15:17 GMT
There is nothing useful in those logs. As I mentioned, CloudStack is only
requesting the 3 interfaces. It has no idea about the 4th. I need to get a
chance to walk through the code to see what CloudStack passes to libvirt to
fire the VM off. I am thinking there may be a bug some where in the
communication between CloudStack and libvirt as this environment is running
Ubuntu 14.04 and the latest stable libvirt.

However, to satisfy your request. Here is cmdfile:

template=domP name=r-11453-VM eth2ip=X.X.X.X eth2mask=255.255.255.192 gatee

way=X.X.X.X eth0ip=10.1.2.1 eth0mask=255.255.255.0 domain=cs2cloud.internaa

l cidrsize=24 dhcprange=10.1.2.1 eth1ip=169.254.0.53 eth1mask=255.255.0.0
type=rr

outer disable_rp_filter=true dns1=X.X.X.X dns2=X.X.X.X


cloud.out:

Thu Feb 26 20:37:58 UTC 2015 Executing cloud-early-config

Thu Feb 26 20:37:58 UTC 2015 Detected that we are running inside kvm guest

Thu Feb 26 20:37:58 UTC 2015 Patching  scripts
oldmd5=160745b757a94262072104caaff

e686a5 newmd5=bbe4d9d7f343e5299e0730fad16d6b94

Thu Feb 26 20:37:58 UTC 2015 Patching  cloud service

Thu Feb 26 20:37:59 UTC 2015 Rebooting system since we patched init scripts

Thu Feb 26 20:38:01 UTC 2015 Updating log4j-cloud.xml

Thu Feb 26 20:38:02 UTC 2015 Setting up virtual router system vm

Thu Feb 26 20:38:02 UTC 2015 checking that eth0 has IP

Thu Feb 26 20:38:02 UTC 2015 checking that eth1 has IP

Thu Feb 26 20:38:02 UTC 2015 checking that eth2 has IP

Thu Feb 26 20:38:13 UTC 2015 Executing cloud-early-config

Thu Feb 26 20:38:13 UTC 2015 Detected that we are running inside kvm guest

Thu Feb 26 20:38:13 UTC 2015 Patching  cloud service

Thu Feb 26 20:38:14 UTC 2015 Updating log4j-cloud.xml

Thu Feb 26 20:38:14 UTC 2015 Setting up virtual router system vm

Thu Feb 26 20:38:14 UTC 2015 checking that eth0 has IP

Thu Feb 26 20:38:15 UTC 2015 waiting for eth0 interface setup with ip
timer=0

Thu Feb 26 20:38:15 UTC 2015 checking that eth1 has IP

Thu Feb 26 20:38:15 UTC 2015 checking that eth2 has IP

Thu Feb 26 20:38:18 UTC 2015 Checking udev NIC assignment order changes

Thu Feb 26 20:38:18 UTC 2015 Setting up dnsmasq

Thu Feb 26 20:38:18 UTC 2015 Setting up apache web server

Thu Feb 26 20:38:18 UTC 2015 Enable service dnsmasq = 1

Thu Feb 26 20:38:18 UTC 2015 Enable service haproxy = 1

Thu Feb 26 20:38:18 UTC 2015 Processors = 1  Enable service  = 0

Thu Feb 26 20:38:18 UTC 2015 Enable service cloud-passwd-srvr = 1

Thu Feb 26 20:38:18 UTC 2015 Enable service cloud = 0

Thu Feb 26 20:38:18 UTC 2015 cloud: Tuning rp_filter on public interfaces

Thu Feb 26 20:38:18 UTC 2015 rpfilter public interfaces :  eth2

Thu Feb 26 20:38:18 UTC 2015 cloud: disable rp_filter on public interfaces

Thu Feb 26 20:38:18 UTC 2015 cloud: disable rp_filter on public interface:
eth2

Thu Feb 26 20:38:18 UTC 2015 cloud: Enabling rp_filter on Non-public
interfaces((

eth0,eth1,lo)

Thu Feb 26 20:38:18 UTC 2015 cloud: enable_fwding = 1

Thu Feb 26 20:38:18 UTC 2015 enable_fwding = 1


As I said, from the output here, everything is fine. the ipassoc.sh script
is actually responsible for breaking NAT by assigning the postrouting rules
to the eth3 interface which shouldnt exist. The broken part of this is on
VM creation, not on boot or post boot.


Paul, I did confirm that the template we have in the other zone is the same
one as we are running in this zone. We are not seeing this issue in the
other zone. The difference are Ubuntu 14.04 vs CentOS 6.4 and Open vSwitch
vs Linux Bridging.

On Fri, Feb 27, 2015 at 10:59 AM, Jayapal Reddy Uradi <
jayapalreddy.uradi@citrix.com> wrote:

> How system vm template update solved the problem.
> Is VR getting more interfaces from the template ?
>
> Thanks,
> Jayapal
> On 27-Feb-2015, at 9:19 PM, Paul Angus <paul.angus@shapeblue.com>
>  wrote:
>
> > I have seen this behaviour in 4.3.2 - I ended up with at least 5
> interfaces on my VRs.  I resolved it by updating my systemVM template as I
> recall.
> >
> >
> > Regards
> >
> > Paul Angus
> > Cloud Architect
> > S: +44 20 3603 0540 | M: +447711418784 | T: CloudyAngus
> > paul.angus@shapeblue.com
> >
> > -----Original Message-----
> > From: Jayapal Reddy Uradi [mailto:jayapalreddy.uradi@citrix.com]
> > Sent: 27 February 2015 03:42
> > To: <users@cloudstack.apache.org>
> > Subject: Re: CS 4.3.2 virtual router issue
> >
> >
> > As previously pointed out, Can you please send VR start command,
> /var/cache/cloud/cmdline file in VR also /var/log/cloud.log.
> > These logs helps us to understand where the problem is.
> >
> > Thanks,
> > Jayapal
> >
> > On 27-Feb-2015, at 2:54 AM, John Skinner <john.skinner@appcore.com>
> wrote:
> >
> >> This happens with all routers in that specific zone, the other zone is
> ok.
> >> Messages and cloudstack logs on the router do not offer any insight.
> >> You can see the results of ipassoc.sh in messages and that it is
> >> clearly setting up postrouting rules for eth3 which should not exist.
> >> CloudStack agent log on the host is useless per always, and the qemu
> >> log for the routers just display the start command. Virsh dumpxml of
> >> the router does show 4 interfaces for the router. I am trying to
> >> discover at what point the VM thinks it needs to create that 4th
> >> interface. User VMs are being created just fine, as well as the
> >> console proxy and secondary storage VMs for that zone.
> >>
> >> On Thu, Feb 26, 2015 at 2:29 PM, Somesh Naidu
> >> <Somesh.Naidu@citrix.com>
> >> wrote:
> >>
> >>> Strange, I haven't seen that happen before.
> >>>
> >>> Is this happening with all routers in the environment or only a
> >>> specific one?
> >>>
> >>> If latter we might just try recreating it. If former, then probably
> >>> have to look at /var/log/messages on the router.
> >>>
> >>> Somesh
> >>> CloudPlatform Escalations
> >>> Citrix Systems, Inc.
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: John Skinner [mailto:john.skinner@appcore.com]
> >>> Sent: Thursday, February 26, 2015 3:19 PM
> >>> To: users@cloudstack.apache.org
> >>> Subject: Re: CS 4.3.2 virtual router issue
> >>>
> >>> Yeah. As far as CloudStack is concerned, it has 3 NICs - the UI and
> >>> the database confirms. The API call also only calls for 3 NICs. How
> >>> this 4th NIC is getting created is a mystery. It has the same IP as
> >>> the 3rd NIC, it doesn't make any sense.
> >>>
> >>> On Thu, Feb 26, 2015 at 2:16 PM, Somesh Naidu
> >>> <Somesh.Naidu@citrix.com>
> >>> wrote:
> >>>
> >>>> Did you check the entries in the nics table for that router? That
> >>>> might give you some pointers.
> >>>>
> >>>> Somesh
> >>>> CloudPlatform Escalations
> >>>> Citrix Systems, Inc.
> >>>>
> >>>> -----Original Message-----
> >>>> From: John Skinner [mailto:john.skinner@appcore.com]
> >>>> Sent: Thursday, February 26, 2015 10:59 AM
> >>>> To: users@cloudstack.apache.org
> >>>> Subject: CS 4.3.2 virtual router issue
> >>>>
> >>>> Running CloudStack 4.3.2 with advanced networking using Open vSwitch
> >>>> on KVM. I am having an issue where my virtual router is coming up
> >>>> with 2 public interfaces when there should only be 1. CloudStack is
> >>>> only requesting the 3 interfaces (private, control, and public) but
> >>>> the VM is coming up with 4. The second public interface comes up
> >>>> with the same IP
> >>> as
> >>>> the first public interface. This obviously really breaks routing and
> >>>> ip forwarding as the postrouting rules in iptables get created for
> >>>> the
> >>> second
> >>>> interface when the traffic is coming in on the 1st interface.
> >>>>
> >>>> Has any one seen anything like this? I can't figure out why it is
> >>>> getting created with 2 public interfaces.
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>>
> >>> *John Skinner*
> >>>
> >>> Senior Cloud Engineer - Ops Lead | Appcore
> >>>
> >>>
> >>> Office +1.800.735.7104 | Direct +1.515.612.7783
> >>>
> >>> john.skinner@appcore.com  |  www.appcore.com
> >>>
> >>> ---------------------------------------------------------------------
> >>> -
> >>>
> >>> The information in this message is intended for the named recipients
> only.
> >>> It may contain information that is privileged, confidential or
> >>> otherwise protected from disclosure. If you are not the intended
> >>> recipient, you are hereby notified that any disclosure, copying,
> >>> distribution, or the taking of any action in reliance on the contents
> >>> of this message is strictly prohibited. If you have received this
> >>> e-mail in error, do not print it or disseminate it or its contents.
> >>> In such event, please notify the sender by return e-mail and delete
> the e-mail file immediately thereafter. Thank you.
> >>>
> >>
> >>
> >>
> >> --
> >>
> >> *John Skinner*
> >>
> >> Senior Cloud Engineer - Ops Lead | Appcore
> >>
> >>
> >> Office +1.800.735.7104 | Direct +1.515.612.7783
> >>
> >> john.skinner@appcore.com  |  www.appcore.com
> >>
> >> ----------------------------------------------------------------------
> >>
> >> The information in this message is intended for the named recipients
> only.
> >> It may contain information that is privileged, confidential or
> >> otherwise protected from disclosure. If you are not the intended
> >> recipient, you are hereby notified that any disclosure, copying,
> >> distribution, or the taking of any action in reliance on the contents
> >> of this message is strictly prohibited. If you have received this
> >> e-mail in error, do not print it or disseminate it or its contents. In
> >> such event, please notify the sender by return e-mail and delete the
> e-mail file immediately thereafter. Thank you.
> >
> > Find out more about ShapeBlue and our range of CloudStack related
> services
> >
> > IaaS Cloud Design & Build<
> http://shapeblue.com/iaas-cloud-design-and-build//>
> > CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
> > CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
> > CloudStack Software Engineering<
> http://shapeblue.com/cloudstack-software-engineering/>
> > CloudStack Infrastructure Support<
> http://shapeblue.com/cloudstack-infrastructure-support/>
> > CloudStack Bootcamp Training Courses<
> http://shapeblue.com/cloudstack-training/>
> >
> > This email and any attachments to it may be confidential and are
> intended solely for the use of the individual to whom it is addressed. Any
> views or opinions expressed are solely those of the author and do not
> necessarily represent those of Shape Blue Ltd or related companies. If you
> are not the intended recipient of this email, you must neither take any
> action based upon its contents, nor copy or show it to anyone. Please
> contact the sender if you believe you have received this email in error.
> Shape Blue Ltd is a company incorporated in England & Wales. ShapeBlue
> Services India LLP is a company incorporated in India and is operated under
> license from Shape Blue Ltd. Shape Blue Brasil Consultoria Ltda is a
> company incorporated in Brasil and is operated under license from Shape
> Blue Ltd. ShapeBlue SA Pty Ltd is a company registered by The Republic of
> South Africa and is traded under license from Shape Blue Ltd. ShapeBlue is
> a registered trademark.
>
>


-- 

*John Skinner*

Senior Cloud Engineer - Ops Lead | Appcore


Office +1.800.735.7104 | Direct +1.515.612.7783

john.skinner@appcore.com  |  www.appcore.com

----------------------------------------------------------------------

The information in this message is intended for the named recipients only.
It may contain information that is privileged, confidential or otherwise
protected from disclosure. If you are not the intended recipient, you are
hereby notified that any disclosure, copying, distribution, or the taking
of any action in reliance on the contents of this message is strictly
prohibited. If you have received this e-mail in error, do not print it or
disseminate it or its contents. In such event, please notify the sender by
return e-mail and delete the e-mail file immediately thereafter. Thank you.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message