cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrei Mikhailovsky <and...@arhont.com.INVALID>
Subject Re: VR routing issues in Advanced Mode
Date Fri, 23 Feb 2018 16:17:29 GMT
Bump.

Any ideas anyone? This issue is really annoying.

Thanks

----- Original Message -----
> From: "Andrei Mikhailovsky" <andrei@arhont.com.INVALID>
> To: "users" <users@cloudstack.apache.org>, "users" <users@cloudstack.apache.org>
> Sent: Wednesday, 21 February, 2018 22:10:25
> Subject: Re: VR routing issues in Advanced Mode

> Dag,
> 
> 
> 
> 
> Yeah, we have egress traffic enabled. We also use VPCs on some of the networks
> and they are also effected by this issue along with None VPC networks.
> 
> 
> 
> 
> Any thoughts?
> 
> 
> 
> 
> Andrei Mikhailovsky
> 
> 
> 
> 
> 
> 
> 
> From: Dag Sonstebo
> 
> 
> Sent: Wednesday 21 February, 18:30
> 
> 
> Subject: Re: VR routing issues in Advanced Mode
> 
> 
> To: users@cloudstack.apache.org
> 
> 
> 
> 
> 
> 
> Hi Andrei,
> 
> 
> 
> 
> Understand. To get all the obvious things out the way – have you allowed egress
> traffic on the two networks (you mention ACLs which we only use on VPCs and
> basic networks)?
> 
> 
> 
> 
> Regards,
> 
> 
> Dag Sonstebo
> 
> 
> Cloud Architect
> 
> 
> ShapeBlue
> 
> 
> 
> 
> On 21/02/2018, 14:51, "Andrei Mikhailovsky" <andrei@arhont.com.INVALID> wrote:
> 
> 
> 
> 
> Hi Dag,
> 
> 
> 
> 
> Please see my comments below:
> 
> 
> 
> 
>> Hi Andrei,
> 
> 
>> 
> 
> 
>> You’re confusing the matter with your masking of public IP ranges. You said you
> 
> 
>> have “2 x Public IP ranges with /26 netmask” – but since you are masking them
> 
> 
>> out with X’s your email doesn’t make sense. If all the X’s are the same then
a
> 
> 
>> .10 and a .20 IP address would be on the same /26 network.
> 
> 
>> 
> 
> 
>> I will assume that you do in fact have 2 x 26-bit networks, e.g.:
> 
> 
>> 
> 
> 
>> 192.168.0.0/26 – with default gateway 192.168.0.1
> 
> 
>> 192.168.0.64/26 – with default gateway 192.168.0.65
> 
> 
>> 
> 
> 
> 
> 
> 
> 
> That is correct. I do have two separate /26 networks similar to what you've
> described above. However, one /26 is used for direct public IP service
> offering, where VRs are not involved in networking at all and the second /26 is
> used for the service offering where VRs are used to provide the networking
> function.
> 
> 
> 
> 
> 
> 
>> If your two guest networks have VRs on separate public IP ranges you will have
> 
> 
>> e.g.
> 
> 
>> 
> 
> 
>> VR1: public IP 192.168.0.10
> 
> 
>> VR2: public IP 192.168.0.70
> 
> 
>> 
> 
> 
> 
> 
> 
> 
> Nope, the guest vms with VRs that can't talk to each other are on the same /26
> network. (in your example that would be on the same 192.168.0.0/26)
> 
> 
> 
> 
> 
> 
>> For a VM hosted behind VR1 to reach a service NAT’ed on VR2 you need to set up
> 
> 
>> routing and possibly firewalling on the data centre device which handles the
> 
> 
>> default gateway for the two networks – i.e. the top of rack switch or router
> 
> 
>> which hosts default gateways 192.168.0.1 and 192.168.0.65. The fact that you
> 
> 
>> can reach services on both networks from outside this range makes sense.
> 
> 
>> 
> 
> 
> 
> 
> These has been set up and vms between separate /26 networks CAN talk to each
> other. The VMs on the same /26 network that doesn't use the VR service can also
> talk to each other. The problem with VMs on the same /26 that use VRs can't
> talk to each other using their public IP addresses.
> 
> 
> 
> 
> 
> 
>> So once you have fixed this you will have VM1 > VR1 > DC_SWITCH_OR_ROUTER >
VR2
> 
> 
>> > VM2.
> 
> 
>> 
> 
> 
>> 
> 
> 
>> Regards,
> 
> 
>> Dag Sonstebo
> 
> 
>> Cloud Architect
> 
> 
>> ShapeBlue
> 
> 
>> 
> 
> 
>> On 21/02/2018, 12:27, "Andrei Mikhailovsky" <andrei@arhont.com.INVALID> wrote:
> 
> 
>> 
> 
> 
>> Hello
> 
> 
>> 
> 
> 
>> Could someone help me to identify the routing issues that we have. The problem
> 
> 
>> is the traffic from different guest networks can not reach each other via the
> 
> 
>> public IPs.
> 
> 
>> 
> 
> 
>> Here is my ACS setup:
> 
> 
>> ACS 4.9.3.0 (both management and agents)
> 
> 
>> KVM Hypervisor based on Ubuntu 16.04
> 
> 
>> Ceph as primary storage. NFS as secondary storage
> 
> 
>> Advanced Networking with vlan separation
> 
> 
>> 2 x Public IP ranges with /26 netmask.
> 
> 
>> 
> 
> 
>> 
> 
> 
>> 
> 
> 
>> Here is an example when routing DOES NOT work:
> 
> 
>> 
> 
> 
>> Case 1 - Advanced Networking, vlan separation, VRs route all traffic and provide
> 
> 
>> all networking services (dhcp, fw, port forwarding, load balancing, etc)
> 
> 
>> 
> 
> 
>> Guest Network 1:
> 
> 
>> 
> 
> 
>> Public IP: XXX.XXX.XXX.10/26
> 
> 
>> Private IP range: 10.1.1.0/24
> 
> 
>> guest vm1 IP: 10.1.1.100/24
> 
> 
>> 
> 
> 
>> Guest Network 2:
> 
> 
>> Public IP: XXX.XXX.XXX.20/26
> 
> 
>> Private IP range: 10.1.1.0/24
> 
> 
>> guest vm2 IP: 10.1.1.200/24
> 
> 
>> 
> 
> 
>> 
> 
> 
>> I've created ACLs on both guest networks to allow traffic from 0.0.0.0/0 on port
> 
> 
>> 80. I've created the port forwarding rules to forward port 80 from public
> 
> 
>> XXX.XXX.XXX.10 and XXX.XXX.XXX.XXX.20 onto 10.1.1.100 and 10.1.1.200
> 
> 
>> respectively.
> 
> 
>> 
> 
> 
>> This setup works perfectly well when I am initiating the connections from
> 
> 
>> outside of our CloudStack. However, vm2 can't reach vm1 on port 80 using the
> 
> 
>> public IP XXX.XXX.XXX.10 and vice versa, vm1 can't reach vm2 on public IP
> 
> 
>> XXX.XXX.XXX.20.
> 
> 
>> 
> 
> 
>> 
> 
> 
>> 
> 
> 
>> 
> 
> 
>> Here is an example when the routing DOES work:
> 
> 
>> 
> 
> 
>> Case 2 - Advanced Networking, vlan separation, VRs are not used. Public IPs are
> 
> 
>> given directly to a guest vm
> 
> 
>> 
> 
> 
>> Guest Network 1:
> 
> 
>> 
> 
> 
>> guest vm1 Public IP: XXX.XXX.XXX.100/26
> 
> 
>> 
> 
> 
>> Guest Network 2:
> 
> 
>> 
> 
> 
>> guest vm2 Public IP: XXX.XXX.XXX.110/26
> 
> 
>> 
> 
> 
>> In the Case 2, the guest vm has a public IP address directly assigned to its
> 
> 
>> network interface. VRs are not used for this networking. Each guest has a fw
> 
> 
>> rule to allow incoming traffic on port 80 from 0.0.0.0/0. Both vm1 and vm2 can
> 
> 
>> access each other on port 80. Also, vms from Case 1 above can access port 80 on
> 
> 
>> vms from Case 2, similarly, vms from Case 2 can access port 80 on vms from Case
> 
> 
>> 1.
> 
> 
>> 
> 
> 
>> 
> 
> 
>> 
> 
> 
>> So, it seems that the rules on the VR in Case 1 do not allow traffic that
> 
> 
>> originates from other VRs within the same public network range. The trace route
> 
> 
>> shows the last hop being the VR's private IP address. How do I change that
> 
> 
>> behaviour and fix the networking issue?
> 
> 
>> 
> 
> 
>> Thanks
> 
> 
>> 
> 
> 
>> Andrei
> 
> 
>> 
> 
> 
>> 
> 
> 
>> 
> 
> 
>> Dag.Sonstebo@shapeblue.com
> 
> 
>> www.shapeblue.com
> 
> 
>> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> 
> 
>> @shapeblue
> 
> 
> 
> 
> 
> 
> 
> 
> Dag.Sonstebo@shapeblue.com
> 
> 
> www.shapeblue.com
> 
> 
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> 
> 
> @shapeblue

Mime
View raw message