cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chip Childers <>
Subject Re: [RFC] QinQ vlans support
Date Thu, 18 Oct 2012 14:05:03 GMT
On Thu, Oct 18, 2012 at 12:42 AM, Marcus Sorensen <> wrote:
> Sorry, I've been up to my ears. I've attached the simple patch that
> makes this all happen, if anyone wants to take a look. This is the
> code that looks for physical devices. It's passed a bridge and then
> determines the parent of that bridge, then whether that parent is a
> tagged device and goes one more step and finds its parent. This just
> circumvents the last lookup if the parent of the bridge is a "vlan"
> device (single tagged, e.g. vlan100) but not a double-tagged one (e.g.
> vlan100.10), and the rest of cloudstack treats vlan100 as though it
> were a physical device, creates tagged bridges on it if it has guest
> traffic type, etc. I've been using it in our test bed for about a
> month, and have only run into the MTU issue.

Hey Marcus,

Attachments get stripped.  Can you post it somewhere?

> If people still think it's a good idea, I'll create a functional spec
> and additional info on how it works.
>  I've also got a small patch to, but I'm debating
> whether or not it's necessary. It detects whether the "physical
> interface" is actually a vlan tagged interface, and if so it subtracts
> the necessary bytes from the MTU when it sets up the double-tagged
> bridges. It's technically not necessary, as the important part is
> whether the guest MTUs fit inside the MTU that the switch allows once
> the extra tag is added. But it just makes it a bit more obvious as to
> what's needed. However it also breaks the admin's ability to bump the
> switch MTUs up just a bit, say 1532, to account for the excess without
> having to go up to 9000 or full jumbo. If anyone is a network guru and
> has any feedback it would be appreciated, but I'm inclined to leave
> the MTUs alone and write it into the functional spec that a switch
> with a 1500 MTU supports double tags up to 1468, and a switch with a
> 9000 MTU supports VM guest networks up to 8968 MTU.
> On Mon, Oct 15, 2012 at 1:43 PM, Marcus Sorensen <> wrote:
>> Ok, I'll pull out the changes and let people see them. Cloudstack
>> seems to let me put the same vlan ranges on multiple physicals, though
>> I haven't done much actual testing with large numbers of vlans. I
>> imagine there would be other bottlenecks if they all needed to be up
>> on the same host at once. Luckily we only create bridges for the
>> actual VMs on the box so it should scale reasonably.
>> The only caveat I've run into so far is that you either need to be
>> running jumbo frames on your switches, or turn down the MTU on the
>> guests a bit to accommodate the space taken by extra tag.  If you
>> wanted to run jumbo fames on the guests as well, you'd run into the
>> same situation and have to use slightly less than the 9000 (although
>> the virtual router would require a patch too for the new size).
>> On Mon, Oct 15, 2012 at 9:56 AM, Ahmad Emneina <> wrote:
>>> On 10/15/12 8:35 AM, "Kelceydamage@bbits" <> wrote:
>>>>That's a far more elegant way then I tried, which was creating tagged
>>>>interfaces within guests.
>>>>Sent from my iPhone
>>>>On Oct 15, 2012, at 12:54 AM, Chiradeep Vittal
>>>><> wrote:
>>>>> This sounds like it can be modeled as multiple physical networks? That
>>>>> each "outer" vlan (400, 401, etc) is a separate physical network in the
>>>>> same zone. That could work, although it is probable that the zone
>>>>> configuration API bits prevent more than 4k VLANs per zone (that can
>>>>> changed to per physical network).
>>>>> As long as communication between guests on different physical networks
>>>>> happens via the public network, it should be Ok.
>>>>> I'd like to see the patch.
>>>>> Thanks
>>>>> On 10/12/12 1:09 AM, "Marcus Sorensen" <> wrote:
>>>>>> Guys, in looking for a free and scalable way to provide private
>>>>>> for customers I've been running a QinQ setup that has been working
>>>>>> well. I've sort of laid the groundwork for it already in changing
>>>>>> bridge naming conventions about a month ago for KVM(to names that
>>>>>> collide if the same vlans is used twice on different phys).
>>>>>> Basically the way it works is like this. Linux has two ways of creating
>>>>>> tagged networks, the eth#.# and the less used vlan# network devices.
>>>>>> have
>>>>>> a tiny patch that causes cloudstack to treat vlan# devs as though
>>>>>> were
>>>>>> physical NICs. In this way, you can do something like physical devices
>>>>>> eth0,eth1,and vlan400. management traffic on eth0's bridge, storage
>>>>>> eth1.102's bridge, maybe eth1.103 for public/guest, then create say
>>>>>> vlan400 that is tag 400 on eth1. You add a traffic type of guest
to it
>>>>>> give it a vlan range, say 10-4000. Then you end up with cloudstack
>>>>>> out vlan400.10, vlan400.11, etc for guest networks. Works great for
>>>>>> network
>>>>>> isolation without burning through a bunch of your "real" vlans. In
>>>>>> unlikely event that you run out, you just create a physical vlan401
>>>>>> start over with the vlan numbers.
>>>>>> In theory all-you-can-eat isolated networks without having to configure
>>>>>> hundreds of vlans on your networking equipment. This may require
>>>>>> additional
>>>>>> config on any upstream switches to pass the double tags around, but
>>>>>> general from what I've seen the inner tags just pass through on
>>>>>> layer 2, it should only get tricky if you try to tunnel, route or
>>>>>> tags.
>>>>>> This is especially nice with system VM routers and VPC (cloudstack
>>>>>> care of everything), but admittedly external routers probably will
>>>>>> spotty support for being able to route double tagged stuff. I'm also
>>>>>> afraid that if I were to get it merged in that it would just become
>>>>>> undocumented hack thing that few know about and nobody uses. So I'm
>>>>>> looking
>>>>>> for feedback on whether this sounds useful enough to commit, how
>>>>>> be documented, and whether it makes sense to hint at this in the
>>>>>> somehow.
>>> +1
>>> This actually sounds amazing Marcus. I'd love to see and use this
>>> implementation.
>>> --
>>> Æ

View raw message