Mailing-List: contact cloudstack-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: cloudstack-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of shadowsor@gmail.com designates
 209.85.212.169 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <C66C814C1ABFA8449FF65CB44953A8B4012CBB65BC13@SJCPMAILBOX01.citrite.net>
References: 
 <CALFpzo6SLQQ4PDRncMW1cNY8+ORL9F4oZsQr7+NV62d7=1u3YQ@mail.gmail.com>
	<C66C814C1ABFA8449FF65CB44953A8B4012CBB65BC0E@SJCPMAILBOX01.citrite.net>
	<CALFpzo4MrVfZCSuA3zKYfvNvW9YeEC00=Y_qkAm0CoMcMxgY+g@mail.gmail.com>
	<CALFpzo7Afyk6=WW=Rd8gbrJHSQ6a-JHpu9Zs-OpEgKO8Bcvq5g@mail.gmail.com>
	<C66C814C1ABFA8449FF65CB44953A8B4012CBB65BC11@SJCPMAILBOX01.citrite.net>
	<CALFpzo4t6tMJTaLmWzsiof3NROiyrJWHOQOkOfQ5kci5BLpfQw@mail.gmail.com>
	<C66C814C1ABFA8449FF65CB44953A8B4012CBB65BC13@SJCPMAILBOX01.citrite.net>
Date: Tue, 28 Aug 2012 15:00:23 -0600
Message-ID: 
 <CALFpzo7suh8iJ2zjnP8V8GfSAzo0kAcd3xqdPK7joDLsNPKfzg@mail.gmail.com>
Subject: Re: VM router spawning multiple public nics
From: Marcus Sorensen <shadowsor@gmail.com>
To: cloudstack-dev@incubator.apache.org
Content-Type: text/plain; charset=ISO-8859-1

 I thought about this solution myself, but below this portion of code
it looks like it uses the hash map to determine which nic number to
add the IP to, so with multiple 'untagged' networks it would have no
way of knowing which nicnum in the router corresponds with the correct
untagged vlan.

                nicNum = vlanAllocatedToVM.get(ip.getVlanId());
                networkUsage(routerIp, "addVif", "eth" + nicNum);
                result = _virtRouterResource.assignPublicIpAddress(routerName,
                        routerIp, ip.getPublicIp(), ip.isAdd(), ip.isFirstIP(),
                        ip.isSourceNat(), ip.getVlanId(), ip.getVlanGateway(),
                        ip.getVlanNetmask(), ip.getVifMacAddress(),
                        ip.getGuestIp(), nicNum);

if ip.getVlanId() returns untagged (as it does on networks with no
vlan id), and we tried to put multiple untagged keys in
vlanAllocatedToVM (as with multiple untagged networks), we get the
wrong nicNum, no?

On Tue, Aug 28, 2012 at 2:48 PM, Edison Su <Edison.su@citrix.com> wrote:
>
>
>> -----Original Message-----
>> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> Sent: Tuesday, August 28, 2012 1:40 PM
>> To: cloudstack-dev@incubator.apache.org
>> Subject: Re: VM router spawning multiple public nics
>>
>> Yes, that looks like it would work for me, however that's not
>> something that would ever make it into master, right? Essentially
>> killing tagging for the public, private, and guest traffic labels?
>> There's also still the issue of not being able to differentiate
>> between multiple untagged networks, if we wanted to add an IP to a
>> router it might not know which untagged interface to apply it to.
>
> Physically, all the "untagged" network will be created on public/guest/private bridge(the name we put in private/public/guest.bridge.name in agent.properties").
> Because, there is no way to create a new untagged bridge by agent itself. Agent code only knows how to create a new tagged(vlan) bridge.
> So the fix should be pushed into master.
>
>>
>> On Tue, Aug 28, 2012 at 2:32 PM, Edison Su <Edison.su@citrix.com> wrote:
>> >
>> >
>> >> -----Original Message-----
>> >> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> >> Sent: Tuesday, August 28, 2012 12:23 PM
>> >> To: cloudstack-dev@incubator.apache.org
>> >> Subject: Re: VM router spawning multiple public nics
>> >>
>> >> Thanks for pointing me in the right direction. I've reviewed this
>> code
>> >> in a bit more detail, and it seems like it's accomplishing the
>> >> following:
>> >>
>> >> 1. get all network interfaces currently connected to the running VM
>> >> (a.k.a vnet devices via libvirt)
>> >> 2. find out which vlans these network interfaces are bridged to,
>> store
>> >> this in a hash map of vlan ids and nics
>> >> 3. get all ip addresses to be added to the VM
>> >> 4. for each ip, get the configured vlan id for the ip, compare it to
>> >> the hash map of existing vlan ids and nics
>> >> 5. if the required vlan id is not found in the hash map, create a
>> new
>> >> nic
>> >> 6. assign the ip to the nic identified by the vlan id key in the
>> hash
>> >> map
>> >>
>> >> In this case, we're getting a vlan id returned in step 2 for a
>> bridged
>> >> nic whose network is defined as untagged in the cloudstack db,
>> >> therefore in step 5 we never match as already having a nic for
>> >> 'untagged'. I wrote a big long response discussing this issue, but
>> as
>> >> I began to dig further I realized that aside from my particular case,
>> >> untagged vlans in general are just broken (for example they can't be
>> >> dealt with uniquely in the current IpAssocCommand code, given the
>> hash
>> >> map) and it would require more effort than I have time for now to
>> make
>> >> things work. If the code were already in place to differentiate
>> >> between multiple untagged nics I think that fixing my problem would
>> be
>> >> trivial, but since its not, I'll just find an alternative solution.
>> >>
>> >
>> > The untagged network usually means "untagged", no vlan on the
>> bridge...
>> > In your case, the untagged network actually has vlan(tagged) on the
>> bridge, thus getting things confused.
>> > Will this patch(http://pastebin.com/HJXzZwKp) work for you?
>> >
>> >> On Mon, Aug 27, 2012 at 10:46 PM, Marcus Sorensen
>> <shadowsor@gmail.com>
>> >> wrote:
>> >> > ...
>> >> >             Integer nicPos = 0;
>> >> >             for (InterfaceDef nic : nics) {
>> >> >                 if
>> >> (nic.getBrName().equalsIgnoreCase(_linkLocalBridgeName)) {
>> >> >                     vlanAllocatedToVM.put("LinkLocal", nicPos);
>> >> >                 } else {
>> >> >                     String vlanId =
>> >> getVlanIdFromBridge(nic.getBrName());
>> >> >                     if (vlanId != null) {
>> >> >                         vlanAllocatedToVM.put(vlanId, nicPos);
>> >> >                     } else {
>> >> >                         vlanAllocatedToVM.put(Vlan.UNTAGGED,
>> nicPos);
>> >> >                     }
>> >> >                 }
>> >> >                 nicPos++;
>> >> >             }
>> >> >             IpAddressTO[] ips = cmd.getIpAddresses();
>> >> >             int i = 0;
>> >> >             String result = null;
>> >> >             int nicNum = 0;
>> >> >             for (IpAddressTO ip : ips) {
>> >> >                 if (!vlanAllocatedToVM.containsKey(ip.getVlanId()))
>> {
>> >> >                     /* plug a vif into router */
>> >> >                     VifHotPlug(conn, routerName, ip.getVlanId(),
>> >> >                             ip.getVifMacAddress());
>> >> >                     vlanAllocatedToVM.put(ip.getVlanId(),
>> nicPos++);
>> >> >                 }
>> >> > ...
>> >> >
>> >> > Looks like the getVlanIdFromBridge might be a bit misleading. I am
>> >> > running my guest public traffic on a 'cloudbr470', which is a
>> bridge
>> >> > to eth2.470, yet I configured this network as 'untagged' because I
>> >> > have a vlan 470 available on eth3 for cloudstack to autoassign
>> (eth3
>> >> > is where all of my stuff will be autoassigned). So I'm not 100%
>> sure
>> >> > yet what's going on here but it seems as though the above is not
>> >> > setting any 'Vlan.UNTAGGED', since it finds a vlan number for
>> >> > eth2.470, but when it enumerates the IPs for the router, it then
>> runs
>> >> > ip.getVlanId() and doesn't find a nic for the untagged IP and
>> creates
>> >> > one.
>> >> >
>> >> >
>> >> > I realize this is perhaps an uncommon case, but a bug nonetheless.
>> >> > I'll play with the code a bit and see if I can come up with a
>> >> > solution. I'm thinking I can look at the nic's broadcast URI and
>> see
>> >> > if it's supposed to be untagged, then add to vlanAllocatedToVM
>> >> > appropriately, off the top of my head something like:
>> >> >
>> >> >                     String vlanId =
>> >> getVlanIdFromBridge(nic.getBrName());
>> >> >                     if (vlanId != null &&
>> >> > !nic.getBroadcastUri().toString().contains("untagged") {
>> >> >                         vlanAllocatedToVM.put(vlanId, nicPos);
>> >> >                     } else {
>> >> >                         vlanAllocatedToVM.put(Vlan.UNTAGGED,
>> nicPos);
>> >> >                     }
>> >> >
>> >> >
>> >> >
>> >> > On Mon, Aug 27, 2012 at 6:42 PM, Edison Su <Edison.su@citrix.com>
>> >> wrote:
>> >> >> Possible bug in in kvm code: LibvirtComputingResource-
>> >> >execute(IpAssocCommand cmd)-> VifHotPlug, which is only place
>> adding
>> >> nic into router vm.
>> >> >> Turn on agent log, then take a look what happened.
>> >> >>
>> >> >>> -----Original Message-----
>> >> >>> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> >> >>> Sent: Monday, August 27, 2012 5:10 PM
>> >> >>> To: cloudstack-dev@incubator.apache.org
>> >> >>> Subject: VM router spawning multiple public nics
>> >> >>>
>> >> >>> I've got two zones running the same build of cloudstack (a
>> recent
>> >> copy
>> >> >>> of master). One of them creates routers that turn into ugly
>> >> >>> multi-headed beasts, and by that I mean that any time I create a
>> >> port
>> >> >>> forwarding or iptables rule for that router I get a new public
>> NIC
>> >> >>> with an identical IP address, I have an instance with a few tens
>> of
>> >> >>> NICs.  My guess is that some script isn't detecting that there's
>> >> >>> already a NIC with the public IP on it.  It looks fine in the
>> >> >>> database, there is only one public NIC defined in the nics table.
>> >> >>> I'll troubleshoot it tomorrow, but if anyone knows where I
>> should
>> >> >>> begin the headstart would be appreciated.
>> >> >>>
>> >> >>> Thanks