incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcus Sorensen <shadow...@gmail.com>
Subject Re: VM router spawning multiple public nics
Date Wed, 29 Aug 2012 13:48:56 GMT
I can confirm that the patch has fixed my particular issue.

This is likely unrelated and I think it doesn't even use the same
code, but I began to play with the VPC stuff a bit and noticed that I
don't get any interfaces except for link local.  I'd probably chalk
that up to it not being ready for KVM, but I thought it was worth a
mention.  I'd be happy to try to help get it ready if someone has time
to nudge me in the right direction.

On Tue, Aug 28, 2012 at 3:15 PM, Edison Su <Edison.su@citrix.com> wrote:
>
>
>> -----Original Message-----
>> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> Sent: Tuesday, August 28, 2012 2:00 PM
>> To: cloudstack-dev@incubator.apache.org
>> Subject: Re: VM router spawning multiple public nics
>>
>>  I thought about this solution myself, but below this portion of code
>> it looks like it uses the hash map to determine which nic number to
>> add the IP to, so with multiple 'untagged' networks it would have no
>> way of knowing which nicnum in the router corresponds with the correct
>> untagged vlan.
>>
>>                 nicNum = vlanAllocatedToVM.get(ip.getVlanId());
>>                 networkUsage(routerIp, "addVif", "eth" + nicNum);
>>                 result =
>> _virtRouterResource.assignPublicIpAddress(routerName,
>>                         routerIp, ip.getPublicIp(), ip.isAdd(),
>> ip.isFirstIP(),
>>                         ip.isSourceNat(), ip.getVlanId(),
>> ip.getVlanGateway(),
>>                         ip.getVlanNetmask(), ip.getVifMacAddress(),
>>                         ip.getGuestIp(), nicNum);
>>
>> if ip.getVlanId() returns untagged (as it does on networks with no
>> vlan id), and we tried to put multiple untagged keys in
>> vlanAllocatedToVM (as with multiple untagged networks), we get the
>> wrong nicNum, no?
>
> In the ipassoc case, if there are multiple untagged networks, all of them are use the
same
> Public bridge. Then multiple ip address will be added on eth2 inside router VM.
> If it works physically, then it works.
>
>>
>> On Tue, Aug 28, 2012 at 2:48 PM, Edison Su <Edison.su@citrix.com> wrote:
>> >
>> >
>> >> -----Original Message-----
>> >> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> >> Sent: Tuesday, August 28, 2012 1:40 PM
>> >> To: cloudstack-dev@incubator.apache.org
>> >> Subject: Re: VM router spawning multiple public nics
>> >>
>> >> Yes, that looks like it would work for me, however that's not
>> >> something that would ever make it into master, right? Essentially
>> >> killing tagging for the public, private, and guest traffic labels?
>> >> There's also still the issue of not being able to differentiate
>> >> between multiple untagged networks, if we wanted to add an IP to a
>> >> router it might not know which untagged interface to apply it to.
>> >
>> > Physically, all the "untagged" network will be created on
>> public/guest/private bridge(the name we put in
>> private/public/guest.bridge.name in agent.properties").
>> > Because, there is no way to create a new untagged bridge by agent
>> itself. Agent code only knows how to create a new tagged(vlan) bridge.
>> > So the fix should be pushed into master.
>> >
>> >>
>> >> On Tue, Aug 28, 2012 at 2:32 PM, Edison Su <Edison.su@citrix.com>
>> wrote:
>> >> >
>> >> >
>> >> >> -----Original Message-----
>> >> >> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> >> >> Sent: Tuesday, August 28, 2012 12:23 PM
>> >> >> To: cloudstack-dev@incubator.apache.org
>> >> >> Subject: Re: VM router spawning multiple public nics
>> >> >>
>> >> >> Thanks for pointing me in the right direction. I've reviewed this
>> >> code
>> >> >> in a bit more detail, and it seems like it's accomplishing the
>> >> >> following:
>> >> >>
>> >> >> 1. get all network interfaces currently connected to the running
>> VM
>> >> >> (a.k.a vnet devices via libvirt)
>> >> >> 2. find out which vlans these network interfaces are bridged to,
>> >> store
>> >> >> this in a hash map of vlan ids and nics
>> >> >> 3. get all ip addresses to be added to the VM
>> >> >> 4. for each ip, get the configured vlan id for the ip, compare
it
>> to
>> >> >> the hash map of existing vlan ids and nics
>> >> >> 5. if the required vlan id is not found in the hash map, create
a
>> >> new
>> >> >> nic
>> >> >> 6. assign the ip to the nic identified by the vlan id key in the
>> >> hash
>> >> >> map
>> >> >>
>> >> >> In this case, we're getting a vlan id returned in step 2 for a
>> >> bridged
>> >> >> nic whose network is defined as untagged in the cloudstack db,
>> >> >> therefore in step 5 we never match as already having a nic for
>> >> >> 'untagged'. I wrote a big long response discussing this issue,
>> but
>> >> as
>> >> >> I began to dig further I realized that aside from my particular
>> case,
>> >> >> untagged vlans in general are just broken (for example they can't
>> be
>> >> >> dealt with uniquely in the current IpAssocCommand code, given the
>> >> hash
>> >> >> map) and it would require more effort than I have time for now
to
>> >> make
>> >> >> things work. If the code were already in place to differentiate
>> >> >> between multiple untagged nics I think that fixing my problem
>> would
>> >> be
>> >> >> trivial, but since its not, I'll just find an alternative
>> solution.
>> >> >>
>> >> >
>> >> > The untagged network usually means "untagged", no vlan on the
>> >> bridge...
>> >> > In your case, the untagged network actually has vlan(tagged) on
>> the
>> >> bridge, thus getting things confused.
>> >> > Will this patch(http://pastebin.com/HJXzZwKp) work for you?
>> >> >
>> >> >> On Mon, Aug 27, 2012 at 10:46 PM, Marcus Sorensen
>> >> <shadowsor@gmail.com>
>> >> >> wrote:
>> >> >> > ...
>> >> >> >             Integer nicPos = 0;
>> >> >> >             for (InterfaceDef nic : nics) {
>> >> >> >                 if
>> >> >> (nic.getBrName().equalsIgnoreCase(_linkLocalBridgeName)) {
>> >> >> >                     vlanAllocatedToVM.put("LinkLocal", nicPos);
>> >> >> >                 } else {
>> >> >> >                     String vlanId =
>> >> >> getVlanIdFromBridge(nic.getBrName());
>> >> >> >                     if (vlanId != null) {
>> >> >> >                         vlanAllocatedToVM.put(vlanId, nicPos);
>> >> >> >                     } else {
>> >> >> >                         vlanAllocatedToVM.put(Vlan.UNTAGGED,
>> >> nicPos);
>> >> >> >                     }
>> >> >> >                 }
>> >> >> >                 nicPos++;
>> >> >> >             }
>> >> >> >             IpAddressTO[] ips = cmd.getIpAddresses();
>> >> >> >             int i = 0;
>> >> >> >             String result = null;
>> >> >> >             int nicNum = 0;
>> >> >> >             for (IpAddressTO ip : ips) {
>> >> >> >                 if
>> (!vlanAllocatedToVM.containsKey(ip.getVlanId()))
>> >> {
>> >> >> >                     /* plug a vif into router */
>> >> >> >                     VifHotPlug(conn, routerName, ip.getVlanId(),
>> >> >> >                             ip.getVifMacAddress());
>> >> >> >                     vlanAllocatedToVM.put(ip.getVlanId(),
>> >> nicPos++);
>> >> >> >                 }
>> >> >> > ...
>> >> >> >
>> >> >> > Looks like the getVlanIdFromBridge might be a bit misleading.
I
>> am
>> >> >> > running my guest public traffic on a 'cloudbr470', which is
a
>> >> bridge
>> >> >> > to eth2.470, yet I configured this network as 'untagged'
>> because I
>> >> >> > have a vlan 470 available on eth3 for cloudstack to autoassign
>> >> (eth3
>> >> >> > is where all of my stuff will be autoassigned). So I'm not
100%
>> >> sure
>> >> >> > yet what's going on here but it seems as though the above
is
>> not
>> >> >> > setting any 'Vlan.UNTAGGED', since it finds a vlan number
for
>> >> >> > eth2.470, but when it enumerates the IPs for the router, it
>> then
>> >> runs
>> >> >> > ip.getVlanId() and doesn't find a nic for the untagged IP
and
>> >> creates
>> >> >> > one.
>> >> >> >
>> >> >> >
>> >> >> > I realize this is perhaps an uncommon case, but a bug
>> nonetheless.
>> >> >> > I'll play with the code a bit and see if I can come up with
a
>> >> >> > solution. I'm thinking I can look at the nic's broadcast URI
>> and
>> >> see
>> >> >> > if it's supposed to be untagged, then add to vlanAllocatedToVM
>> >> >> > appropriately, off the top of my head something like:
>> >> >> >
>> >> >> >                     String vlanId =
>> >> >> getVlanIdFromBridge(nic.getBrName());
>> >> >> >                     if (vlanId != null &&
>> >> >> > !nic.getBroadcastUri().toString().contains("untagged") {
>> >> >> >                         vlanAllocatedToVM.put(vlanId, nicPos);
>> >> >> >                     } else {
>> >> >> >                         vlanAllocatedToVM.put(Vlan.UNTAGGED,
>> >> nicPos);
>> >> >> >                     }
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > On Mon, Aug 27, 2012 at 6:42 PM, Edison Su
>> <Edison.su@citrix.com>
>> >> >> wrote:
>> >> >> >> Possible bug in in kvm code: LibvirtComputingResource-
>> >> >> >execute(IpAssocCommand cmd)-> VifHotPlug, which is only
place
>> >> adding
>> >> >> nic into router vm.
>> >> >> >> Turn on agent log, then take a look what happened.
>> >> >> >>
>> >> >> >>> -----Original Message-----
>> >> >> >>> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> >> >> >>> Sent: Monday, August 27, 2012 5:10 PM
>> >> >> >>> To: cloudstack-dev@incubator.apache.org
>> >> >> >>> Subject: VM router spawning multiple public nics
>> >> >> >>>
>> >> >> >>> I've got two zones running the same build of cloudstack
(a
>> >> recent
>> >> >> copy
>> >> >> >>> of master). One of them creates routers that turn
into ugly
>> >> >> >>> multi-headed beasts, and by that I mean that any time
I
>> create a
>> >> >> port
>> >> >> >>> forwarding or iptables rule for that router I get
a new
>> public
>> >> NIC
>> >> >> >>> with an identical IP address, I have an instance with
a few
>> tens
>> >> of
>> >> >> >>> NICs.  My guess is that some script isn't detecting
that
>> there's
>> >> >> >>> already a NIC with the public IP on it.  It looks
fine in the
>> >> >> >>> database, there is only one public NIC defined in
the nics
>> table.
>> >> >> >>> I'll troubleshoot it tomorrow, but if anyone knows
where I
>> >> should
>> >> >> >>> begin the headstart would be appreciated.
>> >> >> >>>
>> >> >> >>> Thanks

Mime
View raw message