incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alena Prokharchyk <Alena.Prokharc...@citrix.com>
Subject Re: VM router spawning multiple public nics
Date Wed, 29 Aug 2012 21:19:06 GMT
On 8/29/12 1:34 PM, "Edison Su" <Edison.su@citrix.com> wrote:

>Hi Anthony & Alena,
>   Could you help to provide information about VPC, how it works,


Here is the functional spec on the feature:

wiki.cloudstack.org/display/RelOps/Inter-VLAN+Routing+functional+spec


VpcVirtualNetworkApplianceManagerImpl is the manager responsible for VPC
Virtual router operations (plug/unplug nics, etc)



> which commands needed to implemented on the hypervisor side?



1) PlugNicCommand/UnplugNicCommand - does Nic hotplug/unplug (currently
works for VR vm only). In VPC called when add nic for Public/Guest
networks.
2) SetupGuestNetworkCommand - sets up dhcp range, dns information,
networkDomain information on the Nic to make it
3) SetNetworkACLCommand - creates network ACL on the virtual router
4) SetSourceNatCommand - used for setting source nat on the Public IP on
the VPC VR.
5) Site2SiteVpnCfgCommand - for setting up S2S VPN


Anthony/Kelven/Vijay did implementation for Xen/vmWare resources, they can
help you answering all hypervisor related questions. If you need more
details on business logic + Vpc VR management, I can help with that.


-Alena.

>
>> -----Original Message-----
>> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> Sent: Wednesday, August 29, 2012 10:16 AM
>> To: cloudstack-dev@incubator.apache.org
>> Subject: Re: VM router spawning multiple public nics
>>
>> I'd be willing to give it a shot if someone could point me in the
>> right direction and be available to answer questions.
>>
>> On Wed, Aug 29, 2012 at 10:39 AM, Edison Su <Edison.su@citrix.com>
>> wrote:
>> > Yah, KVM doesn't support VPC yet. Will you help to add VPC support on
>> KVM?:) Just implement a few VPC related commands...
>> >
>> >> -----Original Message-----
>> >> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> >> Sent: Wednesday, August 29, 2012 6:49 AM
>> >> To: cloudstack-dev@incubator.apache.org
>> >> Subject: Re: VM router spawning multiple public nics
>> >>
>> >> I can confirm that the patch has fixed my particular issue.
>> >>
>> >> This is likely unrelated and I think it doesn't even use the same
>> >> code, but I began to play with the VPC stuff a bit and noticed that
>> I
>> >> don't get any interfaces except for link local.  I'd probably chalk
>> >> that up to it not being ready for KVM, but I thought it was worth a
>> >> mention.  I'd be happy to try to help get it ready if someone has
>> time
>> >> to nudge me in the right direction.
>> >>
>> >> On Tue, Aug 28, 2012 at 3:15 PM, Edison Su <Edison.su@citrix.com>
>> wrote:
>> >> >
>> >> >
>> >> >> -----Original Message-----
>> >> >> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> >> >> Sent: Tuesday, August 28, 2012 2:00 PM
>> >> >> To: cloudstack-dev@incubator.apache.org
>> >> >> Subject: Re: VM router spawning multiple public nics
>> >> >>
>> >> >>  I thought about this solution myself, but below this portion of
>> >> code
>> >> >> it looks like it uses the hash map to determine which nic number
>> to
>> >> >> add the IP to, so with multiple 'untagged' networks it would have
>> no
>> >> >> way of knowing which nicnum in the router corresponds with the
>> >> correct
>> >> >> untagged vlan.
>> >> >>
>> >> >>                 nicNum = vlanAllocatedToVM.get(ip.getVlanId());
>> >> >>                 networkUsage(routerIp, "addVif", "eth" + nicNum);
>> >> >>                 result =
>> >> >> _virtRouterResource.assignPublicIpAddress(routerName,
>> >> >>                         routerIp, ip.getPublicIp(), ip.isAdd(),
>> >> >> ip.isFirstIP(),
>> >> >>                         ip.isSourceNat(), ip.getVlanId(),
>> >> >> ip.getVlanGateway(),
>> >> >>                         ip.getVlanNetmask(),
>> ip.getVifMacAddress(),
>> >> >>                         ip.getGuestIp(), nicNum);
>> >> >>
>> >> >> if ip.getVlanId() returns untagged (as it does on networks with
>> no
>> >> >> vlan id), and we tried to put multiple untagged keys in
>> >> >> vlanAllocatedToVM (as with multiple untagged networks), we get
>> the
>> >> >> wrong nicNum, no?
>> >> >
>> >> > In the ipassoc case, if there are multiple untagged networks, all
>> of
>> >> them are use the same
>> >> > Public bridge. Then multiple ip address will be added on eth2
>> inside
>> >> router VM.
>> >> > If it works physically, then it works.
>> >> >
>> >> >>
>> >> >> On Tue, Aug 28, 2012 at 2:48 PM, Edison Su <Edison.su@citrix.com>
>> >> wrote:
>> >> >> >
>> >> >> >
>> >> >> >> -----Original Message-----
>> >> >> >> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> >> >> >> Sent: Tuesday, August 28, 2012 1:40 PM
>> >> >> >> To: cloudstack-dev@incubator.apache.org
>> >> >> >> Subject: Re: VM router spawning multiple public nics
>> >> >> >>
>> >> >> >> Yes, that looks like it would work for me, however that's
not
>> >> >> >> something that would ever make it into master, right?
>> Essentially
>> >> >> >> killing tagging for the public, private, and guest traffic
>> labels?
>> >> >> >> There's also still the issue of not being able to
>> differentiate
>> >> >> >> between multiple untagged networks, if we wanted to add
an IP
>> to
>> >> a
>> >> >> >> router it might not know which untagged interface to apply
it
>> to.
>> >> >> >
>> >> >> > Physically, all the "untagged" network will be created on
>> >> >> public/guest/private bridge(the name we put in
>> >> >> private/public/guest.bridge.name in agent.properties").
>> >> >> > Because, there is no way to create a new untagged bridge by
>> agent
>> >> >> itself. Agent code only knows how to create a new tagged(vlan)
>> >> bridge.
>> >> >> > So the fix should be pushed into master.
>> >> >> >
>> >> >> >>
>> >> >> >> On Tue, Aug 28, 2012 at 2:32 PM, Edison Su
>> <Edison.su@citrix.com>
>> >> >> wrote:
>> >> >> >> >
>> >> >> >> >
>> >> >> >> >> -----Original Message-----
>> >> >> >> >> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> >> >> >> >> Sent: Tuesday, August 28, 2012 12:23 PM
>> >> >> >> >> To: cloudstack-dev@incubator.apache.org
>> >> >> >> >> Subject: Re: VM router spawning multiple public
nics
>> >> >> >> >>
>> >> >> >> >> Thanks for pointing me in the right direction.
I've
>> reviewed
>> >> this
>> >> >> >> code
>> >> >> >> >> in a bit more detail, and it seems like it's
accomplishing
>> the
>> >> >> >> >> following:
>> >> >> >> >>
>> >> >> >> >> 1. get all network interfaces currently connected
to the
>> >> running
>> >> >> VM
>> >> >> >> >> (a.k.a vnet devices via libvirt)
>> >> >> >> >> 2. find out which vlans these network interfaces
are
>> bridged
>> >> to,
>> >> >> >> store
>> >> >> >> >> this in a hash map of vlan ids and nics
>> >> >> >> >> 3. get all ip addresses to be added to the VM
>> >> >> >> >> 4. for each ip, get the configured vlan id for
the ip,
>> compare
>> >> it
>> >> >> to
>> >> >> >> >> the hash map of existing vlan ids and nics
>> >> >> >> >> 5. if the required vlan id is not found in the
hash map,
>> >> create a
>> >> >> >> new
>> >> >> >> >> nic
>> >> >> >> >> 6. assign the ip to the nic identified by the
vlan id key
>> in
>> >> the
>> >> >> >> hash
>> >> >> >> >> map
>> >> >> >> >>
>> >> >> >> >> In this case, we're getting a vlan id returned
in step 2
>> for a
>> >> >> >> bridged
>> >> >> >> >> nic whose network is defined as untagged in the
cloudstack
>> db,
>> >> >> >> >> therefore in step 5 we never match as already
having a nic
>> for
>> >> >> >> >> 'untagged'. I wrote a big long response discussing
this
>> issue,
>> >> >> but
>> >> >> >> as
>> >> >> >> >> I began to dig further I realized that aside
from my
>> >> particular
>> >> >> case,
>> >> >> >> >> untagged vlans in general are just broken (for
example they
>> >> can't
>> >> >> be
>> >> >> >> >> dealt with uniquely in the current IpAssocCommand
code,
>> given
>> >> the
>> >> >> >> hash
>> >> >> >> >> map) and it would require more effort than I
have time for
>> now
>> >> to
>> >> >> >> make
>> >> >> >> >> things work. If the code were already in place
to
>> >> differentiate
>> >> >> >> >> between multiple untagged nics I think that fixing
my
>> problem
>> >> >> would
>> >> >> >> be
>> >> >> >> >> trivial, but since its not, I'll just find an
alternative
>> >> >> solution.
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> > The untagged network usually means "untagged", no
vlan on
>> the
>> >> >> >> bridge...
>> >> >> >> > In your case, the untagged network actually has vlan(tagged)
>> on
>> >> >> the
>> >> >> >> bridge, thus getting things confused.
>> >> >> >> > Will this patch(http://pastebin.com/HJXzZwKp) work
for you?
>> >> >> >> >
>> >> >> >> >> On Mon, Aug 27, 2012 at 10:46 PM, Marcus Sorensen
>> >> >> >> <shadowsor@gmail.com>
>> >> >> >> >> wrote:
>> >> >> >> >> > ...
>> >> >> >> >> >             Integer nicPos = 0;
>> >> >> >> >> >             for (InterfaceDef nic : nics)
{
>> >> >> >> >> >                 if
>> >> >> >> >> (nic.getBrName().equalsIgnoreCase(_linkLocalBridgeName))
{
>> >> >> >> >> >                     vlanAllocatedToVM.put("LinkLocal",
>> >> nicPos);
>> >> >> >> >> >                 } else {
>> >> >> >> >> >                     String vlanId =
>> >> >> >> >> getVlanIdFromBridge(nic.getBrName());
>> >> >> >> >> >                     if (vlanId != null)
{
>> >> >> >> >> >                         vlanAllocatedToVM.put(vlanId,
>> >> nicPos);
>> >> >> >> >> >                     } else {
>> >> >> >> >> >
>> vlanAllocatedToVM.put(Vlan.UNTAGGED,
>> >> >> >> nicPos);
>> >> >> >> >> >                     }
>> >> >> >> >> >                 }
>> >> >> >> >> >                 nicPos++;
>> >> >> >> >> >             }
>> >> >> >> >> >             IpAddressTO[] ips = cmd.getIpAddresses();
>> >> >> >> >> >             int i = 0;
>> >> >> >> >> >             String result = null;
>> >> >> >> >> >             int nicNum = 0;
>> >> >> >> >> >             for (IpAddressTO ip : ips) {
>> >> >> >> >> >                 if
>> >> >> (!vlanAllocatedToVM.containsKey(ip.getVlanId()))
>> >> >> >> {
>> >> >> >> >> >                     /* plug a vif into router
*/
>> >> >> >> >> >                     VifHotPlug(conn, routerName,
>> >> ip.getVlanId(),
>> >> >> >> >> >                             ip.getVifMacAddress());
>> >> >> >> >> >                     vlanAllocatedToVM.put(ip.getVlanId(),
>> >> >> >> nicPos++);
>> >> >> >> >> >                 }
>> >> >> >> >> > ...
>> >> >> >> >> >
>> >> >> >> >> > Looks like the getVlanIdFromBridge might
be a bit
>> misleading.
>> >> I
>> >> >> am
>> >> >> >> >> > running my guest public traffic on a 'cloudbr470',
which
>> is
>> >> a
>> >> >> >> bridge
>> >> >> >> >> > to eth2.470, yet I configured this network
as 'untagged'
>> >> >> because I
>> >> >> >> >> > have a vlan 470 available on eth3 for cloudstack
to
>> >> autoassign
>> >> >> >> (eth3
>> >> >> >> >> > is where all of my stuff will be autoassigned).
So I'm
>> not
>> >> 100%
>> >> >> >> sure
>> >> >> >> >> > yet what's going on here but it seems as
though the above
>> is
>> >> >> not
>> >> >> >> >> > setting any 'Vlan.UNTAGGED', since it finds
a vlan number
>> >> for
>> >> >> >> >> > eth2.470, but when it enumerates the IPs
for the router,
>> it
>> >> >> then
>> >> >> >> runs
>> >> >> >> >> > ip.getVlanId() and doesn't find a nic for
the untagged IP
>> >> and
>> >> >> >> creates
>> >> >> >> >> > one.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > I realize this is perhaps an uncommon case,
but a bug
>> >> >> nonetheless.
>> >> >> >> >> > I'll play with the code a bit and see if
I can come up
>> with
>> >> a
>> >> >> >> >> > solution. I'm thinking I can look at the
nic's broadcast
>> URI
>> >> >> and
>> >> >> >> see
>> >> >> >> >> > if it's supposed to be untagged, then add
to
>> >> vlanAllocatedToVM
>> >> >> >> >> > appropriately, off the top of my head something
like:
>> >> >> >> >> >
>> >> >> >> >> >                     String vlanId =
>> >> >> >> >> getVlanIdFromBridge(nic.getBrName());
>> >> >> >> >> >                     if (vlanId != null &&
>> >> >> >> >> > !nic.getBroadcastUri().toString().contains("untagged")
{
>> >> >> >> >> >                         vlanAllocatedToVM.put(vlanId,
>> >> nicPos);
>> >> >> >> >> >                     } else {
>> >> >> >> >> >
>> vlanAllocatedToVM.put(Vlan.UNTAGGED,
>> >> >> >> nicPos);
>> >> >> >> >> >                     }
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > On Mon, Aug 27, 2012 at 6:42 PM, Edison
Su
>> >> >> <Edison.su@citrix.com>
>> >> >> >> >> wrote:
>> >> >> >> >> >> Possible bug in in kvm code: LibvirtComputingResource-
>> >> >> >> >> >execute(IpAssocCommand cmd)-> VifHotPlug,
which is only
>> place
>> >> >> >> adding
>> >> >> >> >> nic into router vm.
>> >> >> >> >> >> Turn on agent log, then take a look
what happened.
>> >> >> >> >> >>
>> >> >> >> >> >>> -----Original Message-----
>> >> >> >> >> >>> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> >> >> >> >> >>> Sent: Monday, August 27, 2012 5:10
PM
>> >> >> >> >> >>> To: cloudstack-dev@incubator.apache.org
>> >> >> >> >> >>> Subject: VM router spawning multiple
public nics
>> >> >> >> >> >>>
>> >> >> >> >> >>> I've got two zones running the same
build of cloudstack
>> (a
>> >> >> >> recent
>> >> >> >> >> copy
>> >> >> >> >> >>> of master). One of them creates
routers that turn into
>> >> ugly
>> >> >> >> >> >>> multi-headed beasts, and by that
I mean that any time I
>> >> >> create a
>> >> >> >> >> port
>> >> >> >> >> >>> forwarding or iptables rule for
that router I get a new
>> >> >> public
>> >> >> >> NIC
>> >> >> >> >> >>> with an identical IP address, I
have an instance with a
>> >> few
>> >> >> tens
>> >> >> >> of
>> >> >> >> >> >>> NICs.  My guess is that some script
isn't detecting
>> that
>> >> >> there's
>> >> >> >> >> >>> already a NIC with the public IP
on it.  It looks fine
>> in
>> >> the
>> >> >> >> >> >>> database, there is only one public
NIC defined in the
>> nics
>> >> >> table.
>> >> >> >> >> >>> I'll troubleshoot it tomorrow, but
if anyone knows
>> where I
>> >> >> >> should
>> >> >> >> >> >>> begin the headstart would be appreciated.
>> >> >> >> >> >>>
>> >> >> >> >> >>> Thanks
>



Mime
View raw message