cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcus Sorensen <shadow...@gmail.com>
Subject Re: VM router spawning multiple public nics
Date Wed, 29 Aug 2012 23:34:14 GMT
Great, thanks. I've already got a working implementation of
adding/removing nics from regular instances that I've been playing
with, so I'm getting vaguely familiar with the various data types and
things surrounding the networking. I don't know if this is quite
within my reach just yet but I'll see how far I get.

On Wed, Aug 29, 2012 at 3:19 PM, Alena Prokharchyk
<Alena.Prokharchyk@citrix.com> wrote:
> On 8/29/12 1:34 PM, "Edison Su" <Edison.su@citrix.com> wrote:
>
>>Hi Anthony & Alena,
>>   Could you help to provide information about VPC, how it works,
>
>
> Here is the functional spec on the feature:
>
> wiki.cloudstack.org/display/RelOps/Inter-VLAN+Routing+functional+spec
>
>
> VpcVirtualNetworkApplianceManagerImpl is the manager responsible for VPC
> Virtual router operations (plug/unplug nics, etc)
>
>
>
>> which commands needed to implemented on the hypervisor side?
>
>
>
> 1) PlugNicCommand/UnplugNicCommand - does Nic hotplug/unplug (currently
> works for VR vm only). In VPC called when add nic for Public/Guest
> networks.
> 2) SetupGuestNetworkCommand - sets up dhcp range, dns information,
> networkDomain information on the Nic to make it
> 3) SetNetworkACLCommand - creates network ACL on the virtual router
> 4) SetSourceNatCommand - used for setting source nat on the Public IP on
> the VPC VR.
> 5) Site2SiteVpnCfgCommand - for setting up S2S VPN
>
>
> Anthony/Kelven/Vijay did implementation for Xen/vmWare resources, they can
> help you answering all hypervisor related questions. If you need more
> details on business logic + Vpc VR management, I can help with that.
>
>
> -Alena.
>
>>
>>> -----Original Message-----
>>> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>>> Sent: Wednesday, August 29, 2012 10:16 AM
>>> To: cloudstack-dev@incubator.apache.org
>>> Subject: Re: VM router spawning multiple public nics
>>>
>>> I'd be willing to give it a shot if someone could point me in the
>>> right direction and be available to answer questions.
>>>
>>> On Wed, Aug 29, 2012 at 10:39 AM, Edison Su <Edison.su@citrix.com>
>>> wrote:
>>> > Yah, KVM doesn't support VPC yet. Will you help to add VPC support on
>>> KVM?:) Just implement a few VPC related commands...
>>> >
>>> >> -----Original Message-----
>>> >> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>>> >> Sent: Wednesday, August 29, 2012 6:49 AM
>>> >> To: cloudstack-dev@incubator.apache.org
>>> >> Subject: Re: VM router spawning multiple public nics
>>> >>
>>> >> I can confirm that the patch has fixed my particular issue.
>>> >>
>>> >> This is likely unrelated and I think it doesn't even use the same
>>> >> code, but I began to play with the VPC stuff a bit and noticed that
>>> I
>>> >> don't get any interfaces except for link local.  I'd probably chalk
>>> >> that up to it not being ready for KVM, but I thought it was worth a
>>> >> mention.  I'd be happy to try to help get it ready if someone has
>>> time
>>> >> to nudge me in the right direction.
>>> >>
>>> >> On Tue, Aug 28, 2012 at 3:15 PM, Edison Su <Edison.su@citrix.com>
>>> wrote:
>>> >> >
>>> >> >
>>> >> >> -----Original Message-----
>>> >> >> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>>> >> >> Sent: Tuesday, August 28, 2012 2:00 PM
>>> >> >> To: cloudstack-dev@incubator.apache.org
>>> >> >> Subject: Re: VM router spawning multiple public nics
>>> >> >>
>>> >> >>  I thought about this solution myself, but below this portion
of
>>> >> code
>>> >> >> it looks like it uses the hash map to determine which nic number
>>> to
>>> >> >> add the IP to, so with multiple 'untagged' networks it would
have
>>> no
>>> >> >> way of knowing which nicnum in the router corresponds with
the
>>> >> correct
>>> >> >> untagged vlan.
>>> >> >>
>>> >> >>                 nicNum = vlanAllocatedToVM.get(ip.getVlanId());
>>> >> >>                 networkUsage(routerIp, "addVif", "eth" + nicNum);
>>> >> >>                 result =
>>> >> >> _virtRouterResource.assignPublicIpAddress(routerName,
>>> >> >>                         routerIp, ip.getPublicIp(), ip.isAdd(),
>>> >> >> ip.isFirstIP(),
>>> >> >>                         ip.isSourceNat(), ip.getVlanId(),
>>> >> >> ip.getVlanGateway(),
>>> >> >>                         ip.getVlanNetmask(),
>>> ip.getVifMacAddress(),
>>> >> >>                         ip.getGuestIp(), nicNum);
>>> >> >>
>>> >> >> if ip.getVlanId() returns untagged (as it does on networks
with
>>> no
>>> >> >> vlan id), and we tried to put multiple untagged keys in
>>> >> >> vlanAllocatedToVM (as with multiple untagged networks), we
get
>>> the
>>> >> >> wrong nicNum, no?
>>> >> >
>>> >> > In the ipassoc case, if there are multiple untagged networks, all
>>> of
>>> >> them are use the same
>>> >> > Public bridge. Then multiple ip address will be added on eth2
>>> inside
>>> >> router VM.
>>> >> > If it works physically, then it works.
>>> >> >
>>> >> >>
>>> >> >> On Tue, Aug 28, 2012 at 2:48 PM, Edison Su <Edison.su@citrix.com>
>>> >> wrote:
>>> >> >> >
>>> >> >> >
>>> >> >> >> -----Original Message-----
>>> >> >> >> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>>> >> >> >> Sent: Tuesday, August 28, 2012 1:40 PM
>>> >> >> >> To: cloudstack-dev@incubator.apache.org
>>> >> >> >> Subject: Re: VM router spawning multiple public nics
>>> >> >> >>
>>> >> >> >> Yes, that looks like it would work for me, however
that's not
>>> >> >> >> something that would ever make it into master, right?
>>> Essentially
>>> >> >> >> killing tagging for the public, private, and guest
traffic
>>> labels?
>>> >> >> >> There's also still the issue of not being able to
>>> differentiate
>>> >> >> >> between multiple untagged networks, if we wanted to
add an IP
>>> to
>>> >> a
>>> >> >> >> router it might not know which untagged interface
to apply it
>>> to.
>>> >> >> >
>>> >> >> > Physically, all the "untagged" network will be created
on
>>> >> >> public/guest/private bridge(the name we put in
>>> >> >> private/public/guest.bridge.name in agent.properties").
>>> >> >> > Because, there is no way to create a new untagged bridge
by
>>> agent
>>> >> >> itself. Agent code only knows how to create a new tagged(vlan)
>>> >> bridge.
>>> >> >> > So the fix should be pushed into master.
>>> >> >> >
>>> >> >> >>
>>> >> >> >> On Tue, Aug 28, 2012 at 2:32 PM, Edison Su
>>> <Edison.su@citrix.com>
>>> >> >> wrote:
>>> >> >> >> >
>>> >> >> >> >
>>> >> >> >> >> -----Original Message-----
>>> >> >> >> >> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>>> >> >> >> >> Sent: Tuesday, August 28, 2012 12:23 PM
>>> >> >> >> >> To: cloudstack-dev@incubator.apache.org
>>> >> >> >> >> Subject: Re: VM router spawning multiple
public nics
>>> >> >> >> >>
>>> >> >> >> >> Thanks for pointing me in the right direction.
I've
>>> reviewed
>>> >> this
>>> >> >> >> code
>>> >> >> >> >> in a bit more detail, and it seems like it's
accomplishing
>>> the
>>> >> >> >> >> following:
>>> >> >> >> >>
>>> >> >> >> >> 1. get all network interfaces currently connected
to the
>>> >> running
>>> >> >> VM
>>> >> >> >> >> (a.k.a vnet devices via libvirt)
>>> >> >> >> >> 2. find out which vlans these network interfaces
are
>>> bridged
>>> >> to,
>>> >> >> >> store
>>> >> >> >> >> this in a hash map of vlan ids and nics
>>> >> >> >> >> 3. get all ip addresses to be added to the
VM
>>> >> >> >> >> 4. for each ip, get the configured vlan id
for the ip,
>>> compare
>>> >> it
>>> >> >> to
>>> >> >> >> >> the hash map of existing vlan ids and nics
>>> >> >> >> >> 5. if the required vlan id is not found in
the hash map,
>>> >> create a
>>> >> >> >> new
>>> >> >> >> >> nic
>>> >> >> >> >> 6. assign the ip to the nic identified by
the vlan id key
>>> in
>>> >> the
>>> >> >> >> hash
>>> >> >> >> >> map
>>> >> >> >> >>
>>> >> >> >> >> In this case, we're getting a vlan id returned
in step 2
>>> for a
>>> >> >> >> bridged
>>> >> >> >> >> nic whose network is defined as untagged
in the cloudstack
>>> db,
>>> >> >> >> >> therefore in step 5 we never match as already
having a nic
>>> for
>>> >> >> >> >> 'untagged'. I wrote a big long response discussing
this
>>> issue,
>>> >> >> but
>>> >> >> >> as
>>> >> >> >> >> I began to dig further I realized that aside
from my
>>> >> particular
>>> >> >> case,
>>> >> >> >> >> untagged vlans in general are just broken
(for example they
>>> >> can't
>>> >> >> be
>>> >> >> >> >> dealt with uniquely in the current IpAssocCommand
code,
>>> given
>>> >> the
>>> >> >> >> hash
>>> >> >> >> >> map) and it would require more effort than
I have time for
>>> now
>>> >> to
>>> >> >> >> make
>>> >> >> >> >> things work. If the code were already in
place to
>>> >> differentiate
>>> >> >> >> >> between multiple untagged nics I think that
fixing my
>>> problem
>>> >> >> would
>>> >> >> >> be
>>> >> >> >> >> trivial, but since its not, I'll just find
an alternative
>>> >> >> solution.
>>> >> >> >> >>
>>> >> >> >> >
>>> >> >> >> > The untagged network usually means "untagged",
no vlan on
>>> the
>>> >> >> >> bridge...
>>> >> >> >> > In your case, the untagged network actually has
vlan(tagged)
>>> on
>>> >> >> the
>>> >> >> >> bridge, thus getting things confused.
>>> >> >> >> > Will this patch(http://pastebin.com/HJXzZwKp)
work for you?
>>> >> >> >> >
>>> >> >> >> >> On Mon, Aug 27, 2012 at 10:46 PM, Marcus
Sorensen
>>> >> >> >> <shadowsor@gmail.com>
>>> >> >> >> >> wrote:
>>> >> >> >> >> > ...
>>> >> >> >> >> >             Integer nicPos = 0;
>>> >> >> >> >> >             for (InterfaceDef nic :
nics) {
>>> >> >> >> >> >                 if
>>> >> >> >> >> (nic.getBrName().equalsIgnoreCase(_linkLocalBridgeName))
{
>>> >> >> >> >> >                     vlanAllocatedToVM.put("LinkLocal",
>>> >> nicPos);
>>> >> >> >> >> >                 } else {
>>> >> >> >> >> >                     String vlanId =
>>> >> >> >> >> getVlanIdFromBridge(nic.getBrName());
>>> >> >> >> >> >                     if (vlanId != null)
{
>>> >> >> >> >> >                         vlanAllocatedToVM.put(vlanId,
>>> >> nicPos);
>>> >> >> >> >> >                     } else {
>>> >> >> >> >> >
>>> vlanAllocatedToVM.put(Vlan.UNTAGGED,
>>> >> >> >> nicPos);
>>> >> >> >> >> >                     }
>>> >> >> >> >> >                 }
>>> >> >> >> >> >                 nicPos++;
>>> >> >> >> >> >             }
>>> >> >> >> >> >             IpAddressTO[] ips = cmd.getIpAddresses();
>>> >> >> >> >> >             int i = 0;
>>> >> >> >> >> >             String result = null;
>>> >> >> >> >> >             int nicNum = 0;
>>> >> >> >> >> >             for (IpAddressTO ip : ips)
{
>>> >> >> >> >> >                 if
>>> >> >> (!vlanAllocatedToVM.containsKey(ip.getVlanId()))
>>> >> >> >> {
>>> >> >> >> >> >                     /* plug a vif into
router */
>>> >> >> >> >> >                     VifHotPlug(conn,
routerName,
>>> >> ip.getVlanId(),
>>> >> >> >> >> >                             ip.getVifMacAddress());
>>> >> >> >> >> >                     vlanAllocatedToVM.put(ip.getVlanId(),
>>> >> >> >> nicPos++);
>>> >> >> >> >> >                 }
>>> >> >> >> >> > ...
>>> >> >> >> >> >
>>> >> >> >> >> > Looks like the getVlanIdFromBridge might
be a bit
>>> misleading.
>>> >> I
>>> >> >> am
>>> >> >> >> >> > running my guest public traffic on a
'cloudbr470', which
>>> is
>>> >> a
>>> >> >> >> bridge
>>> >> >> >> >> > to eth2.470, yet I configured this network
as 'untagged'
>>> >> >> because I
>>> >> >> >> >> > have a vlan 470 available on eth3 for
cloudstack to
>>> >> autoassign
>>> >> >> >> (eth3
>>> >> >> >> >> > is where all of my stuff will be autoassigned).
So I'm
>>> not
>>> >> 100%
>>> >> >> >> sure
>>> >> >> >> >> > yet what's going on here but it seems
as though the above
>>> is
>>> >> >> not
>>> >> >> >> >> > setting any 'Vlan.UNTAGGED', since it
finds a vlan number
>>> >> for
>>> >> >> >> >> > eth2.470, but when it enumerates the
IPs for the router,
>>> it
>>> >> >> then
>>> >> >> >> runs
>>> >> >> >> >> > ip.getVlanId() and doesn't find a nic
for the untagged IP
>>> >> and
>>> >> >> >> creates
>>> >> >> >> >> > one.
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> > I realize this is perhaps an uncommon
case, but a bug
>>> >> >> nonetheless.
>>> >> >> >> >> > I'll play with the code a bit and see
if I can come up
>>> with
>>> >> a
>>> >> >> >> >> > solution. I'm thinking I can look at
the nic's broadcast
>>> URI
>>> >> >> and
>>> >> >> >> see
>>> >> >> >> >> > if it's supposed to be untagged, then
add to
>>> >> vlanAllocatedToVM
>>> >> >> >> >> > appropriately, off the top of my head
something like:
>>> >> >> >> >> >
>>> >> >> >> >> >                     String vlanId =
>>> >> >> >> >> getVlanIdFromBridge(nic.getBrName());
>>> >> >> >> >> >                     if (vlanId != null
&&
>>> >> >> >> >> > !nic.getBroadcastUri().toString().contains("untagged")
{
>>> >> >> >> >> >                         vlanAllocatedToVM.put(vlanId,
>>> >> nicPos);
>>> >> >> >> >> >                     } else {
>>> >> >> >> >> >
>>> vlanAllocatedToVM.put(Vlan.UNTAGGED,
>>> >> >> >> nicPos);
>>> >> >> >> >> >                     }
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> >
>>> >> >> >> >> > On Mon, Aug 27, 2012 at 6:42 PM, Edison
Su
>>> >> >> <Edison.su@citrix.com>
>>> >> >> >> >> wrote:
>>> >> >> >> >> >> Possible bug in in kvm code: LibvirtComputingResource-
>>> >> >> >> >> >execute(IpAssocCommand cmd)-> VifHotPlug,
which is only
>>> place
>>> >> >> >> adding
>>> >> >> >> >> nic into router vm.
>>> >> >> >> >> >> Turn on agent log, then take a look
what happened.
>>> >> >> >> >> >>
>>> >> >> >> >> >>> -----Original Message-----
>>> >> >> >> >> >>> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>>> >> >> >> >> >>> Sent: Monday, August 27, 2012
5:10 PM
>>> >> >> >> >> >>> To: cloudstack-dev@incubator.apache.org
>>> >> >> >> >> >>> Subject: VM router spawning
multiple public nics
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> I've got two zones running the
same build of cloudstack
>>> (a
>>> >> >> >> recent
>>> >> >> >> >> copy
>>> >> >> >> >> >>> of master). One of them creates
routers that turn into
>>> >> ugly
>>> >> >> >> >> >>> multi-headed beasts, and by
that I mean that any time I
>>> >> >> create a
>>> >> >> >> >> port
>>> >> >> >> >> >>> forwarding or iptables rule
for that router I get a new
>>> >> >> public
>>> >> >> >> NIC
>>> >> >> >> >> >>> with an identical IP address,
I have an instance with a
>>> >> few
>>> >> >> tens
>>> >> >> >> of
>>> >> >> >> >> >>> NICs.  My guess is that some
script isn't detecting
>>> that
>>> >> >> there's
>>> >> >> >> >> >>> already a NIC with the public
IP on it.  It looks fine
>>> in
>>> >> the
>>> >> >> >> >> >>> database, there is only one
public NIC defined in the
>>> nics
>>> >> >> table.
>>> >> >> >> >> >>> I'll troubleshoot it tomorrow,
but if anyone knows
>>> where I
>>> >> >> >> should
>>> >> >> >> >> >>> begin the headstart would be
appreciated.
>>> >> >> >> >> >>>
>>> >> >> >> >> >>> Thanks
>>
>
>

Mime
View raw message