incubator-cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcus Sorensen <shadow...@gmail.com>
Subject VPC for KVM (was VM router spawning multiple public nics)
Date Thu, 30 Aug 2012 15:45:07 GMT
I notice there's also an 'IpAssocVpcCommand', I'm assuming that should
be added to the list?

On Wed, Aug 29, 2012 at 6:37 PM, Edison Su <Edison.su@citrix.com> wrote:
> You can find the VPC reference implementation from CitrixResourceBase.java, which is
the implementation for Xenserver. Just take a look at how the VPC related commands are implemented.
> Take SetNetworkACLCommand as an example:
> The function execute(SetNetworkACLCommand cmd) in citrixResourceBase:
> 1. parse SetNetworkACLCommand
> 2. call scripts/vm/hypervisor/xenserver/vmops, function routerProxy
> 3. routerproxy will call scripts/network/domr/router_proxy.sh
> 4. router_proxy.sh will login into router vm, execute a shell script inside router VM
to program rules.
>
> In KVM, we can directly call router_proxy.sh or directly login into router vm, we just
need to prepare the parameters for this script.
> The reference code is in VirtualRoutingResource.java, all the network related command(extended
from NetworkElementCommand) will be redirected to VirtualRoutingResource.
> We can add implementation of SetupGuestNetworkCommand/SetNetworkACLCommand/SetSourceNatCommand/Site2SiteVpnCfgCommand
in VirtualRoutingResource.
>
>
>> -----Original Message-----
>> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> Sent: Wednesday, August 29, 2012 4:34 PM
>> To: cloudstack-dev@incubator.apache.org
>> Cc: Edison Su; Anthony Xu; Kelven Yang; Vijayendra Bhamidipati
>> Subject: Re: VM router spawning multiple public nics
>>
>> Great, thanks. I've already got a working implementation of
>> adding/removing nics from regular instances that I've been playing
>> with, so I'm getting vaguely familiar with the various data types and
>> things surrounding the networking. I don't know if this is quite
>> within my reach just yet but I'll see how far I get.
>>
>> On Wed, Aug 29, 2012 at 3:19 PM, Alena Prokharchyk
>> <Alena.Prokharchyk@citrix.com> wrote:
>> > On 8/29/12 1:34 PM, "Edison Su" <Edison.su@citrix.com> wrote:
>> >
>> >>Hi Anthony & Alena,
>> >>   Could you help to provide information about VPC, how it works,
>> >
>> >
>> > Here is the functional spec on the feature:
>> >
>> > wiki.cloudstack.org/display/RelOps/Inter-VLAN+Routing+functional+spec
>> >
>> >
>> > VpcVirtualNetworkApplianceManagerImpl is the manager responsible for
>> VPC
>> > Virtual router operations (plug/unplug nics, etc)
>> >
>> >
>> >
>> >> which commands needed to implemented on the hypervisor side?
>> >
>> >
>> >
>> > 1) PlugNicCommand/UnplugNicCommand - does Nic hotplug/unplug
>> (currently
>> > works for VR vm only). In VPC called when add nic for Public/Guest
>> > networks.
>> > 2) SetupGuestNetworkCommand - sets up dhcp range, dns information,
>> > networkDomain information on the Nic to make it
>> > 3) SetNetworkACLCommand - creates network ACL on the virtual router
>> > 4) SetSourceNatCommand - used for setting source nat on the Public IP
>> on
>> > the VPC VR.
>> > 5) Site2SiteVpnCfgCommand - for setting up S2S VPN
>> >
>> >
>> > Anthony/Kelven/Vijay did implementation for Xen/vmWare resources,
>> they can
>> > help you answering all hypervisor related questions. If you need more
>> > details on business logic + Vpc VR management, I can help with that.
>> >
>> >
>> > -Alena.
>> >
>> >>
>> >>> -----Original Message-----
>> >>> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> >>> Sent: Wednesday, August 29, 2012 10:16 AM
>> >>> To: cloudstack-dev@incubator.apache.org
>> >>> Subject: Re: VM router spawning multiple public nics
>> >>>
>> >>> I'd be willing to give it a shot if someone could point me in the
>> >>> right direction and be available to answer questions.
>> >>>
>> >>> On Wed, Aug 29, 2012 at 10:39 AM, Edison Su <Edison.su@citrix.com>
>> >>> wrote:
>> >>> > Yah, KVM doesn't support VPC yet. Will you help to add VPC
>> support on
>> >>> KVM?:) Just implement a few VPC related commands...
>> >>> >
>> >>> >> -----Original Message-----
>> >>> >> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> >>> >> Sent: Wednesday, August 29, 2012 6:49 AM
>> >>> >> To: cloudstack-dev@incubator.apache.org
>> >>> >> Subject: Re: VM router spawning multiple public nics
>> >>> >>
>> >>> >> I can confirm that the patch has fixed my particular issue.
>> >>> >>
>> >>> >> This is likely unrelated and I think it doesn't even use the
>> same
>> >>> >> code, but I began to play with the VPC stuff a bit and noticed
>> that
>> >>> I
>> >>> >> don't get any interfaces except for link local.  I'd probably
>> chalk
>> >>> >> that up to it not being ready for KVM, but I thought it was
>> worth a
>> >>> >> mention.  I'd be happy to try to help get it ready if someone
>> has
>> >>> time
>> >>> >> to nudge me in the right direction.
>> >>> >>
>> >>> >> On Tue, Aug 28, 2012 at 3:15 PM, Edison Su
>> <Edison.su@citrix.com>
>> >>> wrote:
>> >>> >> >
>> >>> >> >
>> >>> >> >> -----Original Message-----
>> >>> >> >> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> >>> >> >> Sent: Tuesday, August 28, 2012 2:00 PM
>> >>> >> >> To: cloudstack-dev@incubator.apache.org
>> >>> >> >> Subject: Re: VM router spawning multiple public nics
>> >>> >> >>
>> >>> >> >>  I thought about this solution myself, but below this
portion
>> of
>> >>> >> code
>> >>> >> >> it looks like it uses the hash map to determine which
nic
>> number
>> >>> to
>> >>> >> >> add the IP to, so with multiple 'untagged' networks
it would
>> have
>> >>> no
>> >>> >> >> way of knowing which nicnum in the router corresponds
with
>> the
>> >>> >> correct
>> >>> >> >> untagged vlan.
>> >>> >> >>
>> >>> >> >>                 nicNum =
>> vlanAllocatedToVM.get(ip.getVlanId());
>> >>> >> >>                 networkUsage(routerIp, "addVif", "eth"
+
>> nicNum);
>> >>> >> >>                 result =
>> >>> >> >> _virtRouterResource.assignPublicIpAddress(routerName,
>> >>> >> >>                         routerIp, ip.getPublicIp(),
>> ip.isAdd(),
>> >>> >> >> ip.isFirstIP(),
>> >>> >> >>                         ip.isSourceNat(), ip.getVlanId(),
>> >>> >> >> ip.getVlanGateway(),
>> >>> >> >>                         ip.getVlanNetmask(),
>> >>> ip.getVifMacAddress(),
>> >>> >> >>                         ip.getGuestIp(), nicNum);
>> >>> >> >>
>> >>> >> >> if ip.getVlanId() returns untagged (as it does on
networks
>> with
>> >>> no
>> >>> >> >> vlan id), and we tried to put multiple untagged keys
in
>> >>> >> >> vlanAllocatedToVM (as with multiple untagged networks),
we
>> get
>> >>> the
>> >>> >> >> wrong nicNum, no?
>> >>> >> >
>> >>> >> > In the ipassoc case, if there are multiple untagged networks,
>> all
>> >>> of
>> >>> >> them are use the same
>> >>> >> > Public bridge. Then multiple ip address will be added
on eth2
>> >>> inside
>> >>> >> router VM.
>> >>> >> > If it works physically, then it works.
>> >>> >> >
>> >>> >> >>
>> >>> >> >> On Tue, Aug 28, 2012 at 2:48 PM, Edison Su
>> <Edison.su@citrix.com>
>> >>> >> wrote:
>> >>> >> >> >
>> >>> >> >> >
>> >>> >> >> >> -----Original Message-----
>> >>> >> >> >> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> >>> >> >> >> Sent: Tuesday, August 28, 2012 1:40 PM
>> >>> >> >> >> To: cloudstack-dev@incubator.apache.org
>> >>> >> >> >> Subject: Re: VM router spawning multiple
public nics
>> >>> >> >> >>
>> >>> >> >> >> Yes, that looks like it would work for me,
however that's
>> not
>> >>> >> >> >> something that would ever make it into master,
right?
>> >>> Essentially
>> >>> >> >> >> killing tagging for the public, private,
and guest traffic
>> >>> labels?
>> >>> >> >> >> There's also still the issue of not being
able to
>> >>> differentiate
>> >>> >> >> >> between multiple untagged networks, if we
wanted to add an
>> IP
>> >>> to
>> >>> >> a
>> >>> >> >> >> router it might not know which untagged interface
to apply
>> it
>> >>> to.
>> >>> >> >> >
>> >>> >> >> > Physically, all the "untagged" network will be
created on
>> >>> >> >> public/guest/private bridge(the name we put in
>> >>> >> >> private/public/guest.bridge.name in agent.properties").
>> >>> >> >> > Because, there is no way to create a new untagged
bridge by
>> >>> agent
>> >>> >> >> itself. Agent code only knows how to create a new
tagged(vlan)
>> >>> >> bridge.
>> >>> >> >> > So the fix should be pushed into master.
>> >>> >> >> >
>> >>> >> >> >>
>> >>> >> >> >> On Tue, Aug 28, 2012 at 2:32 PM, Edison Su
>> >>> <Edison.su@citrix.com>
>> >>> >> >> wrote:
>> >>> >> >> >> >
>> >>> >> >> >> >
>> >>> >> >> >> >> -----Original Message-----
>> >>> >> >> >> >> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> >>> >> >> >> >> Sent: Tuesday, August 28, 2012 12:23
PM
>> >>> >> >> >> >> To: cloudstack-dev@incubator.apache.org
>> >>> >> >> >> >> Subject: Re: VM router spawning
multiple public nics
>> >>> >> >> >> >>
>> >>> >> >> >> >> Thanks for pointing me in the right
direction. I've
>> >>> reviewed
>> >>> >> this
>> >>> >> >> >> code
>> >>> >> >> >> >> in a bit more detail, and it seems
like it's
>> accomplishing
>> >>> the
>> >>> >> >> >> >> following:
>> >>> >> >> >> >>
>> >>> >> >> >> >> 1. get all network interfaces currently
connected to
>> the
>> >>> >> running
>> >>> >> >> VM
>> >>> >> >> >> >> (a.k.a vnet devices via libvirt)
>> >>> >> >> >> >> 2. find out which vlans these network
interfaces are
>> >>> bridged
>> >>> >> to,
>> >>> >> >> >> store
>> >>> >> >> >> >> this in a hash map of vlan ids and
nics
>> >>> >> >> >> >> 3. get all ip addresses to be added
to the VM
>> >>> >> >> >> >> 4. for each ip, get the configured
vlan id for the ip,
>> >>> compare
>> >>> >> it
>> >>> >> >> to
>> >>> >> >> >> >> the hash map of existing vlan ids
and nics
>> >>> >> >> >> >> 5. if the required vlan id is not
found in the hash map,
>> >>> >> create a
>> >>> >> >> >> new
>> >>> >> >> >> >> nic
>> >>> >> >> >> >> 6. assign the ip to the nic identified
by the vlan id
>> key
>> >>> in
>> >>> >> the
>> >>> >> >> >> hash
>> >>> >> >> >> >> map
>> >>> >> >> >> >>
>> >>> >> >> >> >> In this case, we're getting a vlan
id returned in step
>> 2
>> >>> for a
>> >>> >> >> >> bridged
>> >>> >> >> >> >> nic whose network is defined as
untagged in the
>> cloudstack
>> >>> db,
>> >>> >> >> >> >> therefore in step 5 we never match
as already having a
>> nic
>> >>> for
>> >>> >> >> >> >> 'untagged'. I wrote a big long response
discussing this
>> >>> issue,
>> >>> >> >> but
>> >>> >> >> >> as
>> >>> >> >> >> >> I began to dig further I realized
that aside from my
>> >>> >> particular
>> >>> >> >> case,
>> >>> >> >> >> >> untagged vlans in general are just
broken (for example
>> they
>> >>> >> can't
>> >>> >> >> be
>> >>> >> >> >> >> dealt with uniquely in the current
IpAssocCommand code,
>> >>> given
>> >>> >> the
>> >>> >> >> >> hash
>> >>> >> >> >> >> map) and it would require more effort
than I have time
>> for
>> >>> now
>> >>> >> to
>> >>> >> >> >> make
>> >>> >> >> >> >> things work. If the code were already
in place to
>> >>> >> differentiate
>> >>> >> >> >> >> between multiple untagged nics I
think that fixing my
>> >>> problem
>> >>> >> >> would
>> >>> >> >> >> be
>> >>> >> >> >> >> trivial, but since its not, I'll
just find an
>> alternative
>> >>> >> >> solution.
>> >>> >> >> >> >>
>> >>> >> >> >> >
>> >>> >> >> >> > The untagged network usually means "untagged",
no vlan
>> on
>> >>> the
>> >>> >> >> >> bridge...
>> >>> >> >> >> > In your case, the untagged network actually
has
>> vlan(tagged)
>> >>> on
>> >>> >> >> the
>> >>> >> >> >> bridge, thus getting things confused.
>> >>> >> >> >> > Will this patch(http://pastebin.com/HJXzZwKp)
work for
>> you?
>> >>> >> >> >> >
>> >>> >> >> >> >> On Mon, Aug 27, 2012 at 10:46 PM,
Marcus Sorensen
>> >>> >> >> >> <shadowsor@gmail.com>
>> >>> >> >> >> >> wrote:
>> >>> >> >> >> >> > ...
>> >>> >> >> >> >> >             Integer nicPos
= 0;
>> >>> >> >> >> >> >             for (InterfaceDef
nic : nics) {
>> >>> >> >> >> >> >                 if
>> >>> >> >> >> >> (nic.getBrName().equalsIgnoreCase(_linkLocalBridgeName))
>> {
>> >>> >> >> >> >> >                     vlanAllocatedToVM.put("LinkLocal",
>> >>> >> nicPos);
>> >>> >> >> >> >> >                 } else {
>> >>> >> >> >> >> >                     String
vlanId =
>> >>> >> >> >> >> getVlanIdFromBridge(nic.getBrName());
>> >>> >> >> >> >> >                     if (vlanId
!= null) {
>> >>> >> >> >> >> >                         vlanAllocatedToVM.put(vlanId,
>> >>> >> nicPos);
>> >>> >> >> >> >> >                     } else
{
>> >>> >> >> >> >> >
>> >>> vlanAllocatedToVM.put(Vlan.UNTAGGED,
>> >>> >> >> >> nicPos);
>> >>> >> >> >> >> >                     }
>> >>> >> >> >> >> >                 }
>> >>> >> >> >> >> >                 nicPos++;
>> >>> >> >> >> >> >             }
>> >>> >> >> >> >> >             IpAddressTO[] ips
= cmd.getIpAddresses();
>> >>> >> >> >> >> >             int i = 0;
>> >>> >> >> >> >> >             String result =
null;
>> >>> >> >> >> >> >             int nicNum = 0;
>> >>> >> >> >> >> >             for (IpAddressTO
ip : ips) {
>> >>> >> >> >> >> >                 if
>> >>> >> >> (!vlanAllocatedToVM.containsKey(ip.getVlanId()))
>> >>> >> >> >> {
>> >>> >> >> >> >> >                     /* plug
a vif into router */
>> >>> >> >> >> >> >                     VifHotPlug(conn,
routerName,
>> >>> >> ip.getVlanId(),
>> >>> >> >> >> >> >                           
 ip.getVifMacAddress());
>> >>> >> >> >> >> >
>> vlanAllocatedToVM.put(ip.getVlanId(),
>> >>> >> >> >> nicPos++);
>> >>> >> >> >> >> >                 }
>> >>> >> >> >> >> > ...
>> >>> >> >> >> >> >
>> >>> >> >> >> >> > Looks like the getVlanIdFromBridge
might be a bit
>> >>> misleading.
>> >>> >> I
>> >>> >> >> am
>> >>> >> >> >> >> > running my guest public traffic
on a 'cloudbr470',
>> which
>> >>> is
>> >>> >> a
>> >>> >> >> >> bridge
>> >>> >> >> >> >> > to eth2.470, yet I configured
this network as
>> 'untagged'
>> >>> >> >> because I
>> >>> >> >> >> >> > have a vlan 470 available on
eth3 for cloudstack to
>> >>> >> autoassign
>> >>> >> >> >> (eth3
>> >>> >> >> >> >> > is where all of my stuff will
be autoassigned). So
>> I'm
>> >>> not
>> >>> >> 100%
>> >>> >> >> >> sure
>> >>> >> >> >> >> > yet what's going on here but
it seems as though the
>> above
>> >>> is
>> >>> >> >> not
>> >>> >> >> >> >> > setting any 'Vlan.UNTAGGED',
since it finds a vlan
>> number
>> >>> >> for
>> >>> >> >> >> >> > eth2.470, but when it enumerates
the IPs for the
>> router,
>> >>> it
>> >>> >> >> then
>> >>> >> >> >> runs
>> >>> >> >> >> >> > ip.getVlanId() and doesn't
find a nic for the
>> untagged IP
>> >>> >> and
>> >>> >> >> >> creates
>> >>> >> >> >> >> > one.
>> >>> >> >> >> >> >
>> >>> >> >> >> >> >
>> >>> >> >> >> >> > I realize this is perhaps an
uncommon case, but a bug
>> >>> >> >> nonetheless.
>> >>> >> >> >> >> > I'll play with the code a bit
and see if I can come
>> up
>> >>> with
>> >>> >> a
>> >>> >> >> >> >> > solution. I'm thinking I can
look at the nic's
>> broadcast
>> >>> URI
>> >>> >> >> and
>> >>> >> >> >> see
>> >>> >> >> >> >> > if it's supposed to be untagged,
then add to
>> >>> >> vlanAllocatedToVM
>> >>> >> >> >> >> > appropriately, off the top
of my head something like:
>> >>> >> >> >> >> >
>> >>> >> >> >> >> >                     String
vlanId =
>> >>> >> >> >> >> getVlanIdFromBridge(nic.getBrName());
>> >>> >> >> >> >> >                     if (vlanId
!= null &&
>> >>> >> >> >> >> > !nic.getBroadcastUri().toString().contains("untagged")
>> {
>> >>> >> >> >> >> >                         vlanAllocatedToVM.put(vlanId,
>> >>> >> nicPos);
>> >>> >> >> >> >> >                     } else
{
>> >>> >> >> >> >> >
>> >>> vlanAllocatedToVM.put(Vlan.UNTAGGED,
>> >>> >> >> >> nicPos);
>> >>> >> >> >> >> >                     }
>> >>> >> >> >> >> >
>> >>> >> >> >> >> >
>> >>> >> >> >> >> >
>> >>> >> >> >> >> > On Mon, Aug 27, 2012 at 6:42
PM, Edison Su
>> >>> >> >> <Edison.su@citrix.com>
>> >>> >> >> >> >> wrote:
>> >>> >> >> >> >> >> Possible bug in in kvm
code:
>> LibvirtComputingResource-
>> >>> >> >> >> >> >execute(IpAssocCommand cmd)->
VifHotPlug, which is
>> only
>> >>> place
>> >>> >> >> >> adding
>> >>> >> >> >> >> nic into router vm.
>> >>> >> >> >> >> >> Turn on agent log, then
take a look what happened.
>> >>> >> >> >> >> >>
>> >>> >> >> >> >> >>> -----Original Message-----
>> >>> >> >> >> >> >>> From: Marcus Sorensen
[mailto:shadowsor@gmail.com]
>> >>> >> >> >> >> >>> Sent: Monday, August
27, 2012 5:10 PM
>> >>> >> >> >> >> >>> To: cloudstack-dev@incubator.apache.org
>> >>> >> >> >> >> >>> Subject: VM router
spawning multiple public nics
>> >>> >> >> >> >> >>>
>> >>> >> >> >> >> >>> I've got two zones
running the same build of
>> cloudstack
>> >>> (a
>> >>> >> >> >> recent
>> >>> >> >> >> >> copy
>> >>> >> >> >> >> >>> of master). One of
them creates routers that turn
>> into
>> >>> >> ugly
>> >>> >> >> >> >> >>> multi-headed beasts,
and by that I mean that any
>> time I
>> >>> >> >> create a
>> >>> >> >> >> >> port
>> >>> >> >> >> >> >>> forwarding or iptables
rule for that router I get a
>> new
>> >>> >> >> public
>> >>> >> >> >> NIC
>> >>> >> >> >> >> >>> with an identical IP
address, I have an instance
>> with a
>> >>> >> few
>> >>> >> >> tens
>> >>> >> >> >> of
>> >>> >> >> >> >> >>> NICs.  My guess is
that some script isn't detecting
>> >>> that
>> >>> >> >> there's
>> >>> >> >> >> >> >>> already a NIC with
the public IP on it.  It looks
>> fine
>> >>> in
>> >>> >> the
>> >>> >> >> >> >> >>> database, there is
only one public NIC defined in
>> the
>> >>> nics
>> >>> >> >> table.
>> >>> >> >> >> >> >>> I'll troubleshoot it
tomorrow, but if anyone knows
>> >>> where I
>> >>> >> >> >> should
>> >>> >> >> >> >> >>> begin the headstart
would be appreciated.
>> >>> >> >> >> >> >>>
>> >>> >> >> >> >> >>> Thanks
>> >>
>> >
>> >

Mime
View raw message