cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcus Sorensen <shadow...@gmail.com>
Subject Re: VPC for KVM (was VM router spawning multiple public nics)
Date Tue, 04 Sep 2012 17:11:51 GMT
Ok, the SetPortForwardingRulesVpcCommand and SetStaticRouteCommand are
working and have been added to the review request.

I'm going to generate a new e-mail on various non-platform-specific
issues I've come across.  I really haven't seen much on the current
state of Vpc in general.

On Fri, Aug 31, 2012 at 3:22 PM, Edison Su <Edison.su@citrix.com> wrote:
>
>
>> -----Original Message-----
>> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> Sent: Friday, August 31, 2012 1:58 PM
>> To: Edison Su
>> Cc: cloudstack-dev@incubator.apache.org; Anthony Xu; Kelven Yang;
>> Vijayendra Bhamidipati
>> Subject: Re: VPC for KVM (was VM router spawning multiple public nics)
>>
>> Ok, I've gotten to most of the list and things seem to be working.
>> Nics plug, Gateway IPs add, networks create, SNATs working, ACLs
>> working. The ones I haven't gotten to are:
>>
>> *Site2SiteVpnCfgCommand
>>
>> and two which weren't on the original list that I just came across
>> (will check for others):
>> *SetPortForwardingRulesVpcCommand
>> *CheckS2SVpnConnectionsCommand
>> *SetStaticRouteCommand
>
> The commands related to site2site VPN can be finished later, as it needs more complicated setup to test.
> But I'd like to get SetPortForwardingRulesVpcCommand and SetStaticRouteCommand working.
>
>>
>> In addition, the following observations have been made and not yet
>> fixed. If anyone has feedback on these (even a 'yeah, that doesn't
>> work', or 'hmm, that should be working') so I can gauge whether I
>> should dig into them or not I'd appreciate it. I don't want to break
>> something that should be working or is working for someone else.
>>
>> * listNetworkACLs always returns an empty listnetworkaclsresponse
>> * not getting default ACLs, not getting any with new tiers/vms,
>> however setting ACLs via API works
>> * password server is broken (saw the other thread where it's a dev
>> binding issue... is someone actively looking at this?)
>> * static route button in UI doesn't do anything when clicked (I would
>> expect at least a dialogue or a fail message because the
>> SetStaticRouteCommand is missing)
>
> That's ok, seems all of these "issues" are on the management server side.
> Alena, could you help to take a look at, are they real issue?
>
>>
>>
>> On Fri, Aug 31, 2012 at 1:40 PM, Edison Su <Edison.su@citrix.com> wrote:
>> >
>> >
>> >> -----Original Message-----
>> >> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> >> Sent: Thursday, August 30, 2012 10:21 PM
>> >> To: Edison Su
>> >> Cc: cloudstack-dev@incubator.apache.org; Anthony Xu; Kelven Yang;
>> >> Vijayendra Bhamidipati
>> >> Subject: Re: VPC for KVM (was VM router spawning multiple public
>> nics)
>> >>
>> >> I've made some good progress today, I'll probably have it all
>> working
>> >> tomorrow, but I have a question.  Looking at the API command
>> >> associateIpAddress and the functional spec, if I pass a vpcId it
>> then
>> >> calls associateIPToVpc. Unlike its brother, associateIPToNetwork, I
>> >> don't clearly see where the ip is actually put on the router, I only
>> >> see it marking the ip allocated in the database. The functional spec
>> >> reads as though this IP should be on the VPC. Maybe I'm just not
>> >> getting that bolded statement 'All ip addresses get allocated to the
>> >> VPC', my first thought is that of course they would, any IP I
>> allocate
>> >> gets put on the router... I guess I just need to play with it a bit
>> >> more.
>> >
>> > After reading the functional spec, my guess is that:
>> > One VPC has multiple guest networks(one guest network is a tenant,
>> e.g VLAN). On the guest network, there are a lot of VMs running on it.
>> > All of these VMs will use ip address range allocated from guest
>> network(createnetwork api), usually they are using internal ip address.
>> > Regarding to these commands:
>> > 1. AssociateIPAddrCmd
>> > "in VPC setup as a result of associateIpAddress all ip addresses get
>> allocated to VPC " means public ip address(when creating a zone, you'd
>> specify public ip address range) are belong to a VPC, you can associate
>> a public address to a guest network which attached to a VPC.
>> > In the router, it will add a public ip address on an ethX(which is
>> created on public.network.device on kvm host), add a routing
>> table(routing guest network gateway to that public ip). So after that,
>> user VMs created on that guest network should be able to access public
>> network through that public ip.
>> > 2. There are other commands(netwrok), which do some magic inside
>> router vm, basically implement the features like, who can access this
>> guest network, how to set up a portforwording rule for guest network
>> etc.
>> >
>> >>
>> >> Not in front of the code at the moment, I just thought of a few more
>> >> quick questions that would help... What is the expected outcome of
>> the
>> >> default rules? no access between private networks, but public access
>> >> for every network?
>> >
>> > By default, all incoming traffic to guest networks is blocked, but
>> outgoing traffic should work.
>> >
>> >>
>> >> The functional spec mentions public and private gateways, does
>> public
>> >> refer to giving someone say a /24 of public address space and then
>> >> creating a route on our router to their VPC router, i.e. public
>> >> networks instead of NATed ones?
>> >
>> > All of these gateways are referring to gateways inside router VM, not
>> any physical router/gateway.
>> > The public refer to the network created on public.network.device on
>> your kvm host.
>> > The private gateways referring to network created on
>> guest.network.device on your kvm host.
>> > In a VPC setup, the router can have multiple guest network(a.k.
>> multiple private gateways) and multiple public networks.
>> > You can call AssociateIPAddrCmd to bind a public ip to a guest
>> network, then can implement portforwording etc.
>> >
>> > I am not VPC expert, my answer is not authoritative as it looks like:)
>> Correct me if I am wrong.
>> >
>> >>
>> >> On Thu, Aug 30, 2012 at 11:54 AM, Marcus Sorensen
>> <shadowsor@gmail.com>
>> >> wrote:
>> >> > Yes, thanks for clarifying.
>> >> >
>> >> > On Thu, Aug 30, 2012 at 11:53 AM, Edison Su <Edison.su@citrix.com>
>> >> wrote:
>> >> >> Oh, I see, there are two kinds of commands:
>> >> >> One doesn't need to get information from libvirt, which just
>> login
>> >> into router vm and program rules. For these commands, we can put
>> them
>> >> into virtualRoutingResource.
>> >> >> Another does need to get infor from libvirt, such ipassoc, which
>> >> need to get nic infor from libvirt, and plug in vif if needed. For
>> >> these commands, need to be implemented in libvirtcomputingresource.
>> >> While if it needs to access/program router, then add put that code
>> in
>> >> virtualroutingresource.
>> >> >>
>> >> >> Does it make sense?
>> >> >>
>> >> >>> -----Original Message-----
>> >> >>> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> >> >>> Sent: Thursday, August 30, 2012 10:47 AM
>> >> >>> To: Edison Su
>> >> >>> Cc: cloudstack-dev@incubator.apache.org; Anthony Xu; Kelven Yang;
>> >> >>> Vijayendra Bhamidipati
>> >> >>> Subject: Re: VPC for KVM (was VM router spawning multiple public
>> >> nics)
>> >> >>>
>> >> >>> Ok, I think I get it. For example when I'm looking at
>> >> IpAssocCommand
>> >> >>> it calls _virtRouterResource.assignPublicIpAddress to actually
>> run
>> >> the
>> >> >>> sh.  So with SetupGuestNetworkCommand, I'll create that in
>> >> >>> LibvirtComputingResource, and within that I'll make a call to
>> >> >>> VirtualRoutingResource to do the work... is that right?
>> >> >>>
>> >> >>> On Thu, Aug 30, 2012 at 11:31 AM, Marcus Sorensen
>> >> <shadowsor@gmail.com>
>> >> >>> wrote:
>> >> >>> > Just want to clarify:
>> >> >>> >
>> >> >>> > "We can add implementation of
>> >> >>> >
>> >> >>>
>> >>
>> SetupGuestNetworkCommand/SetNetworkACLCommand/SetSourceNatCommand/Site2
>> >> >>> SiteVpnCfgCommand
>> >> >>> > in VirtualRoutingResource."
>> >> >>> >
>> >> >>> > So the Xen implementation of SetupGuestNetworkCommand is in
>> >> >>> > CitrixResourceBase.java, and the VMware one is in
>> >> VmwareResource.java,
>> >> >>> > but you're saying I should put an implementation of these in
>> >> >>> > VirtualRoutingResource.java for KVM? Sorry, still trying to
>> piece
>> >> >>> > together how everything relates.
>> >> >>> >
>> >> >>> > On Thu, Aug 30, 2012 at 10:42 AM, Edison Su
>> >> <Edison.su@citrix.com>
>> >> >>> wrote:
>> >> >>> >> Yes, it is.
>> >> >>> >>
>> >> >>> >>> -----Original Message-----
>> >> >>> >>> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> >> >>> >>> Sent: Thursday, August 30, 2012 8:45 AM
>> >> >>> >>> To: Edison Su
>> >> >>> >>> Cc: cloudstack-dev@incubator.apache.org; Anthony Xu; Kelven
>> >> Yang;
>> >> >>> >>> Vijayendra Bhamidipati
>> >> >>> >>> Subject: VPC for KVM (was VM router spawning multiple public
>> >> nics)
>> >> >>> >>>
>> >> >>> >>> I notice there's also an 'IpAssocVpcCommand', I'm assuming
>> that
>> >> >>> should
>> >> >>> >>> be added to the list?
>> >> >>> >>>
>> >> >>> >>> On Wed, Aug 29, 2012 at 6:37 PM, Edison Su
>> >> <Edison.su@citrix.com>
>> >> >>> wrote:
>> >> >>> >>> > You can find the VPC reference implementation from
>> >> >>> >>> CitrixResourceBase.java, which is the implementation for
>> >> Xenserver.
>> >> >>> >>> Just take a look at how the VPC related commands are
>> >> implemented.
>> >> >>> >>> > Take SetNetworkACLCommand as an example:
>> >> >>> >>> > The function execute(SetNetworkACLCommand cmd) in
>> >> >>> citrixResourceBase:
>> >> >>> >>> > 1. parse SetNetworkACLCommand
>> >> >>> >>> > 2. call scripts/vm/hypervisor/xenserver/vmops, function
>> >> >>> routerProxy
>> >> >>> >>> > 3. routerproxy will call
>> scripts/network/domr/router_proxy.sh
>> >> >>> >>> > 4. router_proxy.sh will login into router vm, execute a
>> shell
>> >> >>> script
>> >> >>> >>> inside router VM to program rules.
>> >> >>> >>> >
>> >> >>> >>> > In KVM, we can directly call router_proxy.sh or directly
>> >> login
>> >> >>> into
>> >> >>> >>> router vm, we just need to prepare the parameters for this
>> >> script.
>> >> >>> >>> > The reference code is in VirtualRoutingResource.java, all
>> the
>> >> >>> network
>> >> >>> >>> related command(extended from NetworkElementCommand) will be
>> >> >>> redirected
>> >> >>> >>> to VirtualRoutingResource.
>> >> >>> >>> > We can add implementation of
>> >> >>> >>>
>> >> >>>
>> >>
>> SetupGuestNetworkCommand/SetNetworkACLCommand/SetSourceNatCommand/Site2
>> >> >>> >>> SiteVpnCfgCommand in VirtualRoutingResource.
>> >> >>> >>> >
>> >> >>> >>> >
>> >> >>> >>> >> -----Original Message-----
>> >> >>> >>> >> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> >> >>> >>> >> Sent: Wednesday, August 29, 2012 4:34 PM
>> >> >>> >>> >> To: cloudstack-dev@incubator.apache.org
>> >> >>> >>> >> Cc: Edison Su; Anthony Xu; Kelven Yang; Vijayendra
>> >> Bhamidipati
>> >> >>> >>> >> Subject: Re: VM router spawning multiple public nics
>> >> >>> >>> >>
>> >> >>> >>> >> Great, thanks. I've already got a working implementation
>> of
>> >> >>> >>> >> adding/removing nics from regular instances that I've
>> been
>> >> >>> playing
>> >> >>> >>> >> with, so I'm getting vaguely familiar with the various
>> data
>> >> >>> types
>> >> >>> >>> and
>> >> >>> >>> >> things surrounding the networking. I don't know if this
>> is
>> >> quite
>> >> >>> >>> >> within my reach just yet but I'll see how far I get.
>> >> >>> >>> >>
>> >> >>> >>> >> On Wed, Aug 29, 2012 at 3:19 PM, Alena Prokharchyk
>> >> >>> >>> >> <Alena.Prokharchyk@citrix.com> wrote:
>> >> >>> >>> >> > On 8/29/12 1:34 PM, "Edison Su" <Edison.su@citrix.com>
>> >> wrote:
>> >> >>> >>> >> >
>> >> >>> >>> >> >>Hi Anthony & Alena,
>> >> >>> >>> >> >>   Could you help to provide information about VPC, how
>> it
>> >> >>> works,
>> >> >>> >>> >> >
>> >> >>> >>> >> >
>> >> >>> >>> >> > Here is the functional spec on the feature:
>> >> >>> >>> >> >
>> >> >>> >>> >> > wiki.cloudstack.org/display/RelOps/Inter-
>> >> >>> >>> VLAN+Routing+functional+spec
>> >> >>> >>> >> >
>> >> >>> >>> >> >
>> >> >>> >>> >> > VpcVirtualNetworkApplianceManagerImpl is the manager
>> >> >>> responsible
>> >> >>> >>> for
>> >> >>> >>> >> VPC
>> >> >>> >>> >> > Virtual router operations (plug/unplug nics, etc)
>> >> >>> >>> >> >
>> >> >>> >>> >> >
>> >> >>> >>> >> >
>> >> >>> >>> >> >> which commands needed to implemented on the hypervisor
>> >> side?
>> >> >>> >>> >> >
>> >> >>> >>> >> >
>> >> >>> >>> >> >
>> >> >>> >>> >> > 1) PlugNicCommand/UnplugNicCommand - does Nic
>> >> hotplug/unplug
>> >> >>> >>> >> (currently
>> >> >>> >>> >> > works for VR vm only). In VPC called when add nic for
>> >> >>> Public/Guest
>> >> >>> >>> >> > networks.
>> >> >>> >>> >> > 2) SetupGuestNetworkCommand - sets up dhcp range, dns
>> >> >>> information,
>> >> >>> >>> >> > networkDomain information on the Nic to make it
>> >> >>> >>> >> > 3) SetNetworkACLCommand - creates network ACL on the
>> >> virtual
>> >> >>> >>> router
>> >> >>> >>> >> > 4) SetSourceNatCommand - used for setting source nat on
>> >> the
>> >> >>> Public
>> >> >>> >>> IP
>> >> >>> >>> >> on
>> >> >>> >>> >> > the VPC VR.
>> >> >>> >>> >> > 5) Site2SiteVpnCfgCommand - for setting up S2S VPN
>> >> >>> >>> >> >
>> >> >>> >>> >> >
>> >> >>> >>> >> > Anthony/Kelven/Vijay did implementation for Xen/vmWare
>> >> >>> resources,
>> >> >>> >>> >> they can
>> >> >>> >>> >> > help you answering all hypervisor related questions. If
>> >> you
>> >> >>> need
>> >> >>> >>> more
>> >> >>> >>> >> > details on business logic + Vpc VR management, I can
>> help
>> >> with
>> >> >>> >>> that.
>> >> >>> >>> >> >
>> >> >>> >>> >> >
>> >> >>> >>> >> > -Alena.
>> >> >>> >>> >> >
>> >> >>> >>> >> >>
>> >> >>> >>> >> >>> -----Original Message-----
>> >> >>> >>> >> >>> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> >> >>> >>> >> >>> Sent: Wednesday, August 29, 2012 10:16 AM
>> >> >>> >>> >> >>> To: cloudstack-dev@incubator.apache.org
>> >> >>> >>> >> >>> Subject: Re: VM router spawning multiple public nics
>> >> >>> >>> >> >>>
>> >> >>> >>> >> >>> I'd be willing to give it a shot if someone could
>> point
>> >> me
>> >> >>> in
>> >> >>> >>> the
>> >> >>> >>> >> >>> right direction and be available to answer questions.
>> >> >>> >>> >> >>>
>> >> >>> >>> >> >>> On Wed, Aug 29, 2012 at 10:39 AM, Edison Su
>> >> >>> >>> <Edison.su@citrix.com>
>> >> >>> >>> >> >>> wrote:
>> >> >>> >>> >> >>> > Yah, KVM doesn't support VPC yet. Will you help to
>> add
>> >> VPC
>> >> >>> >>> >> support on
>> >> >>> >>> >> >>> KVM?:) Just implement a few VPC related commands...
>> >> >>> >>> >> >>> >
>> >> >>> >>> >> >>> >> -----Original Message-----
>> >> >>> >>> >> >>> >> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
>> >> >>> >>> >> >>> >> Sent: Wednesday, August 29, 2012 6:49 AM
>> >> >>> >>> >> >>> >> To: cloudstack-dev@incubator.apache.org
>> >> >>> >>> >> >>> >> Subject: Re: VM router spawning multiple public
>> nics
>> >> >>> >>> >> >>> >>
>> >> >>> >>> >> >>> >> I can confirm that the patch has fixed my
>> particular
>> >> >>> issue.
>> >> >>> >>> >> >>> >>
>> >> >>> >>> >> >>> >> This is likely unrelated and I think it doesn't
>> even
>> >> use
>> >> >>> the
>> >> >>> >>> >> same
>> >> >>> >>> >> >>> >> code, but I began to play with the VPC stuff a bit
>> >> and
>> >> >>> >>> noticed
>> >> >>> >>> >> that
>> >> >>> >>> >> >>> I
>> >> >>> >>> >> >>> >> don't get any interfaces except for link local.
>> I'd
>> >> >>> probably
>> >> >>> >>> >> chalk
>> >> >>> >>> >> >>> >> that up to it not being ready for KVM, but I
>> thought
>> >> it
>> >> >>> was
>> >> >>> >>> >> worth a
>> >> >>> >>> >> >>> >> mention.  I'd be happy to try to help get it ready
>> if
>> >> >>> someone
>> >> >>> >>> >> has
>> >> >>> >>> >> >>> time
>> >> >>> >>> >> >>> >> to nudge me in the right direction.
>> >> >>> >>> >> >>> >>
>> >> >>> >>> >> >>> >> On Tue, Aug 28, 2012 at 3:15 PM, Edison Su
>> >> >>> >>> >> <Edison.su@citrix.com>
>> >> >>> >>> >> >>> wrote:
>> >> >>> >>> >> >>> >> >
>> >> >>> >>> >> >>> >> >
>> >> >>> >>> >> >>> >> >> -----Original Message-----
>> >> >>> >>> >> >>> >> >> From: Marcus Sorensen
>> [mailto:shadowsor@gmail.com]
>> >> >>> >>> >> >>> >> >> Sent: Tuesday, August 28, 2012 2:00 PM
>> >> >>> >>> >> >>> >> >> To: cloudstack-dev@incubator.apache.org
>> >> >>> >>> >> >>> >> >> Subject: Re: VM router spawning multiple public
>> >> nics
>> >> >>> >>> >> >>> >> >>
>> >> >>> >>> >> >>> >> >>  I thought about this solution myself, but
>> below
>> >> this
>> >> >>> >>> portion
>> >> >>> >>> >> of
>> >> >>> >>> >> >>> >> code
>> >> >>> >>> >> >>> >> >> it looks like it uses the hash map to determine
>> >> which
>> >> >>> nic
>> >> >>> >>> >> number
>> >> >>> >>> >> >>> to
>> >> >>> >>> >> >>> >> >> add the IP to, so with multiple 'untagged'
>> >> networks it
>> >> >>> >>> would
>> >> >>> >>> >> have
>> >> >>> >>> >> >>> no
>> >> >>> >>> >> >>> >> >> way of knowing which nicnum in the router
>> >> corresponds
>> >> >>> with
>> >> >>> >>> >> the
>> >> >>> >>> >> >>> >> correct
>> >> >>> >>> >> >>> >> >> untagged vlan.
>> >> >>> >>> >> >>> >> >>
>> >> >>> >>> >> >>> >> >>                 nicNum =
>> >> >>> >>> >> vlanAllocatedToVM.get(ip.getVlanId());
>> >> >>> >>> >> >>> >> >>                 networkUsage(routerIp, "addVif",
>> >> "eth"
>> >> >>> +
>> >> >>> >>> >> nicNum);
>> >> >>> >>> >> >>> >> >>                 result =
>> >> >>> >>> >> >>> >> >>
>> >> _virtRouterResource.assignPublicIpAddress(routerName,
>> >> >>> >>> >> >>> >> >>                         routerIp,
>> ip.getPublicIp(),
>> >> >>> >>> >> ip.isAdd(),
>> >> >>> >>> >> >>> >> >> ip.isFirstIP(),
>> >> >>> >>> >> >>> >> >>                         ip.isSourceNat(),
>> >> >>> ip.getVlanId(),
>> >> >>> >>> >> >>> >> >> ip.getVlanGateway(),
>> >> >>> >>> >> >>> >> >>                         ip.getVlanNetmask(),
>> >> >>> >>> >> >>> ip.getVifMacAddress(),
>> >> >>> >>> >> >>> >> >>                         ip.getGuestIp(),
>> nicNum);
>> >> >>> >>> >> >>> >> >>
>> >> >>> >>> >> >>> >> >> if ip.getVlanId() returns untagged (as it does
>> on
>> >> >>> networks
>> >> >>> >>> >> with
>> >> >>> >>> >> >>> no
>> >> >>> >>> >> >>> >> >> vlan id), and we tried to put multiple untagged
>> >> keys
>> >> >>> in
>> >> >>> >>> >> >>> >> >> vlanAllocatedToVM (as with multiple untagged
>> >> networks),
>> >> >>> we
>> >> >>> >>> >> get
>> >> >>> >>> >> >>> the
>> >> >>> >>> >> >>> >> >> wrong nicNum, no?
>> >> >>> >>> >> >>> >> >
>> >> >>> >>> >> >>> >> > In the ipassoc case, if there are multiple
>> untagged
>> >> >>> >>> networks,
>> >> >>> >>> >> all
>> >> >>> >>> >> >>> of
>> >> >>> >>> >> >>> >> them are use the same
>> >> >>> >>> >> >>> >> > Public bridge. Then multiple ip address will be
>> >> added
>> >> >>> on
>> >> >>> >>> eth2
>> >> >>> >>> >> >>> inside
>> >> >>> >>> >> >>> >> router VM.
>> >> >>> >>> >> >>> >> > If it works physically, then it works.
>> >> >>> >>> >> >>> >> >
>> >> >>> >>> >> >>> >> >>
>> >> >>> >>> >> >>> >> >> On Tue, Aug 28, 2012 at 2:48 PM, Edison Su
>> >> >>> >>> >> <Edison.su@citrix.com>
>> >> >>> >>> >> >>> >> wrote:
>> >> >>> >>> >> >>> >> >> >
>> >> >>> >>> >> >>> >> >> >
>> >> >>> >>> >> >>> >> >> >> -----Original Message-----
>> >> >>> >>> >> >>> >> >> >> From: Marcus Sorensen
>> >> [mailto:shadowsor@gmail.com]
>> >> >>> >>> >> >>> >> >> >> Sent: Tuesday, August 28, 2012 1:40 PM
>> >> >>> >>> >> >>> >> >> >> To: cloudstack-dev@incubator.apache.org
>> >> >>> >>> >> >>> >> >> >> Subject: Re: VM router spawning multiple
>> public
>> >> >>> nics
>> >> >>> >>> >> >>> >> >> >>
>> >> >>> >>> >> >>> >> >> >> Yes, that looks like it would work for me,
>> >> however
>> >> >>> >>> that's
>> >> >>> >>> >> not
>> >> >>> >>> >> >>> >> >> >> something that would ever make it into
>> master,
>> >> >>> right?
>> >> >>> >>> >> >>> Essentially
>> >> >>> >>> >> >>> >> >> >> killing tagging for the public, private, and
>> >> guest
>> >> >>> >>> traffic
>> >> >>> >>> >> >>> labels?
>> >> >>> >>> >> >>> >> >> >> There's also still the issue of not being
>> able
>> >> to
>> >> >>> >>> >> >>> differentiate
>> >> >>> >>> >> >>> >> >> >> between multiple untagged networks, if we
>> >> wanted to
>> >> >>> add
>> >> >>> >>> an
>> >> >>> >>> >> IP
>> >> >>> >>> >> >>> to
>> >> >>> >>> >> >>> >> a
>> >> >>> >>> >> >>> >> >> >> router it might not know which untagged
>> >> interface
>> >> >>> to
>> >> >>> >>> apply
>> >> >>> >>> >> it
>> >> >>> >>> >> >>> to.
>> >> >>> >>> >> >>> >> >> >
>> >> >>> >>> >> >>> >> >> > Physically, all the "untagged" network will
>> be
>> >> >>> created
>> >> >>> >>> on
>> >> >>> >>> >> >>> >> >> public/guest/private bridge(the name we put in
>> >> >>> >>> >> >>> >> >> private/public/guest.bridge.name in
>> >> agent.properties").
>> >> >>> >>> >> >>> >> >> > Because, there is no way to create a new
>> >> untagged
>> >> >>> bridge
>> >> >>> >>> by
>> >> >>> >>> >> >>> agent
>> >> >>> >>> >> >>> >> >> itself. Agent code only knows how to create a
>> new
>> >> >>> >>> tagged(vlan)
>> >> >>> >>> >> >>> >> bridge.
>> >> >>> >>> >> >>> >> >> > So the fix should be pushed into master.
>> >> >>> >>> >> >>> >> >> >
>> >> >>> >>> >> >>> >> >> >>
>> >> >>> >>> >> >>> >> >> >> On Tue, Aug 28, 2012 at 2:32 PM, Edison Su
>> >> >>> >>> >> >>> <Edison.su@citrix.com>
>> >> >>> >>> >> >>> >> >> wrote:
>> >> >>> >>> >> >>> >> >> >> >
>> >> >>> >>> >> >>> >> >> >> >
>> >> >>> >>> >> >>> >> >> >> >> -----Original Message-----
>> >> >>> >>> >> >>> >> >> >> >> From: Marcus Sorensen
>> >> >>> [mailto:shadowsor@gmail.com]
>> >> >>> >>> >> >>> >> >> >> >> Sent: Tuesday, August 28, 2012 12:23 PM
>> >> >>> >>> >> >>> >> >> >> >> To: cloudstack-dev@incubator.apache.org
>> >> >>> >>> >> >>> >> >> >> >> Subject: Re: VM router spawning multiple
>> >> public
>> >> >>> nics
>> >> >>> >>> >> >>> >> >> >> >>
>> >> >>> >>> >> >>> >> >> >> >> Thanks for pointing me in the right
>> >> direction.
>> >> >>> I've
>> >> >>> >>> >> >>> reviewed
>> >> >>> >>> >> >>> >> this
>> >> >>> >>> >> >>> >> >> >> code
>> >> >>> >>> >> >>> >> >> >> >> in a bit more detail, and it seems like
>> it's
>> >> >>> >>> >> accomplishing
>> >> >>> >>> >> >>> the
>> >> >>> >>> >> >>> >> >> >> >> following:
>> >> >>> >>> >> >>> >> >> >> >>
>> >> >>> >>> >> >>> >> >> >> >> 1. get all network interfaces currently
>> >> >>> connected to
>> >> >>> >>> >> the
>> >> >>> >>> >> >>> >> running
>> >> >>> >>> >> >>> >> >> VM
>> >> >>> >>> >> >>> >> >> >> >> (a.k.a vnet devices via libvirt)
>> >> >>> >>> >> >>> >> >> >> >> 2. find out which vlans these network
>> >> interfaces
>> >> >>> are
>> >> >>> >>> >> >>> bridged
>> >> >>> >>> >> >>> >> to,
>> >> >>> >>> >> >>> >> >> >> store
>> >> >>> >>> >> >>> >> >> >> >> this in a hash map of vlan ids and nics
>> >> >>> >>> >> >>> >> >> >> >> 3. get all ip addresses to be added to
>> the
>> >> VM
>> >> >>> >>> >> >>> >> >> >> >> 4. for each ip, get the configured vlan
>> id
>> >> for
>> >> >>> the
>> >> >>> >>> ip,
>> >> >>> >>> >> >>> compare
>> >> >>> >>> >> >>> >> it
>> >> >>> >>> >> >>> >> >> to
>> >> >>> >>> >> >>> >> >> >> >> the hash map of existing vlan ids and
>> nics
>> >> >>> >>> >> >>> >> >> >> >> 5. if the required vlan id is not found
>> in
>> >> the
>> >> >>> hash
>> >> >>> >>> map,
>> >> >>> >>> >> >>> >> create a
>> >> >>> >>> >> >>> >> >> >> new
>> >> >>> >>> >> >>> >> >> >> >> nic
>> >> >>> >>> >> >>> >> >> >> >> 6. assign the ip to the nic identified by
>> >> the
>> >> >>> vlan
>> >> >>> >>> id
>> >> >>> >>> >> key
>> >> >>> >>> >> >>> in
>> >> >>> >>> >> >>> >> the
>> >> >>> >>> >> >>> >> >> >> hash
>> >> >>> >>> >> >>> >> >> >> >> map
>> >> >>> >>> >> >>> >> >> >> >>
>> >> >>> >>> >> >>> >> >> >> >> In this case, we're getting a vlan id
>> >> returned
>> >> >>> in
>> >> >>> >>> step
>> >> >>> >>> >> 2
>> >> >>> >>> >> >>> for a
>> >> >>> >>> >> >>> >> >> >> bridged
>> >> >>> >>> >> >>> >> >> >> >> nic whose network is defined as untagged
>> in
>> >> the
>> >> >>> >>> >> cloudstack
>> >> >>> >>> >> >>> db,
>> >> >>> >>> >> >>> >> >> >> >> therefore in step 5 we never match as
>> >> already
>> >> >>> having
>> >> >>> >>> a
>> >> >>> >>> >> nic
>> >> >>> >>> >> >>> for
>> >> >>> >>> >> >>> >> >> >> >> 'untagged'. I wrote a big long response
>> >> >>> discussing
>> >> >>> >>> this
>> >> >>> >>> >> >>> issue,
>> >> >>> >>> >> >>> >> >> but
>> >> >>> >>> >> >>> >> >> >> as
>> >> >>> >>> >> >>> >> >> >> >> I began to dig further I realized that
>> aside
>> >> >>> from my
>> >> >>> >>> >> >>> >> particular
>> >> >>> >>> >> >>> >> >> case,
>> >> >>> >>> >> >>> >> >> >> >> untagged vlans in general are just broken
>> >> (for
>> >> >>> >>> example
>> >> >>> >>> >> they
>> >> >>> >>> >> >>> >> can't
>> >> >>> >>> >> >>> >> >> be
>> >> >>> >>> >> >>> >> >> >> >> dealt with uniquely in the current
>> >> >>> IpAssocCommand
>> >> >>> >>> code,
>> >> >>> >>> >> >>> given
>> >> >>> >>> >> >>> >> the
>> >> >>> >>> >> >>> >> >> >> hash
>> >> >>> >>> >> >>> >> >> >> >> map) and it would require more effort
>> than I
>> >> >>> have
>> >> >>> >>> time
>> >> >>> >>> >> for
>> >> >>> >>> >> >>> now
>> >> >>> >>> >> >>> >> to
>> >> >>> >>> >> >>> >> >> >> make
>> >> >>> >>> >> >>> >> >> >> >> things work. If the code were already in
>> >> place
>> >> >>> to
>> >> >>> >>> >> >>> >> differentiate
>> >> >>> >>> >> >>> >> >> >> >> between multiple untagged nics I think
>> that
>> >> >>> fixing
>> >> >>> >>> my
>> >> >>> >>> >> >>> problem
>> >> >>> >>> >> >>> >> >> would
>> >> >>> >>> >> >>> >> >> >> be
>> >> >>> >>> >> >>> >> >> >> >> trivial, but since its not, I'll just
>> find
>> >> an
>> >> >>> >>> >> alternative
>> >> >>> >>> >> >>> >> >> solution.
>> >> >>> >>> >> >>> >> >> >> >>
>> >> >>> >>> >> >>> >> >> >> >
>> >> >>> >>> >> >>> >> >> >> > The untagged network usually means
>> "untagged",
>> >> no
>> >> >>> >>> vlan
>> >> >>> >>> >> on
>> >> >>> >>> >> >>> the
>> >> >>> >>> >> >>> >> >> >> bridge...
>> >> >>> >>> >> >>> >> >> >> > In your case, the untagged network
>> actually
>> >> has
>> >> >>> >>> >> vlan(tagged)
>> >> >>> >>> >> >>> on
>> >> >>> >>> >> >>> >> >> the
>> >> >>> >>> >> >>> >> >> >> bridge, thus getting things confused.
>> >> >>> >>> >> >>> >> >> >> > Will this
>> patch(http://pastebin.com/HJXzZwKp)
>> >> >>> work
>> >> >>> >>> for
>> >> >>> >>> >> you?
>> >> >>> >>> >> >>> >> >> >> >
>> >> >>> >>> >> >>> >> >> >> >> On Mon, Aug 27, 2012 at 10:46 PM, Marcus
>> >> >>> Sorensen
>> >> >>> >>> >> >>> >> >> >> <shadowsor@gmail.com>
>> >> >>> >>> >> >>> >> >> >> >> wrote:
>> >> >>> >>> >> >>> >> >> >> >> > ...
>> >> >>> >>> >> >>> >> >> >> >> >             Integer nicPos = 0;
>> >> >>> >>> >> >>> >> >> >> >> >             for (InterfaceDef nic :
>> nics)
>> >> {
>> >> >>> >>> >> >>> >> >> >> >> >                 if
>> >> >>> >>> >> >>> >> >> >> >>
>> >> >>> >>> (nic.getBrName().equalsIgnoreCase(_linkLocalBridgeName))
>> >> >>> >>> >> {
>> >> >>> >>> >> >>> >> >> >> >> >
>> >> >>> >>> vlanAllocatedToVM.put("LinkLocal",
>> >> >>> >>> >> >>> >> nicPos);
>> >> >>> >>> >> >>> >> >> >> >> >                 } else {
>> >> >>> >>> >> >>> >> >> >> >> >                     String vlanId =
>> >> >>> >>> >> >>> >> >> >> >> getVlanIdFromBridge(nic.getBrName());
>> >> >>> >>> >> >>> >> >> >> >> >                     if (vlanId != null)
>> {
>> >> >>> >>> >> >>> >> >> >> >> >
>> >> >>> >>> vlanAllocatedToVM.put(vlanId,
>> >> >>> >>> >> >>> >> nicPos);
>> >> >>> >>> >> >>> >> >> >> >> >                     } else {
>> >> >>> >>> >> >>> >> >> >> >> >
>> >> >>> >>> >> >>> vlanAllocatedToVM.put(Vlan.UNTAGGED,
>> >> >>> >>> >> >>> >> >> >> nicPos);
>> >> >>> >>> >> >>> >> >> >> >> >                     }
>> >> >>> >>> >> >>> >> >> >> >> >                 }
>> >> >>> >>> >> >>> >> >> >> >> >                 nicPos++;
>> >> >>> >>> >> >>> >> >> >> >> >             }
>> >> >>> >>> >> >>> >> >> >> >> >             IpAddressTO[] ips =
>> >> >>> >>> cmd.getIpAddresses();
>> >> >>> >>> >> >>> >> >> >> >> >             int i = 0;
>> >> >>> >>> >> >>> >> >> >> >> >             String result = null;
>> >> >>> >>> >> >>> >> >> >> >> >             int nicNum = 0;
>> >> >>> >>> >> >>> >> >> >> >> >             for (IpAddressTO ip : ips)
>> {
>> >> >>> >>> >> >>> >> >> >> >> >                 if
>> >> >>> >>> >> >>> >> >> (!vlanAllocatedToVM.containsKey(ip.getVlanId()))
>> >> >>> >>> >> >>> >> >> >> {
>> >> >>> >>> >> >>> >> >> >> >> >                     /* plug a vif into
>> >> router
>> >> >>> */
>> >> >>> >>> >> >>> >> >> >> >> >                     VifHotPlug(conn,
>> >> >>> routerName,
>> >> >>> >>> >> >>> >> ip.getVlanId(),
>> >> >>> >>> >> >>> >> >> >> >> >
>> >> >>> ip.getVifMacAddress());
>> >> >>> >>> >> >>> >> >> >> >> >
>> >> >>> >>> >> vlanAllocatedToVM.put(ip.getVlanId(),
>> >> >>> >>> >> >>> >> >> >> nicPos++);
>> >> >>> >>> >> >>> >> >> >> >> >                 }
>> >> >>> >>> >> >>> >> >> >> >> > ...
>> >> >>> >>> >> >>> >> >> >> >> >
>> >> >>> >>> >> >>> >> >> >> >> > Looks like the getVlanIdFromBridge
>> might
>> >> be a
>> >> >>> bit
>> >> >>> >>> >> >>> misleading.
>> >> >>> >>> >> >>> >> I
>> >> >>> >>> >> >>> >> >> am
>> >> >>> >>> >> >>> >> >> >> >> > running my guest public traffic on a
>> >> >>> 'cloudbr470',
>> >> >>> >>> >> which
>> >> >>> >>> >> >>> is
>> >> >>> >>> >> >>> >> a
>> >> >>> >>> >> >>> >> >> >> bridge
>> >> >>> >>> >> >>> >> >> >> >> > to eth2.470, yet I configured this
>> network
>> >> as
>> >> >>> >>> >> 'untagged'
>> >> >>> >>> >> >>> >> >> because I
>> >> >>> >>> >> >>> >> >> >> >> > have a vlan 470 available on eth3 for
>> >> >>> cloudstack
>> >> >>> >>> to
>> >> >>> >>> >> >>> >> autoassign
>> >> >>> >>> >> >>> >> >> >> (eth3
>> >> >>> >>> >> >>> >> >> >> >> > is where all of my stuff will be
>> >> autoassigned).
>> >> >>> So
>> >> >>> >>> >> I'm
>> >> >>> >>> >> >>> not
>> >> >>> >>> >> >>> >> 100%
>> >> >>> >>> >> >>> >> >> >> sure
>> >> >>> >>> >> >>> >> >> >> >> > yet what's going on here but it seems
>> as
>> >> >>> though
>> >> >>> >>> the
>> >> >>> >>> >> above
>> >> >>> >>> >> >>> is
>> >> >>> >>> >> >>> >> >> not
>> >> >>> >>> >> >>> >> >> >> >> > setting any 'Vlan.UNTAGGED', since it
>> >> finds a
>> >> >>> vlan
>> >> >>> >>> >> number
>> >> >>> >>> >> >>> >> for
>> >> >>> >>> >> >>> >> >> >> >> > eth2.470, but when it enumerates the
>> IPs
>> >> for
>> >> >>> the
>> >> >>> >>> >> router,
>> >> >>> >>> >> >>> it
>> >> >>> >>> >> >>> >> >> then
>> >> >>> >>> >> >>> >> >> >> runs
>> >> >>> >>> >> >>> >> >> >> >> > ip.getVlanId() and doesn't find a nic
>> for
>> >> the
>> >> >>> >>> >> untagged IP
>> >> >>> >>> >> >>> >> and
>> >> >>> >>> >> >>> >> >> >> creates
>> >> >>> >>> >> >>> >> >> >> >> > one.
>> >> >>> >>> >> >>> >> >> >> >> >
>> >> >>> >>> >> >>> >> >> >> >> >
>> >> >>> >>> >> >>> >> >> >> >> > I realize this is perhaps an uncommon
>> case,
>> >> >>> but a
>> >> >>> >>> bug
>> >> >>> >>> >> >>> >> >> nonetheless.
>> >> >>> >>> >> >>> >> >> >> >> > I'll play with the code a bit and see
>> if I
>> >> can
>> >> >>> >>> come
>> >> >>> >>> >> up
>> >> >>> >>> >> >>> with
>> >> >>> >>> >> >>> >> a
>> >> >>> >>> >> >>> >> >> >> >> > solution. I'm thinking I can look at
>> the
>> >> nic's
>> >> >>> >>> >> broadcast
>> >> >>> >>> >> >>> URI
>> >> >>> >>> >> >>> >> >> and
>> >> >>> >>> >> >>> >> >> >> see
>> >> >>> >>> >> >>> >> >> >> >> > if it's supposed to be untagged, then
>> add
>> >> to
>> >> >>> >>> >> >>> >> vlanAllocatedToVM
>> >> >>> >>> >> >>> >> >> >> >> > appropriately, off the top of my head
>> >> >>> something
>> >> >>> >>> like:
>> >> >>> >>> >> >>> >> >> >> >> >
>> >> >>> >>> >> >>> >> >> >> >> >                     String vlanId =
>> >> >>> >>> >> >>> >> >> >> >> getVlanIdFromBridge(nic.getBrName());
>> >> >>> >>> >> >>> >> >> >> >> >                     if (vlanId != null
>> &&
>> >> >>> >>> >> >>> >> >> >> >>
>> >> >>> >>> > !nic.getBroadcastUri().toString().contains("untagged")
>> >> >>> >>> >> {
>> >> >>> >>> >> >>> >> >> >> >> >
>> >> >>> >>> vlanAllocatedToVM.put(vlanId,
>> >> >>> >>> >> >>> >> nicPos);
>> >> >>> >>> >> >>> >> >> >> >> >                     } else {
>> >> >>> >>> >> >>> >> >> >> >> >
>> >> >>> >>> >> >>> vlanAllocatedToVM.put(Vlan.UNTAGGED,
>> >> >>> >>> >> >>> >> >> >> nicPos);
>> >> >>> >>> >> >>> >> >> >> >> >                     }
>> >> >>> >>> >> >>> >> >> >> >> >
>> >> >>> >>> >> >>> >> >> >> >> >
>> >> >>> >>> >> >>> >> >> >> >> >
>> >> >>> >>> >> >>> >> >> >> >> > On Mon, Aug 27, 2012 at 6:42 PM, Edison
>> Su
>> >> >>> >>> >> >>> >> >> <Edison.su@citrix.com>
>> >> >>> >>> >> >>> >> >> >> >> wrote:
>> >> >>> >>> >> >>> >> >> >> >> >> Possible bug in in kvm code:
>> >> >>> >>> >> LibvirtComputingResource-
>> >> >>> >>> >> >>> >> >> >> >> >execute(IpAssocCommand cmd)-> VifHotPlug,
>> >> which
>> >> >>> is
>> >> >>> >>> >> only
>> >> >>> >>> >> >>> place
>> >> >>> >>> >> >>> >> >> >> adding
>> >> >>> >>> >> >>> >> >> >> >> nic into router vm.
>> >> >>> >>> >> >>> >> >> >> >> >> Turn on agent log, then take a look
>> what
>> >> >>> happened.
>> >> >>> >>> >> >>> >> >> >> >> >>
>> >> >>> >>> >> >>> >> >> >> >> >>> -----Original Message-----
>> >> >>> >>> >> >>> >> >> >> >> >>> From: Marcus Sorensen
>> >> >>> >>> [mailto:shadowsor@gmail.com]
>> >> >>> >>> >> >>> >> >> >> >> >>> Sent: Monday, August 27, 2012 5:10 PM
>> >> >>> >>> >> >>> >> >> >> >> >>> To: cloudstack-
>> dev@incubator.apache.org
>> >> >>> >>> >> >>> >> >> >> >> >>> Subject: VM router spawning multiple
>> >> public
>> >> >>> nics
>> >> >>> >>> >> >>> >> >> >> >> >>>
>> >> >>> >>> >> >>> >> >> >> >> >>> I've got two zones running the same
>> >> build of
>> >> >>> >>> >> cloudstack
>> >> >>> >>> >> >>> (a
>> >> >>> >>> >> >>> >> >> >> recent
>> >> >>> >>> >> >>> >> >> >> >> copy
>> >> >>> >>> >> >>> >> >> >> >> >>> of master). One of them creates
>> routers
>> >> that
>> >> >>> >>> turn
>> >> >>> >>> >> into
>> >> >>> >>> >> >>> >> ugly
>> >> >>> >>> >> >>> >> >> >> >> >>> multi-headed beasts, and by that I
>> mean
>> >> that
>> >> >>> any
>> >> >>> >>> >> time I
>> >> >>> >>> >> >>> >> >> create a
>> >> >>> >>> >> >>> >> >> >> >> port
>> >> >>> >>> >> >>> >> >> >> >> >>> forwarding or iptables rule for that
>> >> router
>> >> >>> I
>> >> >>> >>> get a
>> >> >>> >>> >> new
>> >> >>> >>> >> >>> >> >> public
>> >> >>> >>> >> >>> >> >> >> NIC
>> >> >>> >>> >> >>> >> >> >> >> >>> with an identical IP address, I have
>> an
>> >> >>> instance
>> >> >>> >>> >> with a
>> >> >>> >>> >> >>> >> few
>> >> >>> >>> >> >>> >> >> tens
>> >> >>> >>> >> >>> >> >> >> of
>> >> >>> >>> >> >>> >> >> >> >> >>> NICs.  My guess is that some script
>> >> isn't
>> >> >>> >>> detecting
>> >> >>> >>> >> >>> that
>> >> >>> >>> >> >>> >> >> there's
>> >> >>> >>> >> >>> >> >> >> >> >>> already a NIC with the public IP on
>> it.
>> >> It
>> >> >>> >>> looks
>> >> >>> >>> >> fine
>> >> >>> >>> >> >>> in
>> >> >>> >>> >> >>> >> the
>> >> >>> >>> >> >>> >> >> >> >> >>> database, there is only one public
>> NIC
>> >> >>> defined
>> >> >>> >>> in
>> >> >>> >>> >> the
>> >> >>> >>> >> >>> nics
>> >> >>> >>> >> >>> >> >> table.
>> >> >>> >>> >> >>> >> >> >> >> >>> I'll troubleshoot it tomorrow, but if
>> >> anyone
>> >> >>> >>> knows
>> >> >>> >>> >> >>> where I
>> >> >>> >>> >> >>> >> >> >> should
>> >> >>> >>> >> >>> >> >> >> >> >>> begin the headstart would be
>> appreciated.
>> >> >>> >>> >> >>> >> >> >> >> >>>
>> >> >>> >>> >> >>> >> >> >> >> >>> Thanks
>> >> >>> >>> >> >>
>> >> >>> >>> >> >
>> >> >>> >>> >> >

Mime
View raw message