cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcus Sorensen <shadow...@gmail.com>
Subject Re: IPv6 in VPC (was Re: IPv6 plan - questions)
Date Thu, 09 Jan 2014 16:28:31 GMT
Good info. I think seeing another use case helps us narrow the
requirements a bit more.

On Thu, Jan 9, 2014 at 8:04 AM, Daan Hoogland
<DHoogland@schubergphilis.com> wrote:
> H Marcus, We had a small session on how we plan to go about ipv6 configuration with your
bullet list present. Comments are still brainstorming phase quality but hopefully they will
lead to something.
>
>> * VPC has no global IPv6 prefix (super CIDR as current private space), it's simply
IPv6 enabled or not. Admins can choose to route a /60 or a /48 to a vpc and carve it up among
the networks there, but it's not required or enforced by cloudstack.
> The idea at Schuberg Philis is that VPCs should have a standard size network assigned
to be taken from a pool. We should be able to configure the width of the network.  The idea
is that per level (Physical network/vpc/tier) only one width will be used but the size of
it is configurable. Default block for a VR / VPC would be a /56 and each individual network
would use a /64. Note that in a typical isolated network we only use the first.

Ok, good. Just to clarify, this is your use case, you're not saying
cloudstack should enforce a static size, just that you would deploy it
that way, correct?  Do you have any specific reasoning or need for the
VPC itself to have a configured contiguous block, rather than just
assign the /64s the networks?

>
>> * VPC networks get assigned one or multiple IPv6 prefixes, each prefix can be marked
'SLAAC', 'DHCP', or 'Manual'.
> For networks we are in favor of using SLAAC for configuring routing and addressing, but
would advise DHCPv6 to configure DNS and potentially other things (NTP?). No need to register
the addresses in CloudStack. We already store the MAC of the NIC so we know which address
will be configured on the interface with the configured prefix. Only the router will receive
a specific address (prefix::1/64)

I agree that we would want to publish DNS via DHCPv6, but is it
necessary, do you think? I assume we're not currently planning IPv6
*only* networks, thus AAAA can be looked up via the VR's DHCPv4
resolver. To me it seems like we'd do it for the sake of completeness,
I suppose.  I'm just trying to sort things into stages, what do we
need to have working IPv6, what is nice to have but not necessary, and
which dependent features will we want to add. If we start off and get
everything but that done, everything would still work, I think.

>
> As mentioned earlier first implementation will do SLAAC. How will SLAAC be configured
with ACL's?

I assume ACLs would work as they do with ipv4, where you can select
'incoming from X location and A-B ports with this protocol to this
network' and outgoing. I don't think it has any bearing on SLAAC,
DHCP, or manual configurations as currently ACLs apply to a whole
prefix.

>
>> * Mgmt server allows calling external plugins for pushing routes to routers upstream
of VPCs (SDN or router API), but could be manually done by admins as well
> We discussed the routing issues in front of the VPC / VR. The goal is to provide cloudstack
with a way to deal with routable addresses inside an isolated network, as actually this problem
is relevant for both IPv4 and IPv6. However for IPv4 we have a perfectly workable solution
with NAT and in the IPv6 world we don't/. So we need to address this issue on how to route
a network from the provider edge into an isolated network.
>
> For IPv6 we have several options:
> *       Dynamic routing protocols, here we can think of iBGP, OSPF or any other protocol.
Basically cloudstack assigns a block to a VR / VPC and the VPC / VR tells the outside world
to route that block to the outside IP of the router.
> *       Block allocator, cloudstack knows about the supernet (/48 for example) and assigns
a block (/56) to a VR or VPC. Cloudstack tells the block allocator to allocate (route) this
block to the VR/VPC. Block allocater needs to know about the device being managed (upstream
router like cisco/juniper) and configured the route using telnet or an API
> *       Broker, We introduce a new systemvm, the block broker. The admin routes the /48
to the block broker and CloudStack tells the blok broker to reroute a specific subblock to
this VR / VPC
> *       Static, Admin configures a list of block/ip pairs in the upstream router. CloudStack
knows about these pairs and assigns the ip to the VR / VPC and the block will be routed.
>

We've got some experience in this dept, as we currently route IPv4
public addresses into the VR as well via a little plugin. The simplest
implementation would simply be 1) assign a /64 to a network via mgmt
server, this goes in a table 2) when assignment occurs, and when
router reboots, we send a command to the VR to program the route for
that network. All it takes is for the upstream routers to know to send
the prefix to that VR and you're done.

I may be wrong, and please someone step in and help if needed, but I
think BGP and the like would be a lot of work to automate, and
wouldn't scale. Assuming you'd want everything under the same ASN or
split between a few, the advertisements would scale quadratically
(every VR needs to publish to every other VR), that doesn't work for a
cloud provider who expects thousands or tens of thousands of VPCs. If
you wanted to assign ASNs to VPCs instead, that would be hard to
automate, I think. If that's not the case, then yeah, let's just set
up quagga and go.

The block broker sounds interesting, but ultimately it's going to be a
big bottleneck as a VPC router to rule all VPCs (so to speak).
Actually, this is not dissimilar from what I was suggesting, which was
to allow for plugins so that external routers could be programmed with
the relevant route info during assignment and vpc router start/stop,
just that we take the burden of doing the upstream routing away from a
cloudstack VR and allow whatever hardware or setup the local admins
choose. We could even make it easier to integrate; in addition to the
plugin capability we could publish route info via the message bus, so
any sysadmin is just a python script away from grabbing that data and
programming their upstream routers through whatever tools they use.
We could also do both, for example right now we allow people to bring
up isolated networks without a VR, assuming they'll have their own. We
could both publish the data via plugin/message bus and allow for a
broker VM, and those could be done separately. Honestly, I have no
interest in the block broker, but I think it would be a nice addition
for small setups, and I do think not having that or automated route
advertisement (BGP) is a big hurdle for small and/or less experienced
teams to deploy IPv6.

>
>> * Work could be done in stages, e.g. SLAAC/manual network ranges would be fairly
straightforward, whereas DHCP ranges would require programming scripts and ip allocation code.
>
>> * An issue was raised about privacy concerns with SLAAC using MAC, but we think this
revolves more around clients than servers; that is, a client moving around the country would
be traceable because of the MAC, but a server always has the same address anyway.
>
>> * Still need to figure out what to do about ACLs, that's a whole separate issue,
but a plan needs to be in place.
> Are we going to put ACL's on networks or hosts or both? What is easiest to implement
and enforce?
>
> Do we care about portforwarding, static NAT, Load Balancing etc as well? Or at least
think
> about what impact these decisions have.
> This is not a version 1 consideration?!?

Of course it is. None of these are things that need to be done in
version 1, I don't think, but we need to spec it out such that we
don't have to rebuild it to do these things.

>
>> * We assume there will be an ipv6 public range assignable, allocated for VPC routers/static
nat/loadbalancers to pull from. How will this range be made available?
> Is a cidr configured?
> Will we preconfigure all ranges or have a dynamic availability check?
>
> Radvd package is added to the systemvm template. (Done by Hugo this morning)
>
> Some standard UI components are needed.
>
> (test-)attention needs to go to:
>
> HaProxy version/feature matrix has to be reviewed to check for ipv6 compatability.
>
> Does dnsmasq work with ipv6?
>
> Does keepalived work seemslessly?
>
> Ipv6 tables configuration needs to be pushed to the VRs
>
> -----Original Message-----
> From: Marcus Sorensen [mailto:shadowsor@gmail.com]
> Sent: maandag 6 januari 2014 9:11
> To: cloudstack-dev@incubator.apache.org
> Subject: IPv6 in VPC (was Re: IPv6 plan - questions)
>
> I've discussed this a bit with various subject matter experts at our datacenters/business,
and so far we're leaning toward a rollout like
> this:
>
> * VPC has no global IPv6 prefix (super CIDR as current private space), it's simply IPv6
enabled or not. Admins can choose to route a /60 or a
> /48 to a vpc and carve it up among the networks there, but it's not required or enforced
by cloudstack.
>
> * VPC networks get assigned one or multiple IPv6 prefixes, each prefix can be marked
'SLAAC', 'DHCP', or 'Manual'.
>
> * Mgmt server allows calling external plugins for pushing routes to routers upstream
of VPCs (SDN or router API), but could be manually done by admins as well
>
> * Work could be done in stages, e.g. SLAAC/manual network ranges would be fairly straightforward,
whereas DHCP ranges would require programming scripts and ip allocation code.
>
> * An issue was raised about privacy concerns with SLAAC using MAC, but we think this
revolves more around clients than servers; that is, a client moving around the country would
be traceable because of the MAC, but a server always has the same address anyway.
>
> * Still need to figure out what to do about ACLs, that's a whole separate issue, but
a plan needs to be in place. Do we care about port forwarding, static NAT, Load Balancing
etc as well? Or at least think about what impact these decisions have.
>
> * We assume there will be an ipv6 public range assignable, allocated for VPC routers/static
nat/loadbalancers to pull from.
>
> On Sat, Jan 4, 2014 at 6:11 AM, Marcus Sorensen <shadowsor@gmail.com> wrote:
>> I've put together a rough draft spec:
>>
>> https://cwiki.apache.org/confluence/display/CLOUDSTACK/IPv6+in+VPC+Rou
>> ter
>>
>> I basically just laid out some rough ideas. I know there has been a
>> lot of discussion in the past about DHCPv6, etc. My hope is that we
>> can at least decide on a spec, for future reference.
>>
>>
>> On Fri, Jan 3, 2014 at 9:53 PM, Marcus Sorensen <shadowsor@gmail.com> wrote:
>>> It's been a long time since I've heard anything in regards to IPv6,
>>> let alone VPC support. Does anyone have plans for this at all?  We'd
>>> like to support IPv6, and we have enough CS knowledge and external
>>> tools to hack something together, but I'd much prefer to build with
>>> the community and/or be forward compatible with what it deploys.
>>>
>>> I'd like to start with something simple, like perhaps optionally
>>> providing a /64 or larger as a parameter when creating a VPC (or a
>>> separate call to add an IPV6 block), and network on the vpc. Then it
>>> sounds like there's already a mechanism in place for tracking ipv6
>>> assignments to nics, that could be leveraged to pass dhcp assignments
>>> to routers.
>>>
>>> Then there's the whole acl thing, that seems like at least as big of
>>> a project as mentioned previously.
>>>
>>> On Mon, Aug 12, 2013 at 3:47 PM, Marcus Sorensen <shadowsor@gmail.com>
wrote:
>>>> has there been any further discussion that I might have missed
>>>> around
>>>> ipv6 in VPC?
>>>>
>>>> On Thu, Mar 7, 2013 at 12:09 PM, Sheng Yang <sheng@yasker.org> wrote:
>>>>> Hi Dave,
>>>>>
>>>>> I am glad it fits your need. That's our target. :)
>>>>>
>>>>> --Sheng
>>>>>
>>>>> On Thu, Mar 7, 2013 at 2:14 AM, Dave Cahill <dcahill@midokura.com>
wrote:
>>>>>> Hi Sheng,
>>>>>>
>>>>>> Thanks for the quick reply, that helps a lot.
>>>>>>
>>>>>> My main purpose was to figure out how these changes affect virtual
>>>>>> networking and pluggability. Having read through the IPv6 code
>>>>>> today, it looks like it will work very nicely with virtual networks.
>>>>>>
>>>>>> For example, when VMs are assigned an IPv6 address, the IPv6
>>>>>> address is stored in the NicProfile object. So, taking DHCP as an
>>>>>> example, if the MidoNet plugin implements the DHCPServiceProvider
>>>>>> interface, it will receive the NicProfile as one of the parameters
of addDhcpEntry.
>>>>>> If we want to implement IPv6, we can then take the IPv6 address
>>>>>> from the NicProfile, and just use it as needed.
>>>>>>
>>>>>> Thanks again for taking the time to respond, and for the detailed
FS.
>>>>>>
>>>>>> Dave.
>>>>>>
>>>>>> On Thu, Mar 7, 2013 at 4:57 AM, Sheng Yang <sheng@yasker.org>
wrote:
>>>>>>
>>>>>>> On Wed, Mar 6, 2013 at 1:36 AM, Dave Cahill <dcahill@midokura.com>
wrote:
>>>>>>> > Hi,
>>>>>>>
>>>>>>> Hi Dave,
>>>>>>> >
>>>>>>> > I've been catching up on IPv6 plans by reading the functional
>>>>>>> > specs and Jira tickets - it's great to have so much material
to refer to.
>>>>>>> >
>>>>>>> > I still have a few questions though, and I'm hoping someone
>>>>>>> > involved with the feature can enlighten me.
>>>>>>> >
>>>>>>> > *[Support for Providers other than Virtual Router]* In [3],
the
>>>>>>> > spec says "No external device support in plan."
>>>>>>> > What does this mean exactly?
>>>>>>>
>>>>>>> Because CloudStack also supports using external devices as
>>>>>>> network controller e.g. Juniper SRX as firewall and NetScaler
as
>>>>>>> load balancer. The words here said is just we don't support these
>>>>>>> devices when using IPv6.
>>>>>>> >
>>>>>>> > For example, if using Providers other than the Virtual Router,
>>>>>>> > does the UI still allow setting IPv6 addresses?
>>>>>>> >
>>>>>>> > If so, do we attempt to pass IPv6 addresses to the Providers
no
>>>>>>> > matter what, or do we check whether the Provider has IPv6
support?
>>>>>>>
>>>>>>> Yes, we checked it when you try to create a IPv6
>>>>>>> network(currently only support advance shared network).
>>>>>>>
>>>>>>> >
>>>>>>> > *[Networking Modes]*
>>>>>>> > Advanced Shared mode and Basic mode are mentioned in the
Jira
>>>>>>> > ticket [1] - "Isolated Network" is mentioned briefly in
[2],
>>>>>>> > but I wanted to check if the Advanced Isolated and VPC modes
>>>>>>> > are on the roadmap?
>>>>>>>
>>>>>>> There is no "basic isolated" network, so "Isolated" network is
>>>>>>> what we're talking about. We haven't got plan for VPC yet.
>>>>>>>
>>>>>>> And one correction: we didn't support "basic" mode for phase
1.
>>>>>>> We support only "advance shared network" in phase 1. The
>>>>>>> supported cases are described in FS. Jira ticket only provided
a
>>>>>>> rough idea at the time.
>>>>>>> >
>>>>>>> > *[IP Address Management / IPAM]* From [1], re: handing out
IPv6
>>>>>>> > addresses: "One way could be that the network admin creates
a
>>>>>>> > static route for a /48 towards a Virtual Router and then
the VR
>>>>>>> > can hand out /64s to Instances."
>>>>>>> >
>>>>>>> > With IPv4, IPAM is handled by the CloudStack management
server,
>>>>>>> > and the VR is told which IP address to give to the VM over
>>>>>>> > DHCP. Would this change with IPv6? "The VR can hand out
/64s to
>>>>>>> > instances" sounds like the VR is handling IPAM to some extent.
>>>>>>>
>>>>>>> Well, it's not how it works now. Please refer to the FS. The
>>>>>>> current implementation works like before. VR get a /64 then
>>>>>>> handle out IPv6 addresses to VM.
>>>>>>> >
>>>>>>> > From [3], "Router advertisement should be sent by public
>>>>>>> > gateway in the network." - to double-check, does this mean
the
>>>>>>> > router outside the CloudStack network should send RAs, but
the VR won't send RAs?
>>>>>>>
>>>>>>> Yes. Because in phase 1, we support only "advance shared
>>>>>>> network", in which case, VR is NOT the gateway. So we assume
the
>>>>>>> gateway router outside CloudStack should send out RA to the VMs.
>>>>>>>
>>>>>>> But in the phase 2, VR would acting as gateway, then it would
send out RAs.
>>>>>>>
>>>>>>> --Sheng
>>>>>>> >
>>>>>>> > Thanks,
>>>>>>> > Dave.
>>>>>>> >
>>>>>>> > [1] IPv6 Support main Jira ticket
>>>>>>> > https://issues.apache.org/jira/browse/CLOUDSTACK-452
>>>>>>> >
>>>>>>> > [2] IPv6 Support in CloudStack FS
>>>>>>> > https://cwiki.apache.org/CLOUDSTACK/ipv6-support-in-cloudstack.
>>>>>>> > html
>>>>>>> >
>>>>>>> > [3] IPv6 Support FS
>>>>>>> > https://cwiki.apache.org/CLOUDSTACK/ipv6-support.html
>>>>>>>

Mime
View raw message