cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daan Hoogland <daan.hoogl...@gmail.com>
Subject Re: HA redundant virtual router
Date Mon, 16 Sep 2013 09:25:01 GMT
H Sheng,

>From your doc I don't read the reasons for disabling HA for RVR.

I do see some possible glitches though. One is that a newly installed
router should always have higher prio if the master cannot be reached.
Something similar should happen on rule programming.

To deal with the 255 limit on prios they could be reset to the minimum
as soon as both routers are found again.

Am I correct?

thanks,
Daan

On Fri, Sep 6, 2013 at 12:27 AM, Sheng Yang <sheng@yasker.org> wrote:
> Here is the doc.
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/Redundant+Virtual+Router+Functional+Spec
>
> It's not extremely detail, but describe today's design generally.
>
> --Sheng
>
>
> On Thu, Aug 29, 2013 at 8:17 AM, Daan Hoogland <daan.hoogland@gmail.com>
> wrote:
>>
>> ok,
>>
>> let's postpone the discussion till you are at least halve done. We
>> will of course continue to deliberate on what we need internally.
>>
>> Daan
>>
>> On Thu, Aug 29, 2013 at 5:08 PM, Sheng Yang <sheng@yasker.org> wrote:
>> > Hi Daan,
>> >
>> > As I said, I am writing a design doc to describe the current redundant
>> > router policy, to help understanding redundant router. Current it
>> > doesn't
>> > support VPC, so how to implement it in VPC is still open to discuss.
>> >
>> > --Sheng
>> >
>> >
>> > On Thu, Aug 29, 2013 at 4:26 AM, Daan Hoogland <daan.hoogland@gmail.com>
>> > wrote:
>> >>
>> >> Sheng,
>> >>
>> >> just to make sure; You are going to write this document? I see Roeland
>> >> understood your mail like this.
>> >>
>> >> When you do, I'd like you to keep in mind that we also want redundant
>> >> routers within a VPC to ensure ACS upgrades are more seamless for
>> >> customer application groups and - dtap streets. If you need any help
>> >> on writing such a doc, let me know.
>> >>
>> >> kind regards,
>> >> Daan
>> >>
>> >> On Thu, Aug 29, 2013 at 1:13 PM, Roeland Kuipers
>> >> <RKuipers@schubergphilis.com> wrote:
>> >> > Hi Sheng,
>> >> >
>> >> > Thanks for the info. Looking forward to the design doc, I trust this
>> >> > will make things clearer.
>> >> > In the meantime will be doing some research and thinking too, to see
>> >> > how
>> >> > we can improve things to also have HA on the RvR in a safe way.
>> >> > We will share this once ready.
>> >> >
>> >> > Thanks,
>> >> > Roeland
>> >> >
>> >> >
>> >> > From: Sheng Yang [mailto:sheng@yasker.org]
>> >> > Sent: donderdag 29 augustus 2013 0:19
>> >> > To: <dev@cloudstack.apache.org>
>> >> > Cc: int-cloud; Daan Hoogland
>> >> > Subject: Re: HA redundant virtual router
>> >> >
>> >> > Hi Roeland,
>> >> >
>> >> > I would write a design doc to explain how redundant router works
>> >> > currently. For example, for the point 2, we have to force BACKUP
>> >> > become
>> >> > MASTER because:
>> >> >
>> >> > 1. CS cannot communicate with MASTER at the time
>> >> > 2. CS can communicate with BACKUP.
>> >> > 3. Rule has to be programmed immediately.
>> >> > 4. In case old MASTER come back, it should yield to the VR with
>> >> > updated
>> >> > rule, rather than preempt the updated VR.
>> >> >
>> >> > In this case, CS need to communicate with RvR to program the new
>> >> > rule,
>> >> > thus it need to intervene the RvR to ensure that if there is only one
>> >> > VR got
>> >> > the rule, it should become MASTER.
>> >> >
>> >> > Still, I would write a doc later to try to cover every concern of RvR
>> >> > design.
>> >> >
>> >> > --Sheng
>> >> >
>> >> > On Tue, Aug 27, 2013 at 3:40 AM, Roeland Kuipers
>> >> > <RKuipers@schubergphilis.com<mailto:RKuipers@schubergphilis.com>>
>> >> > wrote:
>> >> > Hi Sheng,
>> >> >
>> >> > Thanks for your reply. I'll see if we can replay this scenario.
>> >> >
>> >> > With respect to point 1: a good principal IMHO.
>> >> >
>> >> > Point 2: Why do we force a keepalived node to become master and not
>> >> > wait
>> >> > for keepalived to become master? This way there is less reason to
>> >> > intervene
>> >> > and less risk of multiple masters? As we have seen this behavior with
>> >> > RvR
>> >> > without HA in the past. The downside that updates to rules do not
>> >> > function
>> >> > until backup becomes master. But maybe this is wise anyways since
>> >> > there is
>> >> > something wrong. This conflicts a bit with point 2 as we do intervene
>> >> > here.
>> >> >
>> >> > Point 3: In my opinion keepalived is solid enough to leave this
>> >> > responsibility with keepalived and that CS just should check the
>> >> > state and
>> >> > not fiddle with priorities to force masters. Because there is
>> >> > obviously a
>> >> > reason why BACKUP refuses to become master.
>> >> > I think we should let keepalived prevent multiple master as is
>> >> > designed
>> >> > to prevent this. Or do I miss something here?
>> >> > Actually in the scenario you described, with a functioning guest
>> >> > network, keepalived should be able to handle this situation if we
>> >> > make sure
>> >> > all routers have different prios.
>> >> >
>> >> > I still have the opinion HA and RvR are different mechanisms.
>> >> >
>> >> > So what do you think is necessary to have the possibility of HA icw
>> >> > RvR?
>> >> > We have a clear business requirement to have this implement on CS.
>> >> > And we
>> >> > have Developers willing to create these changes to make this
>> >> > possible.
>> >> > We also like to see RvR on VPC's and are also willing to contribute
>> >> > this
>> >> > functionality.
>> >> >
>> >> > Thanks for your feedback!
>> >> >
>> >> > Cheers,
>> >> > Roeland
>> >> >
>> >> > -----Original Message-----
>> >> > From: Sheng Yang [mailto:sheng@yasker.org<mailto:sheng@yasker.org>]
>> >> > Sent: vrijdag 23 augustus 2013 23:25
>> >> > To: <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
>> >> > Subject: Re: HA redundant virtual router
>> >> >
>> >> > Hi Roeland,
>> >> >
>> >> > Thank you for your testing!
>> >> >
>> >> > Power off is not an concern right now, because at that time the VM
>> >> > would
>> >> > disappear anyway.
>> >> >
>> >> > Our concern is more about if VM is still alive but we cannot detect
>> >> > it
>> >> > for a while. For example, a network glitch happened, CS lost
>> >> > connection to
>> >> > the host temporarily(control network), but the guest network is still
>> >> > working.
>> >> > HA would start another VR, which would possible result in 3 routers
>> >> > in
>> >> > the guest network(at least for a moment). Many of the policy focus
on
>> >> > dealing these intermediate status. Also if you plug off the network
>> >> > cable of
>> >> > one host many things should happen...
>> >> >
>> >> >
>> >> > In RvR we want to make sure:
>> >> > 1. The status are self-governed, no need for CS to intervene.
>> >> > 2. MASTER would always get the latest rules. That means, if we cannot
>> >> > communicate with MASTER, we would turn to BACKUP and program the rule
>> >> > on it
>> >> > and make it MASTER - even we cannot communicate with MASTER at this
>> >> > time.
>> >> > And BACKUP should able to become MASTER if we request. This is
>> >> > achieved
>> >> > by using a script to bump up the priority of BACKUP.
>> >> > 3. Trying best to prevent the dual-MASTER situation. So we would
>> >> > program
>> >> > different priority for VRs and the MASTER/BACKUP status completely
>> >> > depends
>> >> > on priority.
>> >> >
>> >> > And if you take RvR as an alternative to VM's HA mechanism., it's not
>> >> > that counter intuitive in fact.
>> >> >
>> >> > --Sheng
>> >> >
>> >> >
>> >> > On Fri, Aug 23, 2013 at 1:56 AM, Roeland Kuipers <
>> >> > RKuipers@schubergphilis.com<mailto:RKuipers@schubergphilis.com>>
>> >> > wrote:
>> >> >
>> >> >> Hi Sheng,
>> >> >>
>> >> >> So far our testing showed no big problems. I've marked a redundant
>> >> >> set
>> >> >> of routers to be ha_enabled by setting ha_enabled bit in the
>> >> >> vm_instance table. (This is our workaround ATM) We tested HA icw
RvR
>> >> >> in the scenarios ,shutdown / force power off VM. In these scenarios
>> >> >> HA
>> >> >> worked a treat and did restore the redundant pair as it should.
And
>> >> >> keepalived nicely negotiated MASTER & BACKUP.
>> >> >> These are obviously basic tests, but we are happy to do some more
>> >> >> testing.
>> >> >>
>> >> >> I understand your concerns and am totally in favour of the KISS
>> >> >> principle.
>> >> >> What could be the scenario to end up with 3 routers?
>> >> >> Why is the situation complex to deal with? These are separate
>> >> >> mechanisms.
>> >> >> HA just making sure the router is up and alive. And keepalived
>> >> >> negotatiating MASTER-BACUP states according to keepalived
>> >> >> configuration, unless there a 3 routers with conflicting configs.
>> >> >> But
>> >> >> so far I do not understand the scenario where we could end up with
3
>> >> >> routers, so I cannot judge end/or test this.
>> >> >>
>> >> >> We like to see the hardcoded denial of HA in a redundant router
>> >> >> setup
>> >> >> go for several reasons:
>> >> >> 1. It's counter intuitive - we configured an HA service offering
on
>> >> >> purpose for the RvR's. And found out by accident that it was not
>> >> >> enabled at all.
>> >> >> 2. CS could implement a default offering without HA for this setup
>> >> >> (to
>> >> >> keep it simple by default and keep currently forced behaviour),
but
>> >> >> if
>> >> >> users, like us, deliberately like to have HA, users can create
a
>> >> >> custom offering with HA enabled
>> >> >>
>> >> >> This way it's configurable, doesn't change default behavior and
is
>> >> >> more intuitive.
>> >> >>
>> >> >> Thanks & Cheers,
>> >> >> Roeland
>> >> >>
>> >> >>
>> >> >>
>> >> >> -----Original Message-----
>> >> >> From: Sheng Yang [mailto:sheng@yasker.org<mailto:sheng@yasker.org>]
>> >> >> Sent: vrijdag 23 augustus 2013 3:03
>> >> >> To: <dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org>>
>> >> >> Subject: Re: HA redundant virtual router
>> >> >>
>> >> >> It's a design choice, the only reason is it would be a very complex
>> >> >> situation to deal with. In fact the redundant router itself's policy
>> >> >> has already been very complex...
>> >> >>
>> >> >> We didn't look into details at the time of implementing redundant
>> >> >> router, but there are lots of concerns e.g. a network glitch may
>> >> >> result in 3 routers running in the network and potentially two
of
>> >> >> them
>> >> >> are in MASTER state.
>> >> >>
>> >> >> Of course discussion is welcome. We just want to keep it as simple
>> >> >> as
>> >> >> possible at the time.
>> >> >>
>> >> >> --Sheng
>> >> >>
>> >> >>
>> >> >> On Thu, Aug 22, 2013 at 3:31 AM, Daan Hoogland <
>> >> >> DHoogland@schubergphilis.com<mailto:DHoogland@schubergphilis.com>
>> >> >> > wrote:
>> >> >>
>> >> >> > LS,
>> >> >> >
>> >> >> > Schuberg Philis guarantees 100% functional uptime for their
>> >> >> > customers.
>> >> >> > Infrastructure is of course part of this promise and the easier
>> >> >> > factor to provide strong levels of resiliency. For this reason
we
>> >> >> > want to make use of redundant virtual routers together with
HA
>> >> >> > functionality.
>> >> >> >
>> >> >> > We see HA and redundant routers as to different methods to
provide
>> >> >> > higher levels of uptime.
>> >> >> >
>> >> >> >
>> >> >> > 1.      The redundant router setup takes care of seamless
failover
>> >> >> without
>> >> >> > lengthy hick-ups in the case of a single router failure.
>> >> >> >
>> >> >> > 2.      HA takes care of restarting a failed VM or router.
>> >> >> > Restoring
>> >> >> > connectivity in the case of single router or restoring 2n
>> >> >> > resiliency
>> >> >> > in the case of a redundant router setup.
>> >> >> >
>> >> >> > The combination of these two methods will help us to meet
our 100%
>> >> >> > promise; .We need to restore 2N redundancy ASAP in the case
of
>> >> >> > single component failure e.g. a router. With these two methods
>> >> >> > combined the system is more autonomous and doesn't need human
>> >> >> > intervention to restore redundancy.
>> >> >> >
>> >> >> > In the current situation we need to send a page to an on call
>> >> >> > engineer to restore redundancy asap, because of the tight
SLA's.
>> >> >> > While if we could use HA icw redundant routers. The on-call
guy
>> >> >> > can
>> >> >> > enjoy his sleep and will be a more happy guy :) The present
code
>> >> >> > forces the HA offering to off on redundant routers which seems
>> >> >> > odd.
>> >> >> >
>> >> >> > So my question is: Why is it forced to off; Is there a technical
>> >> >> > restraint or is this a design choice we can discuss and maybe
>> >> >> > revise?
>> >> >> >
>> >> >> > Cheers,
>> >> >> >
>> >> >> >
>> >> >>
>> >> >
>> >
>> >
>
>

Mime
View raw message