stratos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Imesh Gunaratne <im...@apache.org>
Subject Re: MemberFault event is lost forever when MB is down
Date Wed, 30 Jul 2014 13:54:59 GMT
As I understood its not just the Member Fault event that is affected in
this scenario, any event that CEP publishes to message broker will
encounter the same problem.


On Wed, Jul 30, 2014 at 5:49 AM, Michiel Blokzijl (mblokzij) <
mblokzij@cisco.com> wrote:

> +1.
>
> If Stratos, or any component it relies on, fails, and eventually returns
> to service, Stratos should "orchestrate" the cloud back to the desired
> state. If any cartridges went missing and after some time T (post failure)
> Stratos hasn’t re-discovered them, they should be respawned.
>
> Best regards,
>
> Michiel
>
>
> On 30 Jul 2014, at 05:51, Isuru Haththotuwa <isuruh@apache.org> wrote:
>
>
>
>
> On Wed, Jul 30, 2014 at 9:45 AM, Akila Ravihansa Perera <
> ravihansa@wso2.com> wrote:
>
>> Hi Devs,
>>
>> Current Stratos architecture relies heavily on high availability of
>> the message broker. We faced a situation when MB is down, some of the
>> messages published will get lost forever and the system state will
>> never be recovered.
>>
>> One such example is, when a cartridge instance goes down the CEP
>> component will identify this event and publish a MemberFault event to
>> the MB's summarized-health-stat topic. But the problem is CEP
>> component creates its own list of cartridge instance members by
>> looking at health-stats published to MB - it does not consider the
>> topology. Hence, when a cartridge instance goes down, MemberFault
>> event will get fired only once. But if the MB is down at this time, it
>> will cause this message to be lost forever resulting in an un-stable
>> system state in which Stratos thinks a member exists but in reality it
>> is not the case.
>>
>> We can introduce a simple house keeping task to check whether every
>> member is alive. Ideally this should be auto-scaler's responsibility.
>> It will allow the system to recover itself from an un-stable
>> situation. I think this is a critical bug and should be given high
>> priority.
>>
>> Please share your thoughts.
>>
> +1. We would need to decide what is the best method for this though. If we
> consider CEP the central point of decision making, another option is to
> make it listen to topology and get the correct decision. Or else, we can
> use a health check mechanism for the MB which can detect if the MB is down
> and replay any of the messages. This IMO can be very useful since the
> primary communication mechanism in Stratos is the MB.
>
> One other important thing is to have fail-over/HA for MB. There can be
> many other occasion if the MB is down, the system going to a undefined
> state due to loss of messages.
>
>>
>> --
>> Akila Ravihansa Perera
>> Software Engineer
>> WSO2 Inc.
>> http://wso2.com
>>
>> Blog: http://ravihansa3000.blogspot.com
>>
>> --
>> <http://ravihansa3000.blogspot.com/>
>> Thanks and Regards,
>>
>> Isuru H.
>> <http://ravihansa3000.blogspot.com/>
>> +94 716 358 048 <http://ravihansa3000.blogspot.com/>* <http://wso2.com/>*
>>
>>
>> * <http://wso2.com/>*
>>
>>
>>
>


-- 
Imesh Gunaratne

Technical Lead, WSO2
Committer & PPMC Member, Apache Stratos

Mime
View raw message