stratos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lahiru Sandaruwan <lahi...@wso2.com>
Subject Re: Avoid burdening IaaS if there is an error while activating the instances
Date Thu, 09 Jan 2014 05:31:15 GMT
On Thu, Jan 9, 2014 at 10:56 AM, Nirmal Fernando <nirmal070125@gmail.com>wrote:

> Again the same issue right? We keep starting and terminating instances
> (it's just an increase of time).
>

Yes, even that saves a lot of stuff :)

Also absolute +1 for alerting. We can use already existing CEP to alert.

If we have the alerting system, actually we can STOP keep starting and
termination thing and wait until sys admin to fix it. Then he let the
system know that things are normal.





>
>
>
> On Thu, Jan 9, 2014 at 10:53 AM, Lahiru Sandaruwan <lahirus@wso2.com>wrote:
>
>> Hi Nirmal,
>>
>>
>> On Thu, Jan 9, 2014 at 10:25 AM, Nirmal Fernando <nirmal070125@gmail.com>wrote:
>>
>>> Hi Lahiru,
>>>
>>> I'm sorry, but I don't get your point here and also I don't understand
>>> how the proposed solution would solve this problem
>>>
>>
>> mmm, Any specific point that you are not clear?
>>
>> Basically it solves the problem of recursively starting and terminating
>> instances if there is a long running issue.
>>
>>
>>
>>
>>>  (what if there's an error and that made member activation event to not
>>> sent ever).
>>>
>>>
>> If this happens only for one instance(Not sending member activation event
>> for ever), it will only wait the given timeout(say 5 mins). So there is no
>> issue.
>>
>> If this happens to consecutive instances(say 3), we can say that there is
>> some long running issue. So we will wait more than the initial delay after
>> 3 consecutive failures...
>>
>> And this is continuing, we can keep increasing the wait up to a ceiling...
>>
>> Thanks.
>>
>>
>>
>>
>>> I wonder whether we need to be concerned on this kind of issues.
>>>
>>>
>>> On Thu, Jan 9, 2014 at 10:00 AM, Isuru Haththotuwa <isuruh@wso2.com>wrote:
>>>
>>>> +1. We can make the waiting time x2 each time, but there should be a
>>>> ceiling value as well.
>>>>
>>>>
>>>> On Wed, Jan 8, 2014 at 11:28 PM, Lahiru Sandaruwan <lahirus@wso2.com>wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> Currently if an instance is not joined after a timeout, we will
>>>>> terminate the instance and it will be removed from the pending state.
>>>>> Then the Autoscaler will decide to spawn more instances according to
>>>>> the rules, to cover terminated instances.
>>>>> If there is an error which blocks sending member activate event( in
>>>>> the cartridge, network or at any other place), system will be terminating
>>>>> and starting instances continuously, which is an utter waste of resources.
>>>>>
>>>>> So I suggest following scenario,
>>>>>
>>>>> We keep a count of unactivated instances per cluster. If this count
>>>>> exceeds a limit( say 3 - should be configurable), we will increase waiting
>>>>> time on the next instance activation.  May be we keep increasing.
>>>>> We can reset the count when ever a member activation  received.
>>>>>
>>>>> Wdyt?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Sent from my mobile.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks and Regards,
>>>>
>>>> Isuru H.
>>>> Software Engineer, WSO2 Inc.
>>>> +94 716 358 048* <http://wso2.com/>*
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Nirmal
>>>
>>> Nirmal Fernando.
>>> PPMC Member & Committer of Apache Stratos,
>>> Senior Software Engineer, WSO2 Inc.
>>>
>>> Blog: http://nirmalfdo.blogspot.com/
>>>
>>
>>
>>
>> --
>> --
>> Lahiru Sandaruwan
>> Software Engineer,
>> Platform Technologies,
>> WSO2 Inc., http://wso2.com
>> lean.enterprise.middleware
>>
>> email: lahirus@wso2.com cell: (+94) 773 325 954
>> blog: http://lahiruwrites.blogspot.com/
>> twitter: http://twitter.com/lahirus
>> linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146
>>
>>
>
>
> --
> Best Regards,
> Nirmal
>
> Nirmal Fernando.
> PPMC Member & Committer of Apache Stratos,
> Senior Software Engineer, WSO2 Inc.
>
> Blog: http://nirmalfdo.blogspot.com/
>



-- 
--
Lahiru Sandaruwan
Software Engineer,
Platform Technologies,
WSO2 Inc., http://wso2.com
lean.enterprise.middleware

email: lahirus@wso2.com cell: (+94) 773 325 954
blog: http://lahiruwrites.blogspot.com/
twitter: http://twitter.com/lahirus
linked-in: http://lk.linkedin.com/pub/lahiru-sandaruwan/16/153/146

Mime
View raw message