stratos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akila Ravihansa Perera <raviha...@wso2.com>
Subject Re: [Discuss] Cloud Controller Clustering Model
Date Wed, 26 Nov 2014 12:00:36 GMT
Hi  Imesh,


> - The time it takes to propagate a modificaiton from one instance to
> another would be higher compared to using a distributed map (instance 1
>  persist changes, send a cluster message to invalidate the cache, other
> instances read changes from registry database, refresh in memory data
> structure, etc)
>

We don't have to refresh the whole data structure, but only the invalidated
object(s). We have to have a good registry structure for the topology in
order for this to be effective.


> - If above happens, when serving multiple requests (with a high frequency)
> on different instances of CC, the system might come to an inconsistent
> state because the in memory data structures have not updated properly.
>

Good point, thanks for pointing out. But the same problem will occur when
using cluster messages to replicate the state, right? I guess we will have
to resort to a distributed lock in either case to avoid consistencies. I
remember Isuru was working on a hierarchical locking approach for the
topology. Perhaps we can incorporate that for this scenario as well.

If you're going to use a distributed map, how do you plan to utilize that
to distribute the topology data structure? I think replicating the complete
topology object on change would be very expensive, wdyt?

Thanks.


> On Wed, Nov 26, 2014 at 3:10 PM, Akila Ravihansa Perera <
> ravihansa@wso2.com> wrote:
>
>> Hi Imesh,
>>
>> According to the model you've described, we will have to use a
>> distributed lock to synchronize read/write operations into the topology
>> data structure. This could be expensive.
>>
>> I'd like to propose that we actually store and read the topology from the
>> registry. And to optimize the performance, use a distributed cache. We have
>> to store topology as a collection of objects and if a write operation
>> occurs against an object, that cache object has to be invalidated. This way
>> we can get rid of having to replicate topology using cluster messages,
>> which could cause many inconsistent states, IMHO.
>>
>> Thanks.
>>
>> On Wed, Nov 26, 2014 at 2:50 PM, Imesh Gunaratne <imesh@apache.org>
>> wrote:
>>
>>> +1 Yes a good point Lakmal, to reduce the latency for a modification to
>>> propagate to all the instances we might need to replicate topology as well.
>>>
>>> Thanks
>>>
>>> On Wed, Nov 26, 2014 at 2:44 PM, Lakmal Warusawithana <lakmal@wso2.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Wed, Nov 26, 2014 at 1:16 PM, Imesh Gunaratne <imesh@apache.org>
>>>> wrote:
>>>>
>>>>> ​Hi Devs,
>>>>>
>>>>> This is to discuss the clustering model of the cloud controller:
>>>>>
>>>>>
>>>>> ​
>>>>>
>>>>> As shown in the above diagram the idea is to have a coordinator node
>>>>> to handle data persistence logic and message publishing (topology, instance
>>>>> status, etc). The coordinator will be selected randomly and at a given
time
>>>>> there will be only one coordinator. If the existing coordinator node
goes
>>>>> down, another member will become the coordinator automatically (similar
to
>>>>> carbon clustering agent).
>>>>>
>>>>> According to this design Autoscaler (AS)/Stratos Manager (SM) will
>>>>> talk to Cloud Controller (CC) via the Cloud Controller Service endpoint
>>>>> exposed via the load balancer.
>>>>>
>>>>> *Data Replication*
>>>>> When a request comes into one of the CC instances it will execute the
>>>>> necessary actions and update the data holder and/or topology which is
in
>>>>> memory. At this point the data holder changes will be replicated to other
>>>>> instances using a distributed map. Once the coordinator receives the
above
>>>>> updates it will persist the changes to the registry database.
>>>>>
>>>>> In this design we might not need to replicate the topology since it is
>>>>> already there in the message broker. The idea is to let coordinator publish
>>>>> the topology changes and the other members to listen to it.
>>>>>
>>>>>
>>>> IMO, we may need to replicate topology also, otherwise it may occur
>>>> some inconsistency.
>>>>
>>>>
>>>>> Please add your thoughts.
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>> --
>>>>> Imesh Gunaratne
>>>>>
>>>>> Technical Lead, WSO2
>>>>> Committer & PMC Member, Apache Stratos
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Lakmal Warusawithana
>>>> Vice President, Apache Stratos
>>>> Director - Cloud Architecture; WSO2 Inc.
>>>> Mobile : +94714289692
>>>> Blog : http://lakmalsview.blogspot.com/
>>>>
>>>>
>>>
>>>
>>> --
>>> Imesh Gunaratne
>>>
>>> Technical Lead, WSO2
>>> Committer & PMC Member, Apache Stratos
>>>
>>
>>
>>
>> --
>> Akila Ravihansa Perera
>> Software Engineer, WSO2
>>
>> Blog: http://ravihansa3000.blogspot.com
>>
>
>
>
> --
> Imesh Gunaratne
>
> Technical Lead, WSO2
> Committer & PMC Member, Apache Stratos
>



-- 
Akila Ravihansa Perera
Software Engineer, WSO2

Blog: http://ravihansa3000.blogspot.com

Mime
View raw message