airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lahiru Gunathilake <glah...@gmail.com>
Subject Re: Internal messaging system in Airavata
Date Thu, 24 Jul 2014 15:44:58 GMT
No I was curious about the Kafka use-case here.


On Thu, Jul 24, 2014 at 11:39 AM, Saminda Wijeratne <samindaw@gmail.com>
wrote:

> That is what I meant. we use ZK messaging system to share the messages
> among the nodes.
>
>
> On Thu, Jul 24, 2014 at 10:49 AM, Lahiru Gunathilake <glahiru@gmail.com>
> wrote:
>
>> Why not just use ZK ?
>>
>>
>> On Wed, Jul 23, 2014 at 3:51 PM, Saminda Wijeratne <samindaw@gmail.com>
>> wrote:
>>
>>> Hi Devs,
>>>
>>> We currently (unofficially) use google guava eventbus[1] to propagate
>>> experiment/node/task/job status change events. The use of eventbus is
>>> hidden behind the class org.apache.airavata.common.utils.MonitorPublisher
>>> and we have not yet fully evaluated the said framework. All subscriber
>>> classes has to register with the MonitorPublisher in order to receive
>>> messages. This framework can be used to notify of any event occurred within
>>> Airavata components.
>>>
>>> However currently this framework does not work across JVMs. This is a
>>> potential issue since we can have multiple GFac instances and/or different
>>> thrift services hosted in different JVM instances. We are currently using
>>> Apache Zookeeper to manage FT and load balancing in Airavata. Combined with
>>> Apache Kafka, Zookeeper can help delegate solving the problem of sharing
>>> messages between nodes (JVMs) in a quick a simple implementation as follows,
>>>
>>>
>>>    1. Each JVM instance will have a MonitorPublisher instance which
>>>    listeners in that JVM will subscribe to.
>>>    2. Each instance of MonitorPublisher instance will subscribe to
>>>    Kafka messages
>>>    3. When a MonitorPublisher receives a message to publish it will
>>>    also forward the message to Kafka. Kafka will then relay the message to
>>>    other MonitorPublishers in other zookeeper nodes (other JVM instances).
>>>    4. When MonitorPublisher receives a message from kafka it'll publish
>>>    the said message in the guava eventbus.
>>>
>>> However this is not efficient since it leads message flooding. We are
>>> thinking of going ahead of this until we find a better solution. wdyt?
>>>
>>>
>>> 1. http://code.google.com/p/guava-libraries/wiki/EventBusExplained
>>>
>>
>>
>>
>> --
>> System Analyst Programmer
>> PTI Lab
>> Indiana University
>>
>
>


-- 
System Analyst Programmer
PTI Lab
Indiana University

Mime
View raw message