airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saminda Wijeratne <samin...@gmail.com>
Subject Re: Internal messaging system in Airavata
Date Thu, 24 Jul 2014 15:39:09 GMT
That is what I meant. we use ZK messaging system to share the messages
among the nodes.


On Thu, Jul 24, 2014 at 10:49 AM, Lahiru Gunathilake <glahiru@gmail.com>
wrote:

> Why not just use ZK ?
>
>
> On Wed, Jul 23, 2014 at 3:51 PM, Saminda Wijeratne <samindaw@gmail.com>
> wrote:
>
>> Hi Devs,
>>
>> We currently (unofficially) use google guava eventbus[1] to propagate
>> experiment/node/task/job status change events. The use of eventbus is
>> hidden behind the class org.apache.airavata.common.utils.MonitorPublisher
>> and we have not yet fully evaluated the said framework. All subscriber
>> classes has to register with the MonitorPublisher in order to receive
>> messages. This framework can be used to notify of any event occurred within
>> Airavata components.
>>
>> However currently this framework does not work across JVMs. This is a
>> potential issue since we can have multiple GFac instances and/or different
>> thrift services hosted in different JVM instances. We are currently using
>> Apache Zookeeper to manage FT and load balancing in Airavata. Combined with
>> Apache Kafka, Zookeeper can help delegate solving the problem of sharing
>> messages between nodes (JVMs) in a quick a simple implementation as follows,
>>
>>
>>    1. Each JVM instance will have a MonitorPublisher instance which
>>    listeners in that JVM will subscribe to.
>>    2. Each instance of MonitorPublisher instance will subscribe to Kafka
>>    messages
>>    3. When a MonitorPublisher receives a message to publish it will also
>>    forward the message to Kafka. Kafka will then relay the message to other
>>    MonitorPublishers in other zookeeper nodes (other JVM instances).
>>    4. When MonitorPublisher receives a message from kafka it'll publish
>>    the said message in the guava eventbus.
>>
>> However this is not efficient since it leads message flooding. We are
>> thinking of going ahead of this until we find a better solution. wdyt?
>>
>>
>> 1. http://code.google.com/p/guava-libraries/wiki/EventBusExplained
>>
>
>
>
> --
> System Analyst Programmer
> PTI Lab
> Indiana University
>

Mime
View raw message