airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lahiru Gunathilake <>
Subject Re: Internal messaging system in Airavata
Date Thu, 24 Jul 2014 14:49:34 GMT
Why not just use ZK ?

On Wed, Jul 23, 2014 at 3:51 PM, Saminda Wijeratne <>

> Hi Devs,
> We currently (unofficially) use google guava eventbus[1] to propagate
> experiment/node/task/job status change events. The use of eventbus is
> hidden behind the class org.apache.airavata.common.utils.MonitorPublisher
> and we have not yet fully evaluated the said framework. All subscriber
> classes has to register with the MonitorPublisher in order to receive
> messages. This framework can be used to notify of any event occurred within
> Airavata components.
> However currently this framework does not work across JVMs. This is a
> potential issue since we can have multiple GFac instances and/or different
> thrift services hosted in different JVM instances. We are currently using
> Apache Zookeeper to manage FT and load balancing in Airavata. Combined with
> Apache Kafka, Zookeeper can help delegate solving the problem of sharing
> messages between nodes (JVMs) in a quick a simple implementation as follows,
>    1. Each JVM instance will have a MonitorPublisher instance which
>    listeners in that JVM will subscribe to.
>    2. Each instance of MonitorPublisher instance will subscribe to Kafka
>    messages
>    3. When a MonitorPublisher receives a message to publish it will also
>    forward the message to Kafka. Kafka will then relay the message to other
>    MonitorPublishers in other zookeeper nodes (other JVM instances).
>    4. When MonitorPublisher receives a message from kafka it'll publish
>    the said message in the guava eventbus.
> However this is not efficient since it leads message flooding. We are
> thinking of going ahead of this until we find a better solution. wdyt?
> 1.

System Analyst Programmer
Indiana University

View raw message