airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saminda Wijeratne <>
Subject Internal messaging system in Airavata
Date Wed, 23 Jul 2014 19:51:19 GMT
Hi Devs,

We currently (unofficially) use google guava eventbus[1] to propagate
experiment/node/task/job status change events. The use of eventbus is
hidden behind the class org.apache.airavata.common.utils.MonitorPublisher
and we have not yet fully evaluated the said framework. All subscriber
classes has to register with the MonitorPublisher in order to receive
messages. This framework can be used to notify of any event occurred within
Airavata components.

However currently this framework does not work across JVMs. This is a
potential issue since we can have multiple GFac instances and/or different
thrift services hosted in different JVM instances. We are currently using
Apache Zookeeper to manage FT and load balancing in Airavata. Combined with
Apache Kafka, Zookeeper can help delegate solving the problem of sharing
messages between nodes (JVMs) in a quick a simple implementation as follows,

   1. Each JVM instance will have a MonitorPublisher instance which
   listeners in that JVM will subscribe to.
   2. Each instance of MonitorPublisher instance will subscribe to Kafka
   3. When a MonitorPublisher receives a message to publish it will also
   forward the message to Kafka. Kafka will then relay the message to other
   MonitorPublishers in other zookeeper nodes (other JVM instances).
   4. When MonitorPublisher receives a message from kafka it'll publish the
   said message in the guava eventbus.

However this is not efficient since it leads message flooding. We are
thinking of going ahead of this until we find a better solution. wdyt?


View raw message