airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saminda Wijeratne <samin...@gmail.com>
Subject Re: Internal messaging system in Airavata
Date Thu, 24 Jul 2014 19:02:47 GMT
hmm.... I thought Kafka was the natural selection for ZK. Is there a
built-in one in ZK which we can use instead?


On Thu, Jul 24, 2014 at 11:44 AM, Lahiru Gunathilake <glahiru@gmail.com>
wrote:

> No I was curious about the Kafka use-case here.
>
>
> On Thu, Jul 24, 2014 at 11:39 AM, Saminda Wijeratne <samindaw@gmail.com>
> wrote:
>
>> That is what I meant. we use ZK messaging system to share the messages
>> among the nodes.
>>
>>
>> On Thu, Jul 24, 2014 at 10:49 AM, Lahiru Gunathilake <glahiru@gmail.com>
>> wrote:
>>
>>> Why not just use ZK ?
>>>
>>>
>>> On Wed, Jul 23, 2014 at 3:51 PM, Saminda Wijeratne <samindaw@gmail.com>
>>> wrote:
>>>
>>>> Hi Devs,
>>>>
>>>> We currently (unofficially) use google guava eventbus[1] to propagate
>>>> experiment/node/task/job status change events. The use of eventbus is
>>>> hidden behind the class org.apache.airavata.common.utils.MonitorPublisher
>>>> and we have not yet fully evaluated the said framework. All subscriber
>>>> classes has to register with the MonitorPublisher in order to receive
>>>> messages. This framework can be used to notify of any event occurred within
>>>> Airavata components.
>>>>
>>>> However currently this framework does not work across JVMs. This is a
>>>> potential issue since we can have multiple GFac instances and/or different
>>>> thrift services hosted in different JVM instances. We are currently using
>>>> Apache Zookeeper to manage FT and load balancing in Airavata. Combined with
>>>> Apache Kafka, Zookeeper can help delegate solving the problem of sharing
>>>> messages between nodes (JVMs) in a quick a simple implementation as follows,
>>>>
>>>>
>>>>    1. Each JVM instance will have a MonitorPublisher instance which
>>>>    listeners in that JVM will subscribe to.
>>>>    2. Each instance of MonitorPublisher instance will subscribe to
>>>>    Kafka messages
>>>>    3. When a MonitorPublisher receives a message to publish it will
>>>>    also forward the message to Kafka. Kafka will then relay the message to
>>>>    other MonitorPublishers in other zookeeper nodes (other JVM instances).
>>>>    4. When MonitorPublisher receives a message from kafka it'll
>>>>    publish the said message in the guava eventbus.
>>>>
>>>> However this is not efficient since it leads message flooding. We are
>>>> thinking of going ahead of this until we find a better solution. wdyt?
>>>>
>>>>
>>>> 1. http://code.google.com/p/guava-libraries/wiki/EventBusExplained
>>>>
>>>
>>>
>>>
>>> --
>>> System Analyst Programmer
>>> PTI Lab
>>> Indiana University
>>>
>>
>>
>
>
> --
> System Analyst Programmer
> PTI Lab
> Indiana University
>

Mime
View raw message