airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pierce, Marlon" <marpi...@iu.edu>
Subject Re: Using Redis or Zookeeper for Email Monitoring
Date Thu, 20 Jul 2017 18:10:04 GMT
Agreed. I think some architecture diagrams are in order.  

 

Marlon

 

From: "Shenoy, Gourav Ganesh" <goshenoy@indiana.edu>
Reply-To: "dev@airavata.apache.org" <dev@airavata.apache.org>
Date: Thursday, July 20, 2017 at 1:35 PM
To: "dev@airavata.apache.org" <dev@airavata.apache.org>
Subject: Re: Using Redis or Zookeeper for Email Monitoring

 

Apoorv,

 

Good thought about making the email monitoring service scalable and reliable. But I was wondering
if you considered the Helix angle? Helix inherently uses Zookeeper to manage state and the
cluster as a whole. You might want to consider leveraging some of Helix’s data handling
capabilities. Nothing against Redis, Zookeeper looks like a definite candidate to me just
because of Helix. 

 

Also, could you chalk out and share an architecture of how this email monitoring service would
integrate with Airavata (assume a Helix architecture). Drawing an architecture would help
you better understand the problem and even find smaller issues which you didn’t consider.

 

Good work so far!

 

Thanks and Regards,

Gourav Shenoy 

 

From: Apoorv Palkar <apoorv_palkar@aol.com>
Reply-To: "dev@airavata.apache.org" <dev@airavata.apache.org>
Date: Wednesday, July 19, 2017 at 1:59 PM
To: "dev@airavata.apache.org" <dev@airavata.apache.org>
Subject: Using Redis or Zookeeper for Email Monitoring

 

Currently I'm trying to solve the problem of done emails coming before start emails and making
the system scalable. Today, we have only one GFaC that handles the email monitoring system.
As we are moving towards a microservices approach from the monolithic code we have, this email
monitoring system also needs to adapt to these changes. Currently, in the gfac code, a concurrent
hashmap is kept to keep track of start/end of emails using their respective experiment ID's.
Instead of keeping the hashmap locally, we should keep it in a global state so in the future
multiple GFaCs can handle the map. Supun has suggested to use Zookeeper for this as it has
high avaliability and realibility. I was also thinking that since these experiment IDs are
a key value pair, Redis would be a good option for such a use case. What do you guys think
about each one. I understand airavata currently uses zookeeper, so development wise there
seems to be an edge toward it. Would Redis be a good for such use case?


Mime
View raw message