airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siddharth Jain <ja...@umail.iu.edu>
Subject Re: [GSoC Proposal] Apache Airavata Monitoring Module
Date Fri, 25 Mar 2016 10:24:49 GMT
Hi Marlon,

Thank you for the comments :)

*Monitoring-direct communication-complex topology*
Right now the way I have it in proposal is, the GFAC binds it selves with
some queue. The producer sends a message to the broker with routing key
(which the target identification module will determine). The broker's
exchange routes the message on basis of the given routing key and delivers
it to the apt queue. Now if that queue is bound to a GFAC, it will be
delivered to it, if its bound to something else it will be delivered to
that

I understand what you are saying, but right now I made the solution
according to the discussion we had that day. If we really want to implement
what you are saying, we will have to be more concrete with what is actually
required.

   - If we just need capability of having multiple consumers for a job
   status message, we could have a "topic" exchange setup, wherein all the
   consumers tell the exchange which topic (herein, it might be something like
   jobID or maybe something more generic depending on our requirement) they
   want to subscribe to. Then whenever a message arrives at exchange for a
   specific topic, it will be distributed to all its subscribers.
   - If we have a complex topology, wherein some consumers want all the
   messages and some only need on the basis of key and maybe there are some
   more who want on the basis of topic. RabbitMq has a feature of "exchange of
   exchange" bindings, where we can have different type of exchanges binded to
   an exchange in addition to the normal queues, so we can pretty much design
   whatever we can think of, but we will have to come up with exactly what we
   want.

A simple solution to cope up with node failures that I have proposed in the
proposal is delayed acknowledgement. If we have a "reliable" queue (its a
feature in RabbitMQ, which allows setting up queues which survive server
restarts) setup, the consumer fetches the message from this type of queue,
but does not acknowledge it until its done processing the message. The way
this will address our problem is, if the consumer crashes for some reason
and restarts and fetches a message from its queue, it will get the same old
message, because the message was never acknowledged and and hence not
deleted from the queue. If the broker server restarts, the queue survives
anyways.


* Added the figures

* Added new smaller milestones

* Added a point under risk and mitigation, don't really have much to put in
there


I have the revised version ready and up for review. I have have submitted a
pdf version of the same for now on gsoc portal.


Thank you for taking the time out to review my proposal :)

Best,
Siddharth Jain

On Thu, Mar 24, 2016 at 10:34 AM, Pierce, Marlon <marpierc@iu.edu> wrote:

> Hi Sidd,
>
> This is a well written proposal. Some comments:
>
> * The monitoring component doesn’t need to directly communicate with GFAC.
> There can be multiple GFACs, so the original GFAC that submitted the
> request may no longer be running.  Also, other components may want to
> receive updates, including external subscribers.  GFAC subscribers can pick
> up the message and act.  There is the issue of managing race conditions
> between multiple GFACs and other distributed computing issues. If a GFAC
> instance picks up the message and wants to act on it but then becomes
> unresponsive, how do we handle this situation? This is my opinion, anyway.
> It is harder to implement than the direct approach.
>
> * You still need your high level architecture diagram and other figures
> where you have placeholders currently.
>
> * You need 2 week deliverables in your timeline. Add discussion of risk
> and risk mitigation, as I mentioned in my general email earlier today.
>
> Marlon
>
>
> From: Siddharth Jain <jain8@umail.iu.edu>
> Reply-To: "dev@airavata.apache.org" <dev@airavata.apache.org>
> Date: Wednesday, March 23, 2016 at 12:52 AM
> To: "dev@airavata.apache.org" <dev@airavata.apache.org>
> Subject: Re: [GSoC Proposal] Apache Airavata Monitoring Module
>
> Correction, the cwiki.apache.org link is:
>
> https://cwiki.apache.org/confluence/display/AIRAVATA/[GSoC+Proposal]+Apache+Airavata+Monitoring+Module
> <https://cwiki.apache.org/confluence/display/AIRAVATA/%5BGSoC+Proposal%5D+Apache+Airavata+Monitoring+Module>
>
> On Tue, Mar 22, 2016 at 6:35 PM, Siddharth Jain <jain8@umail.iu.edu>
> wrote:
>
>> Hello all,
>> I have the first draft of my GSoC proposal on Apache Airavata Monitoring
>> Module ready. I will appreciate if you could give any comments or
>> suggestions to improve this proposal.
>>
>> The proposal is available on:
>> i)
>> https://cwiki.apache.org/confluence/display/AIRAVATA/GSoC+Proposal+-+In+Situ+Simulation+Analysis+Using+Airavata
>>
>> ii)
>> https://docs.google.com/document/d/1rm_U-51NzdqfBfTNjOyCzc2eTR2SEg5K1tijhtijcz0/edit?usp=sharing
>>
>>
>> The cwiki.apache.org link points to what it will actually look like, the
>> google docs version is just for convenience of commenting.
>>
>>
>> Best regards,
>> Siddharth Jain
>>
>
>

Mime
View raw message