airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pierce, Marlon" <marpi...@iu.edu>
Subject Re: Job Monitor POC
Date Mon, 21 Mar 2016 13:23:32 GMT
Thanks, Siddhartha. We will take a look but don’t wait for our evaluation before submitting
your GSOC proposal draft.

Marlon


From: Siddharth Jain <jain8@umail.iu.edu<mailto:jain8@umail.iu.edu>>
Reply-To: "dev@airavata.apache.org<mailto:dev@airavata.apache.org>" <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Date: Saturday, March 19, 2016 at 12:37 AM
To: "dev@airavata.apache.org<mailto:dev@airavata.apache.org>" <dev@airavata.apache.org<mailto:dev@airavata.apache.org>>
Subject: Job Monitor POC

Hello all,
I have completed creating a stand-alone implementation/POC of job monitor in the monitoring
module of the codebase.

I did fork from a dev branch and code, but I was unable to commit. I tried creating patch,
but the patch did not aggregate all the changes, so I went ahead and just created a copy of
the module (I think it should be as good as a patch) for now and attached it to the JIRA issues:
https://issues.apache.org/jira/browse/AIRAVATA-1912
https://issues.apache.org/jira/browse/AIRAVATA-1914
https://issues.apache.org/jira/browse/AIRAVATA-1915

This code addresses the sub-task:

·       AIRAVATA-1914

·       AIRAVATA-1915
I have also flagged the above two sub-tasks as patch.

Briefly this is what stand-alone implementation does right now:

·       Creates a fanout exchange with durable queues (will persist after server restarts)
and auto-recovery option enabled; declares and binds all the consumer queues

·       Fetch e-mails from a dummy email address

·       Converts the non-serializable javax Message objects into custom serializable MessageExtract
objects, which consists of only the necessary information, viz. “from”, “to”, “subject”
and “content” fields, which are being used in the existing codebase

·       Serialize the messages

·       Send the message to the exchange, which in turn will deliver the message to all the
queues bound to it

·       Two consumers threads fetch the messages from their respective queues

·       Deserialize the message and convert it into javax Message object, so that it can
be used with the existing code

·       Print the received message content

I have tried to keep all the components object-oriented. Following is the explanation of the
components,
Core Components:

·       RabbitMQEmailPublisher: RabbitMQ publisher for e-mail messages

·       EmailReceiver: Represents a consumer, which spawns a thread and keeps on receiving
messages and process them. Multiple EmailReceiver can be instantiated and they will automatically
run in different threads.

·       GmailSMTPMailBox: represents gmail mail box

·       EmailConsumer: This has the implementation of call back for received messages.

·       MessageExtract: Represents the extract from e-mail, which is serializable
Simulation Components:

·       FetchPublish: Fetches emails with GmailSMTPMailBox and publishes to message bus with
RabbitMQEmailPublisher

·       Simulator (also the main class to run):

o   Instantiates two consumers (EmailReceiver) in separate threads

o   Fetches email and publishes it with FetchPublish

o   Shuts down the consumers after a minute

Following is the TO-DO list:

·       Use the actual email server configs

o   Currently I am using my own dummy account for testing, did not want to accidentally mess
up the production mail-boxes

·       Use the codebase broker config

o   Currently I am using a local broker url and config

·       Fetch the actual email text/body

o   Currently I am using a hard-coded string for email text, as somehow when I extract the
content, I always get multi-part content. The code you guys already have in the repository
seems to be written with an assumption that the received message content will be text only,
 I tried creating an email like that, but it always has multi-part component. So once I have
integrated the code with the existing monitor, I can get back to fetching the actual content

·       Open the inbox in read_write mode

o   Currently I am opening the inbox in read mode, so that the messages are not marked as
read and so I don’t have to repeatedly create new dummy messages

·       Probably implement a  RabbitMQ Factory singleton

·       Disable message(from message queue) auto-ack and only ack after processing has been
completed, this might help making the workflow more robust

·       Use the codebase properties file for configuration of broker and email service, right
now creating properties on run time

·       Create test-cases
I looked into integrating this into the live work-flow, but following things kind of need
time:

·       The message processing work-flow will have to be inverted, it’s kind of different
than we assumed. Currently the messages are fetched explicitly and then processing is done.
The way I have implemented the consumers is, the queue messages will be delivered asynchronously,
so we don’t have to poll for it. So all the processing part will have to be wired with call-back
function.

·       Setup PGA and its dependencies on local for testing

·       Understand in which part of the life-cycle the consumer(or equivalent of that) threads
are to be invocated and what specific shutdown interrupts are they to follow and respond to

·       Understand properties files conventions
Hence the aforementioned list with the TO-DO list will be part of milestone 1.

Best regards,
Siddharth Jain
Mime
View raw message