airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siddharth Jain <ja...@umail.iu.edu>
Subject Re: Job Monitor POC
Date Wed, 06 Apr 2016 02:37:35 GMT
Hi Shameera,
Thank you for testing it :)

As much as I would like to get started with integrating the POC this month,
I cannot, because I do not have the bandwidth to work on it. This being the
last month of the semester, there are too many things going on, especially
with the subjects I have taken. Apart from the regular assignments and
classes, I have two (might become three) programming projects to develop
and deliver; three end-term exams to write in this month. My last exam is
on May 6th. I can start working on it from May 7th on wards.

Best regards,
Siddharth Jain

On Sun, Apr 3, 2016 at 11:14 PM, Shameera Rathnayaka <shameerainfo@gmail.com
> wrote:

> Hi Siddharth,
>
> I tested your poc and it is working. Your next move should be to
> intregrate this with airavata, i saw you have created a subtask for this.
> Let's work on that according to your availability. You can do this part by
> part, create your own working branch and work on it, let's do code review
> once you finish each part. By the way nice work, seems you have understood
> the problem correctly.
>
> Thanks,
> Shameera.
>
>
> On Sat, Mar 19, 2016 at 12:37 AM Siddharth Jain <jain8@umail.iu.edu>
> wrote:
>
>> Hello all,
>>
>> I have completed creating a stand-alone implementation/POC of job monitor in
>> the monitoring module of the codebase.
>>
>>
>>
>> I did fork from a dev branch and code, but I was unable to commit. I
>> tried creating patch, but the patch did not aggregate all the changes, so I
>> went ahead and just created a copy of the module (I think it should be as
>> good as a patch) for now and attached it to the JIRA issues:
>>
>> https://issues.apache.org/jira/browse/AIRAVATA-1912
>>
>> https://issues.apache.org/jira/browse/AIRAVATA-1914
>>
>> https://issues.apache.org/jira/browse/AIRAVATA-1915
>>
>>
>>
>> This code addresses the sub-task:
>>
>> ·       AIRAVATA-1914
>>
>> ·       AIRAVATA-1915
>>
>> I have also flagged the above two sub-tasks as patch.
>>
>>
>>
>> Briefly this is what stand-alone implementation does right now:
>>
>> ·       Creates a fanout exchange with durable queues (will persist
>> after server restarts) and auto-recovery option enabled; declares and binds
>> all the consumer queues
>>
>> ·       Fetch e-mails from a dummy email address
>>
>> ·       Converts the *non-serializable* javax Message objects into
>> custom *serializable* MessageExtract objects, which consists of only the
>> necessary information, viz. “from”, “to”, “subject” and “content”
fields,
>> which are being used in the existing codebase
>>
>> ·       Serialize the messages
>>
>> ·       Send the message to the exchange, which in turn will deliver the
>> message to all the queues bound to it
>>
>> ·       Two consumers threads fetch the messages from their respective
>> queues
>>
>> ·       Deserialize the message and convert it into javax Message
>> object, so that it can be used with the existing code
>>
>> ·       Print the received message content
>>
>>
>>
>> I have tried to keep all the components object-oriented. Following is the
>> explanation of the components,
>>
>> Core Components:
>>
>> ·       RabbitMQEmailPublisher: RabbitMQ publisher for e-mail messages
>>
>> ·       EmailReceiver: Represents a consumer, which spawns a thread and
>> keeps on receiving messages and process them. Multiple EmailReceiver can be
>> instantiated and they will automatically run in different threads.
>>
>> ·       GmailSMTPMailBox: represents gmail mail box
>>
>> ·       EmailConsumer: This has the implementation of call back for
>> received messages.
>>
>> ·       MessageExtract: Represents the extract from e-mail, which is
>> serializable
>>
>> Simulation Components:
>>
>> ·       FetchPublish: Fetches emails with GmailSMTPMailBox and publishes
>> to message bus with RabbitMQEmailPublisher
>>
>> ·       Simulator (also the main class to run):
>>
>> o   Instantiates two consumers (EmailReceiver) in separate threads
>>
>> o   Fetches email and publishes it with FetchPublish
>>
>> o   Shuts down the consumers after a minute
>>
>>
>>
>> Following is the TO-DO list:
>>
>> ·       Use the actual email server configs
>>
>> o   Currently I am using my own dummy account for testing, did not want
>> to accidentally mess up the production mail-boxes
>>
>> ·       Use the codebase broker config
>>
>> o   Currently I am using a local broker url and config
>>
>> ·       Fetch the actual email text/body
>>
>> o   Currently I am using a hard-coded string for email text, as somehow
>> when I extract the content, I always get multi-part content. The code you
>> guys already have in the repository seems to be written with an assumption
>> that the received message content will be text only,  I tried creating
>> an email like that, but it always has multi-part component. So once I have
>> integrated the code with the existing monitor, I can get back to fetching
>> the actual content
>>
>> ·       Open the inbox in read_write mode
>>
>> o   Currently I am opening the inbox in read mode, so that the messages
>> are not marked as read and so I don’t have to repeatedly create new dummy
>> messages
>>
>> ·       Probably implement a  RabbitMQ Factory singleton
>>
>> ·       Disable message(from message queue) auto-ack and only ack after
>> processing has been completed, this might help making the workflow more
>> robust
>>
>> ·       Use the codebase properties file for configuration of broker and
>> email service, right now creating properties on run time
>>
>> ·       Create test-cases
>>
>> I looked into integrating this into the live work-flow, but following
>> things kind of need time:
>>
>> ·       The message processing work-flow will have to be inverted, it’s
>> kind of different than we assumed. Currently the messages are fetched
>> explicitly and then processing is done. The way I have implemented the
>> consumers is, the queue messages will be delivered asynchronously, so we
>> don’t have to poll for it. So all the processing part will have to be wired
>> with call-back function.
>>
>> ·       Setup PGA and its dependencies on local for testing
>>
>> ·       Understand in which part of the life-cycle the consumer(or
>> equivalent of that) threads are to be invocated and what specific shutdown
>> interrupts are they to follow and respond to
>>
>> ·       Understand properties files conventions
>>
>> Hence the aforementioned list with the TO-DO list will be part of
>> milestone 1.
>>
>>
>>
>> Best regards,
>>
>> Siddharth Jain
>>
> --
> Shameera Rathnayaka
>

Mime
View raw message