airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shameera Rathnayaka <shameerai...@gmail.com>
Subject Re: Airavata Workflow Interpreter Emulation Framework
Date Tue, 26 Mar 2013 13:36:02 GMT
With the knowledge gained from the paper, let me briefly explain how
emulation process works and it states. The paper suggests 9 steps to an
emulated process but i have come up with a 7 steps flow which is sufficient
to do our works according to my knowledge.

Each job submission to application emulator from workflow engine can
consider application service execution flow.
We can divide such a flow into 7 steps which will call state of the flow,

Flow start-->Input Data transfer --> Pre-computation --> Computation -->
Post-computation --> Output data transfer --> Flow complete

Flow start - In this state it processes initialization activity like
resource and flow information gathering.
Input data transfer - Captures the data transfers, this is not required
when inputs are absent.
Pre computation - Input data products registered with meta-data catalog.
Computation - Emulates the computation or invoke external grid/cloud
emulators via plugins
Post computation - Data production registration
Output data transfer - Capture the data transfers that might be required
for the output data generated by the emulated flow.
Task complete - This is the final stage, execution flow results are sent
back to the workflow engine.

We can use pipeline/chain implementation to run this flow steps. Some
states are not valid for some flows, such states can be interrupted by
user. eg: Output data transfer state is not valid when there is no any
output from emulated computation.

Inside the Computation state user can define failure rate.

As for the ease of recording, I will be updating this in the JIRA too.


On Tue, Mar 26, 2013 at 12:05 AM, Shameera Rathnayaka <
shameerainfo@gmail.com> wrote:

> Hi Suresh,
>
> Of course, i will update the JIRA with these points and hopefully let's
> try to cover up the most important things in the project under the GSoC
> scope. It is my pleasure to be engaged is discussing this project.
>
> Thanks again!
>
>
> On Mon, Mar 25, 2013 at 11:45 PM, Suresh Marru <smarru@apache.org> wrote:
>
>>
>> On Mar 25, 2013, at 2:10 PM, Shameera Rathnayaka <shameerainfo@gmail.com>
>> wrote:
>>
>> Hi Suresh/Saminda,
>>
>> After reading the paper, i have come up with this,
>>
>> 1. we need a configurable interface to customize the data required to
>> start emulator.
>> 2. Application emulation -  Dummy applications(proxy services) can
>> register via Xbaya, once the dummy application receives the input message,
>> it invokes the task state simulation chain and returns the output to the
>> workflow.
>> 3. Application availability and Failure model - to provide a better real
>> world scenario, user can config the failure rate of each task state as
>> user's requirement.
>>  4. Pluggable interface to grid emulators and cloud emulators.
>>
>> We need to find/develop a good Failure level management algorithm to
>> provide better real world simulation interface, and
>>  scalable to handle large number of workflows.
>>
>> However, may be, implementing all above mentioned features will not be
>> the best fit with scope of GSoC.
>>
>>
>> Hi Shameera,
>>
>> Thank you for taking time to digest the paper and come up with these
>> excellent points. Can you please add these to JIRA so we don't loose them.
>> I would agree with you these may not be well scoped for a GSoC project.
>> Please stand by a day or two, we will try to recast all GSoC projects.
>>
>> Cheers,
>> Suresh
>>
>>
>>
>> This is just my initial idea after reading the paper. I would like to
>> know all of your ideas in improving this further. WDYT?
>>
>> Thanks,
>> Shameera
>>
>>
>>
>>
>> On Sun, Mar 24, 2013 at 4:27 AM, Lavanya Ramakrishnan <
>> lramakrishnan@lbl.gov> wrote:
>>
>>> Hi Suresh,
>>>
>>>  That paper is 3 yrs old :) that was the paper summarized the emulator
>>> idea that I used for my thesis. Glad to see it is still being looked at.
>>> Some one in Beth's lab was also looking at the code recently as well.
>>>
>>> Lavanya
>>>
>>> On Mar 23, 2013, at 12:43 PM, Suresh Marru <smarru@apache.org> wrote:
>>>
>>> > Hi Lavanya,
>>> >
>>> > I was pleasantly surprised to see you are still furthering the idea of
>>> workflow emulator, this paper seem to be very recent [1]. We had unfinished
>>> workflow emulators which we offered as GSoC projects last year and we are
>>> resurrecting it now. I will advice the students to shamelessly steal ideas
>>> from your paper.
>>> >
>>> > Hi Shameera,
>>> >
>>> > I was about to re-write Saminda's task description on the emulators,
>>> but Lavanya's paper [1] saved me some time. Why don't you read through the
>>> paper and propose back concrete GSoC tasks you see as applicable to current
>>> Airavata Workflow Interpreter. Do not worry to take the approaches
>>> literally and be liberal in your thinking and implementation. But the paper
>>> should give you a good head start. As you explore Airavata, just throw some
>>> ideas and we will help you refine them.
>>> >
>>> > Cheers,
>>> > Suresh
>>> >
>>> > [1]  - http://escholarship.org/uc/item/7050q8mk
>>> >
>>> > On Mar 23, 2013, at 3:11 PM, "Shameera Rathnayaka (JIRA)" <
>>> jira@apache.org> wrote:
>>> >
>>> >>
>>> >>   [
>>> https://issues.apache.org/jira/browse/AIRAVATA-346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13611833#comment-13611833]
>>> >>
>>> >> Shameera Rathnayaka commented on AIRAVATA-346:
>>> >> ----------------------------------------------
>>> >>
>>> >> Hi devs,
>>> >>
>>> >> As I mentioned in dev mailing list I would like to do this as my GSOC
>>> project. Saminda/Suresh/devs can you provide me more details or shall we
>>> discuss in improving this project?
>>> >>
>>> >> Thanks,
>>> >> Shameera.
>>> >>
>>> >>> [GSoC] Framework to run/monitor workflows locally with
>>> simulated/dummy BE
>>> >>>
>>> -------------------------------------------------------------------------
>>> >>>
>>> >>>               Key: AIRAVATA-346
>>> >>>               URL:
>>> https://issues.apache.org/jira/browse/AIRAVATA-346
>>> >>>           Project: Airavata
>>> >>>        Issue Type: New Feature
>>> >>>          Reporter: Saminda Wijeratne
>>> >>>          Assignee: Saminda Wijeratne
>>> >>>            Labels: gsoc2012, gsoc2013, mentor
>>> >>>
>>> >>> It is common to need to test something you've created on a test
>>> environment rather than on a real setup. This is because,
>>> >>> 1. The tasks are trivial and it'll be faster to run locally to
>>> verify correct flow/execution.
>>> >>> 2. It can cost to run them on a real setup at development stage.
>>> >>> 3. Proper access to the real setup is not available
>>> >>> 4. Unit testing on a real setup is unfeasible
>>> >>> Thus it would be easy for the user if there is a way to setup a
>>> simulated environment without much difficulty may be perhaps with some
>>> dummy data from the user itself. Effectively it will be behaving exactly as
>>> a GFac server/message box/registry etc but with hard wired data. As an
>>> example ideally it would be like the Azure simulator I guess.
>>> >>
>>> >> --
>>> >> This message is automatically generated by JIRA.
>>> >> If you think it was sent incorrectly, please contact your JIRA
>>> administrators
>>> >> For more information on JIRA, see:
>>> http://www.atlassian.com/software/jira
>>> >
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Shameera Rathnayaka.
>>
>> Blog : http://shameerarathnayaka.blogspot.com/
>>
>>
>>
>
>
> --
> Best Regards,
> Shameera Rathnayaka.
>
> Blog : http://shameerarathnayaka.blogspot.com/
>



-- 
Best Regards,
Shameera Rathnayaka.

Blog : http://shameerarathnayaka.blogspot.com/

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message