airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From DImuthu Upeksha <dimuthu.upeks...@gmail.com>
Subject Re: Linked Container Services for Apache Airavata Components - Phase 2 - Initial Prototype
Date Thu, 02 Nov 2017 12:51:04 GMT
Hi Gaurav,

Thanks a lot for the comments and please find my comments inline.

On Thu, Nov 2, 2017 at 11:41 AM, Shenoy, Gourav Ganesh <goshenoy@indiana.edu
> wrote:

> Hi Dimuthu,
>
>
>
> First of all, I must say this is really impressive – the way you grasped
> the problem and built a working prototype in such a short span – this is
> phenomenal. I haven’t yet installed the prototype, but I did look through
> the design document.
>
>
>
> Here are some thoughts/questions. Please feel free to comment or correct
> anything I’ve misunderstood.
>
>
>
>    - You are assuming a “message oriented” approach to performing
>    distributed task execution. Rather than using a third-party framework
>    (e.g.: Helix).
>    - Granted this gives more control in managing workloads, but this also
>    adds the overhead of:
>       - (1) creating, defining and persisting DAGs,
>       - (2) orchestrating through the DAG (although you aim to leverage
>       message broker),
>       - (3) managing the state of these DAGs as they progress,
>       - (4) handling errors related to workflow execution, and related to
>       broker dependency.
>
> Yes. You are correct. We take the burden of executing the DAG but as you
have mentioned, it provides a better control on the flow to handle edge
cases properly. However we have to implement this Task scheduler in more
generic way (I'm not happy with the current implementation but try to
improve this in future).

>
>    -
>       -
>    - I did not understand how you are defining the DAGs. I remember when
>    we were trying to solve this problem in class using the MQ approach
>    (similar but using RabbitMQ instead of Kafka), one of the challenges was
>    the allow a way to dynamically create DAGs and persist them if necessary.
>    How are you defining DAGs in your prototype?
>
> Defining a DAG has no direct tie with the message broker. Workflow
Generator is the one which generates the DAG for a Experiment. Currently
this is a static method specific to Experiments (pre job commands -> data
staging -> launch -> post job commands) but I suggested a new approach to
define a DAG in more generic way in another mail (you can find the subject
at the end). Once the DAG is saved in the database, Scheduler is the one
who fetches and executes it. To be clear, I followed current Airavata data
model with minor changes. DAG is equivalent to a Process. Process can have
multiple Tasks. For each task there is a DAG index. Task Scheduler fetches
the Tasks for a particular Process and sorts those task according to the
DAG index. I have proposed an improved design to compose DAGs both using
static configurations and user interfaces in an another mail (Subject :
"Customizable workflow design using reusable components").


>
>    -
>    - There might be situations when a workflow needs to be re-run (say
>    something went wrong initially, or needed corrections, or target resource
>    was down, etc). Does the design accommodate these scenarios?
>
> Yes. If you want to re run an Workflow (Experiment in this design), create
a new Process and send the process id to Task Executor. If you try this in
the User Interface, you can launch the Experiment multiple times. Each time
you launch the experiment, new Process is created with a new DAG and
resources locations in the compute hosts (input file paths, output file
paths) are generated by being specific to the Process id
(/tmp/{experimentid}/{processid}/inputs).


>    -
>    - One of the bigger goals is to be able to orchestrate Airavata
>    components, and not just the tasks involved in an experiment. As I
>    understand, the design relies on messaging to orchestrate, but is messaging
>    going to be the only communication paradigm within Airavata microservices?
>    If there are 2 components communicating via Thrift, how does the
>    architecture handle them?
>
> No I used Kafka to coordinate the consumers in the message flow other than
actual messaging. Except that, orchestration is performed through
Kubernetes. You can definitely use Thrift or some other mechanism to
communicate among microservices. In fact in this solution, microservices
communicate with the API server through HTTP/REST.


>    -
>    - Another point which comes to my mind is about moving away from this
>    big “API Server” block and segregating into smaller service level first
>    class SDKs. For e.g.: We now have a first class “Profile Service” which
>    allows isolated interactions pertaining to Users, Tenants and Groups. We
>    might want to keep these SDKs / services as independent as possible, which
>    also means no reliance on messaging. Will the architecture support these
>    SDKs?
>
> Agreed. What we need is to split the API server in to clearly defined
domains. I'll try to implement it in next versions.


>
>    -
>    - As Marlon pointed out and something we looked at last Spring, about
>    “database-per-microservice”. Currently we have a Registry which is shared
>    among different components like Orchestrator and GFac. Ideally each
>    microservice will own its database, and for any intersections in data
>    between microservices, we would sync up using events (messages). I can see
>    the Kafka broker come in handy for this.
>
> Yes. But in this case only API server has a database. Others are stateless
services. If they want any information, it will query API Server to fetch
them. However in future I guess Task Scheduler might want a direct database
access to guarantee some transactional operations.


>
>    -
>    - As far as possible, we would like to adopt a generic method to
>    define/maintain/execute the 3 types of workflows (or maybe more):
>       - External – user defined multi-application experiment; which would
>       mean a parent experiment constituting child experiments.
>       - Internal – component level workflows between Airavata
>       microservices. One such use-case I can think about is about dynamic
>       resource binding. Eg: provisioning a container (as target resource) if it
>       does not exist, or spinning up a VM and deploying an application at runtime.
>       - Experiment – typical experiment level task execution workflows
>       needed to complete the experiment.
>
>
Good idea. First and third scenarios can be handled in the new approach
that I have proposed. Can you please explain more about the second point?
Sorry, I'm not sure that I have understood it properly.

I hope I have addressed to all your comments, Please let me know if
anything is not clear enough.

>
>    -
>       -
>
>
>
> I apologize for this lengthy email, and I really appreciate the work you
> did. Some of these points might not make sense, so I would encourage
> discussions from the mailing list. Keep up the good work!
>

I really value your feedback and it helped me to look at the problems at
different angle. Thanks a lot for your time for going through all these. :)


>
>
> Thanks and Regards,
>
> Gourav Shenoy
>
>
>
> *From: *"Pierce, Marlon" <marpierc@iu.edu>
> *Reply-To: *<dev@airavata.apache.org>
> *Date: *Wednesday, November 1, 2017 at 5:22 PM
> *To: *"dev@airavata.apache.org" <dev@airavata.apache.org>
> *Subject: *Re: Linked Container Services for Apache Airavata Components -
> Phase 2 - Initial Prototype
>
>
>
> Hi Dimuthu,
>
>
>
> Thanks for sending this very thoughtful document. A couple of comments:
>
>
>
> * Use of Kafka instead of RabbitMQ is interesting. Can you say more about
> how this approach can handle Kafka client failures?  For RabbitMQ, for
> example, there is the simple “Work Queue” approach in which the broker
> pushes a task to a worker. The task remains in queue until the worker sends
> an acknowledgement that the job has been handled, not just received.
> “Handled” may mean for example that the job has been submitted to an
> external batch scheduler over SSH, which may require some retries, etc.
> If the worker crashes before the job has been submitted, then the broker
> can resend the message to another worker.   I’m wondering how your
> Kafka-based solution would handle the same issue.
>
>
>
> * A simpler but more common failure is communicating with external
> resources. A task executor may need to SSH to a remote resource, which can
> fail (the resource is slow to communicate, usually). How do you handle this
> case?
>
>
>
> * Your design focuses on Airavata’s experiment execution handling.
> Airavata’s registry is another important component: this is where
> experiment objects get persistently stored. The registry stores metadata
> about both “live” experiments that are currently executing as well as
> archived experiments that have completed.
>
>
>
> How would you extend your architecture to include the registry?
>
>
>
> Marlon
>
>
>
>
>
> *From: *"dimuthu.upeksha2@gmail.com" <dimuthu.upeksha2@gmail.com>
> *Reply-To: *"dev@airavata.apache.org" <dev@airavata.apache.org>
> *Date: *Monday, October 30, 2017 at 10:45 AM
> *To: *"dev@airavata.apache.org" <dev@airavata.apache.org>
> *Subject: *Linked Container Services for Apache Airavata Components -
> Phase 2 - Initial Prototype
>
>
>
> Hi All,
>
>
>
> Based on the analysis of Phase 1, within past two weeks I have been
> working on implementing a task execution workflow following the
> microservices deployment pattern and Kubernetes as the deployment platform.
>
>
>
> Please find attached design document that explains the components and
> messaging interactions between components. Based on that design, I have
> implemented following components
>
>
>
> 1. Set of microservices to compose the workflow
>
> 2. A simple Web Console to  deploy and monitor workflows on the framework
>
>
>
> I used Kakfa as the primary messaging medium to communicate among the
> microservices due to its simplicity and powerful features like partitions
> and consumer groups.
>
>
>
> I have attached a user guide so that you can install and try this in your
> local machine. And source code for each component can be found from [1]
>
>
>
> Please share you ideas and suggestions.
>
>
>
> Thanks
>
> Dimuthu
>
>
>
> [1] https://github.com/DImuthuUpe/airavata/tree/master/
> sandbox/airavata-kubernetes
>
> [2] https://docs.google.com/document/d/1R1xrmuPldHiWVDn4xNVa
> y9Vnxn9FODQZXtF55JxJpSY/edit?usp=sharing
>
> [3] https://docs.google.com/document/d/1A5eRIZiuUj4ShZVMS0Nd
> AxjAxtOTZXculaYDCZ7IMQ8/edit?usp=sharing
>

Mime
View raw message