airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shenoy, Gourav Ganesh" <goshe...@indiana.edu>
Subject Re: [#Spring17-Airavata-Courses] : Distributed Workload Management for Airavata
Date Fri, 03 Feb 2017 00:57:14 GMT
Hello all,

Amila, Sagar, thank you for the response and raising those concerns; and apologies because
my email resonated the topic of workload management in terms of how micro-services communicate.
As Ajinkya rightly mentioned, there exists some sort of correlation between micro-services
communication and it’s impact on how that micro-service performs the work under those circumstances.
The goal is to make sure we have maximum independence between micro-services, and investigate
the workflow pattern in which these micro-services will operate such that we can find the
right balance between availability & consistency. Again, from our preliminary analysis
we can assert that these solutions may not be generic and the specific use-case will have
a big decisive role.

For starters, we are focusing on the following example – and I think this will clarify the
doubts on what we are exactly trying to investigate about.

Our test example
Say we have the following 4 micro-services, which each perform a specific task as mentioned
in the box.

[cid:image001.png@01D27D8E.8B69A3F0]


A state-full pattern to distribute work
[cid:image002.png@01D27D8E.8B69A3F0]

Here each communication between micro-services could be via RPC or Messaging (eg: RabbitMQ).
Obvious disadvantage is that if any micro-service is down, then the system availability is
at stake. In this test example, we can see that Microservice-A coordinates the work and maintains
the state information.

A state-less pattern to distribute work

[cid:image003.png@01D27D8E.8B69A3F0]

Another purely asynchronous approach would be to associate message-queues with each micro-service,
where each micro-service performs it’s task, submits a request (message on bus) to the next
micro-service, and continues to process more requests. This ensures more availability, and
perhaps we might need to handle corner cases for failures such as message broker down, or
message loss, etc.

As mentioned, these are just a few proposals that we are planning to investigate via a prototype
project. Inject corner cases/failures and try and find ways to handle these cases. I would
love to hear more thoughts/questions/suggestions.

Thanks and Regards,
Gourav Shenoy

From: Ajinkya Dhamnaskar <adhamnas@umail.iu.edu>
Reply-To: "dev@airavata.apache.org" <dev@airavata.apache.org>
Date: Thursday, February 2, 2017 at 2:22 AM
To: "dev@airavata.apache.org" <dev@airavata.apache.org>
Subject: Re: [#Spring17-Airavata-Courses] : Distributed Workload Management for Airavata

Hello all,

Just a heads up. Here the name Distributed workload management does not necessarily mean having
different instances of a microservice and then distributing work among these instances.

Apparently, the problem is how to make each microservice work independently with concrete
distributed communication infrastructure. So, think of it as a workflow where each microservice
does its part of work and communicates (how? yet to be decided) output. The next underlying
microservice identifies and picks up that output and takes it further towards the final outcome,
having said that, the crux here is, none of the miscoservices need to worry about other miscoservices
in a pipeline.

Vidya Sagar,
I completely second your opinion of having stateless miscoservices, in fact that is the key.
With stateless miscroservices it is difficult to guarantee consistency in a system but it
solves the availability problem to some extent. I would be interested to understand what do
you mean by "an intelligent job scheduling algorithm, which receives real-time updates from
the microservices with their current state information".

On Wed, Feb 1, 2017 at 11:48 PM, Vidya Sagar Kalvakunta <vkalvaku@umail.iu.edu<mailto:vkalvaku@umail.iu.edu>>
wrote:

On Wed, Feb 1, 2017 at 2:37 PM, Amila Jayasekara <thejaka.amila@gmail.com<mailto:thejaka.amila@gmail.com>>
wrote:
Hi Gourav,

Sorry, I did not understand your question. Specifically I am having trouble relating "work
load management" to options you suggest (RPC, message based etc.).
So what exactly you mean by "workload management" ?
What is work in this context ?

Also, I did not understand what you meant by "the most efficient way". Efficient interms of
what ? Are you looking at speed ?

As per your suggestions, it seems you are trying to find a way to communicate between micro
services. RPC might be troublesome if you need to communicate with processes separated from
a firewall.

Thanks
-Thejaka


On Wed, Feb 1, 2017 at 12:52 PM, Shenoy, Gourav Ganesh <goshenoy@indiana.edu<mailto:goshenoy@indiana.edu>>
wrote:
Hello dev, arch,

As part of this Spring’17 Advanced Science Gateway Architecture course, we are working on
trying to debate and find possible solutions to the issue of managing distributed workloads
in Apache Airavata. This leads to the discussion of finding the most efficient way that different
Airavata micro-services should communicate and distribute work, in such a way that:

1.       We maintain the ability to scale these micro-services whenever needed (autoscale
perhaps?).

2.       Achieve fault tolerance.

3.       We can deploy these micro-services independently, or better in a containerized manner
– keeping in mind the ability to use devops for deployment.

As of now the options we are exploring are:

1.       RPC based communication

2.       Message based – either master-worker, or work-queue, etc

3.       A combination of both these approaches

I am more inclined towards exploring the message based approach, but again there arises the
possibility of handling limitations/corner cases of message broker such as downtimes (may
be more). In my opinion, having asynchronous communication will help us achieve most of the
above-mentioned points. Another debatable issue is making the micro-services implementation
stateless, such that we do not have to pass the state information between micro-services.

I would love to hear any thoughts/suggestions/comments on this topic and open up a discussion
via this mail thread. If there is anything that I have missed which is relevant to this issue,
please let me know.

Thanks and Regards,
Gourav Shenoy


Hi Gourav,

Correct me if I'm wrong, but I think this is a case of the job shop scheduling problem, as
we may have 'n' jobs of varying processing times and memory requirements, and we have 'm'
microservices with possibly different computing and memory capacities, and we are trying to
minimize the makespan<https://en.wikipedia.org/wiki/Makespan>.

For this use-case, I'm in favor a highly available and consistent message broker with an intelligent
job scheduling algorithm, which receives real-time updates from the microservices with their
current state information.

As for the state vs stateless implementation, I think that question depends on the functionality
of a particular microservice. In a broad sense, the stateless implementation should be preferred
as it will scale better horizontally.


Regards,
Vidya Sagar


--
Vidya Sagar Kalvakunta | Graduate MS CS Student | IU School of Informatics and Computing |
Indiana University Bloomington | (812) 691-5002<tel:8126915002> | vkalvaku@iu.edu<mailto:vkalvaku@iu.edu>



--
Thanks and regards,

Ajinkya Dhamnaskar
Student ID : 0003469679
Masters (CS)
+1 (812) 369- 5416
Mime
View raw message