airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shameera Rathnayaka <shameerai...@gmail.com>
Subject Work Stealing is not a good solution for Airavata.
Date Tue, 04 Oct 2016 17:07:22 GMT
Hi Devs,

Airavata has adopted to work stealing design pattern lately and use work
queue approach to distributing works among consumers. There are two work
queues in current Airavata architecture. One in middle of API Server and
Orchestrator and the second one in between Orchestrator and Gfac, Following
is very high-level Airavata architecture.


[image: Highleve Arachitecture- Airavata.png]
Here are the issues we have with above architecture.

1. Low resource utilization in Workers (Gfac/Orchestrator).
We have used AMPQ prefetch count to limit the number of requests served by
a Worker, which is not a good way to load balance in the
heterogeneous environment where different workers have different level of
resources. And it is recommended to keep this prefetch count minimum [1]
and this is valid for work stealing too. If we only have one worker and we
have M ( > N) number of long running jobs, and our prefetch count is N
then, only N jobs will in active mode. As we are run this long-running job
in the async way, we can handle more long running jobs than N. Therefore
workers resources are underutilized.

2. Even though we can easily deal with recovery requirement with work
stealing, it is not easy to handle cancel experiments. When this cancel
experiment comes the worker who works on this experiment should act
immediately. To add this behavior we need to introduce priority queues and
no need say this will add extra layer of complexity. Currently, we use
zookeeper to trigger cancel requests ( Another downside, we are using both
zookeeper and rabbitmq to solve different parts of Distributed systems
issues. Almost all latest Distributed system frameworks have being used
zookeeper to manage distributed system problems, we need to strongly
consider using zookeeper  as a way of managing our components and share the
load according to the resource available in workers)

3. Putting email to a queue is not a good solution with commodity servers
where system failures are expected. This email queue is critical, if we
missed one of the statuses of a job then this job can go to the unknown
state or hang in the old status forever. Due to this, we have serious
scalability issue with GFac at the moment due to a bottleneck of email
monitoring.

I think we need to re-evaluate Airavata architecture and find a good yet
simple solution based on requirements. The new architecture should handle
all existing issues and able to extend future requirement.


[1]
http://www.mariuszwojcik.com/blog/How-to-choose-prefetch-count-value-for-RabbitMQ

-- 
Shameera Rathnayaka

Mime
View raw message