airavata-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amila Jayasekara <thejaka.am...@gmail.com>
Subject Re: Work Stealing is not a good solution for Airavata.
Date Wed, 05 Oct 2016 02:51:15 GMT
First of all, what do you mean by "work stealing design pattern"? I have
not heard such a design pattern and probably what you are trying to explain
is not "work stealing" (to my understanding). Work stealing is a
multi-threaded scheduling mechanism mostly use in systems that implement
lightweight threads. The basic idea is to have a queue per each lightweight
thread, and when a particular thread is idle, it can get work (steal) from
a busy lightweight thread. The architecture scenario you are explaining
does not go into that category of work stealing as per my understanding
(See [2] for more info). It is a more likely distribution of work.

Regarding 1 => To me, this looks like a limitation in the tools that we
use. As per what I understood the M, N issue is because of use of AMQP
prefetch. However, I did not understand, what AMQP prefetch is doing. Just
because the underlying tools has limitations, I don't think we can
criticise the design. Always try to keep the conceptual design and tools
separately.

Regarding 2 => For cancel experiments keeping priority queue make sense but
how long a request reside in the worker queue?
I have no idea what you are trying to explain using zookeeper and rabbitmq
in this context (also I don't know how those function). If you conceptually
describe the problem, i may able to give more feedback.

Regarding 3 => Well.. email monitoring was alway problematic. More
precisely monitoring is problematic IMO. To handle monitoring I think we
should be able to get better feedback from job schedulers but as per my
experience, even those job schedulers are unreliable. Until we have a
better way to get feedback from job scheduler, monitoring is going to be
challenging. However, I don't understand why you have "serious scalability"
issues in GFac because of this.

In summary, to me, none of these are concerns for the architecture at the
moment. Also, you cannot just naively complain an architecture is "not
good". Architecture has to be compared with another design and evaluate and
pros/cons for both. I suggest we first try to improve the existing design
to handle issues you are pointing.

[2] https://en.wikipedia.org/wiki/Work_stealing

Thanks
-Amila


On Tue, Oct 4, 2016 at 1:07 PM, Shameera Rathnayaka <shameerainfo@gmail.com>
wrote:

> Hi Devs,
>
> Airavata has adopted to work stealing design pattern lately and use work
> queue approach to distributing works among consumers. There are two work
> queues in current Airavata architecture. One in middle of API Server and
> Orchestrator and the second one in between Orchestrator and Gfac, Following
> is very high-level Airavata architecture.
>
>
> [image: Highleve Arachitecture- Airavata.png]
> Here are the issues we have with above architecture.
>
> 1. Low resource utilization in Workers (Gfac/Orchestrator).
> We have used AMPQ prefetch count to limit the number of requests served by
> a Worker, which is not a good way to load balance in the
> heterogeneous environment where different workers have different level of
> resources. And it is recommended to keep this prefetch count minimum [1]
> and this is valid for work stealing too. If we only have one worker and we
> have M ( > N) number of long running jobs, and our prefetch count is N
> then, only N jobs will in active mode. As we are run this long-running job
> in the async way, we can handle more long running jobs than N. Therefore
> workers resources are underutilized.
>
> 2. Even though we can easily deal with recovery requirement with work
> stealing, it is not easy to handle cancel experiments. When this cancel
> experiment comes the worker who works on this experiment should act
> immediately. To add this behavior we need to introduce priority queues and
> no need say this will add extra layer of complexity. Currently, we use
> zookeeper to trigger cancel requests ( Another downside, we are using both
> zookeeper and rabbitmq to solve different parts of Distributed systems
> issues. Almost all latest Distributed system frameworks have being used
> zookeeper to manage distributed system problems, we need to strongly
> consider using zookeeper  as a way of managing our components and share the
> load according to the resource available in workers)
>
> 3. Putting email to a queue is not a good solution with commodity servers
> where system failures are expected. This email queue is critical, if we
> missed one of the statuses of a job then this job can go to the unknown
> state or hang in the old status forever. Due to this, we have serious
> scalability issue with GFac at the moment due to a bottleneck of email
> monitoring.
>
> I think we need to re-evaluate Airavata architecture and find a good yet
> simple solution based on requirements. The new architecture should handle
> all existing issues and able to extend future requirement.
>
>
> [1] http://www.mariuszwojcik.com/blog/How-to-choose-prefetch
> -count-value-for-RabbitMQ
> --
> Shameera Rathnayaka
>

Mime
View raw message