reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Weimer <>
Subject Re: Map EvaluatorRequest to AllocatedEvaluator
Date Tue, 22 Mar 2016 00:49:42 GMT
On Mon, Mar 21, 2016 at 4:43 PM, Ashvin A <> wrote:
> Is there a REEF jira I could watch?

I don't think so. Maybe we should check whether YARN has one?

> And are there any known workarounds?

As you mentioned, you can submit sequentially. When you have vastly
different requests, you can send the requests all at once, and then
disambiguate in the `AllocatedEvaluator` handler. This works when you
have one kind of container with let's say 4 cores and another one with
1 core. However, YARN is allowed to "round up" to the nearest
container size. If that is 4 cores on a cluster you use, it will
always give you 4 core containers, even if you request 1 core...

In REEF.NET, there is a (ill-fated, IMHO) attempt at codifying such a
matchmaking approach. But for the reasons outlined here, it is also
doomed to fail in much the same ways :(


View raw message