Mailing-List: contact dev-help@reef.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@reef.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAPbHxKnhkMMXLRbgeuJhKTgJX95j=vPqNQLzNFyAUe=2heWLjQ@mail.gmail.com>
References: 
 <CAPbHxKk5OsbDW5qW3tAXidCgjKnhs_s922L41+T886zb0kw2Fg@mail.gmail.com>
 <CAOFpRGCjuzwmd-gVDHTkmTj0wkrG1mFJZOL9JseiLtuaXDV_+Q@mail.gmail.com>
 <CAPbHxKnhkMMXLRbgeuJhKTgJX95j=vPqNQLzNFyAUe=2heWLjQ@mail.gmail.com>
From: Markus Weimer <markus@weimo.de>
Date: Mon, 21 Mar 2016 17:49:42 -0700
Message-ID: 
 <CAOFpRGCTTXEu6FvtkeYy3TCYjm0amaaL3zzEDcVNKfLKdkbD9Q@mail.gmail.com>
Subject: Re: Map EvaluatorRequest to AllocatedEvaluator
To: aasojj@gmail.com, REEF Developers Mailinglist <dev@reef.apache.org>
Content-Type: text/plain; charset=UTF-8

On Mon, Mar 21, 2016 at 4:43 PM, Ashvin A <aas.ojj@gmail.com> wrote:
> Is there a REEF jira I could watch?

I don't think so. Maybe we should check whether YARN has one?

> And are there any known workarounds?

As you mentioned, you can submit sequentially. When you have vastly
different requests, you can send the requests all at once, and then
disambiguate in the `AllocatedEvaluator` handler. This works when you
have one kind of container with let's say 4 cores and another one with
1 core. However, YARN is allowed to "round up" to the nearest
container size. If that is 4 cores on a cluster you use, it will
always give you 4 core containers, even if you request 1 core...

In REEF.NET, there is a (ill-fated, IMHO) attempt at codifying such a
matchmaking approach. But for the reasons outlined here, it is also
doomed to fail in much the same ways :(

Markus