Mailing-List: contact dev-help@reef.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@reef.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CADq0cj61RK9bygmYg9yA4JYGtRyQ2MHRThG3KvzZspKSASuXGA@mail.gmail.com>
References: 
 <CAPCBsnN2ssH+3SPzt-dSiWxZuutUbH6idxxxcqSNTa+S7mt3Ug@mail.gmail.com>
	<CADq0cj61RK9bygmYg9yA4JYGtRyQ2MHRThG3KvzZspKSASuXGA@mail.gmail.com>
Date: Mon, 7 Dec 2015 15:38:20 -0800
Message-ID: 
 <CAPCBsnNTZ+CsdjJCY-KrSvceDzVM1VWu-DuqRLxKNrimrx79Jg@mail.gmail.com>
Subject: Re: Specifying locality constraints for Driver
From: Tobin Baker <tdbaker@cs.washington.edu>
To: dev@reef.apache.org
Cc: Brandon Haynes <bhaynes@cs.washington.edu>
Content-Type: multipart/alternative; boundary=001a1135ffb031bc5a0526575a3e

--001a1135ffb031bc5a0526575a3e
Content-Type: text/plain; charset=UTF-8

Thanks for the tip about node labels! I'll see if I can get this working at
the YARN level (I did something similar with node locality before that was
added in the last release).  Created a JIRA for this issue and assigned to
myself: https://issues.apache.org/jira/browse/REEF-1048


On Mon, Dec 7, 2015 at 2:59 PM, Byung-Gon Chun <bgchun@gmail.com> wrote:

> Hi Tobin,
>
> This's a requested feature for our customers as well in Korea. Driver runs
> on a more powerful machine in the cluster.
> There's no locality setter in YARN ApplicationSubmissionContext.
>
> https://hadoop.apache.org/docs/r2.6.1/api/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.html#setNodeLabelExpression(java.lang.String)
>
> But we can achieve the effect you want by using node labels. This's not
> currently exposed through REEF, though. A good item to contribute. :)
>
> Cheers,
> Gon
>
> On Tue, Dec 8, 2015 at 7:51 AM, Tobin Baker <tdbaker@cs.washington.edu>
> wrote:
>
> > Hi, our initial users for Myria on REEF would like to be able to launch
> the
> > driver on a particular node (because they have just enough memory on the
> > non-coordinator nodes for a single Task evaluator and don't want to
> > allocate enough slack memory on every node for the driver in case it gets
> > randomly assigned to that node). I spent some time grepping REEF source (
> >
> >
> https://github.com/apache/reef/blob/master/lang/java/reef-common/src/main/java/org/apache/reef/client/DriverConfiguration.java
> > ,
> >
> >
> https://github.com/apache/reef/blob/master/lang/java/reef-runtime-yarn/src/main/java/org/apache/reef/runtime/yarn/driver/YarnDriverConfiguration.java
> > )
> > for such a configuration option, but came up empty. Am I missing
> something?
> > (Note that there may be a YARN bug that prevents this from working even
> if
> > I dropped down to YARN:
> >
> >
> http://stackoverflow.com/questions/29302659/run-my-own-application-master-on-a-specific-node-in-a-yarn-cluster
> > .)
> >
>
>
>
> --
> Byung-Gon Chun
>

--001a1135ffb031bc5a0526575a3e--