reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobin Baker <tdba...@cs.washington.edu>
Subject Re: Specifying locality constraints for Driver
Date Mon, 07 Dec 2015 23:38:20 GMT
Thanks for the tip about node labels! I'll see if I can get this working at
the YARN level (I did something similar with node locality before that was
added in the last release).  Created a JIRA for this issue and assigned to
myself: https://issues.apache.org/jira/browse/REEF-1048



On Mon, Dec 7, 2015 at 2:59 PM, Byung-Gon Chun <bgchun@gmail.com> wrote:

> Hi Tobin,
>
> This's a requested feature for our customers as well in Korea. Driver runs
> on a more powerful machine in the cluster.
> There's no locality setter in YARN ApplicationSubmissionContext.
>
> https://hadoop.apache.org/docs/r2.6.1/api/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.html#setNodeLabelExpression(java.lang.String)
>
> But we can achieve the effect you want by using node labels. This's not
> currently exposed through REEF, though. A good item to contribute. :)
>
> Cheers,
> Gon
>
> On Tue, Dec 8, 2015 at 7:51 AM, Tobin Baker <tdbaker@cs.washington.edu>
> wrote:
>
> > Hi, our initial users for Myria on REEF would like to be able to launch
> the
> > driver on a particular node (because they have just enough memory on the
> > non-coordinator nodes for a single Task evaluator and don't want to
> > allocate enough slack memory on every node for the driver in case it gets
> > randomly assigned to that node). I spent some time grepping REEF source (
> >
> >
> https://github.com/apache/reef/blob/master/lang/java/reef-common/src/main/java/org/apache/reef/client/DriverConfiguration.java
> > ,
> >
> >
> https://github.com/apache/reef/blob/master/lang/java/reef-runtime-yarn/src/main/java/org/apache/reef/runtime/yarn/driver/YarnDriverConfiguration.java
> > )
> > for such a configuration option, but came up empty. Am I missing
> something?
> > (Note that there may be a YARN bug that prevents this from working even
> if
> > I dropped down to YARN:
> >
> >
> http://stackoverflow.com/questions/29302659/run-my-own-application-master-on-a-specific-node-in-a-yarn-cluster
> > .)
> >
>
>
>
> --
> Byung-Gon Chun
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message