reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Byung-Gon Chun <bgc...@gmail.com>
Subject Re: Specifying locality constraints for Driver
Date Mon, 07 Dec 2015 22:59:59 GMT
Hi Tobin,

This's a requested feature for our customers as well in Korea. Driver runs
on a more powerful machine in the cluster.
There's no locality setter in YARN ApplicationSubmissionContext.
https://hadoop.apache.org/docs/r2.6.1/api/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.html#setNodeLabelExpression(java.lang.String)

But we can achieve the effect you want by using node labels. This's not
currently exposed through REEF, though. A good item to contribute. :)

Cheers,
Gon

On Tue, Dec 8, 2015 at 7:51 AM, Tobin Baker <tdbaker@cs.washington.edu>
wrote:

> Hi, our initial users for Myria on REEF would like to be able to launch the
> driver on a particular node (because they have just enough memory on the
> non-coordinator nodes for a single Task evaluator and don't want to
> allocate enough slack memory on every node for the driver in case it gets
> randomly assigned to that node). I spent some time grepping REEF source (
>
> https://github.com/apache/reef/blob/master/lang/java/reef-common/src/main/java/org/apache/reef/client/DriverConfiguration.java
> ,
>
> https://github.com/apache/reef/blob/master/lang/java/reef-runtime-yarn/src/main/java/org/apache/reef/runtime/yarn/driver/YarnDriverConfiguration.java
> )
> for such a configuration option, but came up empty. Am I missing something?
> (Note that there may be a YARN bug that prevents this from working even if
> I dropped down to YARN:
>
> http://stackoverflow.com/questions/29302659/run-my-own-application-master-on-a-specific-node-in-a-yarn-cluster
> .)
>



-- 
Byung-Gon Chun

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message