reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julia Wang (QIUHE)" <>
Subject RE: YARN Request IDs are here
Date Wed, 04 Oct 2017 21:21:46 GMT
This feature is in 2.9.0, 3.0.0-beta1. When can we pick up Hadoop 2.9.0? 


-----Original Message-----
From: Byung-Gon Chun [] 
Sent: Wednesday, October 4, 2017 1:12 AM
Subject: Re: YARN Request IDs are here

Hi Markus,

It's great to hear that YARN request IDs are available finally. :)

The initial proposal for node labels proposed to add Map<String, String> of metadata
in EvaluatorDescriptor. After a couple of review rounds, Seokchan and Gyewon suggested using

Do you think it's better to use Map<String, String> instead of SchedulingConstraints?
If so, we can go back to the route of using Map<String, String> without adding SchedulingConstraints.


On Wed, Oct 4, 2017 at 4:16 PM, Markus Weimer <> wrote:

> Hi,
> the long timers might remember, but this has been a need of ours for a 
> long
> time: YARN now allows you to tag resource requests, and the 
> allocations that come back will have those tags:
> 2d7cd011db47%7C1%7C0%7C636427015482515730&sdata=XOV5rMnucxcedTmU7Hiu%2
> BKe4jLOvG9UiIuDQpWwCDys%3D&reserved=0
> This will greatly simplify the implementation of stuff like IMRU, 
> where there are different resource allocations for different 
> Evaluators. Now it is up to us to support that new API. I believe that 
> the `SchedulingConstraints` approach taken in REEF-1750 should be a 
> workable way to get this new feature into the `EvaluatorRequestor` 
> APIs. For the `AllocatedEvaluator`, we could add a new field 
> `requestId` or such to `EvaluatorDescriptor`. However, I think that 
> we've reached the point where that class might be better served by a 
> `Map<String,String>` of metadata about that Evaluator. That way, 
> runtimes can communicate an open set of metadata to the Driver.
> Markus

Byung-Gon Chun
View raw message