reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julia Wang (QIUHE)" <Qiuhe.W...@microsoft.com.INVALID>
Subject RE: YARN Request IDs are here
Date Wed, 04 Oct 2017 21:21:46 GMT
This feature is in 2.9.0, 3.0.0-beta1. When can we pick up Hadoop 2.9.0? 

-Julia

-----Original Message-----
From: Byung-Gon Chun [mailto:bgchun@gmail.com] 
Sent: Wednesday, October 4, 2017 1:12 AM
To: dev@reef.apache.org
Subject: Re: YARN Request IDs are here

Hi Markus,

It's great to hear that YARN request IDs are available finally. :)

The initial proposal for node labels proposed to add Map<String, String> of metadata
in EvaluatorDescriptor. After a couple of review rounds, Seokchan and Gyewon suggested using
SchedulingConstraints.

Do you think it's better to use Map<String, String> instead of SchedulingConstraints?
If so, we can go back to the route of using Map<String, String> without adding SchedulingConstraints.

Thanks!
-Gon



On Wed, Oct 4, 2017 at 4:16 PM, Markus Weimer <markus@weimo.de> wrote:

> Hi,
>
> the long timers might remember, but this has been a need of ours for a 
> long
> time: YARN now allows you to tag resource requests, and the 
> allocations that come back will have those tags:
>
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissue
> s.apache.org%2Fjira%2Fbrowse%2FYARN-4879&data=02%7C01%7CQiuhe.Wang%40m
> icrosoft.com%7C00f9c69007854b81d4a708d50affa680%7C72f988bf86f141af91ab
> 2d7cd011db47%7C1%7C0%7C636427015482515730&sdata=XOV5rMnucxcedTmU7Hiu%2
> BKe4jLOvG9UiIuDQpWwCDys%3D&reserved=0
>
>
> This will greatly simplify the implementation of stuff like IMRU, 
> where there are different resource allocations for different 
> Evaluators. Now it is up to us to support that new API. I believe that 
> the `SchedulingConstraints` approach taken in REEF-1750 should be a 
> workable way to get this new feature into the `EvaluatorRequestor` 
> APIs. For the `AllocatedEvaluator`, we could add a new field 
> `requestId` or such to `EvaluatorDescriptor`. However, I think that 
> we've reached the point where that class might be better served by a 
> `Map<String,String>` of metadata about that Evaluator. That way, 
> runtimes can communicate an open set of metadata to the Driver.
>
> WDYT?
>
> Markus
>



--
Byung-Gon Chun
Mime
View raw message