reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Weimer <mar...@weimo.de>
Subject YARN Request IDs are here
Date Wed, 04 Oct 2017 07:16:36 GMT
Hi,

the long timers might remember, but this has been a need of ours for a long
time: YARN now allows you to tag resource requests, and the allocations
that come back will have those tags:

https://issues.apache.org/jira/browse/YARN-4879


This will greatly simplify the implementation of stuff like IMRU, where
there are different resource allocations for different Evaluators. Now it
is up to us to support that new API. I believe that the
`SchedulingConstraints` approach taken in REEF-1750 should be a workable
way to get this new feature into the `EvaluatorRequestor` APIs. For the
`AllocatedEvaluator`, we could add a new field `requestId` or such to
`EvaluatorDescriptor`. However, I think that we've reached the point where
that class might be better served by a `Map<String,String>` of metadata
about that Evaluator. That way, runtimes can communicate an open set of
metadata to the Driver.

WDYT?

Markus

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message