hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandy Ryza <sandy.r...@cloudera.com>
Subject scheduler satisfying heterogeneous resource requests at same priority
Date Sat, 29 Dec 2012 00:46:14 GMT
I am trying to understand how YARN schedulers are able to satisfy smaller
requests while larger requests are outstanding (per YARN-289).

Consider the following situation:
An application submits two requests - one for a container with 1024 MB and
one for a container with 2048 MB.  1024 MB frees up on a node.  The
scheduler should (or might wish to) place the smaller container on the
node, instead of placing a reservation for the larger one.

However, currently, if I understand correctly, the larger request is always
serviced first.  AppSchedulingInfo, which is used by all the schedulers to
find a container request when space becomes available, stores a map of
priorities to maps of node/rack/* to ResourceRequests.  A ResourceRequest
contains a single Resource (capability), and the number of containers.  Why
does a ResourceRequest not allow for heterogeneous containers.  Is this
just not supported yet because it hasn't been needed yet?  Or is there a
more fundamental reason I'm missing about why it doesn't make sense?

many thanks for any guidance,

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message