hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From santosh kumar <santosh.adit...@gmail.com>
Subject Yet Another Scheduling Idea
Date Mon, 09 Dec 2013 20:43:00 GMT

I am trying to develop an auction based scheduling algorithm for YARN. I
have few questions about the same.

Idea is to have an "Auction Manager" inside every "Node Manager" who
receive auction requests from "Resource Manager" for every resource request
from "Application Master". The cluster nodes are grouped such that each
rack has one seller and all other as participants. Based on few
parameters/metrics the seller selects a winning participant to serve the
resource request.

Question 1:
I added a new module for auctions in my local code repository and pretty
much covered most of the use cases. But I am stuck at understanding how
communication actually takes place between RM and NM or among NMs. I need
to include something on lines of send/recv commands. Like how the
map/reduce commands are given using some job.xml.

Question 2:
All my code additions are in:
hadoop-yarn-api; hadoop-yarn-common; hadoop-yarn-server-common;
hadoop-yarn-server-nodemanager; hadoop-yarn-server-resourcemanager

Would I need other modules like hadoop-yarn-client etc? My guess is I don't
think so.

Question 3:
Did not really understand the Yarn-Protos. Any place where I can get some
decent documentation about this?

Any response/help is greatly appreciated.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message