hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Re: Ordering of ResourceRequest handling in the RM
Date Mon, 15 Jun 2015 09:41:56 GMT

> On 12 Jun 2015, at 20:20, Navina Ramesh <nramesh@linkedin.com.INVALID> wrote:
> Hi Steve,
> Thanks for your response.
> I think I should have mentioned a couple of design choices we made:
> 1. *Continuous Scheduling*  - I explored the option of requesting
> containers with relaxLocality==false. But this will add to latency and
> there is no way of revoking the resource request.

You can cancel requests. AMRMClientAsync.removeContainerRequest(request);

> An alternate approach
> was to use continuous scheduling with the FairScheduler. Instead of
> waiting on node heartbeats, the RM iterates through the nodes based on the
> latest known states. We enable this and also, associate a rack and node
> level delay. This way if the requested resource is not available, it will
> automatically relaxLocality and return any available resource.
> 2. All the resource request that I make is for running the same component
> and hence, their priority is the same.
>>> that's a bad strategy. What if hostY had capacity before hostX? Would
>>> you expect it to be blocked until hostX was satisifed? as hostY may not
>>> be free then
> It is fine if the hostY gets allocated before hostX. I do maintain a map
> of resource request to requested host. So, if hostY is returned, I will
> know that I expected that host for container-1. The issue is how to
> associate an allocated resource that is not in my list of preferred hosts.
> If a random hostA is allocated, then which container should I assign ?
> This is where I am stuck and I am wondering if there is a way around.

I couldn't find a way here. When things come in to an unexpected host they just get an instance
of that component type assigned to it.

that leaves outstanding requests in the list, I deal with that when eventually all the containers
get satisified -at that point I know everything outstanding is satisified, so cancel all the

I'd have liked a request ID to do the mapping, but that would apparently make aggregating
requests impossible, so no.

View raw message