hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grandl Robert <rgra...@yahoo.com>
Subject Re: Force reducer to execute on a specific host
Date Mon, 25 Nov 2013 04:02:20 GMT
I found the fix. I had to add also a rack local request for reducer. And set the relaxLocality
to false for both ANY and rack requests, in order to force data locality.

robert




On Sunday, November 24, 2013 6:32 PM, Grandl Robert <rgrandl@yahoo.com> wrote:
 
Hi guys, 

I am trying to run an experiment where I want the reducers to be forced executing on a specific
host. The simplest way I found it is to modify in RMContainerAllocator such that whenever
there is an addContainerReq(ContainerRequest) for a reducer, I simply do: addResourceRequest(req.priority,
host_X_name, req.capability) and return, instead of setting the resourceName to ResourceRequest.ANY.


Unfortunately when I pass such a ResourceRequest for reducers through a makeRemoteRequest,
the RM crashes, giving an ugly exception like: 
FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
type NODE_UPDATE to the scheduler
java.lang.NullPointerException
    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:841)
    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:640)
    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:554)
    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:695)
    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:739)
    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:86)
    at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:549)
    at java.lang.Thread.run(Thread.java:722)

Do you have other suggestion how I can enforce reducers execution on a certain host ? Or do
you know what I am doing wrong here ? 

Thanks,
robert
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message