hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grandl Robert <rgra...@yahoo.com>
Subject Force reducer to execute on a specific host
Date Mon, 25 Nov 2013 02:32:19 GMT
Hi guys, 

I am trying to run an experiment where I want the reducers to be forced executing on a specific
host. The simplest way I found it is to modify in RMContainerAllocator such that whenever
there is an addContainerReq(ContainerRequest) for a reducer, I simply do: addResourceRequest(req.priority,
host_X_name, req.capability) and return, instead of setting the resourceName to ResourceRequest.ANY.


Unfortunately when I pass such a ResourceRequest for reducers through a makeRemoteRequest,
the RM crashes, giving an ugly exception like: 
FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in handling event
type NODE_UPDATE to the scheduler
java.lang.NullPointerException
    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:841)
    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:640)
    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:554)
    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:695)
    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:739)
    at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:86)
    at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:549)
    at java.lang.Thread.run(Thread.java:722)

Do you have other suggestion how I can enforce reducers execution on a certain host ? Or do
you know what I am doing wrong here ? 

Thanks,
robert
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message