hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Krishna Kishore Bonagiri <write2kish...@gmail.com>
Subject 100% CPU consumption by Resource Manager process
Date Sun, 10 Aug 2014 15:29:16 GMT
Hi,
  My YARN resource manager is consuming 100% CPU when I am running an
application that is running for about 10 hours, requesting as many as 27000
containers. The CPU consumption was very low at the starting of my
application, and it gradually went high to over 100%. Is this a known issue
or are we doing something wrong?

Every dump of the EVent Processor thread is running
LeafQueue::assignContainers() specifically the for loop below from
LeafQueue.java and seems to be looping through some priority list.

    // Try to assign containers to applications in order
    for (FiCaSchedulerApp application : activeApplications) {
...
        // Schedule in priority order
        for (Priority priority : application.getPriorities()) {

3XMTHREADINFO      "ResourceManager Event Processor"
J9VMThread:0x0000000001D08600, j9thread_t:0x00007F032D2FAA00,
java/lang/Thread:0x000000008341D9A0, state:CW, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x1E, isDaemon:false)
3XMTHREADINFO1            (native thread ID:0x4B64, native priority:0x5,
native policy:UNKNOWN)
3XMTHREADINFO2            (native stack address range
from:0x00007F0313DF8000, to:0x00007F0313E39000, size:0x41000)
3XMCPUTIME               *CPU usage total: 42334.614623696 secs*
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=20456
(0x4FE8)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.assignContainers(LeafQueue.java:850(Compiled
Code))
5XESTACKTRACE                   (entered lock:
org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp@0x000000008360DFE0,
entry count: 1)
5XESTACKTRACE                   (entered lock:
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue@0x00000000833B9280,
entry count: 1)
4XESTACKTRACE                at
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.assignContainersToChildQueues(ParentQueue.java:655(Compiled
Code))
5XESTACKTRACE                   (entered lock:
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue@0x0000000083360A80,
entry count: 2)
4XESTACKTRACE                at
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.assignContainers(ParentQueue.java:569(Compiled
Code))
5XESTACKTRACE                   (entered lock:
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue@0x0000000083360A80,
entry count: 1)
4XESTACKTRACE                at
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:831(Compiled
Code))
5XESTACKTRACE                   (entered lock:
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler@0x00000000834037C8,
entry count: 1)
4XESTACKTRACE                at
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.handle(CapacityScheduler.java:878(Compiled
Code))
4XESTACKTRACE                at
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.handle(CapacityScheduler.java:100(Compiled
Code))
4XESTACKTRACE                at
org/apache/hadoop/yarn/server/resourcemanager/ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:591)
4XESTACKTRACE                at java/lang/Thread.run(Thread.java:853)

3XMTHREADINFO      "ResourceManager Event Processor"
J9VMThread:0x0000000001D08600, j9thread_t:0x00007F032D2FAA00,
java/lang/Thread:0x000000008341D9A0, state:CW, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x1E, isDaemon:false)
3XMTHREADINFO1            (native thread ID:0x4B64, native priority:0x5,
native policy:UNKNOWN)
3XMTHREADINFO2            (native stack address range
from:0x00007F0313DF8000, to:0x00007F0313E39000, size:0x41000)
3XMCPUTIME               CPU usage total: 42379.604203548 secs
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=57280
(0xDFC0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.assignContainers(LeafQueue.java:841(Compiled
Code))
5XESTACKTRACE                   (entered lock:
org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp@0x000000008360DFE0,
entry count: 1)
5XESTACKTRACE                   (entered lock:
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue@0x00000000833B9280,
entry count: 1)
4XESTACKTRACE                at
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.assignContainersToChildQueues(ParentQueue.java:655(Compiled
Code))
5XESTACKTRACE                   (entered lock:
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue@0x0000000083360A80,
entry count: 2)
4XESTACKTRACE                at
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.assignContainers(ParentQueue.java:569(Compiled
Code))
5XESTACKTRACE                   (entered lock:
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue@0x0000000083360A80,
entry count: 1)
4XESTACKTRACE                at
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:831(Compiled
Code))
5XESTACKTRACE                   (entered lock:
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler@0x00000000834037C8,
entry count: 1)
4XESTACKTRACE                at
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.handle(CapacityScheduler.java:878(Compiled
Code))
4XESTACKTRACE                at
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.handle(CapacityScheduler.java:100(Compiled
Code))
4XESTACKTRACE                at
org/apache/hadoop/yarn/server/resourcemanager/ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:591)
4XESTACKTRACE                at java/lang/Thread.run(Thread.java:853)

3XMTHREADINFO      "ResourceManager Event Processor"
J9VMThread:0x0000000001D08600, j9thread_t:0x00007F032D2FAA00,
java/lang/Thread:0x000000008341D9A0, state:CW, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x1E, isDaemon:false)
3XMTHREADINFO1            (native thread ID:0x4B64, native priority:0x5,
native policy:UNKNOWN)
3XMTHREADINFO2            (native stack address range
from:0x00007F0313DF8000, to:0x00007F0313E39000, size:0x41000)
3XMCPUTIME               CPU usage total: 42996.394528764 secs
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=475576
(0x741B8)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at
java/util/TreeMap.successor(TreeMap.java:2001(Compiled Code))
4XESTACKTRACE                at
java/util/TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1127(Compiled
Code))
4XESTACKTRACE                at
java/util/TreeMap$KeyIterator.next(TreeMap.java:1180(Compiled Code))
4XESTACKTRACE                at
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.assignContainers(LeafQueue.java:838(Compiled
Code))
5XESTACKTRACE                   (entered lock:
org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp@0x000000008360DFE0,
entry count: 1)
5XESTACKTRACE                   (entered lock:
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue@0x00000000833B9280,
entry count: 1)
4XESTACKTRACE                at
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.assignContainersToChildQueues(ParentQueue.java:655(Compiled
Code))
5XESTACKTRACE                   (entered lock:
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue@0x0000000083360A80,
entry count: 2)
4XESTACKTRACE                at
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.assignContainers(ParentQueue.java:569(Compiled
Code))
5XESTACKTRACE                   (entered lock:
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue@0x0000000083360A80,
entry count: 1)
4XESTACKTRACE                at
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:831(Compiled
Code))
5XESTACKTRACE                   (entered lock:
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler@0x00000000834037C8,
entry count: 1)
4XESTACKTRACE                at
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.handle(CapacityScheduler.java:878(Compiled
Code))
4XESTACKTRACE                at
org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.handle(CapacityScheduler.java:100(Compiled
Code))
4XESTACKTRACE                at
org/apache/hadoop/yarn/server/resourcemanager/ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:591)
4XESTACKTRACE                at java/lang/Thread.run(Thread.java:853)

Thanks,
Kishore

Mime
View raw message