curator-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cantrell, Curtis" <>
Subject LeaderSelector running away with CPU
Date Wed, 25 Feb 2015 16:58:00 GMT
We have 5 servers in the ensemble with a LeaderSelector in each server.    In our performance
environment, we are noticing that the server that takes leadership causes CPU utilization
to run up.  Test yesterday say CPU utilization at 95%.  Today was better at 33%.  The odd
thing is that even after the test are completed, the CPU utilization does not go down.

We determined what thread was causing the high CPU utilization with

ps -C java -L -o pid,tid,pcpu,state,stime,time,nlwp

The TID is the thread id.   We then converted the TID to hex and searched the javacore thread
dump to see what was causing the CPU to hang.  Every time the consuming thread was running
the LeaderSelector.

3XMTHREADINFO      "LeaderSelector-2" J9VMThread:0x00000000C54D8800, j9thread_t:0x00007F8C51E5CC80,
java/lang/Thread:0x0000000006FAB4D8, state:CW, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x3A8, isDaemon:true)
3XMTHREADINFO1            (native thread ID:0x2BDE, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO2            (native stack address range from:0x00007F8BD298F000, to:0x00007F8BD29D0000,
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at com/lps/mg/inlet/sadevents/Leader.takeLeadership(
4XESTACKTRACE                at org/apache/curator/framework/recipes/leader/LeaderSelector$WrappedListener.takeLeadership(
4XESTACKTRACE                at org/apache/curator/framework/recipes/leader/LeaderSelector.doWork(
4XESTACKTRACE                at org/apache/curator/framework/recipes/leader/LeaderSelector.doWorkLoop(
4XESTACKTRACE                at org/apache/curator/framework/recipes/leader/LeaderSelector.access$100(
4XESTACKTRACE                at org/apache/curator/framework/recipes/leader/LeaderSelector$
4XESTACKTRACE                at org/apache/curator/framework/recipes/leader/LeaderSelector$
4XESTACKTRACE                at java/util/concurrent/FutureTask$Sync.innerRun(
4XESTACKTRACE                at java/util/concurrent/
4XESTACKTRACE                at java/util/concurrent/Executors$
4XESTACKTRACE                at java/util/concurrent/FutureTask$Sync.innerRun(
4XESTACKTRACE                at java/util/concurrent/
4XESTACKTRACE                at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(
4XESTACKTRACE                at java/util/concurrent/ThreadPoolExecutor$
4XESTACKTRACE                at java/lang/ Code))

Is this a reason that the LeaderSelector would run away with the CPU?

Thank you,
Curtis Cantrell

The information contained in this message is proprietary and/or confidential. If you are not
the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose,
distribute or use the message in any manner; and (iii) notify the sender immediately. In addition,
please be aware that any message addressed to our domain is subject to archiving and review
by persons other than the intended recipient. Thank you.

View raw message