hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitry Sivachenko <trtrmi...@gmail.com>
Subject Re: Idle tasktracker eats CPU
Date Wed, 11 Jun 2014 10:58:06 GMT

On 10 июня 2014 г., at 7:24, Dmitry Sivachenko <trtrmitya@gmail.com> wrote:

> 
> On 10 июня 2014 г., at 5:58, Serge Blazhievsky <hadoop.ca@gmail.com> wrote:
> 
>> There are ongoing pings to jobtracker 
>> 
>> But 10% seems high
> 
> 
> Sometimes it is 10%, sometimes it is 200+%.
> Different nodes, different numbers.
> 
> So pings can't explain that.
> 


The more jobs I run, more java threads start to consume CPU after all tasks finished.  After
several job execution, top(1) output looks like this (splitted by thread, the same PID):

  PID USERNAME     PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
79045 hadoop        47    0  1948M   867M uwait   2  20:49  37.50% java{java}
79045 hadoop        31    0  1948M   867M uwait  31   1:45  19.29% java{java}
79045 hadoop        33    0  1948M   867M uwait  21   2:51  19.19% java{java}
79045 hadoop        30    0  1948M   867M uwait  17   2:51  18.65% java{java}
79045 hadoop        30    0  1948M   867M uwait  11   1:52  18.36% java{java}
79045 hadoop        30    0  1948M   867M uwait  22   1:45  18.36% java{java}
79045 hadoop        31    0  1948M   867M uwait  29   2:50  18.26% java{java}
79045 hadoop        31    0  1948M   867M uwait   6   1:57  18.16% java{java}
79045 hadoop        31    0  1948M   867M uwait  13   4:55  17.97% java{java}
79045 hadoop        31    0  1948M   867M uwait  26   3:39  17.77% java{java}
79045 hadoop        33    0  1948M   867M uwait   8   1:21  17.48% java{java}
79045 hadoop        30    0  1948M   867M uwait   1   3:32  16.70% java{java}
79045 hadoop        32    0  1948M   867M uwait  24   3:12  16.70% java{java}
79045 hadoop        26    0  1948M   867M uwait   4   1:27  10.35% java{java}
72417 root          20    0 19828K  3252K CPU21  21   0:00   0.29% top
  836 root          20    0 36104K  1952K select 14   6:51   0.00% snmpd
79045 hadoop        20    0  1948M   867M uwait  20   6:51   0.00% java{java}
79045 hadoop        20    0  1948M   867M uwait  27   3:45   0.00% java{java}
79045 hadoop        20    0  1948M   867M uwait  30   2:37   0.00% java{java}
79045 hadoop        20    0  1948M   867M uwait  15   0:54   0.00% java{java}
79045 hadoop        20    0  1948M   867M uwait   2   0:48   0.00% java{java}
79045 hadoop        20    0  1948M   867M uwait  14   0:48   0.00% java{java}
79045 hadoop        20    0  1948M   867M uwait   2   0:48   0.00% java{java}
<....>

This is on absolutely idle cluster, no single task is running.
I am attaching truss(1) output for that java process.

Thanks.


Mime
View raw message