cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shuo Chen <chenatu2...@gmail.com>
Subject Re: Too many compactions, maybe keyspace system?
Date Tue, 12 Jan 2016 02:21:23 GMT
Thanks Rob!

I restarted the cluster at Jan 11 15:00 and I checked the status at Jan 12
9:30. Only one of the node have large mount pending tasks issue. I
increased the heap size to 12G before restarting and run nodetool stop
COMPACTION,
VALIDATION, CLEANUP, SCRUB, INDEX_BUILD.

Here is the compactionstats:
pending tasks: 112583326
Active compaction remaining time :        n/a

I run grep -i compact /path/to/system.log but the result is tool long. The
result is in the attachment.

Besides I checked iotop of cassandra, It does not consuming too much io.

top result:

   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

129276 cassandr  20   0 16.9g  14g  12m S 100.2 23.7   1743:43 java
 <----    cassandra process
216014 root      20   0  181m  14m 3500 S  7.3  0.0   0:05.00 iotop

185693 mysql     20   0 56.5g  44g 6952 S  5.3 70.1   4372:08 mysqld

 29968 mysql     20   0  103m 1504 1024 S  1.3  0.0   9:02.57 awk

   348 root      39  19     0    0    0 S  1.0  0.0 833:34.10 kipmi0

GCutil result of cassandra for 18 hours:
  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT
100.00   0.00 100.00 100.00  59.57    256  370.527  1574 52600.562 52971.088

nodetool info at glance
[cassandra@whaty181 apache-cassandra-2.0.16]$ bin/nodetool info
Token                  : 0
ID                     : 5d90300f-2fb4-4065-9819-10ece285223d
Gossip active          : true
Thrift active          : true
Native Transport active: true
Load                   : 10.39 GB
Generation No          : 1452506508
Uptime (seconds)       : 58684
Heap Memory (MB)       : 11499.87 / 12083.25
Off Heap Memory (MB)   : 10.98
Data Center            : DC1
Rack                   : RAC1
Exceptions             : 0
Key Cache              : size 33264 (bytes), capacity 104857600 (bytes),
859 hits, 975 requests, 0.881 recent hit rate, 14400 save period in seconds
Row Cache              : size 0 (bytes), capacity 0 (bytes), 0 hits, 0
requests, NaN recent hit rate, 0 save period in seconds

Thanks again!

On Tue, Jan 12, 2016 at 3:54 AM, Robert Coli <rcoli@eventbrite.com> wrote:

> On Sat, Jan 9, 2016 at 8:23 AM, Shuo Chen <chenatu2006@gmail.com> wrote:
>
>> I don't know what exactly compaction logs is like in system.log. But I
>> see logs like this in system.log, I think maybe this is the compaction log
>>
>
> grep -i compact /path/to/system.log
>
> =Rob
>
>



-- 
*陈硕* *Shuo Chen*
chenatu2006@gmail.com
chenshuo@whaty.com

Mime
View raw message