Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CC04F18472 for ; Fri, 12 Feb 2016 19:10:46 +0000 (UTC) Received: (qmail 35888 invoked by uid 500); 12 Feb 2016 19:10:43 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 35838 invoked by uid 500); 12 Feb 2016 19:10:43 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 35824 invoked by uid 99); 12 Feb 2016 19:10:43 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 12 Feb 2016 19:10:43 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 1027FC0BE4 for ; Fri, 12 Feb 2016 19:10:43 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.498 X-Spam-Level: ** X-Spam-Status: No, score=2.498 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, KAM_LINEPADDING=1.2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=pythian-com.20150623.gappssmtp.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id i3kUgIUzMOWc for ; Fri, 12 Feb 2016 19:10:39 +0000 (UTC) Received: from mail-io0-f175.google.com (mail-io0-f175.google.com [209.85.223.175]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 0785031ADD for ; Fri, 12 Feb 2016 19:10:38 +0000 (UTC) Received: by mail-io0-f175.google.com with SMTP id z135so32062946iof.0 for ; Fri, 12 Feb 2016 11:10:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pythian-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=37+Mcbd9YoD5ASOfE8qoiwWYh3sPPbdngT3JlT7c8DY=; b=CyZPDtMlqNl8umvaWb38sLzDP04bNLR857o/hg+sS6yX1rfGs72InJJmBHzl+/T+se MsVV3B9lz95lmjlkDDp7yF9CK960H6Eo8veuguz4Ggaln2sDKRrPIDqRhzExS0KFWKVo nHe6pbraudOtO6qWoU4gkYKx/prz9wLwg0z/1hz3NOGC+fnfcHN4jp+cwFcyWa6IVfMj TNhC/r5YkfX/DRpiaSXT96uKFHpvASd5smGdymuFyb1+B1GBwR+EwcCBj7Dx5cA56isy tHV6KcNKzKgMXFZrgAOr3PNp7Um/WQD7WoQe0uc0oeZptU0AL5sdfQSdsawq3T7eIlbO QiMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:content-type; bh=37+Mcbd9YoD5ASOfE8qoiwWYh3sPPbdngT3JlT7c8DY=; b=FcfxZUfi/ThoACvEEv5mqGcH2FFQtbiDzQoEpk659ekdUQE5PsX5xpdC0uOZ0Ivkzn 6ox6k/kMi/oRr40UzY5EEqOrlSP0Y1Rt0dVcb1iRtLKYpx4FQhHOIaeqvh48RD6qH+0W CXPHexEn7CX3Mak7LSA16W5Vjgby8RjwJ6VnTv++XzOR2SmZVSZ7ZVQumn7ZLmJplvk/ 9DV1AaJ7ISMBg2s2biTUcOJJ/MFUcavEk9AhfCch4wk8U7IqgKJpEFIPTlAmFnqxXqoP myInJ1M8tPd+q+G3UVTjJV6DeKy3v5fDepdfKu+cJkCeTlExev3XHu5bKPTcqAog0+9z KY3A== X-Gm-Message-State: AG10YOSdGRCh41B/xR+kgFXTpkDJkurdRRKr/kalVSSrzt6zFv1izJmGPz/1XltNV0Cs3zNB5Hi6guCo9YLmoWf0q/KZI6CF+1ykOvaUjJ9h+aiOa0PjMhY8W0paKVNuz/fzITo6LXY6HGGuJHNEAto= X-Received: by 10.107.129.215 with SMTP id l84mr5066760ioi.47.1455304236794; Fri, 12 Feb 2016 11:10:36 -0800 (PST) MIME-Version: 1.0 Received: by 10.64.24.70 with HTTP; Fri, 12 Feb 2016 11:10:17 -0800 (PST) In-Reply-To: <233D643E-F1C7-4639-A3EE-C35C2CB98073@anguenot.org> References: <76FEA31B-37F9-4C3C-9CD0-5BE239D5C25C@skvazh.com> <3B833086-8F42-4F58-9ED6-2AD2255F1833@skvazh.com> <0ECE847E-45A0-4DDF-838F-207EF17477EF@skvazh.com> <09BBF630-71E1-4722-9A0B-571A91DEE03A@anguenot.org> <233D643E-F1C7-4639-A3EE-C35C2CB98073@anguenot.org> From: Carlos Rolo Date: Fri, 12 Feb 2016 19:10:17 +0000 Message-ID: Subject: Re: Cassandra eats all cpu cores, high load average To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a113ec6ac13162d052b976c13 --001a113ec6ac13162d052b976c13 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable if you have internode_compression: all, try disabling it. Also I would move to STCS if you have a lot of tombstones. If they get pilled in higher levels you have to wait until those higher levels get compacted before you get them out. For G1 your heap is too small. Bump that to 16GB (or at least 12GB). Or revert back to CMS with 12GB Heap / 3GB NewSize. This are a bit wild guesses since I don't have eyes on the cluster, but from what I've seen, this should bring some improvements. Regards, Carlos Juzarte Rolo Cassandra Consultant / Datastax Certified Architect / Cassandra MVP Pythian - Love your data rolo@pythian | Twitter: @cjrolo | Linkedin: *linkedin.com/in/carlosjuzarter= olo * Mobile: +351 91 891 81 00 | Tel: +1 613 565 8696 x1649 www.pythian.com On Fri, Feb 12, 2016 at 5:24 PM, Julien Anguenot wrote: > If you positive this is not compaction related I would: > > 1. check disk IOPs and latency on the EBS volume. (dstat) > 2. turn GC log on in cassandra-env.sh and use jstat to see what is > happening to your HEAP. > > I have been asking about compactions initially because if you having one > (1) big table written by all nodes and fully replicated to all nodes in t= he > cluster would definitely trigger constant compactions on every nodes > depending on write throughput. > > J. > > On Feb 12, 2016, at 11:03 AM, Skvazh Roman wrote: > > Does the load decrease and the node answers requests =E2=80=9Cnormally=E2= =80=9D when you > do disable auto-compaction? You actually see pending compactions on nodes > having high load correct? > > Nope. > > All seems legit here. Using G1 GC? > > Yes > > Problems also occurred on nodes without pending compactions. > > > > On 12 Feb 2016, at 18:44, Julien Anguenot wrote: > > > On Feb 12, 2016, at 9:24 AM, Skvazh Roman wrote: > > I have disabled autocompaction and stop it on highload node. > > > Does the load decrease and the node answers requests =E2=80=9Cnormally=E2= =80=9D when you > do disable auto-compaction? You actually see pending compactions on nodes > having high load correct? > > Heap is 8Gb. gc_grace is 86400 > > All sstables is about 200-300 Mb. > > > All seems legit here. Using G1 GC? > > $ nodetool compactionstats > pending tasks: 14 > > > Try to increase the compactors from 4 to 6-8 on a node, disable gossip an= d > let it finish compacting and put it back in the ring by enabling gossip. > See what happens. > > The tombstones count growing is because the auto-aucompactions are > disabled on these nodes. Probably not your issue. > > J. > > > > $ dstat -lvnr 10 > ---load-avg--- ---procs--- ------memory-usage----- ---paging-- -dsk/total= - > ---system-- ----total-cpu-usage---- -net/total- --io/total- > 1m 5m 15m |run blk new| used buff cach free| in out | read writ= | > int csw |usr sys idl wai hiq siq| recv send| read writ > 29.4 28.6 23.5|0.0 0 1.2|11.3G 190M 17.6G 407M| 0 0 |7507k > 7330k| 13k 40k| 11 1 88 0 0 0| 0 0 |96.5 64.6 > 29.3 28.6 23.5| 29 0 0.9|11.3G 190M 17.6G 408M| 0 0 | 0 > 189k|9822 2319 | 99 0 0 0 0 0| 138k 120k| 0 4.30 > 29.4 28.6 23.6| 30 0 2.0|11.3G 190M 17.6G 408M| 0 0 | 0 > 26k|8689 2189 |100 0 0 0 0 0| 139k 120k| 0 2.70 > 29.4 28.7 23.6| 29 0 3.0|11.3G 190M 17.6G 408M| 0 0 | 0 > 20k|8722 1846 | 99 0 0 0 0 0| 136k 120k| 0 1.50 ^C > > > JvmTop 0.8.0 alpha - 15:20:37, amd64, 16 cpus, Linux 3.14.44-3, load avg > 28.09 > http://code.google.com/p/jvmtop > > PID 32505: org.apache.cassandra.service.CassandraDaemon > ARGS: > VMARGS: -ea -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar > -XX:+CMSCl[...] > VM: Oracle Corporation Java HotSpot(TM) 64-Bit Server VM 1.8.0_65 > UP: 8:31m #THR: 334 #THRPEAK: 437 #THRCREATED: 4694 USER: cassandra > GC-Time: 0: 8m #GC-Runs: 6378 #TotalLoadedClasses: 5926 > CPU: 97.96% GC: 0.00% HEAP:6049m /7540m NONHEAP: 82m / n/a > > TID NAME STATE CPU TOTALCPU > BLOCKEDBY > 447 SharedPool-Worker-45 RUNNABLE 60.47% 1.03% > 343 SharedPool-Worker-2 RUNNABLE 56.46% 3.07% > 349 SharedPool-Worker-8 RUNNABLE 56.43% 1.61% > 456 SharedPool-Worker-25 RUNNABLE 55.25% 1.06% > 483 SharedPool-Worker-40 RUNNABLE 53.06% 1.04% > 475 SharedPool-Worker-53 RUNNABLE 52.31% 1.03% > 464 SharedPool-Worker-20 RUNNABLE 52.00% 1.11% > 577 SharedPool-Worker-71 RUNNABLE 51.73% 1.02% > 404 SharedPool-Worker-10 RUNNABLE 51.10% 1.29% > 486 SharedPool-Worker-34 RUNNABLE 51.06% 1.03% > Note: Only top 10 threads (according cpu load) are shown! > > > On 12 Feb 2016, at 18:14, Julien Anguenot wrote: > > At the time when the load is high and you have to restart, do you see any > pending compactions when using `nodetool compactionstats`? > > Possible to see a `nodetool compactionstats` taken *when* the load is too > high? Have you checked the size of your SSTables for that big table? Any > large ones in there? What about the Java HEAP configuration on these nod= es? > > If you have too many tombstones I would try to decrease gc_grace_seconds > so they get cleared out earlier during compactions. > > J. > > On Feb 12, 2016, at 8:45 AM, Skvazh Roman wrote: > > There is 1-4 compactions at that moment. > We have many tombstones, which does not removed. > DroppableTombstoneRatio is 5-6 (greater than 1) > > On 12 Feb 2016, at 15:53, Julien Anguenot wrote: > > Hey, > > What about compactions count when that is happening? > > J. > > > On Feb 12, 2016, at 3:06 AM, Skvazh Roman wrote: > > Hello! > We have a cluster of 25 c3.4xlarge nodes (16 cores, 32 GiB) with attached > 1.5 TB 4000 PIOPS EBS drive. > Sometimes one or two nodes user cpu spikes to 100%, load average to 20-30 > - read requests drops of. > Only restart of this cassandra services helps. > Please advice. > > One big table with wide rows. 600 Gb per node. > LZ4Compressor > LeveledCompaction > > concurrent compactors: 4 > compactor throughput: tried from 16 to 128 > Concurrent_readers: from 16 to 32 > Concurrent_writers: 128 > > > https://gist.github.com/rskvazh/de916327779b98a437a6 > > > JvmTop 0.8.0 alpha - 06:51:10, amd64, 16 cpus, Linux 3.14.44-3, load avg > 19.35 > http://code.google.com/p/jvmtop > > Profiling PID 9256: org.apache.cassandra.service.CassandraDa > > 95.73% ( 4.31s) > ....google.common.collect.AbstractIterator.tryToComputeN() > 1.39% ( 0.06s) com.google.common.base.Objects.hashCode() > 1.26% ( 0.06s) io.netty.channel.epoll.Native.epollWait() > 0.85% ( 0.04s) net.jpountz.lz4.LZ4JNI.LZ4_compress_limitedOutput() > 0.46% ( 0.02s) net.jpountz.lz4.LZ4JNI.LZ4_decompress_fast() > 0.26% ( 0.01s) com.google.common.collect.Iterators$7.computeNext() > 0.06% ( 0.00s) io.netty.channel.epoll.Native.eventFdWrite() > > > ttop: > > 2016-02-12T08:20:25.605+0000 Process summary > process cpu=3D1565.15% > application cpu=3D1314.48% (user=3D1354.48% sys=3D-40.00%) > other: cpu=3D250.67% > heap allocation rate 146mb/s > [000405] user=3D76.25% sys=3D-0.54% alloc=3D 0b/s - SharedPool-Worker= -9 > [000457] user=3D75.54% sys=3D-1.26% alloc=3D 0b/s - SharedPool-Worker= -14 > [000451] user=3D73.52% sys=3D 0.29% alloc=3D 0b/s - SharedPool-Worker= -16 > [000311] user=3D76.45% sys=3D-2.99% alloc=3D 0b/s - SharedPool-Worker= -4 > [000389] user=3D70.69% sys=3D 2.62% alloc=3D 0b/s - SharedPool-Worker= -6 > [000388] user=3D86.95% sys=3D-14.28% alloc=3D 0b/s - SharedPool-Worke= r-5 > [000404] user=3D70.69% sys=3D 0.10% alloc=3D 0b/s - SharedPool-Worker= -8 > [000390] user=3D72.61% sys=3D-1.82% alloc=3D 0b/s - SharedPool-Worker= -7 > [000255] user=3D87.86% sys=3D-17.87% alloc=3D 0b/s - SharedPool-Worke= r-1 > [000444] user=3D72.21% sys=3D-2.30% alloc=3D 0b/s - SharedPool-Worker= -12 > [000310] user=3D71.50% sys=3D-2.31% alloc=3D 0b/s - SharedPool-Worker= -3 > [000445] user=3D69.68% sys=3D-0.83% alloc=3D 0b/s - SharedPool-Worker= -13 > [000406] user=3D72.61% sys=3D-4.40% alloc=3D 0b/s - SharedPool-Worker= -10 > [000446] user=3D69.78% sys=3D-1.65% alloc=3D 0b/s - SharedPool-Worker= -11 > [000452] user=3D66.86% sys=3D 0.22% alloc=3D 0b/s - SharedPool-Worker= -15 > [000256] user=3D69.08% sys=3D-2.42% alloc=3D 0b/s - SharedPool-Worker= -2 > [004496] user=3D29.99% sys=3D 0.59% alloc=3D 30mb/s - CompactionExecuto= r:15 > [004906] user=3D29.49% sys=3D 0.74% alloc=3D 39mb/s - CompactionExecuto= r:16 > [010143] user=3D28.58% sys=3D 0.25% alloc=3D 26mb/s - CompactionExecuto= r:17 > [000785] user=3D27.87% sys=3D 0.70% alloc=3D 38mb/s - CompactionExecuto= r:12 > [012723] user=3D 9.09% sys=3D 2.46% alloc=3D 2977kb/s - RMI TCP > Connection(2673)-127.0.0.1 > [000555] user=3D 5.35% sys=3D-0.08% alloc=3D 474kb/s - SharedPool-Worker= -24 > [000560] user=3D 3.94% sys=3D 0.07% alloc=3D 434kb/s - SharedPool-Worker= -22 > [000557] user=3D 3.94% sys=3D-0.17% alloc=3D 339kb/s - SharedPool-Worker= -25 > [000447] user=3D 2.73% sys=3D 0.60% alloc=3D 436kb/s - SharedPool-Worker= -19 > [000563] user=3D 3.33% sys=3D-0.04% alloc=3D 460kb/s - SharedPool-Worker= -20 > [000448] user=3D 2.73% sys=3D 0.27% alloc=3D 414kb/s - SharedPool-Worker= -21 > [000554] user=3D 1.72% sys=3D 0.70% alloc=3D 232kb/s - SharedPool-Worker= -26 > [000558] user=3D 1.41% sys=3D 0.39% alloc=3D 213kb/s - SharedPool-Worker= -23 > [000450] user=3D 1.41% sys=3D-0.03% alloc=3D 158kb/s - SharedPool-Worker= -17 > > > > > > > > -- > Julien Anguenot (@anguenot) > USA +1.832.408.0344 > FR +33.7.86.85.70.44 > > > > --=20 -- --001a113ec6ac13162d052b976c13 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
if you have internode_compression: all, try disa= bling it.
Also I would move to STCS if you have a lot of tombstones.
= If they get pilled in higher levels you have to wait until those higher lev= els get compacted before you get them out.

For G1 your heap is= too small. Bump that to 16GB (or at least 12GB). Or revert back to CMS wit= h 12GB Heap / 3GB NewSize.

This are a bit wild guesses since I= don't have eyes on the cluster, but from what I've seen, this shou= ld bring some improvements.



Regards,

Carlos Juzarte Rolo
<= div>Cassandra Consultant / Datastax Certified Architect / Cassandra MVP
=
=C2=A0
Pythian - Love your data

rolo@pythian | Twitter: @cjrolo | Linkedin: <= a href=3D"http://linkedin.com/in/carlosjuzarterolo" target=3D"_blank">linke= din.com/in/carlosjuzarterolo
Mobile: +351 91 891 8= 1 00 | Tel: +1 613 565 8696 x1649

On Fri, Feb 12, 2016 at 5:24 PM, Julien Angu= enot <julien@anguenot.org> wrote:
If you positive this is not c= ompaction related I would:

=C2=A0 =C2=A01. check disk IO= Ps and latency on the EBS volume. (dstat)
=C2=A0 =C2=A02. turn GC= log on in cassandra-env.sh =C2=A0and use jstat to see what is happening to= your HEAP.

I have been asking about compactions i= nitially because if you having one (1) big table written by all nodes and f= ully replicated to all nodes in the cluster would definitely trigger consta= nt compactions on every nodes depending on write throughput.

=C2=A0 =C2= =A0J.=C2=A0

On Feb 12, 2016, at 11:03 AM, Skvazh Roman <r@skvazh.com> wrote:
Does the load decrease and the node answers requests =E2=80= =9Cnormally=E2=80=9D when you do disable auto-compaction? You actually see = pending compactions on nodes having high load correct?
Nope.

All seems legit here. Using G1 GC?
Yes
<= div>
Problems also occurred on nodes without pending compacti= ons.



On 12 Feb 2016, at 18:44, Julien Anguenot <julien@anguenot.org> wr= ote:


On Feb 12, 2016, at 9:24 A= M, Skvazh Roman <r@skv= azh.com> wrote:

I have disabled autocompaction an= d stop it on highload node.

Does the load decrease and the node answers requests =E2=80=9Cnormally=E2= =80=9D when you do disable auto-compaction? You actually see pending compac= tions on nodes having high load correct?

Heap is 8Gb. gc_grace is 86400
All sstables is about 200-300 Mb.

All seems legit here. Using G1 GC?

<= div style=3D"font-family:Helvetica;font-size:12px;font-style:normal;font-va= riant:normal;font-weight:normal;letter-spacing:normal;text-align:start;text= -indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
$ nodetool compactionstats
pending tasks: = 14

Try to increase the comp= actors from 4 to 6-8 on a node, disable gossip and let it finish compacting= and put it back in the ring by enabling gossip. See what happens.

The tombstones count growing is because the auto-aucompact= ions are disabled on these nodes. Probably not your issue.

=C2=A0 =C2=A0J.



$ dstat -lvnr 10
---load-avg--- ---procs--- ---= ---memory-usage----- ---paging-- -dsk/total- ---system-- ----total-cpu-usag= e---- -net/total- --io/total-
1m =C2=A0=C2=A05m =C2=A015m |run blk new| = used =C2=A0buff =C2=A0cach =C2=A0free| =C2=A0in =C2=A0=C2=A0out | read =C2= =A0writ| int =C2=A0=C2=A0csw |usr sys idl wai hiq siq| recv =C2=A0send| rea= d =C2=A0writ
29.4 28.6 23.5|0.0 =C2=A0=C2=A00 1.2|11.3G =C2=A0190M 17.6G= =C2=A0407M| =C2=A0=C2=A00 =C2=A0=C2=A0=C2=A0=C2=A00 |7507k 7330k| =C2=A013= k =C2=A0=C2=A040k| 11 =C2=A0=C2=A01 =C2=A088 =C2=A0=C2=A00 =C2=A0=C2=A00 = =C2=A0=C2=A00| =C2=A0=C2=A00 =C2=A0=C2=A0=C2=A0=C2=A00 |96.5 =C2=A064.6
= 29.3 28.6 23.5| 29 =C2=A0=C2=A00 0.9|11.3G =C2=A0190M 17.6G =C2=A0408M| =C2= =A0=C2=A00 =C2=A0=C2=A0=C2=A0=C2=A00 | =C2=A0=C2=A00 =C2=A0=C2=A0189k|9822 = =C2=A02319 | 99 =C2=A0=C2=A00 =C2=A0=C2=A00 =C2=A0=C2=A00 =C2=A0=C2=A00 =C2= =A0=C2=A00| 138k =C2=A0120k| =C2=A0=C2=A00 =C2=A04.30
29.4 28.6 23.6| 30= =C2=A0=C2=A00 2.0|11.3G =C2=A0190M 17.6G =C2=A0408M| =C2=A0=C2=A00 =C2=A0= =C2=A0=C2=A0=C2=A00 | =C2=A0=C2=A00 =C2=A0=C2=A0=C2=A026k|8689 =C2=A02189 |= 100 =C2=A0=C2=A00 =C2=A0=C2=A00 =C2=A0=C2=A00 =C2=A0=C2=A00 =C2=A0=C2=A00| = 139k =C2=A0120k| =C2=A0=C2=A00 =C2=A02.70
29.4 28.7 23.6| 29 =C2=A0=C2= =A00 3.0|11.3G =C2=A0190M 17.6G =C2=A0408M| =C2=A0=C2=A00 =C2=A0=C2=A0=C2= =A0=C2=A00 | =C2=A0=C2=A00 =C2=A0=C2=A0=C2=A020k|8722 =C2=A01846 | 99 =C2= =A0=C2=A00 =C2=A0=C2=A00 =C2=A0=C2=A00 =C2=A0=C2=A00 =C2=A0=C2=A00| 136k = =C2=A0120k| =C2=A0=C2=A00 =C2=A01.50 ^C


JvmTop 0.8.0 alpha - 15:= 20:37, =C2=A0amd64, 16 cpus, Linux 3.14.44-3, load avg 28.09
http://code.google.com/p= /jvmtop

PID 32505: org.apache.cassandra.service.CassandraDaemon<= br>ARGS:
VMARGS: -ea -javaagent:/usr/share/cassandra/lib/jamm-0.3.0.jar = -XX:+CMSCl[...]
VM: Oracle Corporation Java HotSpot(TM) 64-Bit Server VM= 1.8.0_65
UP: =C2=A08:31m =C2=A0#THR: 334 =C2=A0#THRPEAK: 437 =C2=A0#THR= CREATED: 4694 USER: cassandra
GC-Time: =C2=A00: 8m =C2=A0=C2=A0#GC-Runs:= 6378 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0#TotalLoadedClasses: 5926
CPU: 97.96= % GC: =C2=A00.00% HEAP:6049m /7540m NONHEAP: =C2=A082m / =C2=A0n/a

= =C2=A0TID =C2=A0=C2=A0NAME =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0STATE =C2=A0=C2=A0=C2=A0CPU =C2=A0TOTALCPU BLOCKEDBY
=C2=A0= =C2=A0=C2=A0447 SharedPool-Worker-45 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0RUNNABLE 60.47%= =C2=A0=C2=A0=C2=A0=C2=A01.03%
=C2=A0=C2=A0=C2=A0343 SharedPool-Worker-2= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0RUNNABLE 56.46% =C2=A0=C2=A0=C2=A0=C2=A03.07%=
=C2=A0=C2=A0=C2=A0349 SharedPool-Worker-8 =C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= RUNNABLE 56.43% =C2=A0=C2=A0=C2=A0=C2=A01.61%
=C2=A0=C2=A0=C2=A0456 Shar= edPool-Worker-25 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0RUNNABLE 55.25% =C2=A0=C2=A0=C2=A0= =C2=A01.06%
=C2=A0=C2=A0=C2=A0483 SharedPool-Worker-40 =C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0RUNNABLE 53.06% =C2=A0=C2=A0=C2=A0=C2=A01.04%
=C2=A0=C2=A0=C2=A047= 5 SharedPool-Worker-53 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0RUNNABLE 52.31% =C2=A0=C2=A0= =C2=A0=C2=A01.03%
=C2=A0=C2=A0=C2=A0464 SharedPool-Worker-20 =C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0RUNNABLE 52.00% =C2=A0=C2=A0=C2=A0=C2=A01.11%
=C2=A0=C2=A0= =C2=A0577 SharedPool-Worker-71 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0RUNNABLE 51.73% =C2= =A0=C2=A0=C2=A0=C2=A01.02%
=C2=A0=C2=A0=C2=A0404 SharedPool-Worker-10 = =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0RUNNABLE 51.10% =C2=A0=C2=A0=C2=A0=C2=A01.29%
=C2= =A0=C2=A0=C2=A0486 SharedPool-Worker-34 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0RUNNABLE 51.= 06% =C2=A0=C2=A0=C2=A0=C2=A01.03%
Note: Only top 10 threads (according c= pu load) are shown!


On 12 Feb 2016, at= 18:14, Julien Anguenot <julien@anguenot.org> wrote:

At the time when the = load is high and you have to restart, do you see any pending compactions wh= en using `nodetool compactionstats`?

Possible to see a `nodetool com= pactionstats` taken *when* the load is too high?=C2=A0 Have you checked the= size of your SSTables for that big table? Any large ones in there?=C2=A0 W= hat about the Java HEAP configuration on these nodes?

If you have to= o many tombstones I would try to decrease gc_grace_seconds so they get clea= red out earlier during compactions.

=C2=A0J.

On Feb 12, 2016, at 8:45 AM, Skvazh Roman <r@skvazh.com> wrote:

There is = 1-4 compactions at that moment.
We have many tombstones, which does not = removed.
DroppableTombstoneRatio is 5-6 (greater than 1)

On 12 Feb 2016, at 15:53, Julien Anguenot <julien@anguenot.org> w= rote:

Hey,=C2=A0

What about compactions count wh= en that is happening?

J.


On Feb= 12, 2016, at 3:06 AM, Skvazh Roman <r@skvazh.com> wrote:

Hello!
We have a clust= er of 25 c3.4xlarge nodes (16 cores, 32 GiB) with attached 1.5 TB 4000 PIOP= S EBS drive.
Sometimes one or two nodes user cpu spikes to 100%, load av= erage to 20-30 - read requests drops of.
Only restart of this cassandra = services helps.
Please advice.

One big table with wide rows. 600 = Gb per node.
LZ4Compressor
LeveledCompaction

concurrent compac= tors: 4
compactor throughput: tried from 16 to 128
Concurrent_readers= : from 16 to 32
Concurrent_writers: 128


https://gist.= github.com/rskvazh/de916327779b98a437a6


JvmTop 0.8.0 alpha -= 06:51:10, =C2=A0amd64, 16 cpus, Linux 3.14.44-3, load avg 19.35
http://code.google.c= om/p/jvmtop

Profiling PID 9256: org.apache.cassandra.service.Cas= sandraDa

95.73% ( =C2=A0=C2=A0=C2=A0=C2=A04.31s) ....google.common.c= ollect.AbstractIterator.tryToComputeN()
1.39% ( =C2=A0=C2=A0=C2=A0=C2=A0= 0.06s) com.google.common.base.Objects.hashCode()
1.26% ( =C2=A0=C2=A0=C2= =A0=C2=A00.06s) io.netty.channel.epoll.Native.epollWait()
0.85% ( =C2=A0= =C2=A0=C2=A0=C2=A00.04s) net.jpountz.lz4.LZ4JNI.LZ4_compress_limitedOutput(= )
0.46% ( =C2=A0=C2=A0=C2=A0=C2=A00.02s) net.jpountz.lz4.LZ4JNI.LZ4_deco= mpress_fast()
0.26% ( =C2=A0=C2=A0=C2=A0=C2=A00.01s) com.google.common.c= ollect.Iterators$7.computeNext()
0.06% ( =C2=A0=C2=A0=C2=A0=C2=A00.00s) = io.netty.channel.epoll.Native.eventFdWrite()


ttop:

2016-0= 2-12T08:20:25.605+0000 Process summary
process cpu=3D1565.15%
applica= tion cpu=3D1314.48% (user=3D1354.48% sys=3D-40.00%)
other: cpu=3D250.67%=
heap allocation rate 146mb/s
[000405] user=3D76.25% sys=3D-0.54% all= oc=3D =C2=A0=C2=A0=C2=A0=C2=A00b/s - SharedPool-Worker-9
[000457] user= =3D75.54% sys=3D-1.26% alloc=3D =C2=A0=C2=A0=C2=A0=C2=A00b/s - SharedPool-W= orker-14
[000451] user=3D73.52% sys=3D 0.29% alloc=3D =C2=A0=C2=A0=C2=A0= =C2=A00b/s - SharedPool-Worker-16
[000311] user=3D76.45% sys=3D-2.99% al= loc=3D =C2=A0=C2=A0=C2=A0=C2=A00b/s - SharedPool-Worker-4
[000389] user= =3D70.69% sys=3D 2.62% alloc=3D =C2=A0=C2=A0=C2=A0=C2=A00b/s - SharedPool-W= orker-6
[000388] user=3D86.95% sys=3D-14.28% alloc=3D =C2=A0=C2=A0=C2=A0= =C2=A00b/s - SharedPool-Worker-5
[000404] user=3D70.69% sys=3D 0.10% all= oc=3D =C2=A0=C2=A0=C2=A0=C2=A00b/s - SharedPool-Worker-8
[000390] user= =3D72.61% sys=3D-1.82% alloc=3D =C2=A0=C2=A0=C2=A0=C2=A00b/s - SharedPool-W= orker-7
[000255] user=3D87.86% sys=3D-17.87% alloc=3D =C2=A0=C2=A0=C2=A0= =C2=A00b/s - SharedPool-Worker-1
[000444] user=3D72.21% sys=3D-2.30% all= oc=3D =C2=A0=C2=A0=C2=A0=C2=A00b/s - SharedPool-Worker-12
[000310] user= =3D71.50% sys=3D-2.31% alloc=3D =C2=A0=C2=A0=C2=A0=C2=A00b/s - SharedPool-W= orker-3
[000445] user=3D69.68% sys=3D-0.83% alloc=3D =C2=A0=C2=A0=C2=A0= =C2=A00b/s - SharedPool-Worker-13
[000406] user=3D72.61% sys=3D-4.40% al= loc=3D =C2=A0=C2=A0=C2=A0=C2=A00b/s - SharedPool-Worker-10
[000446] user= =3D69.78% sys=3D-1.65% alloc=3D =C2=A0=C2=A0=C2=A0=C2=A00b/s - SharedPool-W= orker-11
[000452] user=3D66.86% sys=3D 0.22% alloc=3D =C2=A0=C2=A0=C2=A0= =C2=A00b/s - SharedPool-Worker-15
[000256] user=3D69.08% sys=3D-2.42% al= loc=3D =C2=A0=C2=A0=C2=A0=C2=A00b/s - SharedPool-Worker-2
[004496] user= =3D29.99% sys=3D 0.59% alloc=3D =C2=A0=C2=A030mb/s - CompactionExecutor:15<= br>[004906] user=3D29.49% sys=3D 0.74% alloc=3D =C2=A0=C2=A039mb/s - Compac= tionExecutor:16
[010143] user=3D28.58% sys=3D 0.25% alloc=3D =C2=A0=C2= =A026mb/s - CompactionExecutor:17
[000785] user=3D27.87% sys=3D 0.70% al= loc=3D =C2=A0=C2=A038mb/s - CompactionExecutor:12
[012723] user=3D 9.09%= sys=3D 2.46% alloc=3D 2977kb/s - RMI TCP Connection(2673)-127.0.0.1
[00= 0555] user=3D 5.35% sys=3D-0.08% alloc=3D =C2=A0474kb/s - SharedPool-Worker= -24
[000560] user=3D 3.94% sys=3D 0.07% alloc=3D =C2=A0434kb/s - SharedP= ool-Worker-22
[000557] user=3D 3.94% sys=3D-0.17% alloc=3D =C2=A0339kb/s= - SharedPool-Worker-25
[000447] user=3D 2.73% sys=3D 0.60% alloc=3D =C2= =A0436kb/s - SharedPool-Worker-19
[000563] user=3D 3.33% sys=3D-0.04% al= loc=3D =C2=A0460kb/s - SharedPool-Worker-20
[000448] user=3D 2.73% sys= =3D 0.27% alloc=3D =C2=A0414kb/s - SharedPool-Worker-21
[000554] user=3D= 1.72% sys=3D 0.70% alloc=3D =C2=A0232kb/s - SharedPool-Worker-26
[00055= 8] user=3D 1.41% sys=3D 0.39% alloc=3D =C2=A0213kb/s - SharedPool-Worker-23=
[000450] user=3D 1.41% sys=3D-0.03% alloc=3D =C2=A0158kb/s - SharedPool= -Worker-17






--
Julien Anguenot (@anguenot)




--



--001a113ec6ac13162d052b976c13--