Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 924AD200C63 for ; Thu, 11 May 2017 23:44:32 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 90E01160BC7; Thu, 11 May 2017 21:44:32 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id B94EA160BB3 for ; Thu, 11 May 2017 23:44:30 +0200 (CEST) Received: (qmail 17897 invoked by uid 500); 11 May 2017 21:44:29 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 17882 invoked by uid 99); 11 May 2017 21:44:27 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 May 2017 21:44:27 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 678DD1A7B8A for ; Thu, 11 May 2017 21:44:27 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.397 X-Spam-Level: X-Spam-Status: No, score=-0.397 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-2.796, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id Ic7uEm7TSvzn for ; Thu, 11 May 2017 21:44:21 +0000 (UTC) Received: from mail-ua0-f175.google.com (mail-ua0-f175.google.com [209.85.217.175]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 0E27B60DF0 for ; Thu, 11 May 2017 21:44:20 +0000 (UTC) Received: by mail-ua0-f175.google.com with SMTP id j17so36924959uag.3 for ; Thu, 11 May 2017 14:44:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=ZGAKnGTqeeKFhH5OjTiVmCsRNHxaQ0kToN57X+YoQFQ=; b=Zsng9h9iQeNF+ROHPlayiPkFsP9tJZWy+wDcNR4q/Mo75dWVLVKPOYADCaKUUEIXM2 QvIRhTEXeRT97dD+dY/5dyrVJ7wUgSmKcOvUwCI7prWCAa0Bux1J3LvFOdHYOjK0GD2X 5FwlTNBL8goh70K7z72nP+IKQ/L5+sTxbQ7bIU79uETg/nxkUmpTa/C1WqJPD2mSpmbq cwsZzkNCHaoMpuMGxuAg3bpPjoDuFSNSMP0mCvCHmHG+M3i2cMvuHOxQxQ5uHftaLbN9 bXffuiWLlvBfg5bbnL/pL5a+B0dHEPSlOrpWriHrT9NK5UkRI4sbyhoTbAhGiPS2gnfY HpOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=ZGAKnGTqeeKFhH5OjTiVmCsRNHxaQ0kToN57X+YoQFQ=; b=HLvN9TMbKW3slLGSe8l3/74Aox2Oh5Y02Uswr23sL41HyMpkXz6LNtbyuTWpKDkvlk VuitkDNxx/dt1JY9SHHHF5oU+DGw7/AaWjfWjHCtvT9CeJcDjzyWww5PIOFoO/sZDeAI xPYGmrjF6ip2JziR1c5pfMhBqP8vpvdBMtmbpsm9Dc/OIBhupocPdLojQSwxFqM0UcmQ GBcqJf6KiH10gVwelMK22oW0kR3CbdqcWBNE06yG+04ZZxJ1rmBtayFzXXqaTKUqdpYF K2VMqjFwWparpGjIC2lvhckIEnakPSFlCxSaApkMVMWISoWM3/fJbKP0ahf2VeBXMQhu HeNQ== X-Gm-Message-State: AODbwcDBitQf0Lm+MOm77VBO2ysTjCbCYkpO+v7nkYG+qQg1VqXxvGbh UI9ff+nKi0U6CJpO8eT8bHshGEiCNw== X-Received: by 10.176.76.206 with SMTP id e14mr300384uag.135.1494539058577; Thu, 11 May 2017 14:44:18 -0700 (PDT) MIME-Version: 1.0 Received: by 10.103.83.151 with HTTP; Thu, 11 May 2017 14:43:58 -0700 (PDT) In-Reply-To: References: From: Alain RODRIGUEZ Date: Thu, 11 May 2017 22:43:58 +0100 Message-ID: Subject: Re: Drop tables takes too long To: Bohdan Tantsiura Cc: user@cassandra.apache.org Content-Type: multipart/alternative; boundary="f4030435bb00b08852054f467db6" archived-at: Thu, 11 May 2017 21:44:32 -0000 --f4030435bb00b08852054f467db6 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi We were trying to overcome OOM crashes. Fair enough :-). We changed settings to default on one node. GC times became about two times > smaller on that node. > That's encouraging! Looks like even if the number of tables is really high, there is still space for optimization. Have you made the change on the entire cluster by now? How are things going? Also you can continue to play around with a canary node to find the sweet spot, the right tuning for your use case and hardware. Taking a day to do this is sometimes very worth it ;-). Number of sstables is not constant. During about 2.5 hours number of tables > was changed for 26 tables, e.g. [4, 4, 4, 4, 4, 4, 4, 4, 4, 4] =3D> [6, 6= , > 6, 6, 6, 6, 6, 6, 6, 6] or [4, 7, 7, 7, 5, 5, 4, 4, 4, 4] =3D> [4, 4, 4, = 4, > 4, 4, 4, 4, 4, 4] (each list is number of sstables on each of 10 nodes fo= r > one table). > Number of sstables is balanced for almost all tables. But for some tables > number of sstables is not really balanced, like [11, 2, 4, 4, 4, 2, 2, 2, > 2, 5] or [439, 558, 346, 521, 490, 553, 500, 515, 522, 495] > Sounds like reasonable numbers of SSTables. Imbalances are not that big. I would say compaction is running normally We run incremental repairs We use LCS for all tables and MVs. We don't do manual compactions (or > trigger any anti-compactions) If you run incremental repairs, it means you trigger anti-compactions. Actually the first repair might completely have produce the increase in the number of SSTables. If it was not the first time, you might have run into one of the corner cases still not fixed on incremental repairs. Also, using LCS for such a high number of tables is probably putting some pressure on disks. Is it a real need? Would STCS or TWCS not be a better fit on some of the table? That being said, compactions look good. Are you seeing pending compactions under standard conditions or on peak hours? What number of MemtableFlushWriter are you using? > > We do not specify if, so default is use > https://github.com/apache/cassandra/blob/cassandra-3.10/conf/cassandra.yaml= #L538 So it is 2. If disks IO is not a bottleneck you might want to consider increasing this number to 4, it should be a safe value. To know if Flush Writers is an issue, use 'watch -d nodetool tpstats' and see if there are any 'FlushWriter' pending threads. One column for each node > READ: 2609 7 0 0 2 1 2 0 1 1 > The only non negligible value is about read being dropped and on only one node. If this value is not growing anymore, you might have faced a punctual issue. This cluster looks relatively healthy excepted for GC activity (that can explain read drops). I would persevere on GC tuning, continuing to monitor things you have been sharing with us so far, to see how it evolves. Having a closer look at repairs impact might be worth it as well. Good luck! C*heers, ----------------------- Alain Rodriguez - @arodream - alain@thelastpickle.com France The Last Pickle - Apache Cassandra Consulting http://www.thelastpickle.com 2017-05-08 14:21 GMT+01:00 Bohdan Tantsiura : > Hi, > > > Why did you move from defaults that much? > We were trying to overcome OOM crashes. > > > Would you consider giving defaults a try on a canary node and monitor / > compare GC times to other nodes? > We changed settings to default on one node. GC times became about two > times smaller on that node. > > > What do you mean from time to time? For how long are this task pending, > what frequency is this happening? > CompactionExecutor pending tasks appeared on nodes once during 3 or more > hours. Tasks were pending during about 5-15 minutes. > > > Is the number of sstables constant and balanced between nodes? > Number of sstables is not constant. During about 2.5 hours number of > tables was changed for 26 tables, e.g. [4, 4, 4, 4, 4, 4, 4, 4, 4, 4] =3D= > > [6, 6, 6, 6, 6, 6, 6, 6, 6, 6] or [4, 7, 7, 7, 5, 5, 4, 4, 4, 4] =3D> [4, > 4, 4, 4, 4, 4, 4, 4, 4, 4] (each list is number of sstables on each of 10 > nodes for one table). > Number of sstables is balanced for almost all tables. But for some tables > number of sstables is not really balanced, like [11, 2, 4, 4, 4, 2, 2, 2, > 2, 5] or [439, 558, 346, 521, 490, 553, 500, 515, 522, 495] > > > Also do you run full or incremental repairs? > We run incremental repairs > > > Do you use LCS or do some manual compactions (or trigger any > anti-compactions)? > We use LCS for all tables and MVs. We don't do manual compactions (or > trigger any anti-compactions) > > > How is CPU doing, is there any burst in CPU that could be related to > these errors? > Unfortunately, stats for period when there were InternalResponseState > pending tasks is lost. > > > What number of MemtableFlushWriter are you using > We do not specify if, so default is used > > > What tasks were dropped > One column for each node > READ: 2609 7 0 0 2 1 2 0 1 1 > HINT: 0 0 0 0 1 0 1 0 0 0 > MUTATION: 0 0 1 0 0 0 0 0 0 0 > REQUEST_RESPONSE: 0 0 0 0 2 0 0 0 0 1 > > Thanks > > 2017-05-03 17:00 GMT+03:00 Alain RODRIGUEZ : > >> Hi, >> >> A few comments: >> >> Long GC Pauses take about one minute >>> >> >> This is huge. About JVM config, I haven't played much with G1GC, but the >> following seems to be a bad idea according to comments: >> >> ## Main G1GC tunable: lowering the pause target will lower throughput an= d >> vise versa. >> ## 200ms is the JVM default and lowest viable setting >> ## 1000ms increases throughput. Keep it smaller than the timeouts in >> cassandra.yaml. >> -XX:MaxGCPauseMillis=3D15000 >> >> # Save CPU time on large (>=3D 16GB) heaps by delaying region scanning >> # until the heap is 70% full. The default in Hotspot 8u40 is 40%. >> -XX:InitiatingHeapOccupancyPercent=3D30 >> # For systems with > 8 cores, the default ParallelGCThreads is 5/8 the >> number of logical cores. >> # Otherwise equal to the number of cores when 8 or less. >> # Machines with > 10 cores should try setting these to <=3D full cores. >> -XX:ParallelGCThreads=3D8 >> # By default, ConcGCThreads is 1/4 of ParallelGCThreads. >> # Setting both to the same value can reduce STW durations. >> -XX:ConcGCThreads=3D8 >> >> >> Why did you move from defaults that much? Would you consider >> giving defaults a try on a canary node and monitor / compare GC times to >> other nodes? >> >> 1) ColumnFamilyStore.java:542 - Failed unregistering mbean: >>> org.apache.cassandra.db:type=3DTables,keyspace=3D...,table=3D... >>> from MigrationStage thread >>> >> >> I am not sure about this one... :p >> >> 2) Read 1715 live rows and 1505 tombstone cells for query ... >>> from ReadStage thread >> >> >> Half of what was read for this query was deleted data. With obvious disk >> space, disk throughput and latency consequences. This is an entire topic= ... >> Here is what I know about it: thelastpickle.com/blog/2016/07 >> /27/about-deletes-and-tombstones.html, I hope it will help you solving >> your issue. >> >> 3) GCInspector.java:282 - G1 Young Generation GC in 1725ms >> >> >> This might be related to your GC configuration or some other issues >> mentioned in your last mail. >> >> About 3000-6000 CompactionExecutor pending tasks appeared on all nodes >>> from time to time. >> >> >> Hum that's weird. It's huge, but as you have so many tables I am not >> sure, it might be a 'normal' issue when running with so many tables and = MVs. >> >> What do you mean from time to time? For how long are this task pending, >> what frequency is this happening? >> >> Is the number of sstables constant and balanced between nodes? >> Also do you run full or incremental repairs? >> Do you use LCS or do some manual compactions (or trigger any >> anti-compactions)? >> >> >>> About 1000 MigrationStage pending tasks appeared on 2 nodes. >> >> >> That's pending writes. Meaning this Cassandra node can't cope with what >> what is thrown at it. It can be related to pending flushes (blocking >> writes), huge Garbage Collection (Stop The World, including writes), due= to >> hardware limits (CPU busy with compactions?) or even to a too conservati= ve >> configuration of the concurrent_write. >> >> >>> About 700 InternalResponseState pending tasks appeared on 2 nodes. >> >> >> I never had issues with this one and so didn't knew much about it. But >> according to Chris Lohfink in this post https://www.pythian.com/blog/g >> uide-to-cassandra-thread-pools/#InternalResponseStage, this thread pool >> is responsible for "Responding to non-client initiated messages, includi= ng >> bootstrapping and schema checking". Which again might be related with th= e >> huge number of tables in the cluster. How is CPU doing, is there any bur= st >> in CPU that could be related to these errors? >> >> About 60 MemtableFlushWriter appeared on 3 nodes. >> >> >> What number of MemtableFlushWriter are you using. Consider increasing it >> (or maybe the memtable size). >> >> >>> There were no blocked tasks, but there were "All time blocked" tasks >>> (they were before starting dropping tables) from 3 millions to 20 milli= ons >>> on different nodes. >> >> >> What tasks were dropped. >> >> The cluster doesn't look completely healthy, but I believe it is possibl= e >> to improve things, before thinking about splitting tables in multiples >> cluster. I would definitely not add more tables though... >> >> C*heers, >> ----------------------- >> Alain Rodriguez - @arodream - alain@thelastpickle.com >> France >> >> The Last Pickle - Apache Cassandra Consulting >> http://www.thelastpickle.com >> >> 2017-04-28 14:35 GMT+01:00 Bohdan Tantsiura : >> >>> Thanks Alain, >>> >>> > Or is it on happening during drop table actions? >>> Some other schema changes (e.g. adding columns to tables) also takes to= o >>> much time. >>> >>> Link to complete set of GC options: https://pastebin.com/4qyENeyu >>> >>> > Have you had a look at logs, mainly errors and warnings? >>> In logs I found warnings of 3 types: >>> 1) ColumnFamilyStore.java:542 - Failed unregistering mbean: >>> org.apache.cassandra.db:type=3DTables,keyspace=3D...,table=3D... >>> from MigrationStage thread >>> 2) Read 1715 live rows and 1505 tombstone cells for query ... >>> from ReadStage thread >>> 3) GCInspector.java:282 - G1 Young Generation GC in 1725ms. G1 Eden >>> Space: 38017171456 -> 0; G1 Survivor Space: 2516582400 >>> <(251)%20658-2400> -> 2650800128; from Service Thread >>> >>> > Are they some pending, blocked or dropped tasks in thread pool stats? >>> About 3000-6000 CompactionExecutor pending tasks appeared on all nodes >>> from time to time. About 1000 MigrationStage pending tasks appeared on = 2 >>> nodes. About 700 InternalResponseState pending tasks appeared on 2 node= s. >>> About 60 MemtableFlushWriter appeared on 3 nodes. >>> There were no blocked tasks, but there were "All time blocked" tasks >>> (they were before starting dropping tables) from 3 millions to 20 milli= ons >>> on different nodes. >>> >>> > Are some resources constraint (CPU / disk IO,...)? >>> CPU and disk IO are not constraint >>> >>> Thanks >>> >>> 2017-04-27 11:10 GMT+03:00 Alain RODRIGUEZ : >>> >>>> Hi >>>> >>>> >>>>> Long GC Pauses take about one minute. But why it takes so much time >>>>> and how that can be fixed? >>>> >>>> >>>> This is very long. Looks like you are having a major issue, and it is >>>> not just about dropping tables... Or is it on happening during drop ta= ble >>>> actions? Knowing the complete set of GC options in use could help here= , >>>> could you paste it here (or link to it)? >>>> >>>> Also, GC is often high as a consequence of other issues and not only >>>> when 'badly=E2=80=98 tuned >>>> >>>> >>>> - Have you had a look at logs, mainly errors and warnings? >>>> >>>> $ grep -e "ERROR" -e "WARN" /var/log/cassandra/system.log >>>> >>>> - Are they some pending, blocked or dropped tasks in thread pool >>>> stats? >>>> >>>> $ watch -d nodetool tpstats >>>> >>>> - Are some resources constraint (CPU / disk IO,...)? >>>> >>>> >>>> We have about 60 keyspaces with about 80 tables in each keyspace >>>> >>>> In each keyspace we also have 11 MVs >>>> >>>> >>>> Even if I believe we can dig it and maybe improve things, I agree with >>>> Carlos, this is a lot of Tables (4880) and even more a high number of = MV >>>> (660). It might be interesting splitting it somehow if possible. >>>> >>>> Cannot achieve consistency level ALL >>>> >>>> >>>> Finally you could try to adjust the corresponding request timeout (not >>>> sure if it is the global one or the truncate timeout), so it may succe= ed >>>> even when nodes are having minutes GC, but it is a workaround as this >>>> minute GC will most definitely be an issue for the client queries runn= ing >>>> (default is 10 sec timeout, so many query are probably failing). >>>> >>>> C*heers, >>>> ----------------------- >>>> Alain Rodriguez - @arodream - alain@thelastpickle.com >>>> France >>>> >>>> The Last Pickle - Apache Cassandra Consulting >>>> http://www.thelastpickle.com >>>> >>>> 2017-04-25 13:58 GMT+02:00 Bohdan Tantsiura : >>>> >>>>> Thanks Zhao Yang, >>>>> >>>>> > Could you try some jvm tool to find out which thread are allocating >>>>> memory or gc? maybe the migration stage thread.. >>>>> >>>>> I use Cassandra Cluster Manager to locally reproduce the issue. I >>>>> tried to use VisualVM to find out which threads are allocating >>>>> memory, but VisualVM does not see cassandra processes and says >>>>> "Cannot open application with pid". Then I tried to use YourKit Java >>>>> Profiler. It created snapshot when process of one cassandra node fail= ed. >>>>> http://i.imgur.com/9jBcjcl.png - how CPU is used by threads. >>>>> http://i.imgur.com/ox5Sozy.png - how memory is used by threads, but >>>>> biggest part of memory is used by objects without allocation informat= ion. >>>>> http://i.imgur.com/oqx9crX.png - which objects use biggest part of >>>>> memory. Maybe you know some other good jvm tool that can show by whic= h >>>>> threads biggest part of memory is used? >>>>> >>>>> > BTW, is your cluster under high load while dropping table? >>>>> >>>>> LA5 was <=3D 5 on all nodes almost all time while dropping tables >>>>> >>>>> Thanks >>>>> >>>>> 2017-04-21 19:49 GMT+03:00 Jasonstack Zhao Yang < >>>>> zhaoyangsingapore@gmail.com>: >>>>> >>>>>> Hi Bohdan, Carlos, >>>>>> >>>>>> Could you try some jvm tool to find out which thread are allocating >>>>>> memory or gc? maybe the migration stage thread.. >>>>>> >>>>>> BTW, is your cluster under high load while dropping table? >>>>>> >>>>>> As far as I remember, in older c* version, it applies the schema >>>>>> mutation in memory, ie. DROP, then flush all schema info into sstabl= e, then >>>>>> reads all on disk schema into memory (5k tables info + related colum= n >>>>>> info).. >>>>>> >>>>>> > You also might need to increase the node count if you're resource >>>>>> constrained. >>>>>> >>>>>> More nodes won't help and most probably make it worse due to >>>>>> coordination. >>>>>> >>>>>> >>>>>> Zhao Yang >>>>>> >>>>>> >>>>>> >>>>>> On Fri, 21 Apr 2017 at 21:10 Bohdan Tantsiura >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Problem is still not solved. Does anybody have any idea what to do >>>>>>> with it? >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> 2017-04-20 15:05 GMT+03:00 Bohdan Tantsiura : >>>>>>> >>>>>>>> Thanks Carlos, >>>>>>>> >>>>>>>> In each keyspace we also have 11 MVs. >>>>>>>> >>>>>>>> It is impossible to reduce number of tables now. Long GC Pauses >>>>>>>> take about one minute. But why it takes so much time and how that = can be >>>>>>>> fixed? >>>>>>>> >>>>>>>> Each node in cluster has 128GB RAM, so resources are not >>>>>>>> constrained now >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> 2017-04-20 13:18 GMT+03:00 Carlos Rolo : >>>>>>>> >>>>>>>>> You have 4800 Tables in total? That is a lot of tables, plus MVs? >>>>>>>>> or MVs are already considered in the 60*80 account? >>>>>>>>> >>>>>>>>> I would recommend to reduce the table number. Other thing is that >>>>>>>>> you need to check your log file for GC Pauses, and how long those= pauses >>>>>>>>> take. >>>>>>>>> >>>>>>>>> You also might need to increase the node count if you're resource >>>>>>>>> constrained. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Carlos Juzarte Rolo >>>>>>>>> Cassandra Consultant / Datastax Certified Architect / Cassandra M= VP >>>>>>>>> >>>>>>>>> Pythian - Love your data >>>>>>>>> >>>>>>>>> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin: >>>>>>>>> *linkedin.com/in/carlosjuzarterolo >>>>>>>>> * >>>>>>>>> Mobile: +351 918 918 100 <+351%20918%20918%20100> >>>>>>>>> www.pythian.com >>>>>>>>> >>>>>>>>> On Thu, Apr 20, 2017 at 11:10 AM, Bohdan Tantsiura < >>>>>>>>> bohdantan@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> We are using cassandra 3.10 in a 10 nodes cluster with >>>>>>>>>> replication =3D 3. MAX_HEAP_SIZE=3D64GB on all nodes, G1 GC is u= sed. We have >>>>>>>>>> about 60 keyspaces with about 80 tables in each keyspace. We had= to delete >>>>>>>>>> three tables and two materialized views from each keyspace. It b= egan to >>>>>>>>>> take more and more time for each next keyspace (for some keyspac= es it took >>>>>>>>>> about 30 minutes) and then failed with "Cannot achieve consisten= cy level >>>>>>>>>> ALL". After restarting the same repeated. It seems that cassandr= a hangs on >>>>>>>>>> GC. How that can be solved? >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> >>> >> > --f4030435bb00b08852054f467db6 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi

We were trying to overcome OOM crashes.

Fair enough :-).

We changed settings to default on one node. GC times bec= ame about two times smaller on that node.
That's encouraging! Looks like even if the number of tables= is really high, there is still space for optimization. Have you made the c= hange on the entire cluster by now? How are things going?

Also you can continue to play around with a canary node to find the= sweet spot, the right tuning for your use case and hardware. Taking a day = to do this is sometimes very worth it ;-).

Number of sstables is not constant. During abou= t 2.5 hours number of tables was changed for 26 tables, e.g.=C2=A0[4, 4, 4, 4, 4, 4, 4, 4, 4, 4] =3D> [6, 6= , 6, 6, 6, 6, 6, 6, 6, 6] or=C2=A0[= 4, 7, 7, 7, 5, 5, 4, 4, 4, 4] =3D> [4, 4, 4, 4, 4, 4, 4, 4, 4, 4] (each = list is number of sstables on each of 10 nodes for one table).=C2=A0=
Numb= er of sstables is balanced for almost all tables. But for some tables numbe= r of sstables is not really balanced, like [11, 2, 4, 4, 4, 2, 2, 2, 2, 5] = or=C2=A0[439, 558, 346, 521, 490, 553, 500, 515, 522, 495]

Sounds like reasonable numbers of SSTables. Im= balances are not that big. I would say compaction is running normally
=

=C2=A0We run incremental repairs
=
=C2=A0We use=C2=A0LCS for a= ll tables and MVs. We don't do=C2=A0manual compactions (or trigger any anti-compactions)

If you run incremental repairs, it means you trigger anti-compac= tions. Actually the first repair might completely have produce the increase= in the number of SSTables. If it was not the first time, you might have ru= n into one of the corner cases still not fixed on incremental repairs.

Also, using LCS for such a high number of tables is pr= obably putting some pressure on disks. Is it a real need? Would STCS or TWC= S not be a better fit on some of the table? That being said, compactions lo= ok good. Are you seeing pending compactions under standard conditions or on= peak hours?

=C2=A0<= /span>What number of=C2=A0MemtableFlushWriter are you using?
= We do not specify if, so default is use

<= a href=3D"https://github.com/apache/cassandra/blob/cassandra-3.10/conf/cass= andra.yaml#L538">https://github.com/apache/cassandra/blob/cassandra-3.10/co= nf/cassandra.yaml#L538

So it is 2. If disks IO is not= a bottleneck you might want to consider increasing this number to 4, it sh= ould be a safe value. To know if Flush Writers is an issue, use 'watch = -d nodetool tpstats' and see if there are any 'FlushWriter' pen= ding threads.

One column for each node
READ: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 2609 7 =C2=A0 0 =C2=A0 0 =C2=A0 2 =C2=A0 1 =C2=A0 2 =C2=A0 0 =C2= =A0 1 =C2=A0 1

The only non ne= gligible value is about read being dropped and on only one node. If this va= lue is not growing anymore, you might have faced a punctual issue.

This cluster looks relatively healthy excepted for GC acti= vity (that can explain read drops). I would persevere on GC tuning, continu= ing to monitor things you have been sharing with us so far, to see how it e= volves. Having a closer look at repairs impact might be worth it as well. G= ood luck!

C*heers,
---------------------= --
Alain Rodriguez - @arodream - alain@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting

2017-05-08 14:21 GMT= +01:00 Bohdan Tantsiura <bohdantan@gmail.com>:
Hi,

=
>=C2=A0Why did you move from defau= lts that much?
We = were trying to overcome OOM crashes.
=
>=C2=A0Would you co= nsider giving=C2=A0defaults a try on a canary node and monitor / compare GC= times to other nodes?
We changed settings to default on one node. GC times became about two = times smaller on that node.

>=C2=A0What do you mean from t= ime to time? For how long are this task pending, what frequency is this hap= pening?
=C2=A0Comp= actionExecutor pending tasks appeared on nodes once during 3 or more hours.= Tasks were pending during about 5-15 minutes.

>=C2=A0I= s the number of sstables constant and balanced between nodes?
<= /span>
Number of sstables is not const= ant. During about 2.5 hours number of tables was changed for 26 tables, e.g= .=C2=A0[4, 4, 4, 4, 4, 4, 4, 4, 4, = 4] =3D> [6, 6, 6, 6, 6, 6, 6, 6, 6, 6] or=C2=A0[4, 7, 7, 7, 5, 5, 4, 4, 4, 4] =3D> [4, 4, 4, 4, 4, 4, 4,= 4, 4, 4] (each list is number of sstables on each of 10 nodes for one tabl= e).=C2=A0
Number of sstab= les is balanced for almost all tables. But for some tables number of sstabl= es is not really balanced, like [11, 2, 4, 4, 4, 2, 2, 2, 2, 5] or=C2=A0[43= 9, 558, 346, 521, 490, 553, 500, 515, 522, 495]

>=C2=A0Also do you run full or incremental repairs?<= /span>
We run incremental= repairs

>=C2=A0Do you use LCS or do some manual compactions= (or trigger any anti-compactions)?
We use=C2=A0LCS f= or all tables and MVs. We don't do=C2=A0manual compactions (or trigger any anti-compactions)
<= span class=3D"">

>=C2=A0How is CPU doing, is there any burst in CPU that could be related = to these errors?
U= nfortunately, stats for period when there were InternalResponseState pendin= g tasks is lost.

>=C2=A0<= /span>What number of=C2=A0MemtableFlushWriter are you using
We do not specify if, so default is= used

>=C2=A0What tasks were dropped<= /span>
One column for each node<= /span>
READ: =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 2609 7 =C2=A0 0 =C2=A0 0 =C2=A0 2 =C2=A0= 1 =C2=A0 2 =C2=A0 0 =C2=A0 1 =C2=A0 1
HINT: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 = =C2=A0 =C2=A00 =C2=A0 0 =C2=A0 0 =C2=A0 1 =C2=A0 0 =C2=A0 1 =C2=A0 0 =C2=A0= 0 =C2=A0 0
MUTATION: = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 =C2=A0 =C2=A00 =C2=A0 1 =C2=A0 0 =C2= =A0 0 =C2=A0 0 =C2=A0 0 =C2=A0 0 =C2=A0 0 =C2=A0 0
REQUEST_RESPONSE: =C2=A0 0 =C2=A0 =C2=A00 =C2= =A0 0 =C2=A0 0 =C2=A0 2 =C2=A0 0 =C2=A0 0 =C2=A0 0 =C2=A0 0 =C2=A0 1=

Thanks

20= 17-05-03 17:00 GMT+03:00 Alain RODRIGUEZ <arodrime@gmail.com>:
Hi,

A few com= ments:

Long GC Pauses take about one minute

This is huge. About JVM config, I haven= 't played much with G1GC, but the following seems to be a bad idea acco= rding to comments:

## Main G1GC tunable: lowering the pause tar= get will lower throughput and vise versa.
## 200ms is the JVM def= ault and lowest viable setting
## 1000ms increases throughput. Ke= ep it smaller than the timeouts in cassandra.yaml.
-XX:MaxGCPause= Millis=3D15000

# Save CPU time on large (>=3D 16GB) heaps by= delaying region scanning
# until the heap is 70% full. The defau= lt in Hotspot 8u40 is 40%.
-XX:InitiatingHeapOccupancyPercen= t=3D30
# For systems with > 8 cores, the default ParallelGCThr= eads is 5/8 the number of logical cores.
# Otherwise equal to the= number of cores when 8 or less.
# Machines with > 10 cores sh= ould try setting these to <=3D full cores.
-XX:ParallelGCThrea= ds=3D8
# By default, ConcGCThreads is 1/4 of ParallelGCThreads.
# Setting both to the same value can reduce STW durations.
-XX:ConcGCThreads=3D8


Why did you move from defaults th= at much? Would you consider giving=C2=A0defaults a try on a canary node and= monitor / compare GC times to other nodes?

1)=C2=A0ColumnFamilyStore.java:542 - Failed unregistering mbean: org.apa= che.cassandra.db:type=3DTables= ,keyspace=3D...,table=3D... =C2=A0from=C2=A0MigrationStage thread

I am not sure about this one... :p

2)=C2=A0Read 1715 live rows and 1505 tombstone cells for query = ... =C2=A0from=C2=A0ReadStage thread

Half of what was read for this query was deleted data. With obvious= disk space, disk throughput and latency consequences. This is an entire to= pic... Here is what I know about it: thelastp= ickle.com/blog/2016/07/27/about-deletes-and-tombstones.html, = I hope it will help you solving your issue.

3)=C2=A0G= CInspector.java:282 - G1 Young Generation GC in 1725ms
<= /span>

This might be related to your GC configuration or some other= issues mentioned in your last mail.

=C2=A0About 3000-= 6000 CompactionExecutor pending tasks appeared on all nodes from time to ti= me.

Hum that's weird. It&= #39;s huge, but as you have so many tables I am not sure, it might be a = 9;normal' issue when running with so many tables and MVs.

What d= o you mean from time to time? For how long are this task pending, what freq= uency is this happening?

Is the number of sstables constant a= nd balanced between nodes?
Also do you run full or incremental re= pairs?
Do you use LCS or do some manual compactions (or trigger any anti= -compactions)?
=C2=A0
About 1000 MigrationStag= e pending tasks appeared on 2 nodes.

Tha= t's pending writes. Meaning this Cassandra node can't cope with wha= t what is thrown at it. It can be related to pending flushes (blocking writ= es), huge Garbage Collection (Stop The World, including writes), due to har= dware limits (CPU busy with compactions?) or even to a too conservative con= figuration of the concurrent_write.
=C2=A0
About 700= InternalResponseState pending tasks appeared on 2 nodes.

I never had issues with this one and so didn't knew m= uch about it. But according to Chris Lohfink in this post https://www.pythian.com/blog/guide-to-cassandra-= thread-pools/#InternalResponseStage,=C2=A0this thread pool is resp= onsible for "Responding to non-client initiated messages, including bo= otstrapping and schema checking". Which again might be related with th= e huge number of tables in the cluster. How is CPU doing, is there any burs= t in CPU that could be related to these errors?

= About 60=C2=A0MemtableFlushWriter appeared on 3 nodes.
=

What number of=C2=A0MemtableFlushWriter are you using. Consider increasing it (or maybe th= e memtable size).
=C2=A0
There were no = blocked tasks, but there were "All time blocked" tasks (they were= before starting dropping tables) from 3 millions to 20 millions on differe= nt nodes.

What tasks were dro= pped.

The cluster doesn't look completely heal= thy, but I believe it is possible to improve things, before thinking about = splitting tables in multiples cluster. I would definitely not add more tabl= es though...

C*heers,
=
-----------------------
Alain Rodriguez - @arodream - alain@thelastpickle.= com
France

The Last Pickle - Apache = Cassandra Consulting

2017-04-28 = 14:35 GMT+01:00 Bohdan Tantsiura <bohdantan@gmail.com>:
Thanks Alain,
>=C2=A0Or is it on happening dur= ing drop table actions?
Some other schema changes (e.g. adding columns to tables) also takes = too much time.

Link to complete set of GC opt= ions:=C2=A0http= s://pastebin.com/4qyENeyu

>= =C2=A0Have you had a look at logs, = mainly errors and warnings?
In logs I found warnings of 3 types:
1)=C2=A0ColumnFamilyStore.java:542 - Failed unregiste= ring mbean: org.apache.cassandra.db:type=3DTables,keyspace=3D...,table= =3D... =C2=A0from=C2=A0MigrationStage thread
2)=C2=A0Read 1715 live rows and 1505 tombstone cells = for query ... =C2=A0from=C2=A0ReadStage thread
3)=C2=A0GCInspector.java:282 - G1 Young Generation GC= in 1725ms.=C2=A0 G1 Eden Space: 38017171456 -> 0; G1 Survivor Space: 251= 6582400 -> 2650800128; from=C2=A0Service Thread

>=C2=A0Are they = some pending, blocked or dropped tasks in thread pool stats?
About 3000-6000 CompactionExecut= or pending tasks appeared on all nodes from time to time. About 1000 Migrat= ionStage pending tasks appeared on 2 nodes. About 700 InternalResponseState= pending tasks appeared on 2 nodes. About 60=C2=A0MemtableFlushWriter appea= red on 3 nodes.
There wer= e no blocked tasks, but there were "All time blocked" tasks (they= were before starting dropping tables) from 3 millions to 20 millions on di= fferent nodes.

=
>=C2=A0Are some resources constraint (CPU / disk IO,...)= ?
CPU and=C2=A0disk IO are not constraint

Thanks

2017-04-27 11:10 GMT+03:00 Alain RODRIGUEZ <= ;arodrime@gmail.com= >:
Hi=
=C2=A0
Long GC Pauses take about one minute. = But why it takes so much time and how that can be fixed?

This is very long. Looks like you are having a = major issue, and it is not just about dropping tables... Or is it on happen= ing during drop table actions? Knowing the complete set of GC options in us= e could help here, could you paste it here (or link to it)?=C2=A0

Also, GC is often high as a consequence of other issues and= not only when 'badly=E2=80=98 tuned

  • H= ave you had a look at logs, mainly errors and warnings?

    $ grep -e &q= uot;ERROR" -e "WARN" /var/log/cassandra/system.log

  • Are they some pending, blocked or dropped tasks in thread pool stats= ?=C2=A0

    $ watch -d nodetool tpstats

  • Are some resourc= es constraint (CPU / disk IO,...)?

We have about 60 keyspaces with about 80 tables in each keyspace= =C2=A0
In each keyspace we also have 11 MVs

Even if I believe we can dig it and maybe improve t= hings, I agree with Carlos, this is a lot of Tables (4880) and even more a = high number of MV (660). It might be interesting splitting it somehow if po= ssible.

Cannot achieve consistency = level ALL

Finally you could t= ry to adjust the corresponding request timeout (not sure if it is the globa= l one or the truncate timeout), so it may succeed even when nodes are havin= g minutes GC, but it is a workaround as this minute GC will most definitely= be an issue for the client queries running (default is 10 sec timeout, so = many query are probably failing).

C*heers,
-----------------------
Alain Rodriguez - @arodream - alain@thelastpick= le.com
France

The Last Pickle - Apac= he Cassandra Consulting

2017= -04-25 13:58 GMT+02:00 Bohdan Tantsiura <bohdantan@gmail.com>:
Thanks Zhao Yang,

>=C2=A0Could you try some j= vm tool to find out which thread are allocating memory or gc? maybe the mig= ration stage thread..
I use Cassandra Cluster Manager to locally reproduce the issue. I = tried to use VisualVM=C2=A0to find out whi= ch threads are allocating memory, but=C2=A0VisualVM does not see cas= sandra processes and says "Cannot open application with pid". The= n I tried to use YourKit Java Profiler. It created snapshot when process of= one cassandra node failed.=C2=A0http://i.imgur.com/9jBcjcl.png - how CPU is use= d by threads.=C2=A0http://i.imgur.com/ox5Sozy.png - how memory is used by thread= s, but biggest part of memory is used by objects without allocation informa= tion.=C2=A0htt= p://i.imgur.com/oqx9crX.png - which objects use biggest part of me= mory. Maybe you know some other good jvm tool that can show by which thread= s biggest part of memory is used?

>=C2=A0= BTW, is your cluster under high load while= dropping table?

LA5 was <=3D 5 on = all nodes almost all time while dropping tables

Thanks

20= 17-04-21 19:49 GMT+03:00 Jasonstack Zhao Yang <zhaoyangsingapore= @gmail.com>:
Hi Bohdan, Carlos,

Could you try some jvm tool to fin= d out which thread are allocating memory or gc? maybe the migration stage t= hread..

BTW, is your cluster under high load while= dropping table?

As far as I remember, in older c*= version, it applies the schema mutation in memory, ie. DROP, then flush al= l schema info into sstable, then reads all on disk schema into memory (5k t= ables info + related column info)..

>=C2= =A0You also might need to increase the = node count if you're resource constrained.

More nodes won't help and most probably make it worse du= e to coordination.


Zhao= Yang



On Fri, 21= Apr 2017 at 21:10 Bohdan Tantsiura <bohdantan@gmail.com> wrote:
Hi,

Problem is st= ill not solved. Does anybody have any idea what to do with it?
Thanks

2017-04-20 15:05 GMT+03:00 Bohdan Tantsiura <bohdantan@gma= il.com>:
<= div>Thanks Carlos,

In each keyspace we also have 11 MVs= .

It is impossible to reduce number of tables now. Long = GC Pauses take about one minute. But why it takes so much time and how that= can be fixed?

Each node in cluster has 128GB RAM,= so=C2=A0resources are not constrained now=

Thanks
=

2017-04-20 13:18 GMT+03:00 Carlos Rolo <rolo@pythian.com>:=
You have 4800 Tables in= total? That is a lot of tables, plus MVs? or MVs are already considered in= the 60*80 account?

I would recommend to reduce the tabl= e number. Other thing is that you need to check your log file for GC Pauses= , and how long those pauses take.=C2=A0

You also m= ight need to increase the node count if you're resource constrained.

Regards,

Carlos Juzarte Rolo
Cassandra Consultant / D= atastax Certified Architect / Cassandra MVP
=C2=A0
= Pythian - Love your data

rolo@pythian | Twitter: @= cjrolo | Skype: cjr2k3 | Linkedin: linkedin.com/in/c= arlosjuzarterolo
<= /div>

On Thu, Apr 20, 2017 at 11:10 AM, Bohdan Tan= tsiura <bohdantan@gmail.com> wrote:
Hi,

We a= re using cassandra 3.10 in a 10 nodes cluster with replication =3D 3. MAX_H= EAP_SIZE=3D64GB on all nodes, G1 GC is used. We have about 60 keyspaces wit= h about 80 tables in each keyspace. We had to delete three tables and two m= aterialized views from each keyspace. It began to take more and more time f= or each next keyspace (for some keyspaces it took about 30 minutes) and the= n failed with "Cannot achieve consistency level ALL". After resta= rting the same repeated. It seems that cassandra hangs on GC. How that can = be solved?

Thanks


--











--f4030435bb00b08852054f467db6--