Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 373EB200C68 for ; Wed, 3 May 2017 16:00:32 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 313CF160BB5; Wed, 3 May 2017 14:00:32 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id D0429160BAA for ; Wed, 3 May 2017 16:00:29 +0200 (CEST) Received: (qmail 50940 invoked by uid 500); 3 May 2017 14:00:28 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 50928 invoked by uid 99); 3 May 2017 14:00:28 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 May 2017 14:00:28 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id D5B86C3274 for ; Wed, 3 May 2017 14:00:27 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.88 X-Spam-Level: * X-Spam-Status: No, score=1.88 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id k6MqbvjYTP1o for ; Wed, 3 May 2017 14:00:24 +0000 (UTC) Received: from mail-ua0-f179.google.com (mail-ua0-f179.google.com [209.85.217.179]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 18A0C5F5CA for ; Wed, 3 May 2017 14:00:23 +0000 (UTC) Received: by mail-ua0-f179.google.com with SMTP id e55so72282374uaa.2 for ; Wed, 03 May 2017 07:00:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=lDTrEA//DaypVwI/BQPIOAXWUxb6SV+ReJKQCNUCWAQ=; b=LjzcCkT/AeLYqAGEKhD20GJTHz43EYglah2rQBPHTnvm2XmzECtd3BuNoEgnoI0efN PSxSFj6vmGpeoBJ4MywpASrygFbOAGWNZrB54S/UV/uq3UvBfywBSmtRRFlUsfX15+ab HbvmVnns+3eiw/HUYLmndFg6DeIc1f4H4XRe3gs+2vaU2gHyAArjxZuSpaKCfQ+cn4ZE wGYyCXtMQCSOh2+jJjpkmj3M+OrVVjY91oSZrj0IPP65ZLkYjTD6TXdf14PzXbDQy2nm kZNew/BzfDnekVeAA8FTCO5TOPxYPIenG3KKpkyUUzFAZBRyG2KtA+9wjVQ1gPxO66M8 PZgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=lDTrEA//DaypVwI/BQPIOAXWUxb6SV+ReJKQCNUCWAQ=; b=H3ZcGE9OUIB9HC8LSjPmbF/YFBAj2SiGL3V4D420/CnMJXszzL3H87PWU/uer7EvEJ A4SSxTKCbcGIJ6ZEjb7QQ5WiE1vp5A3VMQmNaG84b1I3LlN0eOJFVgn2ainn++kiD1+z p6719xoUNgueZeXZWLY/OkPG90RAyxstABbYnMValqYyIgBdAjUiNM/HFR1kpjNdiC71 wVxwa0M2Ct+YJHvCari5tl3LCTeyGyxMGVjRGYvR467xJqzIEm/hvkF16vhtUDfAcX31 o5WkrIyHKlWwbd3Kv+0JY/x+nyUUP6WWqSO2LSFF/c4i5bhdzN5wYbJYlVhzlUN70yN1 +e2g== X-Gm-Message-State: AN3rC/5i5PEovihvqC5O1M/xIPupaMvHKuxBB4AAr00Y4NLpdfi2qXAg 8ybAAeth/tURmj0u/IJxUWMPHMRJuw== X-Received: by 10.176.83.202 with SMTP id l10mr19312044uaa.14.1493820021862; Wed, 03 May 2017 07:00:21 -0700 (PDT) MIME-Version: 1.0 Received: by 10.103.83.151 with HTTP; Wed, 3 May 2017 07:00:01 -0700 (PDT) In-Reply-To: References: From: Alain RODRIGUEZ Date: Wed, 3 May 2017 15:00:01 +0100 Message-ID: Subject: Re: Drop tables takes too long To: Bohdan Tantsiura Cc: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=f403045e427ec2f6a1054e9f131d archived-at: Wed, 03 May 2017 14:00:32 -0000 --f403045e427ec2f6a1054e9f131d Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi, A few comments: Long GC Pauses take about one minute > This is huge. About JVM config, I haven't played much with G1GC, but the following seems to be a bad idea according to comments: ## Main G1GC tunable: lowering the pause target will lower throughput and vise versa. ## 200ms is the JVM default and lowest viable setting ## 1000ms increases throughput. Keep it smaller than the timeouts in cassandra.yaml. -XX:MaxGCPauseMillis=3D15000 # Save CPU time on large (>=3D 16GB) heaps by delaying region scanning # until the heap is 70% full. The default in Hotspot 8u40 is 40%. -XX:InitiatingHeapOccupancyPercent=3D30 # For systems with > 8 cores, the default ParallelGCThreads is 5/8 the number of logical cores. # Otherwise equal to the number of cores when 8 or less. # Machines with > 10 cores should try setting these to <=3D full cores. -XX:ParallelGCThreads=3D8 # By default, ConcGCThreads is 1/4 of ParallelGCThreads. # Setting both to the same value can reduce STW durations. -XX:ConcGCThreads=3D8 Why did you move from defaults that much? Would you consider giving defaults a try on a canary node and monitor / compare GC times to other nodes? 1) ColumnFamilyStore.java:542 - Failed unregistering mbean: > org.apache.cassandra.db:type=3DTables,keyspace=3D...,table=3D... > from MigrationStage thread > I am not sure about this one... :p 2) Read 1715 live rows and 1505 tombstone cells for query ... > from ReadStage thread Half of what was read for this query was deleted data. With obvious disk space, disk throughput and latency consequences. This is an entire topic... Here is what I know about it: thelastpickle.com/blog/2016/ 07/27/about-deletes-and-tombstones.html, I hope it will help you solving your issue. 3) GCInspector.java:282 - G1 Young Generation GC in 1725ms This might be related to your GC configuration or some other issues mentioned in your last mail. About 3000-6000 CompactionExecutor pending tasks appeared on all nodes > from time to time. Hum that's weird. It's huge, but as you have so many tables I am not sure, it might be a 'normal' issue when running with so many tables and MVs. What do you mean from time to time? For how long are this task pending, what frequency is this happening? Is the number of sstables constant and balanced between nodes? Also do you run full or incremental repairs? Do you use LCS or do some manual compactions (or trigger any anti-compactions)? > About 1000 MigrationStage pending tasks appeared on 2 nodes. That's pending writes. Meaning this Cassandra node can't cope with what what is thrown at it. It can be related to pending flushes (blocking writes), huge Garbage Collection (Stop The World, including writes), due to hardware limits (CPU busy with compactions?) or even to a too conservative configuration of the concurrent_write. > About 700 InternalResponseState pending tasks appeared on 2 nodes. I never had issues with this one and so didn't knew much about it. But according to Chris Lohfink in this post https://www.pythian.com/blog/guide-to-cassandra-thread-pools/#InternalRespo= nseStage, this thread pool is responsible for "Responding to non-client initiated messages, including bootstrapping and schema checking". Which again might be related with the huge number of tables in the cluster. How is CPU doing, is there any burst in CPU that could be related to these errors? About 60 MemtableFlushWriter appeared on 3 nodes. What number of MemtableFlushWriter are you using. Consider increasing it (or maybe the memtable size). > There were no blocked tasks, but there were "All time blocked" tasks (the= y > were before starting dropping tables) from 3 millions to 20 millions on > different nodes. What tasks were dropped. The cluster doesn't look completely healthy, but I believe it is possible to improve things, before thinking about splitting tables in multiples cluster. I would definitely not add more tables though... C*heers, ----------------------- Alain Rodriguez - @arodream - alain@thelastpickle.com France The Last Pickle - Apache Cassandra Consulting http://www.thelastpickle.com 2017-04-28 14:35 GMT+01:00 Bohdan Tantsiura : > Thanks Alain, > > > Or is it on happening during drop table actions? > Some other schema changes (e.g. adding columns to tables) also takes too > much time. > > Link to complete set of GC options: https://pastebin.com/4qyENeyu > > > Have you had a look at logs, mainly errors and warnings? > In logs I found warnings of 3 types: > 1) ColumnFamilyStore.java:542 - Failed unregistering mbean: > org.apache.cassandra.db:type=3DTables,keyspace=3D...,table=3D... > from MigrationStage thread > 2) Read 1715 live rows and 1505 tombstone cells for query ... > from ReadStage thread > 3) GCInspector.java:282 - G1 Young Generation GC in 1725ms. G1 Eden > Space: 38017171456 -> 0; G1 Survivor Space: 2516582400 <(251)%20658-2400> > -> 2650800128; from Service Thread > > > Are they some pending, blocked or dropped tasks in thread pool stats? > About 3000-6000 CompactionExecutor pending tasks appeared on all nodes > from time to time. About 1000 MigrationStage pending tasks appeared on 2 > nodes. About 700 InternalResponseState pending tasks appeared on 2 nodes. > About 60 MemtableFlushWriter appeared on 3 nodes. > There were no blocked tasks, but there were "All time blocked" tasks (the= y > were before starting dropping tables) from 3 millions to 20 millions on > different nodes. > > > Are some resources constraint (CPU / disk IO,...)? > CPU and disk IO are not constraint > > Thanks > > 2017-04-27 11:10 GMT+03:00 Alain RODRIGUEZ : > >> Hi >> >> >>> Long GC Pauses take about one minute. But why it takes so much time and >>> how that can be fixed? >> >> >> This is very long. Looks like you are having a major issue, and it is no= t >> just about dropping tables... Or is it on happening during drop table >> actions? Knowing the complete set of GC options in use could help here, >> could you paste it here (or link to it)? >> >> Also, GC is often high as a consequence of other issues and not only whe= n >> 'badly=E2=80=98 tuned >> >> >> - Have you had a look at logs, mainly errors and warnings? >> >> $ grep -e "ERROR" -e "WARN" /var/log/cassandra/system.log >> >> - Are they some pending, blocked or dropped tasks in thread pool >> stats? >> >> $ watch -d nodetool tpstats >> >> - Are some resources constraint (CPU / disk IO,...)? >> >> >> We have about 60 keyspaces with about 80 tables in each keyspace >> >> In each keyspace we also have 11 MVs >> >> >> Even if I believe we can dig it and maybe improve things, I agree with >> Carlos, this is a lot of Tables (4880) and even more a high number of MV >> (660). It might be interesting splitting it somehow if possible. >> >> Cannot achieve consistency level ALL >> >> >> Finally you could try to adjust the corresponding request timeout (not >> sure if it is the global one or the truncate timeout), so it may succeed >> even when nodes are having minutes GC, but it is a workaround as this >> minute GC will most definitely be an issue for the client queries runnin= g >> (default is 10 sec timeout, so many query are probably failing). >> >> C*heers, >> ----------------------- >> Alain Rodriguez - @arodream - alain@thelastpickle.com >> France >> >> The Last Pickle - Apache Cassandra Consulting >> http://www.thelastpickle.com >> >> 2017-04-25 13:58 GMT+02:00 Bohdan Tantsiura : >> >>> Thanks Zhao Yang, >>> >>> > Could you try some jvm tool to find out which thread are allocating >>> memory or gc? maybe the migration stage thread.. >>> >>> I use Cassandra Cluster Manager to locally reproduce the issue. I tried >>> to use VisualVM to find out which threads are allocating memory, but Vi= sualVM >>> does not see cassandra processes and says "Cannot open application with >>> pid". Then I tried to use YourKit Java Profiler. It created snapshot wh= en >>> process of one cassandra node failed. http://i.imgur.com/9jBcjcl.png - >>> how CPU is used by threads. http://i.imgur.com/ox5Sozy.png - how memory >>> is used by threads, but biggest part of memory is used by objects witho= ut >>> allocation information. http://i.imgur.com/oqx9crX.png - which objects >>> use biggest part of memory. Maybe you know some other good jvm tool tha= t >>> can show by which threads biggest part of memory is used? >>> >>> > BTW, is your cluster under high load while dropping table? >>> >>> LA5 was <=3D 5 on all nodes almost all time while dropping tables >>> >>> Thanks >>> >>> 2017-04-21 19:49 GMT+03:00 Jasonstack Zhao Yang < >>> zhaoyangsingapore@gmail.com>: >>> >>>> Hi Bohdan, Carlos, >>>> >>>> Could you try some jvm tool to find out which thread are allocating >>>> memory or gc? maybe the migration stage thread.. >>>> >>>> BTW, is your cluster under high load while dropping table? >>>> >>>> As far as I remember, in older c* version, it applies the schema >>>> mutation in memory, ie. DROP, then flush all schema info into sstable,= then >>>> reads all on disk schema into memory (5k tables info + related column >>>> info).. >>>> >>>> > You also might need to increase the node count if you're resource >>>> constrained. >>>> >>>> More nodes won't help and most probably make it worse due to >>>> coordination. >>>> >>>> >>>> Zhao Yang >>>> >>>> >>>> >>>> On Fri, 21 Apr 2017 at 21:10 Bohdan Tantsiura >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> Problem is still not solved. Does anybody have any idea what to do >>>>> with it? >>>>> >>>>> Thanks >>>>> >>>>> 2017-04-20 15:05 GMT+03:00 Bohdan Tantsiura : >>>>> >>>>>> Thanks Carlos, >>>>>> >>>>>> In each keyspace we also have 11 MVs. >>>>>> >>>>>> It is impossible to reduce number of tables now. Long GC Pauses take >>>>>> about one minute. But why it takes so much time and how that can be = fixed? >>>>>> >>>>>> Each node in cluster has 128GB RAM, so resources are not constrained >>>>>> now >>>>>> >>>>>> Thanks >>>>>> >>>>>> 2017-04-20 13:18 GMT+03:00 Carlos Rolo : >>>>>> >>>>>>> You have 4800 Tables in total? That is a lot of tables, plus MVs? o= r >>>>>>> MVs are already considered in the 60*80 account? >>>>>>> >>>>>>> I would recommend to reduce the table number. Other thing is that >>>>>>> you need to check your log file for GC Pauses, and how long those p= auses >>>>>>> take. >>>>>>> >>>>>>> You also might need to increase the node count if you're resource >>>>>>> constrained. >>>>>>> >>>>>>> Regards, >>>>>>> >>>>>>> Carlos Juzarte Rolo >>>>>>> Cassandra Consultant / Datastax Certified Architect / Cassandra MVP >>>>>>> >>>>>>> Pythian - Love your data >>>>>>> >>>>>>> rolo@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin: >>>>>>> *linkedin.com/in/carlosjuzarterolo >>>>>>> * >>>>>>> Mobile: +351 918 918 100 <+351%20918%20918%20100> >>>>>>> www.pythian.com >>>>>>> >>>>>>> On Thu, Apr 20, 2017 at 11:10 AM, Bohdan Tantsiura < >>>>>>> bohdantan@gmail.com> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> We are using cassandra 3.10 in a 10 nodes cluster with replication >>>>>>>> =3D 3. MAX_HEAP_SIZE=3D64GB on all nodes, G1 GC is used. We have a= bout 60 >>>>>>>> keyspaces with about 80 tables in each keyspace. We had to delete = three >>>>>>>> tables and two materialized views from each keyspace. It began to = take more >>>>>>>> and more time for each next keyspace (for some keyspaces it took a= bout 30 >>>>>>>> minutes) and then failed with "Cannot achieve consistency level AL= L". After >>>>>>>> restarting the same repeated. It seems that cassandra hangs on GC.= How that >>>>>>>> can be solved? >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>> >> > --f403045e427ec2f6a1054e9f131d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,

A few comments:

Long GC Pauses = take about one minute

This is huge. A= bout JVM config, I haven't played much with G1GC, but the following see= ms to be a bad idea according to comments:

## Main G1GC tunable= : lowering the pause target will lower throughput and vise versa.
## 200ms is the JVM default and lowest viable setting
## 1000ms = increases throughput. Keep it smaller than the timeouts in cassandra.yaml.<= /div>
-XX:MaxGCPauseMillis=3D15000

# Save CPU time on large= (>=3D 16GB) heaps by delaying region scanning
# until the hea= p is 70% full. The default in Hotspot 8u40 is 40%.
-XX:Initiating= HeapOccupancyPercent=3D30
# For systems with > 8 cores, the de= fault ParallelGCThreads is 5/8 the number of logical cores.
# Oth= erwise equal to the number of cores when 8 or less.
# Machines wi= th > 10 cores should try setting these to <=3D full cores.
= -XX:ParallelGCThreads=3D8
# By default, ConcGCThreads is 1/4 of P= arallelGCThreads.
# Setting both to the same value can reduce STW= durations.
-XX:ConcGCThreads=3D8


Why did you mo= ve from defaults that much? Would you consider giving=C2=A0defaults a try o= n a canary node and monitor / compare GC times to other nodes?
<= div>
1)=C2=A0ColumnFamilyStore.java:542 - Failed unregistering m= bean: org.apache.cassandra.db:type=3DTables,keyspace=3D...,table=3D... =C2=A0from=C2=A0MigrationStage thr= ead

I am not sure about this one... :p

2)=C2=A0Read 1715 live rows and 1505 tombstone cells for query = ... =C2=A0from=C2=A0ReadStage thread

Half of what was read for this query was deleted data. With obvious disk s= pace, disk throughput and latency consequences. This is an entire topic... = Here is what I know about it: thelastpickle.c= om/blog/2016/07/27/about-deletes-and-tombstones.html, I hope = it will help you solving your issue.

3)=C2=A0GCInspector.jav= a:282 - G1 Young Generation GC in 1725ms

This m= ight be related to your GC configuration or some other issues mentioned in = your last mail.

= =C2=A0About 3000-6000 CompactionExecutor p= ending tasks appeared on all nodes from time to time.

Hum that's weird. It's huge, but as you have so m= any tables I am not sure, it might be a 'normal' issue when running= with so many tables and MVs.

What do you mean from time to time? Fo= r how long are this task pending, what frequency is this happening?

Is the number of sstables constant and balanced between nodes?
=
Also do you run full or incremental repairs?
Do you use LCS or do s= ome manual compactions (or trigger any anti-compactions)?
=C2=A0<= /div>
About 1000 MigrationStage pending tasks appeared on 2 nodes.

That's pending writes. Meaning this Cassandr= a node can't cope with what what is thrown at it. It can be related to = pending flushes (blocking writes), huge Garbage Collection (Stop The World,= including writes), due to hardware limits (CPU busy with compactions?) or = even to a too conservative configuration of the concurrent_write.
=C2=A0=
About 700 InternalResponseState pending tasks appeared on 2 n= odes.

I never had issues with this one and so d= idn't knew much about it. But according to Chris Lohfink in this post <= a href=3D"https://www.pythian.com/blog/guide-to-cassandra-thread-pools/#Int= ernalResponseStage">https://www.pythian.com/blog/guide-to-cassandra-thread-= pools/#InternalResponseStage,=C2=A0this thread pool is responsible for = "Responding to non-client initiated messages, including bootstrapping = and schema checking". Which again might be related with the huge numbe= r of tables in the cluster. How is CPU doing, is there any burst in CPU tha= t could be related to these errors?

About 60=C2=A0Mem= tableFlushWriter appeared on 3 nodes.

What number of=C2=A0MemtableFlushWriter = are you using. Consider increasing it (or maybe the memtable size).<= /div>
=C2=A0
There were no blocked tasks, but there were &= quot;All time blocked" tasks (they were before starting dropping table= s) from 3 millions to 20 millions on different nodes.

What tasks were dropped.

The clu= ster doesn't look completely healthy, but I believe it is possible to i= mprove things, before thinking about splitting tables in multiples cluster.= I would definitely not add more tables though...
C*heers,
-----------------------
Alain= Rodriguez - @arodream - alain@t= helastpickle.com
France

The Last Pic= kle - Apache Cassandra Consulting

2017-04-28 14:35 GMT+01:00 Bohda= n Tantsiura <bohdantan@gmail.com>:
Thanks Alain,

>= ;=C2=A0Or is it on happening during drop t= able actions?
Some= other schema changes (e.g. adding columns to tables) also takes too much t= ime.

Link to complete set of GC options:=C2= =A0https://past= ebin.com/4qyENeyu

&= gt;=C2=A0Have you had a look at log= s, mainly errors and warnings?
In logs I found warnings of 3 types:
1)=C2=A0ColumnFamilyStore.java:542 - Failed unregi= stering mbean: org.apache.cassandra.db:type=3DTables,keyspace=3D...,ta= ble=3D... =C2=A0from=C2=A0MigrationStage thread
2)=C2=A0Read 1715 live rows and 1505 tombstone cells= for query ... =C2=A0from=C2=A0ReadStage thread
3)=C2=A0GCInspector.java:282 - G1 Young Generation G= C in 1725ms.=C2=A0 G1 Eden Space: 38017171456 -> 0; G1 Survivor Space: <= a href=3D"tel:(251)%20658-2400" value=3D"+12516582400" target=3D"_blank">25= 16582400 -> 2650800128; from=C2=A0Service Thread

>=C2=A0Are they some pending, blocked or dropped tasks in thread pool stats?
About 3000-6000 Comp= actionExecutor pending tasks appeared on all nodes from time to time. About= 1000 MigrationStage pending tasks appeared on 2 nodes. About 700 InternalR= esponseState pending tasks appeared on 2 nodes. About 60=C2=A0MemtableFlush= Writer appeared on 3 nodes.
There were no blocked tasks, but there were "All time blocked"= tasks (they were before starting dropping tables) from 3 millions to 20 mi= llions on different nodes.

&= gt;=C2=A0Are some resources constra= int (CPU / disk IO,...)?
CPU and=C2=A0disk IO are not= constraint

Thanks

2017-04-27 11:10 GMT+03:00 Alain RODRIGUEZ <arodri= me@gmail.com>:
Hi
=C2=A0
Long GC Pauses take about = one minute. But why it takes so much time and how that can be fixed?=

This is very long. Looks like you a= re having a major issue, and it is not just about dropping tables... Or is = it on happening during drop table actions? Knowing the complete set of GC o= ptions in use could help here, could you paste it here (or link to it)?=C2= =A0

Also, GC is often high as a consequence of oth= er issues and not only when 'badly=E2=80=98 tuned

<= div>
  • Have you had a look at logs, mainly errors and warnings?
    $ grep -e "ERROR" -e "WARN" /var/log/cassandra/system.= log

  • Are they some pending, blocked or dropped tasks in thre= ad pool stats?=C2=A0

    $ watch -d nodetool tpstats

  • Are= some resources constraint (CPU / disk IO,...)?
We have about 60 keyspaces with about 80 tables in each keys= pace=C2=A0
In each keyspace we also have = 11 MVs

Even if I believe we can dig it and mayb= e improve things, I agree with Carlos, this is a lot of Tables (4880) and e= ven more a high number of MV (660). It might be interesting splitting it so= mehow if possible.

Cannot achieve c= onsistency level ALL

Finally = you could try to adjust the corresponding request timeout (not sure if it i= s the global one or the truncate timeout), so it may succeed even when node= s are having minutes GC, but it is a workaround as this minute GC will most= definitely be an issue for the client queries running (default is 10 sec t= imeout, so many query are probably failing).

C*hee= rs,
-----------------------
Alain Rodriguez - @aro= dream - alain@= thelastpickle.com
France

The Last Pi= ckle - Apache Cassandra Consulting

2017-0= 4-25 13:58 GMT+02:00 Bohdan Tantsiura <bohdantan@gmail.com>:
Thanks Zhao Yang,

>=C2=A0Could you try some jvm= tool to find out which thread are allocating memory or gc? maybe the migra= tion stage thread..

= I use Cassandra Cluster Manager to locally reproduce the issue. I tr= ied to use VisualVM=C2=A0to find out which= threads are allocating memory, but=C2=A0VisualVM does not see cassa= ndra processes and says "Cannot open application with pid". Then = I tried to use YourKit Java Profiler. It created snapshot when process of o= ne cassandra node failed.=C2=A0http://i.imgur.com/9jBcjcl.png - how CPU is used = by threads.=C2=A0http://i.imgur.com/ox5Sozy.png - how memory is used by threads,= but biggest part of memory is used by objects without allocation informati= on.=C2=A0http:= //i.imgur.com/oqx9crX.png - which objects use biggest part of memo= ry. Maybe you know some other good jvm tool that can show by which threads = biggest part of memory is used?

>=C2=A0BTW, is your cluster under high load while d= ropping table?

LA5 was <=3D 5 on al= l nodes almost all time while dropping tables

Thanks

2017-04-21 19:49= GMT+03:00 Jasonstack Zhao Yang <zhaoyangsingapore@gmail.com= >:
Hi Bohdan, = Carlos,

Could you try some jvm tool to find out which th= read are allocating memory or gc? maybe the migration stage thread..
<= div>
BTW, is your cluster under high load while dropping tabl= e?

As far as I remember, in older c* version, it a= pplies the schema mutation in memory, ie. DROP, then flush all schema info = into sstable, then reads all on disk schema into memory (5k tables info + r= elated column info)..

>=C2=A0You also might need to increase the node count if = you're resource constrained.

More nodes won't help and most probably make it worse due to coordinat= ion.

Zhao Yang



On Fri= , 21 Apr 2017 at 21:10 Bohdan Tantsiura <bohdantan@gmail.com> wrote:
Hi,

Problem i= s still not solved. Does anybody have any idea what to do with it?

Thanks

2017-04-20 15:05 GMT+03:00 Bohdan Tantsiura <bohdantan= @gmail.com>:
Thanks Carlos,

In each keyspace we also have 11= MVs.

It is impossible to reduce number of tables now. L= ong GC Pauses take about one minute. But why it takes so much time and how = that can be fixed?

Each node in cluster has 128GB = RAM, so=C2=A0resources are not constrained= now

Thanks

2017-0= 4-20 13:18 GMT+03:00 Carlos Rolo <rolo@pythian.com>:
You have 4800 Tables in total? T= hat is a lot of tables, plus MVs? or MVs are already considered in the 60*8= 0 account?

I would recommend to reduce the table number.= Other thing is that you need to check your log file for GC Pauses, and how= long those pauses take.=C2=A0

You also might need= to increase the node count if you're resource constrained.
=

=
Regards,

Carlos Juzarte Rolo
Ca= ssandra Consultant / Datastax Certified Architect / Cassandra MVP
=
=C2=A0
Pythian - Love your data

rol= o@pythian | Twitter: @cjrolo | Skype: cjr2k3 | Linkedin: linkedin.com/in/carlosjuzarterolo
M= obile: +351 918 918 100
=

On Thu, Apr 20, 2017 at 11:10 AM, Bohdan Tan= tsiura <bohdantan@gmail.com> wrote:
Hi,

We a= re using cassandra 3.10 in a 10 nodes cluster with replication =3D 3. MAX_H= EAP_SIZE=3D64GB on all nodes, G1 GC is used. We have about 60 keyspaces wit= h about 80 tables in each keyspace. We had to delete three tables and two m= aterialized views from each keyspace. It began to take more and more time f= or each next keyspace (for some keyspaces it took about 30 minutes) and the= n failed with "Cannot achieve consistency level ALL". After resta= rting the same repeated. It seems that cassandra hangs on GC. How that can = be solved?

Thanks


--









--f403045e427ec2f6a1054e9f131d--