From user-return-62003-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Wed Aug 22 11:02:14 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 351AA180662 for ; Wed, 22 Aug 2018 11:02:14 +0200 (CEST) Received: (qmail 73734 invoked by uid 500); 22 Aug 2018 09:02:12 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 73723 invoked by uid 99); 22 Aug 2018 09:02:12 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Aug 2018 09:02:12 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id D79EF1A270C for ; Wed, 22 Aug 2018 09:02:11 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.879 X-Spam-Level: * X-Spam-Status: No, score=1.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=zalando.de Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id gkTbIIGpXaGA for ; Wed, 22 Aug 2018 09:02:09 +0000 (UTC) Received: from mail-qk0-f194.google.com (mail-qk0-f194.google.com [209.85.220.194]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 5B1095F396 for ; Wed, 22 Aug 2018 09:02:09 +0000 (UTC) Received: by mail-qk0-f194.google.com with SMTP id g197-v6so726215qke.5 for ; Wed, 22 Aug 2018 02:02:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zalando.de; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=W6LcE5XkFDYxzKuPQoc2YgCbyU2hSzaIyUPaJY1l3Yw=; b=R7lsHWQvUYEEQaw8+1aQnW2VwDXjDvsAzw/vPtq+tkMFoK3cN9SCHYwDKiu3oiCqax utcTO7E0vST/NsTYKEdQuqcArjqHKRSrixHBX5bq8TX9LLxGMD6oNPyOCPedc3utbalW ibw81IbDim+QT/T35HO8LTuZbHDOlUmizOs6nTpxD2mwjq9SkXTRm5vgQ27QoY07xBc2 GbZm+2eAnNOA+iVsZ7OEvqufFAUkL4xwvtXXz4Ex8TEQysXi6sDRhMHx3WyYgZAkeYMz ggbyNcuCjLA29L0wgiWubBxiK0d0GbENg5umjBQ0f4NmI86lif5DQyrHSEO1Ic0V4Htp i+0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=W6LcE5XkFDYxzKuPQoc2YgCbyU2hSzaIyUPaJY1l3Yw=; b=X2gzm4utOBp99r/ckGg7WlkcGoQ6qE889d8xpJzC/1/B50Ih0Tww2Rs0FSsmiZ7vf6 ffPP/mwwj0MTpJzmm+jvrN7rDmdtAjPHKHWUjZXAvit0QGWrDOja17ImfbrH1XmYDihA gKuDZa4hkpN8eqhbdaTEOV6+sURv+RYt+Q/VCjqhOVn4iRZr138KNpaedd/QRyJ0Z1p3 tAQBTYErg2OtRx0AP2DJfhKKeg81BYeTXTQ3NWhdcY9vZwmSVb5ue2Sm589maBtTIdVU t2NS/tb3AxQ2kAsS6bVl4A9PlLShXW+tqwfzSuG7lKLnhG+EsAoNxm602uPpFuCJHIrW dbkg== X-Gm-Message-State: APzg51B6BmWfVLIwTjuK5YzUaZk3GXQfZujsLizFHUMb2I5d/L6JrPlW ulk0zHXz1hEcJkHF3SkpDQxPqlj338opvMt84tIqOw== X-Google-Smtp-Source: ANB0VdY/OqB0t80nh44WQxNp97xFJlnuMYdk5mJOyg2tNU/2IPZ2PamSqSNVzEHwzPUbb3nDpFfQE9vhRxneQF185/4= X-Received: by 2002:ae9:e64b:: with SMTP id x11-v6mr7452326qkl.222.1534928528880; Wed, 22 Aug 2018 02:02:08 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Oleksandr Shulgin Date: Wed, 22 Aug 2018 11:01:42 +0200 Message-ID: Subject: Re: [Cassandra] nodetool compactionstats not showing pending task. To: arodrime@gmail.com Cc: Abhishek.Maheshwari@timesinternet.in, kurt@instaclustr.com, User Content-Type: multipart/alternative; boundary="000000000000b8776905740265ec" --000000000000b8776905740265ec Content-Type: text/plain; charset="UTF-8" On Fri, May 5, 2017 at 1:20 PM Alain RODRIGUEZ wrote: > Sorry to hear the restart did not help. > Hi, We are hitting the same issue since a few weeks on version 3.0.16. Normally, restarting an affected node helps, but this is something we would like to avoid doing. What makes it worse for us is that Cassandra Reaper stops scheduling new repair jobs if it sees that the node have more than 20 pending compaction tasks. We could bump this threshold, but in general the estimate could be more accurate (or the actual tasks should be started timely). Maybe try to monitor through JMX with 'org.apache.cassandra.db:type=CompactionManager', >> attribute 'Compactions' or 'CompactionsSummary' > > > What is this attribute showing? > For example, I have right now a node showing "pending tasks: 16" and no compaction running. Here is the JMX output (well, via Jolokia): "Compactions": [], "CoreCompactorThreads": 1, "CompactionSummary": [], "MaximumCompactorThreads": 1, "CoreValidationThreads": 1, "MaximumValidatorThreads": 2147483647, Here is the Apache Cassandra Jira: > https://issues.apache.org/jira/browse/CASSANDRA. You search here > > ( > https://issues.apache.org/jira/browse/CASSANDRA-12529?jql=project%20%3D%20CASSANDRA%20AND%20text%20~%20%22pending%20compactions%22%20ORDER%20BY%20created%20DESC), > for example. > I believe this is a separate issue. There some actual compaction tasks are running, but not making progress. And we never TRUNCATEd our tables, and for sure not recently. Any more pointers on how to debug this? Regards, -- Alex --000000000000b8776905740265ec Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Fri, May 5,= 2017 at 1:20 PM Alain RODRIGUEZ <= arodrime@gmail.com> wrote:
Sorry to hear the restart did not h= elp.=C2=A0

Hi,

We are hitting the same issue s= ince a few weeks on version 3.0.16.
Normally, restarting an affected n= ode helps, but this is something we would like to avoid doing.

What makes it wors= e for us is that Cassandra Reaper stops scheduling new repair jobs if it se= es that the node have more than 20 pending compaction tasks.=C2=A0 We could= bump this threshold, but in general the estimate could be more accurate (o= r the actual tasks should be started timely).

<= div>
Maybe try to monitor through JMX with 'org.apache.cassandra= .db:type=3DCompactionManager', = attribute 'Compactions' or 'CompactionsSummary'

What is this attribute showing?
=

For example, I have right now a node= showing "pending tasks: 16" and no compaction running.=C2=A0 Her= e is the JMX output (well, via Jolokia):

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "Compactions": [],=
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 "CoreCompactorThreads": 1,
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 "CompactionSummary": [],
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 "MaximumCom= pactorThreads": 1,
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 "MaximumValidatorThreads": 2147483647= ,


I beli= eve this is a separate issue.=C2=A0 There some actual compaction tasks are = running, but not making progress.=C2=A0 And we never TRUNCATEd our tables, = and for sure not recently.

Any more pointers on ho= w to debug this?

Regards,
--
A= lex

--000000000000b8776905740265ec--