Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 277DE200C7D for ; Tue, 16 May 2017 15:23:15 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 261B6160BAC; Tue, 16 May 2017 13:23:15 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 4489E160B9D for ; Tue, 16 May 2017 15:23:14 +0200 (CEST) Received: (qmail 30106 invoked by uid 500); 16 May 2017 13:23:07 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 30096 invoked by uid 99); 16 May 2017 13:23:07 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 May 2017 13:23:07 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 170BB181944 for ; Tue, 16 May 2017 13:23:07 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.629 X-Spam-Level: ** X-Spam-Status: No, score=2.629 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id dPDE6yytPhsx for ; Tue, 16 May 2017 13:23:04 +0000 (UTC) Received: from mail-it0-f52.google.com (mail-it0-f52.google.com [209.85.214.52]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id B70A35FB55 for ; Tue, 16 May 2017 13:23:03 +0000 (UTC) Received: by mail-it0-f52.google.com with SMTP id g126so87394109ith.0 for ; Tue, 16 May 2017 06:23:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=LczeSDk1Ts49aNaTA2cXO3bTXJ+RQs/X61AY8M0eXek=; b=glqIVSJYHS8LNrxsweu9tcc/MIkAeUXTyEji1x4wKp6TZReICBM9N656Q67Gm3cK38 /4kymhvIt/+cbcGY1rjqEiZHDySlxRIU9Dm8IfePa9D0XShQ41/4z1gfaoTubqOgkTyA LRrHyflOl345ctmqCXL+HWXvjhVMr3BYWirQg7DXZjcznyAZ8k7Cp2cupSK33IRHY9Sv gdlT9ZaibWFCsk3ZvhE8Rby62z9GOkCddekE6H9Vb1BlfizVhz63k4PFKL9wFQhu+zjD r6sJ7XSSjthgK2RMoYbLCNTFKs0+pBxIygwh6RET81VMosOh7bL90HhiN++EqKYNTzgB jZsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=LczeSDk1Ts49aNaTA2cXO3bTXJ+RQs/X61AY8M0eXek=; b=PlsCv7jOHvMnJWItBeUyRnzCutd1TP4cLt3YkcQa8W+JlQ/fWPiXVTBiuYcFk0Qoys b2sLinvdin6CIB+xV78h5/BU0jqCvJKrSDgD6qHHGdb3cyYJqhqojRcrLKaxvyqcnEv9 2LO/ctRzmP5O2YUM/IJVXUrQvgSoghvFUiALsS6FCw7wBANAjoftKSiA/aH0VAUtMEQ1 1O84uPJZrGVT6ggWczCLuqk4Nc6LhcwDSmVPgYTlM6SW4l8K5hnX5IK1XjPUu2ijdbp9 h5PESboMurCwCZMbis9tEkoV7eGh27A2UDjZb24uAmW55p0nPPk+fBBbieqgEw/BpdYm 68Sw== X-Gm-Message-State: AODbwcCHCFXukpfsHDU7RXpNU0qXfGG7csemz+X+Uh5tw4g9iZNKkVcB 6OpmTse0KlIvnURZKDF/ZQf5YIGy5w== X-Received: by 10.36.211.130 with SMTP id n124mr10663640itg.107.1494940983278; Tue, 16 May 2017 06:23:03 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.28.19 with HTTP; Tue, 16 May 2017 06:23:02 -0700 (PDT) In-Reply-To: References: <17A1047A-4FFC-4E47-BC06-2F0F428BC00B@bamlabs.com> From: varun saluja Date: Tue, 16 May 2017 18:53:02 +0530 Message-ID: Subject: Re: Long running compaction on huge hint table. To: Nitan Kainth , user@cassandra.apache.org Content-Type: multipart/alternative; boundary="001a1149ac9644c31f054fa41285" archived-at: Tue, 16 May 2017 13:23:15 -0000 --001a1149ac9644c31f054fa41285 Content-Type: text/plain; charset="UTF-8" Hi, Could see intermittent GCs and mutation drops. *System log reports:* INFO [Service Thread] GCInspector.java:252 - ParNew GC in 3816ms. CMS Old Gen: 4663180720 -> 5520012520; Par Eden Space: 1718091776 -> 0; Par Survivor Space: 0 -> 214695936 INFO [ScheduledTasks:1] MessagingService.java:888 - 228 MUTATION messages dropped in last 5000ms PS: As of now , there is no significant load on our cluster. The only load is of these hints been replayed. Can you Please help. Regards, Varun Saluja On 16 May 2017 at 18:28, varun saluja wrote: > Hi Nitan, > > Thanks for response. > > Yes, I could see mutation drops and increase count in system.hints. Is > there any way , i can proceed to truncate hints like using nodetool > truncatehints. > > > Regards, > Varun Saluja > > On 16 May 2017 at 17:52, Nitan Kainth wrote: > >> Do you see mutation drops? >> Select count from system.hints; is it increasing? >> >> Sent from my iPhone >> >> On May 16, 2017, at 5:52 AM, varun saluja wrote: >> >> Hi Experts, >> >> We are facing issue on production cluster. Compaction on system.hint >> table is running from last 2 days. >> >> >> pending tasks: 1 >> compaction type keyspace table completed total >> unit progress >> Compaction system hints 20623021829 >> *877874092407* bytes 2.35% >> Active compaction remaining time : 0h27m15s >> >> >> Active compaction remaining time shows in minutes. But, this is job is >> running like indefinitely. >> >> We have 3 node cluster V 2.1.7. And we ran write intensive job last week >> on particular table. >> Compaction on this table finished but hint table size is growing >> continuously. >> >> Can someone Please help me. >> >> >> Thanks & Regards, >> Varun Saluja >> >> > --001a1149ac9644c31f054fa41285 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi,

Could see intermittent GCs and muta= tion drops.

System log reports:
<= br>
INFO =C2=A0[Service Thread] =C2=A0GCInspector.java:252 -= ParNew GC in 3816ms.=C2=A0 CMS Old Gen: 4663180720 -> 5520012520; Par E= den Space: 1718091776 -> 0; Par Survivor Space: 0 -> 214695936
<= div>INFO =C2=A0[ScheduledTasks:1] MessagingService.java:888 - 228 MUTATION = messages dropped in last 5000ms

PS: As of no= w , there is no significant load on our cluster. The only load is of these = hints been replayed.

Can you Please help.

Regards,
Varun Saluja

On 16 May 2017 at 18:28, varun s= aluja <saluja50@gmail.com> wrote:
Hi Nitan,

Thanks for response.=

Yes, I could see mutation drops and increase coun= t in system.hints. Is there any way , i can proceed to truncate hints like = using nodetool truncatehints.


Regar= ds,
Varun Saluja

On 16 May 2017= at 17:52, Nitan Kainth <nitan@bamlabs.com> wrote:
Do you see mutation drops?<= /div>
Select count from system.hints; is it increasing?

Sent from my i= Phone

On May 16, = 2017, at 5:52 AM, varun saluja <saluja50@gmail.com> wrote:

Hi Exp= erts,

<= span style=3D"font-size:12.8px">We are facing issue on production cluster. = Compaction on system.hint table is running from last 2 days.


pending tasks: 1
=C2=A0 =C2=A0compact= ion type=C2=A0 =C2=A0keyspace=C2=A0 =C2=A0table=C2=A0 =C2=A0 =C2=A0complete= d=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 total=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 unit=C2=A0 =C2=A0progress=
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Compaction=C2=A0 =C2=A0 =C2=A0system= =C2=A0 =C2=A0hints=C2=A0 =C2=A020623021829=C2=A0 =C2=A0877874092407= =C2=A0 =C2=A0bytes=C2=A0 =C2=A0 =C2=A0 2.35%
Active compaction remaining time := =C2=A0 =C2=A00h27m15s


Active compaction remaining time shows in minutes.=C2=A0 But, this is j= ob is running like indefinitely.

We have 3 node c= luster V 2.1.7. And we ran=C2=A0 write intensive job last week on particula= r table.
Compaction on this table finished but hint table size is growing contin= uously.

Can someone Please help me.


Thanks & Regards,Varun Saluja


--001a1149ac9644c31f054fa41285--