Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 9ECB8200C7D for ; Tue, 16 May 2017 14:58:20 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 9D602160BAC; Tue, 16 May 2017 12:58:20 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id BDCE6160B9D for ; Tue, 16 May 2017 14:58:19 +0200 (CEST) Received: (qmail 50331 invoked by uid 500); 16 May 2017 12:58:18 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 50320 invoked by uid 99); 16 May 2017 12:58:18 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 16 May 2017 12:58:18 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id D04841883A2 for ; Tue, 16 May 2017 12:58:17 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.929 X-Spam-Level: * X-Spam-Status: No, score=1.929 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id QpmehsQja2BR for ; Tue, 16 May 2017 12:58:16 +0000 (UTC) Received: from mail-it0-f50.google.com (mail-it0-f50.google.com [209.85.214.50]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 83F9C5FBC1 for ; Tue, 16 May 2017 12:58:16 +0000 (UTC) Received: by mail-it0-f50.google.com with SMTP id o5so61320783ith.1 for ; Tue, 16 May 2017 05:58:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=rcA6vsDtrMkTZjwcA82Qs7fN2AvceGWbe/zKJH0AWwk=; b=CXv+XNPBSsPPxipLCQJDVzNH1QRvnhbwzI5gj/fchVBsBLH2JMD5qYsTQPnQLCkdku RoMu0Pwaa0aLvi3BN1l0LlobD4+TOwVe18YQst4pC5q8AQevT0Qpuzx+olgWwIwHgy6T IsqstrleDeosGgM9EjzWbsOyJOYttfBWKxJvHn3BBkemtyQBQ/x2ftHAjv4Bg0Pu4FvC OR2GnRDsRQ7M0pgQIYpVAQtyLTvRdC/dcoZ1b7aDrtn9PQZPhl6tYRdW8I0wyBoKPKBi U5gLfFU8qO7eYxyfeI97cPZmWxBeijCJJQPsfQpmgI0bO0sk59jqnS2Py6qgtXq87q4E w0lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=rcA6vsDtrMkTZjwcA82Qs7fN2AvceGWbe/zKJH0AWwk=; b=r2N9goJUmWr06jRKkyJtyQQmA/Gs2ZpJ2Nj+32DOf0FoOl60+p5MsNwjG/9KNfexcW F7upZA9BmUbVyUAt9jImJ52FAi1d8ZLIPa1ULunWt9nrUTw2tochhuUOXFPN3CVxCcCZ tgFkb5P53T5Bh7qciGVXg5noQMQC1pIeyVDm4DUfGNAnnq/IeKokLcZtzMCCHeR3IxxP akohgea2OoviB3ZCmRlKA2AKZcLUkjk2E1GvJTiICX/v5fF1RmbG3W0veHDOzVVLjHpB +VdPIsS6FEU0LvgLG8SFbwwA0iWfBzUmEL9+7aj+ldpti7hhA8i2rg9fpjxm69waIJBP CnlQ== X-Gm-Message-State: AODbwcC64N0NkzLJ3vlIeNCLzcrUEyj0qZ4wpeNcDmM56wBvUZGmBvGO 3SQTEzyzKmicO6pR+bFtiAQlt/Ei5g== X-Received: by 10.36.245.201 with SMTP id k192mr11220142ith.104.1494939490588; Tue, 16 May 2017 05:58:10 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.28.19 with HTTP; Tue, 16 May 2017 05:58:09 -0700 (PDT) In-Reply-To: <17A1047A-4FFC-4E47-BC06-2F0F428BC00B@bamlabs.com> References: <17A1047A-4FFC-4E47-BC06-2F0F428BC00B@bamlabs.com> From: varun saluja Date: Tue, 16 May 2017 18:28:09 +0530 Message-ID: Subject: Re: Long running compaction on huge hint table. To: Nitan Kainth Cc: user@cassandra.apache.org Content-Type: multipart/alternative; boundary="94eb2c041c704c1ea2054fa3b985" archived-at: Tue, 16 May 2017 12:58:20 -0000 --94eb2c041c704c1ea2054fa3b985 Content-Type: text/plain; charset="UTF-8" Hi Nitan, Thanks for response. Yes, I could see mutation drops and increase count in system.hints. Is there any way , i can proceed to truncate hints like using nodetool truncatehints. Regards, Varun Saluja On 16 May 2017 at 17:52, Nitan Kainth wrote: > Do you see mutation drops? > Select count from system.hints; is it increasing? > > Sent from my iPhone > > On May 16, 2017, at 5:52 AM, varun saluja wrote: > > Hi Experts, > > We are facing issue on production cluster. Compaction on system.hint table > is running from last 2 days. > > > pending tasks: 1 > compaction type keyspace table completed total > unit progress > Compaction system hints 20623021829 *877874092407* > bytes 2.35% > Active compaction remaining time : 0h27m15s > > > Active compaction remaining time shows in minutes. But, this is job is > running like indefinitely. > > We have 3 node cluster V 2.1.7. And we ran write intensive job last week > on particular table. > Compaction on this table finished but hint table size is growing > continuously. > > Can someone Please help me. > > > Thanks & Regards, > Varun Saluja > > --94eb2c041c704c1ea2054fa3b985 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Nitan,

Thanks for response.

Yes, I could see mutation drops and increase count in syst= em.hints. Is there any way , i can proceed to truncate hints like using nod= etool truncatehints.


Regards,
=
Varun Saluja

On 16 May 2017 at 17:52, Nitan Kainth <nitan@bamlabs.com&g= t; wrote:
D= o you see mutation drops?
Select count from system.hints; is it increasing?

Sent from= my iPhone

On May 16, 2017, at 5:52 AM= , varun saluja <= saluja50@gmail.com> wrote:

Hi Experts,

We are facing issue on production cluster. Compaction on sy= stem.hint table is running from last 2 days.


pending tasks: 1
=C2=A0 =C2=A0compaction type=C2=A0 = =C2=A0keyspace=C2=A0 =C2=A0table=C2=A0 =C2=A0 =C2=A0completed=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 total=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 unit=C2=A0 =C2=A0progress
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 Compaction=C2=A0 =C2=A0 =C2=A0system=C2=A0 =C2=A0h= ints=C2=A0 =C2=A020623021829=C2=A0 =C2=A0877874092407=C2=A0 =C2=A0by= tes=C2=A0 =C2=A0 =C2=A0 2.35%
Active compaction remaining time :=C2=A0 =C2=A00h2= 7m15s

<= br style=3D"font-size:12.8px">Active compa= ction remaining time shows in minutes.=C2=A0 But, this is job is running li= ke indefinitely.

We have 3 node cluster V 2.1.7. = And we ran=C2=A0 write intensive job last week on particular table.<= br style=3D"font-size:12.8px">Compaction o= n this table finished but hint table size is growing continuously.
Can someone Please help me.


Thanks & Regards,
Varun Saluja

--94eb2c041c704c1ea2054fa3b985--