From user-return-61242-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Mon Jun 4 16:57:53 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 2C985180636 for ; Mon, 4 Jun 2018 16:57:52 +0200 (CEST) Received: (qmail 57061 invoked by uid 500); 4 Jun 2018 14:57:51 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 57051 invoked by uid 99); 4 Jun 2018 14:57:51 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Jun 2018 14:57:51 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 1663018024D for ; Mon, 4 Jun 2018 14:57:51 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.879 X-Spam-Level: * X-Spam-Status: No, score=1.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id AdR0miWzoHjr for ; Mon, 4 Jun 2018 14:57:48 +0000 (UTC) Received: from mail-vk0-f49.google.com (mail-vk0-f49.google.com [209.85.213.49]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id D9E9C5F1EE for ; Mon, 4 Jun 2018 14:57:47 +0000 (UTC) Received: by mail-vk0-f49.google.com with SMTP id o17-v6so7759043vka.2 for ; Mon, 04 Jun 2018 07:57:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=fNvp7BIfsVee1/QsOPCGHPsAAP1CbnDgbWUfTIHz330=; b=ZwSzQPxAatG4HyrcQgnK8MJtBXBh0Mae0Xs2IKLxMUhD4STeP15Y6cz/jQBFWzB8K6 JZcy0l7ECMPu6x00VwQ5GmHztbJB93/zXY/0iiHEDyE2Ip+RPCN4FkOtbqD6C14N2y1Z q5LXUmUaMt7qwPspFrDygtXk/TUKsQWg1zBE6GAPCZhXRcnmRzajh9wUbnZaiTl8WkGL 5qL8HTNpwiQwx8JLmk2zHiqp2fJW40MI0clCDSoe3xi1I1a9w/fSbWuqH9L7a628COom F0XAA7W+FviDBPC22OwLtAqRYWaUe6RKX0Pg0eiiY1SXnlv2wZl8ZyOHouV5Um7PXGpm /BNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=fNvp7BIfsVee1/QsOPCGHPsAAP1CbnDgbWUfTIHz330=; b=fWzmobsT7yYkrdFA4WDtLC5DopIKsl9Fcrwir4ae5xabDu1MEc3WEroB6uiMEVnBdQ CGdN+tP3Yj+FKyVgo80nwMepUqQFVpiR6OTBL0td4lVSNZiTHQ0n1KGVdtqtWr2Vvnml /OHLFnXbqaSlszQKQ9kckIW/G4ahYF4HMCtEkzN/ouuxoTabxpbe5ZNeRQk6HstGciTx EptmhBRASQ3iHX81aqKPzVQprFovuoIrCn7FE4CdwB+6nVMXzhRaOID/E7OW9/KzzHr/ 5N47a8K0A3Hh6FmOeh0qTSG4UK0VzOA5gi0rpxDPE2QSZAFU0FMV73tddpgoJugyKep7 kWMw== X-Gm-Message-State: ALKqPwfAnNmVdQF6M790XDwfyvvpi3eSeVN/XgMsUw/2/cte36bng0RE F+8/Xwees6a8q4zUy/KK5MIpyittWF2Wp/0Fu8BhBWZN X-Google-Smtp-Source: ADUXVKKVd9fHlhpw0RDUExVSpKmF+PMh7lGnbL5TFNyGPOQp0rf0lrqQSuyDwIXnxuQkP+Tuazgwhc7wp5rbzxl2GiI= X-Received: by 2002:a1f:3697:: with SMTP id d145-v6mr13359862vka.79.1528124260423; Mon, 04 Jun 2018 07:57:40 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a67:3602:0:0:0:0:0 with HTTP; Mon, 4 Jun 2018 07:57:20 -0700 (PDT) In-Reply-To: References: <163b6bd710d.f05fcaad628.4539626093092721364@zoho.com> <163beaae995.bf1b293427284.2163883682340031187@zoho.com> From: Alain RODRIGUEZ Date: Mon, 4 Jun 2018 15:57:20 +0100 Message-ID: Subject: Re: how to immediately delete tombstones To: "user cassandra.apache.org" Content-Type: multipart/alternative; boundary="000000000000b71f2b056dd227e2" --000000000000b71f2b056dd227e2 Content-Type: text/plain; charset="UTF-8" Hello, When you don't have any disk space anymore (or not much), there are things you can do : *Make some space:* - Remove snapshots (nodetool clearsnapshot) - Remove any heap dump that might be stored there - Remove *old* -tmp- SSTables that could still be around. - Truncate unused table / data: Truncate does not create tombstone, but removes the files, after creating a snapshot (default behavior). Thus truncating/dropping a table and removing the snapshot could produce immediate disk space availability. - Anything else than SSTable in this disk that can be removed? OR *Add some space:* - Add some disk space temporary (EBS, physical disk) - Make sure to clean tombstones ('uncheck_tombstone_compaction: true' often helped) - Wait for tombstones to be compacted with all the data it shadows and disk space to reduce - Remove extra disk OR *Play around, at the edge*: - Look at the biggest SSTables that you can actually compact (be aware of the compressed vs uncompressed sizes when monitoring - I think the thing was that 'nodetool compactionstats -H' shows the values uncompressed) - Use sstablemetadata to determine the ratio of the data that is droppable - Run user defined compaction on these sstables specifically - If it works and there is more disk space available, reproduce with bigger sstables. In some complex cases removing commit logs helped as well, but this is riskier already as it would be playing with consistency/durability considerations. I'm using cassandra on a single node. > I would not play with commit logs with a single-node setup. But I imagine it is not a production 'cluster' either. C*heers, ----------------------- Alain Rodriguez - @arodream - alain@thelastpickle.com France / Spain The Last Pickle - Apache Cassandra Consulting http://www.thelastpickle.com 2018-06-02 8:29 GMT+01:00 Nitan Kainth : > You can compact selective sstables using jmx Call. > > Sent from my iPhone > > On Jun 2, 2018, at 12:04 AM, onmstester onmstester > wrote: > > Thanks for your replies > But my current situation is that i do not have enough free disk for my > biggest sstable, so i could not run major compaction or nodetool > garbagecollect > > Sent using Zoho Mail > > > ---- On Thu, 31 May 2018 22:32:32 +0430 *Alain RODRIGUEZ > >* wrote ---- > > > --000000000000b71f2b056dd227e2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello,

When you don't have any disk= space anymore (or not much), there are things you can do :

<= b>Make some space:

- Remove snapshots (nodetool=C2=A0= clearsnapshot)
- Remove any heap dump that might be stored there
- Remove old -tmp- SSTables that could still be around.
- Truncate unused table / data: Truncate does not create tombstone, but re= moves the files, after creating a snapshot (default behavior). Thus truncat= ing/dropping a table and removing the snapshot could produce immediate disk= space availability.
- Anything else than SSTable in this disk th= at can be removed?

OR

Add some space:

- Add some disk space temporary (EBS, phy= sical disk)
- Make sure to clean tombstones ('uncheck_tombsto= ne_compaction: true' often helped)
- Wait for tombstones to b= e compacted with all the data it shadows and disk space to reduce
- Remove extra disk

OR

<= b>Play around, at the edge:

- Look at the biggest SSTable= s that you can actually compact (be aware of the compressed vs uncompressed= sizes when monitoring - I think the thing was that 'nodetool compactio= nstats=C2=A0-H' shows the values uncompressed)
- Use sstablem= etadata to determine the ratio of the data that is droppable
- Ru= n user defined=C2=A0compaction on these sstables specifically
- I= f it works and there is more disk space available, reproduce with bigger ss= tables.

In some complex cases removing commit logs= =C2=A0helped as well, but this is riskier already as it would be playing wi= th consistency/durability considerations.

I'm using cassandra on a single node.
=

I would not play with commit logs with a single-node se= tup. But I imagine it is not a production 'cluster' either.

C*heers,
-----------------------
Alain Rodriguez - @arodream - a= lain@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting<= div class=3D"gmail_extra">
2018-06-02 8:29 GM= T+01:00 Nitan Kainth <nitankainth@gmail.com>:
You can compact selective sstable= s using jmx Call.

= Sent from my iPhone

On Jun 2, 2018, at= 12:04 AM, onmstester onmstester <onmstester@zoho.com> wrote:

Thanks for your replies
But my = current situation is that i do not have enough free disk for my biggest sst= able, so i could not run major compaction or nodetool garbagecollect
<= div>

Sent using Z= oho Mail



---- On Thu, 31 May 2018 22:32:32 +0430=C2=A0Alain RODRIGUEZ <= ;arodrime@gmail.com= > wrote ----



--000000000000b71f2b056dd227e2--