Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F060018D39 for ; Tue, 8 Dec 2015 00:14:43 +0000 (UTC) Received: (qmail 18379 invoked by uid 500); 8 Dec 2015 00:14:37 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 18324 invoked by uid 500); 8 Dec 2015 00:14:36 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 18300 invoked by uid 99); 8 Dec 2015 00:14:36 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Dec 2015 00:14:36 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 6CE621A0C2E for ; Tue, 8 Dec 2015 00:14:36 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.899 X-Spam-Level: ** X-Spam-Status: No, score=2.899 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id U8gdv2aM3NZV for ; Tue, 8 Dec 2015 00:14:31 +0000 (UTC) Received: from mail-qg0-f51.google.com (mail-qg0-f51.google.com [209.85.192.51]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id 6021542BCA for ; Tue, 8 Dec 2015 00:14:31 +0000 (UTC) Received: by qgcc31 with SMTP id c31so3340375qgc.3 for ; Mon, 07 Dec 2015 16:14:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=TXNPr240WyD+l01tW6kwXhZel2YfCOxgzS4MYg9Okb8=; b=XnP+y8oFu/hfbdLpe5vjLc8o16VDmkQGRJY3ZYcDqXpcXhClQyjFTizRDSXxfyAxUR a5T8oKQnq6sEfRrj2MAG4xehpXbj//VuRWZcLnL3t1XRfJu6pffRftIeOzuKTNYmffaD mwHhJnPhHRLjz2Xy6BRel7KF3FYJB32qgA2h8JreXgtq8q6+VfZSGJOneIQJc5U31Fjh l1bpMDPDN71s82n+Vpk7bust+UR/JjOnD1Ve04n1jCz30LnkmFABw3r88jv57bfkLJWi PeiB9ZzVd9DSF92D2RwUmBKUDZSZjN9BJai3FRHF9xAQp9bEnzkppkilaeHUjuCKD8T3 MN+g== MIME-Version: 1.0 X-Received: by 10.140.99.86 with SMTP id p80mr310197qge.97.1449533665466; Mon, 07 Dec 2015 16:14:25 -0800 (PST) Received: by 10.55.106.198 with HTTP; Mon, 7 Dec 2015 16:14:25 -0800 (PST) Received: by 10.55.106.198 with HTTP; Mon, 7 Dec 2015 16:14:25 -0800 (PST) In-Reply-To: <7C10EF71-435C-477E-BCE5-8D3720A4088F@crowdstrike.com> References: <7C10EF71-435C-477E-BCE5-8D3720A4088F@crowdstrike.com> Date: Mon, 7 Dec 2015 19:14:25 -0500 Message-ID: Subject: Re: lots of tombstone after compaction From: Kai Wang To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a113ad8463861b8052657db06 --001a113ad8463861b8052657db06 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Rob and Jeff, Thank you. It makes sense. I am on 2.1.10 and will upgrade to 2.1.12. On Dec 7, 2015 7:05 PM, "Jeff Jirsa" wrote: > https://issues.apache.org/jira/browse/CASSANDRA-7953 > > https://issues.apache.org/jira/browse/CASSANDRA-10505 > > There are buggy versions of cassandra that will multiple tombstones durin= g > compaction. 2.1.12 SHOULD correct that, if you=E2=80=99re on 2.1. > > > > From: Kai Wang > Reply-To: "user@cassandra.apache.org" > Date: Monday, December 7, 2015 at 3:46 PM > To: "user@cassandra.apache.org" > Subject: lots of tombstone after compaction > > I bulkloaded a few tables using CQLSStableWrite/sstableloader. The data > are large amount of wide rows with lots of null's. It takes one day or tw= o > for the compaction to complete. sstable count is at single digit. Maximum > partition size is ~50M and mean size is ~5M. However I am seeing frequent > read query timeouts caused by tombstone_failure_threshold (100000). These > tables are basically read-only. There're no writes. > > I just kicked off compaction on those tables using nodetool. Hopefully it > can remove those tombstones. But is it normal to have these many tombston= es > after the initial compactions? Is this related to the fact the original > data has lots of nulls? > > Thanks. > --001a113ad8463861b8052657db06 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

Rob and Jeff,

Thank you. It makes sense. I am on 2.1.10 and will upgrade t= o 2.1.12.

On Dec 7, 2015 7:05 PM, "Jeff Jirsa" &= lt;jeff.jirsa@crowdstrike.com= > wrote:


There are buggy versions of cassandra that wi= ll multiple tombstones during compaction. 2.1.12 SHOULD correct that, if yo= u=E2=80=99re on 2.1.
=

<= /u>

From: Kai Wang
Reply-To: "user@cassandra.apache.org"<= br>Date: Monday, December 7, 2015 = at 3:46 PM
To: "user@cassandra.apache.= org"
Subject: lots of = tombstone after compaction

I bulkloaded a few tables using CQLSStableWrite/sstableloader.= The data are large amount of wide rows with lots of null's. It takes o= ne day or two for the compaction to complete. sstable count is at single di= git. Maximum partition size is ~50M and mean size is ~5M. However I am seeing frequent read query timeouts caused by to= mbstone_failure_threshold (100000). These tables are basically read-only. T= here're no writes.

I just kicked off compaction on those tables using nodetool. Hopefully it c= an remove those tombstones. But is it normal to have these many tombstones = after the initial compactions? Is this related to the fact the original dat= a has lots of nulls?

Thanks.
--001a113ad8463861b8052657db06--