Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C041E18AB6 for ; Mon, 6 Jul 2015 21:47:06 +0000 (UTC) Received: (qmail 88790 invoked by uid 500); 6 Jul 2015 21:47:03 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 88745 invoked by uid 500); 6 Jul 2015 21:47:03 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 88735 invoked by uid 99); 6 Jul 2015 21:47:03 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 06 Jul 2015 21:47:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 403CD1827F2 for ; Mon, 6 Jul 2015 21:47:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3 X-Spam-Level: *** X-Spam-Status: No, score=3 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=3, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id pe5i4jzlk8TM for ; Mon, 6 Jul 2015 21:46:54 +0000 (UTC) Received: from mail-pa0-f43.google.com (mail-pa0-f43.google.com [209.85.220.43]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id 137964C0EA for ; Mon, 6 Jul 2015 21:46:52 +0000 (UTC) Received: by pabvl15 with SMTP id vl15so101453736pab.1 for ; Mon, 06 Jul 2015 14:46:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-type:message-id:mime-version :subject:date:references:to:in-reply-to; bh=016f+4nsyx9+7fMJubDtp1YUIpcJoKGh+3IoAO6NToM=; b=mCou6m7yTns2B4R8tw0AzGuhyZr2tyj15CdV1rBQsGYuRH3XnWc5E0I8rB8+vH23pM tborg2Ta7swEwXgz2G6kBa6UVKGVwxq3lneC6aFlQeyezVuVA7wDxYIT3UZl9xEW6zKG Tqk96hqXysaXfgbnqEDXavBrX+4L6BGVjTRUIr7/3VzcEF7LQAI/Ml3Nuy6mn4vS393/ PmbINBxOeK1ccULWDwdmeCi9JXhPg3BLQB0rD+aPY8bQVVesHSfWxmoRWxWeILrMnxdC oHR+iqJ/OHhFzetnUccRMIZggKzJS9VuAhguM16clpmUPmSmSJRexEJpu13WTApIzw2S 1uyw== X-Gm-Message-State: ALoCoQnMOlP9NaxvUVCrhOw8NCxdp5uNMxAhAvhT1NYl08RkEUu7rEZbJdpZ9tlOqxM9Mbx4aub5 X-Received: by 10.70.1.102 with SMTP id 6mr1889000pdl.32.1436219211330; Mon, 06 Jul 2015 14:46:51 -0700 (PDT) Received: from [172.31.99.243] ([199.87.81.121]) by mx.google.com with ESMTPSA id i10sm19402754pdr.78.2015.07.06.14.46.49 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 06 Jul 2015 14:46:50 -0700 (PDT) From: Jeff Ferland Content-Type: multipart/alternative; boundary="Apple-Mail=_6484BDBD-DE8A-47BE-B43B-0F16EE610343" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2098\)) Subject: Re: Compaction issues, 2.0.12 Date: Mon, 6 Jul 2015 14:46:41 -0700 References: To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.2098) --Apple-Mail=_6484BDBD-DE8A-47BE-B43B-0F16EE610343 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 I=E2=80=99ve seen the same thing: = https://issues.apache.org/jira/browse/CASSANDRA-9577 = I=E2=80=99ve had cases where a restart clears the old tables, and I=E2=80=99= ve had cases where a restart considers the old tables to be live. > On Jul 6, 2015, at 1:51 PM, Robert Coli wrote: >=20 > On Mon, Jul 6, 2015 at 1:46 PM, Jeff Williams = > wrote: > 1) Cassandra version is 2.0.12.=20 > =20 > 2) Interesting. Looking at JMX org.apache.cassandra.db -> = ColumnFamilies -> trackcontent -> track_content -> Attributes, I get: >=20 > LiveDiskSpaceUsed: 17788740448 , i.e. ~17GB > LiveSSTableCount: 3 > TotalDiskSpaceUsed: 55714084629, i.e. ~55GB >=20 > So it obviously knows about the extra disk space even though the = "live" space looks correct. I couldn't find anything to identify the = actual files though. >=20 > That's what I would expect. > =20 > 3) So that was even more interesting. After restarting the cassandra = daemon, the sstables were not deleted and now the same JMX attributes = are: >=20 > LiveDiskSpaceUsed: 55681040579, i.e. ~55GB > LiveSSTableCount: 8 > TotalDiskSpaceUsed: 55681040579, i.e. ~55GB >=20 > So some of my non-live tables are back live again, and obviously some = of the big ones!! >=20 > This is permanently fatal to consistency; sorry if I was not clear = enough that if they were not live, there was some risk of Cassandra = considering them live again upon restart. >=20 > If I were you, I would either stop the node and remove the files you = know shouldn't be live or do a major compaction ASAP.=20 >=20 > The behavior you just encountered sounds like a bug, and it is a = rather serious one. SSTables which should be dead being marked live is = very bad for consistency.=20 >=20 > Do you see any exceptions in your logs or anything? If you can repro, = you should file a JIRA ticket with the apache project... >=20 > =3DRob >=20 --Apple-Mail=_6484BDBD-DE8A-47BE-B43B-0F16EE610343 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
I=E2=80=99ve seen the same thing: https://issues.apache.org/jira/browse/CASSANDRA-9577
<= div class=3D"">
I=E2=80=99ve had = cases where a restart clears the old tables, and I=E2=80=99ve had cases = where a restart considers the old tables to be live.

On = Jul 6, 2015, at 1:51 PM, Robert Coli <rcoli@eventbrite.com> wrote:

On Mon, = Jul 6, 2015 at 1:46 PM, Jeff Williams <jeffw@wherethebitsroam.com> wrote:
1) Cassandra version is = 2.0.12. 
 
2) Interesting. Looking at JMX = org.apache.cassandra.db -> ColumnFamilies -> trackcontent -> = track_content -> Attributes, I get:

LiveDiskSpaceUsed: 17788740448, i.e. ~17GB
LiveSSTableCount: 3
TotalDiskSpaceUsed: = 55714084629, i.e. ~55GB

So it obviously knows about the extra disk space even though = the "live" space looks correct. I couldn't find anything to identify the = actual files though.

That's what I would expect.
 
3) So that was even more interesting. After = restarting the cassandra daemon, the sstables were not deleted and now = the same JMX attributes are:

LiveDiskSpaceUsed: = 55681040579, i.e. ~55GB
LiveSSTableCount: = 8
TotalDiskSpaceUsed: 55681040579, i.e. = ~55GB

So = some of my non-live tables are back live again, and obviously some of = the big ones!!

This is permanently fatal to = consistency; sorry if I was not clear enough that if they were not live, = there was some risk of Cassandra considering them live again upon = restart.

If I = were you, I would either stop the node and remove the files you know = shouldn't be live or do a major compaction ASAP. 

The behavior you just = encountered sounds like a bug, and it is a rather serious one. SSTables = which should be dead being marked live is very bad for = consistency. 

Do you see any exceptions in your logs or anything? If you = can repro, you should file a JIRA ticket with the apache = project...

=3DRob


= --Apple-Mail=_6484BDBD-DE8A-47BE-B43B-0F16EE610343--