Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id F0891115F8 for ; Mon, 21 Jul 2014 07:50:31 +0000 (UTC) Received: (qmail 69231 invoked by uid 500); 21 Jul 2014 07:50:28 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 69196 invoked by uid 500); 21 Jul 2014 07:50:28 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 69184 invoked by uid 99); 21 Jul 2014 07:50:27 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Jul 2014 07:50:27 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_REMOTE_IMAGE X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ud1937@gmail.com designates 209.85.212.178 as permitted sender) Received: from [209.85.212.178] (HELO mail-wi0-f178.google.com) (209.85.212.178) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Jul 2014 07:50:24 +0000 Received: by mail-wi0-f178.google.com with SMTP id hi2so3561673wib.11 for ; Mon, 21 Jul 2014 00:50:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=fggaimn3UCJwU2guhf7tqd5GbispbhhFKxecY//puTM=; b=0PC6ZE7Il5YWS9r81Etsoxml5qObECXcjjmB4Z0+NTdmcWpApsPitRjbPKQ5c6wM+L 7jlNJw908FWvGELO7SvSYmuDk56+NGBsakMw4lDvOYJBjRjKx89Av07KlJcEBSIc6m/y 3YtGrFcvQi9lyY0WYtgdiXYzlDKmxEZrdHS0p3ojJdxumBHZivp5pssO1FOP1qNORV05 OucRr+Yjd8XTfwIHay4I1ocCaH95pMknfNBu6xEzudfn59JmT7HLKN4JNbNaWDVb+vmq oykwEfNmGS5Iy5vsL/UM8dObF5RzrpayPZUc2Ven5kNoSzS/PJn4H9cIjo8Y/ATJ30Vo QKBQ== MIME-Version: 1.0 X-Received: by 10.180.76.68 with SMTP id i4mr1590455wiw.83.1405929002818; Mon, 21 Jul 2014 00:50:02 -0700 (PDT) Received: by 10.194.119.200 with HTTP; Mon, 21 Jul 2014 00:50:02 -0700 (PDT) In-Reply-To: References: Date: Mon, 21 Jul 2014 15:50:02 +0800 Message-ID: Subject: =?UTF-8?Q?Re=3A_=22ghost=22_table_is_breaking_compactions_and_won=27t_?= =?UTF-8?Q?go_away=E2=80=A6_even_during_a_drop=2E?= From: Philo Yang To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=f46d043c814acae3bc04feaf5adb X-Virus-Checked: Checked by ClamAV on apache.org --f46d043c814acae3bc04feaf5adb Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable In my experience, SSTable FileNotFoundException, not only caused by recreate a table but also other operations or even bug, cannot be solved by any nodetool command. However, restart the node for more than one time can make this Exception disappear. I don't know the reason but it does work... Thanks, Philo Yang 2014-07-17 10:32 GMT+08:00 Kevin Burton : > you rock=E2=80=A6 glad it's fixed in 2.1=E2=80=A6 :) > > > > > On Wed, Jul 16, 2014 at 7:05 PM, graham sanderson wrote= : > >> Known issue deleting and recreating a CF with the same name, fixed in 2.= 1 >> (manifests in lots of ways) >> >> https://issues.apache.org/jira/browse/CASSANDRA-5202 >> >> On Jul 16, 2014, at 8:53 PM, Kevin Burton wrote: >> >> looks like a restart of cassandra and a "nodetool compact" fixed this=E2= =80=A6 >> >> >> On Wed, Jul 16, 2014 at 6:45 PM, Kevin Burton wrote= : >> >>> this is really troubling=E2=80=A6 >>> >>> I have a "ghost" table. I dropped it.. but it's not going away. >>> >>> (Cassandra 2.0.8 btw) >>> >>> I ran a 'drop table' on it.. then a 'describe tables' shows that it's >>> not there. >>> >>> However, when I recreated it, with a new schema, all operations on it >>> failed. >>> >>> Looking at why=E2=80=A6 it seems that cassandra had some old SSTables t= hat I >>> imagine are no longer being used but are now in an inconsistent state? >>> >>> This is popping up in the system.log: >>> >>> Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: >>> /d0/cassandra/data/blogindex/content_idx_source_hashcode/blogindex-cont= ent_idx_source_hashcode-jb-1447-Data.db >>> (No such file or directory) >>> >>> so I think what happened=E2=80=A6 is that the original drop table, fail= ed, and >>> then left things in an inconsistent state. >>> >>> I tried a nodetool repair and a nodetool compact=E2=80=A6 those fail on= the >>> same java.io.FileNotFoundException =E2=80=A6 I moved the directories ou= t of the >>> way, same failure issue. >>> >>> =E2=80=A6 any advice on resolving this? >>> >>> -- >>> >>> Founder/CEO Spinn3r.com >>> Location: *San Francisco, CA* >>> blog: http://burtonator.wordpress.com >>> =E2=80=A6 or check out my Google+ profile >>> >>> >>> >>> >> >> >> -- >> >> Founder/CEO Spinn3r.com >> Location: *San Francisco, CA* >> blog: http://burtonator.wordpress.com >> =E2=80=A6 or check out my Google+ profile >> >> >> >> >> > > > -- > > Founder/CEO Spinn3r.com > Location: *San Francisco, CA* > blog: http://burtonator.wordpress.com > =E2=80=A6 or check out my Google+ profile > > > > --f46d043c814acae3bc04feaf5adb Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
In my experience, SSTable=C2=A0FileNotFoundExc= eption, not only caused by recreate a table but also other operations or ev= en bug, =C2=A0cannot be solved by any nodetool command. However, restart th= e node for more than one time can make this Exception=C2=A0disappear. I don= 't know=C2=A0the reason but it does work...=C2=A0

Thanks,
Philo Yang



2014-07-17 10:32 GMT+08:00 Kevin Burton = <burton@spinn3r.com>:
you rock=E2=80=A6 glad it's fixed in 2.1=E2=80=A6 :)

<= div class=3D"gmail_extra">

On Wed, Jul 16= , 2014 at 7:05 PM, graham sanderson <graham@vast.com> wrote:
Kno= wn issue deleting and recreating a CF with the same name, fixed in 2.1 (man= ifests in lots of ways)

https://issues.apache.org/jira/browse/CASSANDRA-5202=

On Jul 16, 2014, at 8:53 PM, Kevin Burton = <burton@spinn3r.= com> wrote:

looks like a restart of cass= andra and a "nodetool compact" fixed this=E2=80=A6=C2=A0


On Wed, Jul 16, = 2014 at 6:45 PM, Kevin Burton <burton@spinn3r.com> wrote:
this is really troubling=E2= =80=A6

I have a "ghost" table. =C2=A0I dropped= it.. but it's not going away. =C2=A0

(Cassandra 2.0.8 btw)

I ran a = 'drop table' on it.. then a 'describe tables' shows that it= 's not there. =C2=A0

However, when I recreated it, with a new schema, all op= erations on it failed.

Looking at why=E2=80=A6 it = seems that cassandra had some old SSTables that I imagine are no longer bei= ng used but are now in an inconsistent state?

This is popping up in the system.log:

Caused by: java.lang.RuntimeException: java.io.FileNotFoundExceptio= n: /d0/cassandra/data/blogindex/content_idx_source_hashcode/blogindex-conte= nt_idx_source_hashcode-jb-1447-Data.db (No such file or directory)

so I think what happened=E2=80=A6 is that the original = drop table, failed, and then left things in an inconsistent state.

I tried a nodetool repair and a nodetool compact=E2=80=A6 = those fail on the same=C2=A0java.io.FileNotFoundException =E2=80=A6 I moved= the directories out of the way, same failure issue.

=E2=80=A6 any advice on resolving this?

--

Founder/CEO=C2=A0Spinn3r.com
Location:=C2=A0San Francisco, CA
=E2=80=A6 or check out my Google+ profile




--

Founder/CEO=C2=A0Spinn3r.com
Location:=C2=A0San Francisco, CA
=E2=80=A6 or check out my Google+ profile





--

Founder/CEO=C2=A0Spinn3r.c= om
Location:=C2=A0San Francisco, CA


--f46d043c814acae3bc04feaf5adb--