Return-Path: 
 <user-return-42229-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org>
X-Original-To: apmail-cassandra-user-archive@www.apache.org
Delivered-To: apmail-cassandra-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id CA94D11AC5
	for <apmail-cassandra-user-archive@www.apache.org>;
 Thu,  7 Aug 2014 09:30:21 +0000 (UTC)
Received: (qmail 34880 invoked by uid 500); 7 Aug 2014 09:30:18 -0000
Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org
Received: (qmail 34844 invoked by uid 500); 7 Aug 2014 09:30:18 -0000
Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:user-help@cassandra.apache.org>
List-Unsubscribe: <mailto:user-unsubscribe@cassandra.apache.org>
List-Post: <mailto:user@cassandra.apache.org>
List-Id: <user.cassandra.apache.org>
Reply-To: user@cassandra.apache.org
Delivered-To: mailing list user@cassandra.apache.org
Received: (qmail 34834 invoked by uid 99); 7 Aug 2014 09:30:18 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Aug 2014 09:30:18 +0000
X-ASF-Spam-Status: No, hits=2.2 required=5.0
	tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_NEUTRAL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Received: from [74.125.82.180] (HELO mail-we0-f180.google.com) (74.125.82.180)
    by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Aug 2014 09:30:14 +0000
Received: by mail-we0-f180.google.com with SMTP id w61so3817118wes.25
        for <user@cassandra.apache.org>; Thu, 07 Aug 2014 02:29:52 -0700 (PDT)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=1e100.net; s=20130820;
        h=x-gm-message-state:mime-version:in-reply-to:references:from:date
         :message-id:subject:to:content-type;
        bh=ahmgy9sUNo0qFf3nBsvQZJLwl2RIOl2mbLIJ7/GaNMk=;
        b=g1VbfhGoBESFJ8DooiZFGCNpzZbQQbFq3nqynL/cJtzDY3FjwTDWrmKihTXv17WZwX
         tjPhaPTuD11JP86Hx8LDN15IH+NFoB1Dkbt7FXETaQQfd0YBvANd7+eBVYMZzTBaLMxv
         jF+KBUqRVyUdfg8vjwKtXlESLnaxPtJi/vRoe+drVs3chvGXZqbAxlEkMVlatZQ+muqx
         CNlLMtw+yMSCXjc05Or2YH99XZK+lz6g43USsHurEaqsexUMBM4/9nec0jjh63u/yLUD
         uTSowx1I8+6DfmvS3GYldvqqIRg0A6nh5QIURVNzRg9vfM6K5yF5yU2gl48fe0WGVePA
         anuw==
X-Gm-Message-State: 
 ALoCoQmLQ7vxJd3niWG5gvDDvHrSKv9RhghTS7DvT2cdZ3iyieTFXseUeonxHDRZtbSi/vwK50Od
X-Received: by 10.180.80.133 with SMTP id r5mr55718204wix.62.1407403792542;
 Thu, 07 Aug 2014 02:29:52 -0700 (PDT)
MIME-Version: 1.0
Received: by 10.194.42.34 with HTTP; Thu, 7 Aug 2014 02:29:32 -0700 (PDT)
In-Reply-To: 
 <CAHvupe8h3J3FJszSk9Ph9qx_rR2h3nfCApjeH1fBXeq6p8MoOQ@mail.gmail.com>
References: 
 <CAHvupe-thEu9ZSW8otvv9mhf5VH+vsrDNF1dXt+H__UtnxOvqA@mail.gmail.com>
 <CAABB5w_x8PAYjyRTX68wqePR3Ajxn7Zo-nLuK0xk9O9+4wCkhA@mail.gmail.com>
 <CAAam9ss=gReLRSaQCHO=b3R67jZ3BEKSexr7siSpmqYo74jy1g@mail.gmail.com>
 <CAFqWSgcwEsqUuGNaVSxDowwveAyURLdermS_-qAZpS_sdjHuXA@mail.gmail.com>
 <CAHvupe8h3J3FJszSk9Ph9qx_rR2h3nfCApjeH1fBXeq6p8MoOQ@mail.gmail.com>
From: Fabrice Larcher <fabrice.larcher@level5.fr>
Date: Thu, 7 Aug 2014 11:29:32 +0200
Message-ID: 
 <CAHvupe9xwt+Yku-DU_LPkLaDjmsOZhwftLxJCEi8whMcLfwWFQ@mail.gmail.com>
Subject: Re: C* 2.1-rc2 gets unstable after a 'DROP KEYSPACE' command ?
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=f46d044403a21c65b7050006bbd5
X-Virus-Checked: Checked by ClamAV on apache.org

--f46d044403a21c65b7050006bbd5
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hello,

After a 'DROP TABLE' command that returns "errors=3D{}, last_host=3D127.0.0=
.1"
(like most DROP commands do) from CQLSH with C* 2.1.0-rc2, I stopped C*.
And I can not start one node. It says :
ERROR 09:18:34 Exception encountered during startup
java.lang.NullPointerException: null
    at org.apache.cassandra.db.Directories.<init>(Directories.java:191)
~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
    at
org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftove=
rs(ColumnFamilyStore.java:553)
~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
    at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:245=
)
[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
    at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:=
455)
[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
    at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:544)
[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
java.lang.NullPointerException
    at org.apache.cassandra.db.Directories.<init>(Directories.java:191)
    at
org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftove=
rs(ColumnFamilyStore.java:553)
    at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:245=
)
    at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:=
455)
    at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:544)
Exception encountered during startup: null

I do not now if it can help.


Fabrice LARCHER


2014-07-18 7:23 GMT+02:00 Fabrice Larcher <fabrice.larcher@level5.fr>:

> Hello,
>
> I still experience a similar issue after a 'DROP KEYSPACE' command with C=
*
> 2.1-rc3. Connection to the node may fail after a 'DROP'.
>
> But I did not see this issue with 2.1-rc1 (-> it seems like to be a
> regression brought with 2.1-rc2).
>
> Fabrice LARCHER
>
>
> 2014-07-17 9:19 GMT+02:00 Benedict Elliott Smith <
> belliottsmith@datastax.com>:
>
> Also https://issues.apache.org/jira/browse/CASSANDRA-7437 and
>> https://issues.apache.org/jira/browse/CASSANDRA-7465 for rc3, although
>> the CounterCacheKey assertion looks like an independent (though
>> comparatively benign) bug I will file a ticket for.
>>
>> Can you try this against rc3 to see if the problem persists? You may see
>> the last exception, but it shouldn't affect the stability of the cluster=
.
>> If either of the other exceptions persist, please file a ticket.
>>
>>
>> On Thu, Jul 17, 2014 at 1:41 AM, Tyler Hobbs <tyler@datastax.com> wrote:
>>
>>> This looks like https://issues.apache.org/jira/browse/CASSANDRA-6959,
>>> but that was fixed for 2.1.0-rc1.
>>>
>>> Is there any chance you can put together a script to reproduce the issu=
e?
>>>
>>>
>>> On Thu, Jul 10, 2014 at 8:51 AM, Pavel Kogan <pavel.kogan@cortica.com>
>>> wrote:
>>>
>>>> It seems that memtable tries to flush itself to SSTable of not existin=
g
>>>> keyspace. I don't know why it is happens, but probably running nodetoo=
l
>>>> flush before drop should prevent this issue.
>>>>
>>>> Pavel
>>>>
>>>>
>>>> On Thu, Jul 10, 2014 at 4:09 AM, Fabrice Larcher <
>>>> fabrice.larcher@level5.fr> wrote:
>>>>
>>>>> =E2=80=8BHello,
>>>>>
>>>>> I am using the 'development' version 2.1-rc2.
>>>>>
>>>>> With one node (=3Dlocalhost), I get timeouts trying to connect to C*
>>>>> after running a 'DROP KEYSPACE' command. I have following error messa=
ges in
>>>>> system.log :
>>>>>
>>>>> INFO  [SharedPool-Worker-3] 2014-07-09 16:29:36,578
>>>>> MigrationManager.java:319 - Drop Keyspace 'test_main'
>>>>> (...)
>>>>> ERROR [MemtableFlushWriter:6] 2014-07-09 16:29:37,178
>>>>> CassandraDaemon.java:166 - Exception in thread
>>>>> Thread[MemtableFlushWriter:6,5,main]
>>>>> java.lang.RuntimeException: Last written key
>>>>> DecoratedKey(91e7f660-076f-11e4-a36d-28d2444c0b1b,
>>>>> 52446dde90244ca49789b41671e4ca7c) >=3D current key
>>>>> DecoratedKey(91e7f660-076f-11e4-a36d-28d2444c0b1b,
>>>>> 52446dde90244ca49789b41671e4ca7c) writing into
>>>>> ./../data/data/test_main/user-911d5360076f11e4812d3d4ba97474ac/test_m=
ain-user.user_account-tmp-ka-1-Data.db
>>>>>         at
>>>>> org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWri=
ter.java:172)
>>>>> ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
>>>>>         at
>>>>> org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.ja=
va:215)
>>>>> ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
>>>>>         at
>>>>> org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Me=
mtable.java:351)
>>>>> ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
>>>>>         at
>>>>> org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:=
314)
>>>>> ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
>>>>>         at
>>>>> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareR=
unnable.java:48)
>>>>> ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
>>>>>         at
>>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:2=
8)
>>>>> ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
>>>>>         at
>>>>> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorSer=
vice.execute(MoreExecutors.java:297)
>>>>> ~[guava-16.0.jar:na]
>>>>>         at
>>>>> org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore=
.java:1054)
>>>>> ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
>>>>>         at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.=
java:1145)
>>>>> ~[na:1.7.0_55]
>>>>>         at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor=
.java:615)
>>>>> ~[na:1.7.0_55]
>>>>>         at java.lang.Thread.run(Thread.java:744) ~[na:1.7.0_55]
>>>>>
>>>>> Then, I can not connect to the Cluster anymore from my app (Java
>>>>> Driver 2.1-SNAPSHOT) and got in application logs :
>>>>>
>>>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All
>>>>> host(s) tried for query failed (tried: /127.0.0.1:9042
>>>>> (com.datastax.driver.core.exceptions.DriverException: Timeout during =
read))
>>>>>     at
>>>>> com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoH=
ostAvailableException.java:65)
>>>>>     at
>>>>> com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecu=
tionException(DefaultResultSetFuture.java:258)
>>>>>     at
>>>>> com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(De=
faultResultSetFuture.java:174)
>>>>>     at
>>>>> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java=
:52)
>>>>>     at
>>>>> com.datastax.driver.core.AbstractSession.execute(AbstractSession.java=
:36)
>>>>>     (...)
>>>>> Caused by:
>>>>> com.datastax.driver.core.exceptions.NoHostAvailableException: All hos=
t(s)
>>>>> tried for query failed (tried: /127.0.0.1:9042
>>>>> (com.datastax.driver.core.exceptions.DriverException: Timeout during =
read))
>>>>>     at
>>>>> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.ja=
va:103)
>>>>>     at
>>>>> com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:175=
)
>>>>>     at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.=
java:1145)
>>>>>     at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor=
.java:615)
>>>>>     at java.lang.Thread.run(Thread.java:745)
>>>>>
>>>>> I can still connect through CQLSH but if I run (again) a "DROP
>>>>> KEYSPACE" command from CQLSH, I get the following error :
>>>>> errors=3D{}, last_host=3D127.0.0.1
>>>>>
>>>>> Now, on a 2 nodes cluster I also have a similar issue but the error's
>>>>> stacktrace is different :
>>>>>
>>>>> From application logs :
>>>>>
>>>>> 17971 [Cassandra Java Driver worker-2] WARN
>>>>> com.datastax.driver.core.Cluster  - No schema agreement from live rep=
licas
>>>>> after 10000 ms. The schema may not be up to date on some nodes.
>>>>>
>>>>> From system.log :
>>>>>
>>>>> INFO  [SharedPool-Worker-2] 2014-07-10 09:04:53,434
>>>>> MigrationManager.java:319 - Drop Keyspace 'test_main'
>>>>> (...)
>>>>> ERROR [MigrationStage:1] 2014-07-10 09:04:56,553
>>>>> CommitLogSegmentManager.java:304 - Failed waiting for a forced recycl=
e of
>>>>> in-use commit log segments
>>>>> java.lang.AssertionError: null
>>>>>         at
>>>>> org.apache.cassandra.db.commitlog.CommitLogSegmentManager.forceRecycl=
eAll(CommitLogSegmentManager.java:299)
>>>>> ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
>>>>>         at
>>>>> org.apache.cassandra.db.commitlog.CommitLog.forceRecycleAllSegments(C=
ommitLog.java:160)
>>>>> [apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
>>>>>         at
>>>>> org.apache.cassandra.db.DefsTables.dropColumnFamily(DefsTables.java:5=
16)
>>>>> [apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
>>>>>         at
>>>>> org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.jav=
a:300)
>>>>> [apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
>>>>>         at
>>>>> org.apache.cassandra.db.DefsTables.mergeSchemaInternal(DefsTables.jav=
a:187)
>>>>> [apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
>>>>>         at
>>>>> org.apache.cassandra.db.DefsTables.mergeSchema(DefsTables.java:164)
>>>>> [apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
>>>>>         at
>>>>> org.apache.cassandra.service.MigrationManager$2.runMayThrow(Migration=
Manager.java:393)
>>>>> [apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
>>>>>         at
>>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:2=
8)
>>>>> [apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
>>>>>         at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:47=
1)
>>>>> [na:1.7.0_45]
>>>>>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>> [na:1.7.0_45]
>>>>>         at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.=
java:1145)
>>>>> [na:1.7.0_45]
>>>>>         at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor=
.java:615)
>>>>> [na:1.7.0_45]
>>>>>         at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
>>>>>
>>>>>
>>>>> In both cases, I have to restart the C* server, so that I can connect
>>>>> again from my application. The connection from CQLSH is still possibl=
e but
>>>>> some key-space related commands fail.
>>>>>
>>>>> NB : I also get sometimes the following error on my nodes (I do not
>>>>> know if there is any link with the topic) :
>>>>> ERROR [CompactionExecutor:62] 2014-07-10 08:49:02,059
>>>>> CassandraDaemon.java:166 - Exception in thread
>>>>> Thread[CompactionExecutor:62,1,main]
>>>>> java.lang.NullPointerException: null
>>>>>         at
>>>>> org.apache.cassandra.cache.CounterCacheKey.getPathInfo(CounterCacheKe=
y.java:53)
>>>>> ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
>>>>>         at
>>>>> org.apache.cassandra.cache.AutoSavingCache$Writer.saveCache(AutoSavin=
gCache.java:215)
>>>>> ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
>>>>>         at
>>>>> org.apache.cassandra.db.compaction.CompactionManager$11.run(Compactio=
nManager.java:1052)
>>>>> ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
>>>>>         at
>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:47=
1)
>>>>> ~[na:1.7.0_55]
>>>>>         at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>>>>> ~[na:1.7.0_55]
>>>>>         at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.=
java:1145)
>>>>> ~[na:1.7.0_55]
>>>>>         at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor=
.java:615)
>>>>> [na:1.7.0_55]
>>>>>         at java.lang.Thread.run(Thread.java:744) [na:1.7.0_55]
>>>>>
>>>>> It seems to be a bug (is it known ?). Or do you have any clue on how =
I
>>>>> can solve that ?
>>>>>
>>>>> Thanks for your help
>>>>>
>>>>> Fabrice
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Tyler Hobbs
>>> DataStax <http://datastax.com/>
>>>
>>
>>
>

--f46d044403a21c65b7050006bbd5
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-family:verdana,=
sans-serif">Hello, <br><br>After a &#39;DROP TABLE&#39; command that return=
s &quot;errors=3D{}, last_host=3D127.0.0.1&quot; (like most DROP commands d=
o) from CQLSH with C* 2.1.0-rc2, I stopped C*. And I can not start one node=
. It says :<br>

ERROR 09:18:34 Exception encountered during startup<br>java.lang.NullPointe=
rException: null<br>=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.db.Directori=
es.&lt;init&gt;(Directories.java:191) ~[apache-cassandra-2.1.0-rc2.jar:2.1.=
0-rc2]<br>
=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.db.ColumnFamilyStore.removeUnfin=
ishedCompactionLeftovers(ColumnFamilyStore.java:553) ~[apache-cassandra-2.1=
.0-rc2.jar:2.1.0-rc2]<br>
=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.service.CassandraDaemon.setup(Ca=
ssandraDaemon.java:245) [apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]<br>=C2=
=A0=C2=A0=C2=A0 at org.apache.cassandra.service.CassandraDaemon.activate(Ca=
ssandraDaemon.java:455) [apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]<br>

=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.service.CassandraDaemon.main(Cas=
sandraDaemon.java:544) [apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]<br>java.l=
ang.NullPointerException<br>=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.db.D=
irectories.&lt;init&gt;(Directories.java:191)<br>

=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.db.ColumnFamilyStore.removeUnfin=
ishedCompactionLeftovers(ColumnFamilyStore.java:553)<br>=C2=A0=C2=A0=C2=A0 =
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:=
245)<br>=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.service.CassandraDaemon.=
activate(CassandraDaemon.java:455)<br>

=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.service.CassandraDaemon.main(Cas=
sandraDaemon.java:544)<br>Exception encountered during startup: null<br><br=
></div><div class=3D"gmail_default" style=3D"font-family:verdana,sans-serif=
">I do not now if it can help.<br>

<br></div></div><div class=3D"gmail_extra"><br clear=3D"all"><div><div dir=
=3D"ltr"><font>Fabrice LARCHER</font><br></div></div>
<br><br><div class=3D"gmail_quote">2014-07-18 7:23 GMT+02:00 Fabrice Larche=
r <span dir=3D"ltr">&lt;<a href=3D"mailto:fabrice.larcher@level5.fr" target=
=3D"_blank">fabrice.larcher@level5.fr</a>&gt;</span>:<br><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex">

<div dir=3D"ltr"><div class=3D"gmail_default" style=3D"font-family:verdana,=
sans-serif">Hello,<br></div><div class=3D"gmail_default" style=3D"font-fami=
ly:verdana,sans-serif"><br>I still experience a similar issue after a &#39;=
DROP KEYSPACE&#39; command with C* 2.1-rc3. Connection to the node may fail=
 after a &#39;DROP&#39;.<br>


</div><div class=3D"gmail_default" style=3D"font-family:verdana,sans-serif"=
><br>But I did not see this issue with 2.1-rc1 (-&gt; it seems like to be a=
 regression brought with 2.1-rc2).<br></div></div><div class=3D"gmail_extra=
">


<br clear=3D"all"><div><div dir=3D"ltr"><font>Fabrice LARCHER</font><br></d=
iv></div>
<br><br><div class=3D"gmail_quote">2014-07-17 9:19 GMT+02:00 Benedict Ellio=
tt Smith <span dir=3D"ltr">&lt;<a href=3D"mailto:belliottsmith@datastax.com=
" target=3D"_blank">belliottsmith@datastax.com</a>&gt;</span>:<div><div cla=
ss=3D"h5">

<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-lef=
t:1px #ccc solid;padding-left:1ex">
<div dir=3D"ltr">Also <a href=3D"https://issues.apache.org/jira/browse/CASS=
ANDRA-7437" target=3D"_blank">https://issues.apache.org/jira/browse/CASSAND=
RA-7437</a>=C2=A0and <a href=3D"https://issues.apache.org/jira/browse/CASSA=
NDRA-7465" target=3D"_blank">https://issues.apache.org/jira/browse/CASSANDR=
A-7465</a> for rc3, although the CounterCacheKey assertion looks like an in=
dependent (though comparatively benign) bug I will file a ticket for.=C2=A0=
<div>



<br></div><div>Can you try this against rc3 to see if the problem persists?=
 You may see the last exception, but it shouldn&#39;t affect the stability =
of the cluster. If either of the other exceptions persist, please file a ti=
cket.<br>



</div></div><div><div><div class=3D"gmail_extra"><br><br><div class=3D"gmai=
l_quote">On Thu, Jul 17, 2014 at 1:41 AM, Tyler Hobbs <span dir=3D"ltr">&lt=
;<a href=3D"mailto:tyler@datastax.com" target=3D"_blank">tyler@datastax.com=
</a>&gt;</span> wrote:<br>



<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div>This looks like <a hre=
f=3D"https://issues.apache.org/jira/browse/CASSANDRA-6959" target=3D"_blank=
">https://issues.apache.org/jira/browse/CASSANDRA-6959</a>, but that was fi=
xed for 2.1.0-rc1.<br>



<br></div>Is there any chance you can put together a script to reproduce th=
e issue?<br>

</div><div class=3D"gmail_extra"><div><div><br><br><div class=3D"gmail_quot=
e">On Thu, Jul 10, 2014 at 8:51 AM, Pavel Kogan <span dir=3D"ltr">&lt;<a hr=
ef=3D"mailto:pavel.kogan@cortica.com" target=3D"_blank">pavel.kogan@cortica=
.com</a>&gt;</span> wrote:<br>





<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr">It seems that memtable trie=
s to flush itself to SSTable of not existing keyspace. I don&#39;t know why=
 it is happens, but probably running nodetool flush before drop should prev=
ent this issue.=C2=A0<span><font color=3D"#888888"><div>





<br></div>
<div>Pavel</div></font></span></div><div><div><div class=3D"gmail_extra"><b=
r><br><div class=3D"gmail_quote">On Thu, Jul 10, 2014 at 4:09 AM, Fabrice L=
archer <span dir=3D"ltr">&lt;<a href=3D"mailto:fabrice.larcher@level5.fr" t=
arget=3D"_blank">fabrice.larcher@level5.fr</a>&gt;</span> wrote:<br>






<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div dir=3D"ltr"><font><div=
 class=3D"gmail_default" style=3D"font-family:verdana,sans-serif;display:in=
line">=E2=80=8BHello,<br>






<br>I am using the &#39;development&#39; version 2.1-rc2.<br><br>With one n=
ode (=3Dlocalhost), I get timeouts trying to connect to C* after running a =
&#39;DROP KEYSPACE&#39; command. I have following error messages in system.=
log :<br>








<br>INFO=C2=A0 [SharedPool-Worker-3] 2014-07-09 16:29:36,578 MigrationManag=
er.java:319 - Drop Keyspace &#39;test_main&#39;<br>(...)<br>ERROR [Memtable=
FlushWriter:6] 2014-07-09 16:29:37,178 CassandraDaemon.java:166 - Exception=
 in thread Thread[MemtableFlushWriter:6,5,main]<br>








java.lang.RuntimeException: Last written key DecoratedKey(91e7f660-076f-11e=
4-a36d-28d2444c0b1b, 52446dde90244ca49789b41671e4ca7c) &gt;=3D current key =
DecoratedKey(91e7f660-076f-11e4-a36d-28d2444c0b1b, 52446dde90244ca49789b416=
71e4ca7c) writing into ./../data/data/test_main/user-911d5360076f11e4812d3d=
4ba97474ac/test_main-user.user_account-tmp-ka-1-Data.db<br>








=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.io.sstab=
le.SSTableWriter.beforeAppend(SSTableWriter.java:172) ~[apache-cassandra-2.=
1.0-rc2.jar:2.1.0-rc2]<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org=
.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:215) ~=
[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]<br>








=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.db.Memta=
ble$FlushRunnable.writeSortedContents(Memtable.java:351) ~[apache-cassandra=
-2.1.0-rc2.jar:2.1.0-rc2]<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at =
org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:314) ~=
[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]<br>








=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.io.util.=
DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) ~[apache-cassandra=
-2.1.0-rc2.jar:2.1.0-rc2]<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at =
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[a=
pache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]<br>








=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at com.google.common.util.concur=
rent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297=
) ~[guava-16.0.jar:na]<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org=
.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:105=
4) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]<br>








=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.util.concurrent.ThreadPo=
olExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_55]<br>=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.util.concurrent.ThreadPoolExec=
utor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_55]<br>=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.lang.Thread.run(Thread.java:744) ~[n=
a:1.7.0_55]<br>








<br>Then, I can not connect to the Cluster anymore from my app (Java Driver=
 2.1-SNAPSHOT) and got in application logs :<br><br>com.datastax.driver.cor=
e.exceptions.NoHostAvailableException: All host(s) tried for query failed (=
tried: /<a href=3D"http://127.0.0.1:9042" target=3D"_blank">127.0.0.1:9042<=
/a> (com.datastax.driver.core.exceptions.DriverException: Timeout during re=
ad))<br>








=C2=A0=C2=A0=C2=A0 at com.datastax.driver.core.exceptions.NoHostAvailableEx=
ception.copy(NoHostAvailableException.java:65)<br>=C2=A0=C2=A0=C2=A0 at com=
.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionExcep=
tion(DefaultResultSetFuture.java:258)<br>








=C2=A0=C2=A0=C2=A0 at com.datastax.driver.core.DefaultResultSetFuture.getUn=
interruptibly(DefaultResultSetFuture.java:174)<br>=C2=A0=C2=A0=C2=A0 at com=
.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52)<br>=
=C2=A0=C2=A0=C2=A0 at com.datastax.driver.core.AbstractSession.execute(Abst=
ractSession.java:36)<br>








=C2=A0=C2=A0=C2=A0 (...)<br>Caused by: com.datastax.driver.core.exceptions.=
NoHostAvailableException: All host(s) tried for query failed (tried: /<a hr=
ef=3D"http://127.0.0.1:9042" target=3D"_blank">127.0.0.1:9042</a> (com.data=
stax.driver.core.exceptions.DriverException: Timeout during read))<br>








=C2=A0=C2=A0=C2=A0 at com.datastax.driver.core.RequestHandler.sendRequest(R=
equestHandler.java:103)<br>=C2=A0=C2=A0=C2=A0 at com.datastax.driver.core.R=
equestHandler$1.run(RequestHandler.java:175)<br>=C2=A0=C2=A0=C2=A0 at java.=
util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)<=
br>








=C2=A0=C2=A0=C2=A0 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Th=
readPoolExecutor.java:615)<br>=C2=A0=C2=A0=C2=A0 at java.lang.Thread.run(Th=
read.java:745)<br><br>I can still connect through CQLSH but if I run (again=
) a &quot;DROP KEYSPACE&quot; command from CQLSH, I get the following error=
 :<br>








errors=3D{}, last_host=3D127.0.0.1<br><br>Now, on a 2 nodes cluster I also =
have a similar issue but the error&#39;s stacktrace is different :<br><br>F=
rom application logs :<br><br>17971 [Cassandra Java Driver worker-2] WARN=
=C2=A0 com.datastax.driver.core.Cluster=C2=A0 - No schema agreement from li=
ve replicas after 10000 ms. The schema may not be up to date on some nodes.=
<br>








<br>From system.log :<br><br>INFO=C2=A0 [SharedPool-Worker-2] 2014-07-10 09=
:04:53,434 MigrationManager.java:319 - Drop Keyspace &#39;test_main&#39;<br=
>(...)<br>ERROR [MigrationStage:1] 2014-07-10 09:04:56,553 CommitLogSegment=
Manager.java:304 - Failed waiting for a forced recycle of in-use commit log=
 segments<br>








java.lang.AssertionError: null<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0 at org.apache.cassandra.db.commitlog.CommitLogSegmentManager.forceRecyc=
leAll(CommitLogSegmentManager.java:299) ~[apache-cassandra-2.1.0-rc2.jar:2.=
1.0-rc2]<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.cassan=
dra.db.commitlog.CommitLog.forceRecycleAllSegments(CommitLog.java:160) [apa=
che-cassandra-2.1.0-rc2.jar:2.1.0-rc2]<br>








=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.db.DefsT=
ables.dropColumnFamily(DefsTables.java:516) [apache-cassandra-2.1.0-rc2.jar=
:2.1.0-rc2]<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.cas=
sandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:300) [apache-cassa=
ndra-2.1.0-rc2.jar:2.1.0-rc2]<br>








=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.db.DefsT=
ables.mergeSchemaInternal(DefsTables.java:187) [apache-cassandra-2.1.0-rc2.=
jar:2.1.0-rc2]<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.=
cassandra.db.DefsTables.mergeSchema(DefsTables.java:164) [apache-cassandra-=
2.1.0-rc2.jar:2.1.0-rc2]<br>








=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.service.=
MigrationManager$2.runMayThrow(MigrationManager.java:393) [apache-cassandra=
-2.1.0-rc2.jar:2.1.0-rc2]<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at =
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) [ap=
ache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]<br>








=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.util.concurrent.Executor=
s$RunnableAdapter.call(Executors.java:471) [na:1.7.0_45]<br>=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.util.concurrent.FutureTask.run(FutureTa=
sk.java:262) [na:1.7.0_45]<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at=
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:=
1145) [na:1.7.0_45]<br>








=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.util.concurrent.ThreadPo=
olExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]<br>=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.lang.Thread.run(Thread.java:74=
4) [na:1.7.0_45]<br><br><br>In both cases, I have to restart the C* server,=
 so that I can connect again from my application. The connection from CQLSH=
 is still possible but some key-space related commands fail.<br>








<br>NB : I also get sometimes the following error on my nodes (I do not kno=
w if there is any link with the topic) :<br>ERROR [CompactionExecutor:62] 2=
014-07-10 08:49:02,059 CassandraDaemon.java:166 - Exception in thread Threa=
d[CompactionExecutor:62,1,main]<br>








java.lang.NullPointerException: null<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 at org.apache.cassandra.cache.CounterCacheKey.getPathInfo(Counter=
CacheKey.java:53) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]<br>=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.cache.AutoSavingC=
ache$Writer.saveCache(AutoSavingCache.java:215) ~[apache-cassandra-2.1.0-rc=
2.jar:2.1.0-rc2]<br>








=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.db.compa=
ction.CompactionManager$11.run(CompactionManager.java:1052) ~[apache-cassan=
dra-2.1.0-rc2.jar:2.1.0-rc2]<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) =
~[na:1.7.0_55]<br>








=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.util.concurrent.FutureTa=
sk.run(FutureTask.java:262) ~[na:1.7.0_55]<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0 at java.util.concurrent.ThreadPoolExecutor.runWorker(Thread=
PoolExecutor.java:1145) ~[na:1.7.0_55]<br>=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolE=
xecutor.java:615) [na:1.7.0_55]<br>








=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.lang.Thread.run(Thread.j=
ava:744) [na:1.7.0_55]<br><br>It seems to be a bug (is it known ?). Or do y=
ou have any clue on how I can solve that ?<br><br>Thanks for your help<span=
><font color=3D"#888888"><br>
<br>Fabrice<br></font></span></div></font></div>


</div>
</blockquote></div><br></div>
</div></div></blockquote></div><br><br clear=3D"all"><br></div></div><span>=
<font color=3D"#888888">-- <br><font color=3D"#888888">Tyler Hobbs<span></s=
pan><br>
<a href=3D"http://datastax.com/" target=3D"_blank">DataStax</a><br></font>
</font></span></div>
</blockquote></div><br></div>
</div></div></blockquote></div></div></div><br></div>
</blockquote></div><br></div>

--f46d044403a21c65b7050006bbd5--