Return-Path: Delivered-To: apmail-incubator-cassandra-user-archive@minotaur.apache.org Received: (qmail 70008 invoked from network); 14 Oct 2009 13:17:45 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 14 Oct 2009 13:17:45 -0000 Received: (qmail 47942 invoked by uid 500); 14 Oct 2009 13:17:44 -0000 Delivered-To: apmail-incubator-cassandra-user-archive@incubator.apache.org Received: (qmail 47926 invoked by uid 500); 14 Oct 2009 13:17:44 -0000 Mailing-List: contact cassandra-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: cassandra-user@incubator.apache.org Delivered-To: mailing list cassandra-user@incubator.apache.org Received: (qmail 47916 invoked by uid 99); 14 Oct 2009 13:17:44 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Oct 2009 13:17:44 +0000 X-ASF-Spam-Status: No, hits=-2.6 required=5.0 tests=BAYES_00 X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [212.192.243.50] (HELO hook.sigaev.ru) (212.192.243.50) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Oct 2009 13:17:42 +0000 Received: by hook.sigaev.ru (Postfix, from userid 1003) id 7EF26398F9; Wed, 14 Oct 2009 17:17:20 +0400 (MSD) X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on hook.sigaev.ru X-Spam-Level: Received: from xor.sai.msu.ru (unknown [195.218.191.68]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: teodor) by hook.sigaev.ru (Postfix) with ESMTPSA id 42F72398F2 for ; Wed, 14 Oct 2009 17:17:10 +0400 (MSD) Message-ID: <4AD5CF54.5080000@sigaev.ru> Date: Wed, 14 Oct 2009 17:17:08 +0400 From: Teodor Sigaev User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.8.1.23) Gecko/20090909 SeaMonkey/1.1.18 MIME-Version: 1.0 To: cassandra-user@incubator.apache.org Subject: Cassandra restart Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 7bit X-Old-Spam-Status: No, score=-104.3 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00, USER_IN_WHITELIST autolearn=ham version=3.2.5 I have 12-nodes cassandra's cluster, after restart (really, upgrade from 0.4.0 to 0.4.1) of one node it writes in log a lot of messages like: ERROR [pool-1-thread-46] 2009-10-14 17:06:40,804 Cassandra.java (line 679) Internal error processing get_slice java.lang.RuntimeException: error reading key 00000000000028344851 at org.apache.cassandra.service.StorageProxy.weakReadRemote(StorageProxy.java:265) at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:312) at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:100) at org.apache.cassandra.service.CassandraServer.getSlice(CassandraServer.java:182) at org.apache.cassandra.service.CassandraServer.multigetSliceInternal(CassandraServer.java:251) at org.apache.cassandra.service.CassandraServer.get_slice(CassandraServer.java:220) at org.apache.cassandra.service.Cassandra$Processor$get_slice.process(Cassandra.java:671) at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:627) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Caused by: java.util.concurrent.TimeoutException: Operation timed out. at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:97) at org.apache.cassandra.service.StorageProxy.weakReadRemote(StorageProxy.java:261) ... 11 more Read from that key from another nodes is ok, but reading from restarted timeouted with that message. Is some rules/recommendation to restart node? -- Teodor Sigaev E-mail: teodor@sigaev.ru WWW: http://www.sigaev.ru/