Return-Path: Delivered-To: apmail-incubator-cassandra-user-archive@minotaur.apache.org Received: (qmail 93064 invoked from network); 14 Oct 2009 14:26:25 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 14 Oct 2009 14:26:25 -0000 Received: (qmail 43780 invoked by uid 500); 14 Oct 2009 14:26:25 -0000 Delivered-To: apmail-incubator-cassandra-user-archive@incubator.apache.org Received: (qmail 43759 invoked by uid 500); 14 Oct 2009 14:26:25 -0000 Mailing-List: contact cassandra-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: cassandra-user@incubator.apache.org Delivered-To: mailing list cassandra-user@incubator.apache.org Received: (qmail 43750 invoked by uid 99); 14 Oct 2009 14:26:25 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Oct 2009 14:26:25 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [212.192.243.50] (HELO hook.sigaev.ru) (212.192.243.50) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 14 Oct 2009 14:26:14 +0000 Received: by hook.sigaev.ru (Postfix, from userid 1003) id 3ECEC398F6; Wed, 14 Oct 2009 18:25:54 +0400 (MSD) X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on hook.sigaev.ru X-Spam-Level: Received: from xor.sai.msu.ru (unknown [195.218.191.68]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: teodor) by hook.sigaev.ru (Postfix) with ESMTPSA id B3582398F2 for ; Wed, 14 Oct 2009 18:25:43 +0400 (MSD) Message-ID: <4AD5DF67.4070707@sigaev.ru> Date: Wed, 14 Oct 2009 18:25:43 +0400 From: Teodor Sigaev User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.8.1.23) Gecko/20090909 SeaMonkey/1.1.18 MIME-Version: 1.0 To: cassandra-user@incubator.apache.org Subject: Re: Cassandra restart References: <4AD5CF54.5080000@sigaev.ru> <4AD5D53B.3090909@sigaev.ru> <4AD5DB2C.9070309@sigaev.ru> In-Reply-To: Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Old-Spam-Status: No, score=-104.3 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00, USER_IN_WHITELIST autolearn=ham version=3.2.5 Sorry, I don't see anything connected to key 00000000000013062900 or message 2691, debug is turned off on that node because 10.3.2.36 wasn't restarted I will try to reproduce problem on smaller test cluster. Fragment of 10.3.2.36 log (time is in sync): ERROR [pool-1-thread-15051] 2009-10-14 18:03:30,562 StorageProxy.java (line 179) error writing key 00000000000176715312 java.util.concurrent.TimeoutException: Operation timed out - received only 0 responses from . at org.apache.cassandra.service.QuorumResponseHandler.get(QuorumResponseHandler.java:88) at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:164) at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:468) at org.apache.cassandra.service.CassandraServer.remove(CassandraServer.java:461) at org.apache.cassandra.service.Cassandra$Processor$remove.process(Cassandra.java:884) at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:627) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) ERROR [pool-1-thread-14947] 2009-10-14 18:03:32,593 StorageProxy.java (line 179) error writing key 00000000000176715312 java.util.concurrent.TimeoutException: Operation timed out - received only 0 responses from . at org.apache.cassandra.service.QuorumResponseHandler.get(QuorumResponseHandler.java:88) at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:164) at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:468) at org.apache.cassandra.service.CassandraServer.remove(CassandraServer.java:461) at org.apache.cassandra.service.Cassandra$Processor$remove.process(Cassandra.java:884) at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:627) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) ERROR [pool-1-thread-14985] 2009-10-14 18:03:34,571 StorageProxy.java (line 179) error writing key 00000000000176715312 java.util.concurrent.TimeoutException: Operation timed out - received only 0 responses from . at org.apache.cassandra.service.QuorumResponseHandler.get(QuorumResponseHandler.java:88) at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:164) at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:468) at org.apache.cassandra.service.CassandraServer.remove(CassandraServer.java:461) at org.apache.cassandra.service.Cassandra$Processor$remove.process(Cassandra.java:884) at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:627) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) ERROR [pool-1-thread-14493] 2009-10-14 18:03:34,571 Cassandra.java (line 679) Internal error processing get_slice java.lang.RuntimeException: error reading key 00000000000106371503 at org.apache.cassandra.service.StorageProxy.weakReadRemote(StorageProxy.java:265) at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:312) at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:100) at org.apache.cassandra.service.CassandraServer.getSlice(CassandraServer.java:182) at org.apache.cassandra.service.CassandraServer.multigetSliceInternal(CassandraServer.java:251) at org.apache.cassandra.service.CassandraServer.get_slice(CassandraServer.java:220) at org.apache.cassandra.service.Cassandra$Processor$get_slice.process(Cassandra.java:671) at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:627) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Caused by: java.util.concurrent.TimeoutException: Operation timed out. at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:97) at org.apache.cassandra.service.StorageProxy.weakReadRemote(StorageProxy.java:261) ... 11 more ERROR [pool-1-thread-14820] 2009-10-14 18:03:35,406 Cassandra.java (line 679) Internal error processing get_slice java.lang.RuntimeException: error reading key 00000000000019586353 at org.apache.cassandra.service.StorageProxy.weakReadRemote(StorageProxy.java:265) at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:312) at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:100) at org.apache.cassandra.service.CassandraServer.getSlice(CassandraServer.java:182) at org.apache.cassandra.service.CassandraServer.multigetSliceInternal(CassandraServer.java:251) at org.apache.cassandra.service.CassandraServer.get_slice(CassandraServer.java:220) at org.apache.cassandra.service.Cassandra$Processor$get_slice.process(Cassandra.java:671) at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:627) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Caused by: java.util.concurrent.TimeoutException: Operation timed out. at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:97) at org.apache.cassandra.service.StorageProxy.weakReadRemote(StorageProxy.java:261) ... 11 more ERROR [pool-1-thread-15051] 2009-10-14 18:03:36,602 StorageProxy.java (line 179) error writing key 00000000000176715312 java.util.concurrent.TimeoutException: Operation timed out - received only 0 responses from . at org.apache.cassandra.service.QuorumResponseHandler.get(QuorumResponseHandler.java:88) at org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:164) at org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:468) at org.apache.cassandra.service.CassandraServer.remove(CassandraServer.java:461) at org.apache.cassandra.service.Cassandra$Processor$remove.process(Cassandra.java:884) at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:627) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) ERROR [pool-1-thread-14658] 2009-10-14 18:03:37,590 Cassandra.java (line 679) Internal error processing get_slice java.lang.RuntimeException: error reading key 00000000000139616600 at org.apache.cassandra.service.StorageProxy.weakReadRemote(StorageProxy.java:265) at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:312) at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:100) at org.apache.cassandra.service.CassandraServer.getSlice(CassandraServer.java:182) at org.apache.cassandra.service.CassandraServer.multigetSliceInternal(CassandraServer.java:251) at org.apache.cassandra.service.CassandraServer.get_slice(CassandraServer.java:220) at org.apache.cassandra.service.Cassandra$Processor$get_slice.process(Cassandra.java:671) at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:627) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Caused by: java.util.concurrent.TimeoutException: Operation timed out. at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:97) at org.apache.cassandra.service.StorageProxy.weakReadRemote(StorageProxy.java:261) ... 11 more -- Teodor Sigaev E-mail: teodor@sigaev.ru WWW: http://www.sigaev.ru/