Return-Path: Delivered-To: apmail-incubator-cassandra-user-archive@minotaur.apache.org Received: (qmail 30929 invoked from network); 16 Dec 2009 18:46:59 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 16 Dec 2009 18:46:59 -0000 Received: (qmail 31105 invoked by uid 500); 16 Dec 2009 18:46:59 -0000 Delivered-To: apmail-incubator-cassandra-user-archive@incubator.apache.org Received: (qmail 31053 invoked by uid 500); 16 Dec 2009 18:46:58 -0000 Mailing-List: contact cassandra-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: cassandra-user@incubator.apache.org Delivered-To: mailing list cassandra-user@incubator.apache.org Received: (qmail 31044 invoked by uid 99); 16 Dec 2009 18:46:58 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Dec 2009 18:46:58 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of rrabah@playdom.com designates 74.125.149.203 as permitted sender) Received: from [74.125.149.203] (HELO na3sys009aog110.obsmtp.com) (74.125.149.203) by apache.org (qpsmtpd/0.29) with SMTP; Wed, 16 Dec 2009 18:46:48 +0000 Received: from source ([209.85.222.193]) by na3sys009aob110.postini.com ([74.125.148.12]) with SMTP ID DSNKSykrA/MTsQWhRoEF/LkFMhwEczgyE051@postini.com; Wed, 16 Dec 2009 10:46:28 PST Received: by pzk31 with SMTP id 31so853112pzk.28 for ; Wed, 16 Dec 2009 10:46:26 -0800 (PST) MIME-Version: 1.0 Received: by 10.140.55.2 with SMTP id d2mr998396rva.75.1260989186658; Wed, 16 Dec 2009 10:46:26 -0800 (PST) Date: Wed, 16 Dec 2009 10:46:26 -0800 Message-ID: Subject: TimedOutException From: Ramzi Rabah To: cassandra-user@incubator.apache.org Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Checked: Checked by ClamAV on apache.org We are observing increasing number of TimedOutExceptions in cassandra 0.5 trunk although the load seems fairly low (about 400 reads/writes per second). cfstats reports that operations are taking less than 2 ms on average. 2 Things I have noticed looking at the source code. 1- TimedOutExceptions are silently swallowed by Cassandra and not reported in the logs even at debug level 2- readstats does not account for these long time running queries that time out. I'm wondering, what could be causing the system to go haywire like this? and how much is a reasonable read/write performance on some of the user's successful deployments before the system starts getting bad response times. Thanks Ramzi