From cassandra-user-return-1592-apmail-incubator-cassandra-user-archive=incubator.apache.org@incubator.apache.org Thu Dec 03 19:03:55 2009 Return-Path: Delivered-To: apmail-incubator-cassandra-user-archive@minotaur.apache.org Received: (qmail 78724 invoked from network); 3 Dec 2009 19:03:55 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 3 Dec 2009 19:03:55 -0000 Received: (qmail 50973 invoked by uid 500); 3 Dec 2009 19:03:54 -0000 Delivered-To: apmail-incubator-cassandra-user-archive@incubator.apache.org Received: (qmail 50930 invoked by uid 500); 3 Dec 2009 19:03:54 -0000 Mailing-List: contact cassandra-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: cassandra-user@incubator.apache.org Delivered-To: mailing list cassandra-user@incubator.apache.org Received: (qmail 50920 invoked by uid 99); 3 Dec 2009 19:03:54 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Dec 2009 19:03:54 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of bburruss@real.com designates 207.188.23.7 as permitted sender) Received: from [207.188.23.7] (HELO cir-el.real.com) (207.188.23.7) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Dec 2009 19:03:46 +0000 Received: from [172.21.131.225] ([::ffff:172.21.131.225]) (TLS: TLSv1/SSLv3,256bits,AES256-SHA) by cir-el.real.com with esmtp; Thu, 03 Dec 2009 11:03:24 -0800 id 0020000B.4B180B7C.00002B5A Subject: Re: Persistently increasing read latency From: "B. Todd Burruss" To: cassandra-user@incubator.apache.org In-Reply-To: References: <59DD1BA8FD3C0F4C90771C18F2B5B53A4C850190C6@GVW0432EXB.americas.hpqcorp.net> <59DD1BA8FD3C0F4C90771C18F2B5B53A4C850C15FE@GVW0432EXB.americas.hpqcorp.net> Content-Type: text/plain; charset="UTF-8" Date: Thu, 03 Dec 2009 11:03:25 -0800 Message-ID: <1259867005.2898.20.camel@btoddb-laptop> Mime-Version: 1.0 X-Mailer: Evolution 2.28.1 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org i am seeing this as well. i did a test with just 1 cassandra node, ReplicationFactor=1, 'get' ConsistencyLevel.ONE, 'put' ConsistencyLevel.QUORUM. The first test was writing and reading random values starting from a fresh database. The put performance is staying reasonabe, but the read performance falls off dramatically as the data grows. The get performance fell from approx 6500 get/sec to 150 get/sec (as reported by my client stats.) The database has grown to approx 500gig. i have the stats recorded on 5 second intervals and i see a very linear drop off as the data grows. i stopped the server and restarted it, let it do its thing during restart and then rerun a read-only test using the exact same data. i am still at about 150 get/sec. via JMX i can see the read latency at about 60, but this varies as the app runs. my keyspace is simple: 0.01 all values are exactly the same and are 2k in length. i've tried to do some tuning to make things faster but don't necessarily understand the options. here are some of the params i've changed in the config file: 256 1024 0.6 1000 1440 hope this data helps, and any help you can provide is much appreciated. On Tue, 2009-12-01 at 20:18 -0600, Jonathan Ellis wrote: > On Tue, Dec 1, 2009 at 7:31 PM, Freeman, Tim wrote: > > Looking at the Cassandra mbean's, the attributes of ROW-MUTATION-STAGE and ROW-READ-STAGE and RESPONSE-STAGE are all less than 10. MINOR-COMPACTION-POOL reports 1218 pending tasks. > > That's probably the culprit right there. Something is wrong if you > have 1200 pending compactions. > > This is something that upgrading to trunk will help with right away > since we parallelize compactions there. > > Another thing you can do is increase the memtable limits so you are > not flushing + compacting so often with your insert traffic. > > -Jonathan