Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BE8E3E1B8 for ; Sat, 19 Jan 2013 18:49:49 +0000 (UTC) Received: (qmail 19803 invoked by uid 500); 19 Jan 2013 18:49:47 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 19776 invoked by uid 500); 19 Jan 2013 18:49:47 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 19768 invoked by uid 99); 19 Jan 2013 18:49:47 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 Jan 2013 18:49:47 +0000 X-ASF-Spam-Status: No, hits=-2.8 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_HI,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jcistaro@netflix.com designates 69.53.237.162 as permitted sender) Received: from [69.53.237.162] (HELO exout103.netflix.com) (69.53.237.162) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 19 Jan 2013 18:49:40 +0000 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; s=s2048;d=netflix.com; h=from:to:subject:date:message-id:in-reply-to:content-type:mime-version; bh=+R/hNRLQX84ya6y9hIgCOmXW4A0=; b=dx/RxPXp4uwJlgxpp8xri4WbVu5W1xvX8yT4VxKUfs4kneRO1NnKVxoW4mn/G7QKd7/AJWbz NVtkl7fubRYsgkeesIqxZvBoA8aJ6qBhdm3KMyGOCUKAp03u9UU8MgY4ByJJWstMJUiHAtSo tJKJpATIETrhbbQoroB+B0ytzvjfRcqjSJB52DmeK1nBY7OIHCL6ATHH1vYuSWA4WaHXwcY1 17ex75MMAdFEqS7Gm54ZGYMrZWIJs6aIMLskcrTxnP93YBWfGmMEoNENvnt+0nARyUHqdcPp Iqeu7tMYhUh1DU4ETu+Roagg12eYMQNYHkqijTz8Yx39pxJ6oILEiw== DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s2048;d=netflix.com; h=from:to:subject:date:message-id:in-reply-to:content-type:mime-version; b=FxU8PkMoKCn5F2Jme4pA5bdKqZpGQRjI144O5UFUalxu8ydQ0tC71aSIon++sIZ5EP2LmyPK zvxTQrX36X5rM5LpCeoEYoYlQ42T2n9s5EWaDuY1X7BWupGw0Lr9uJS5z5PG9iXOkxTJxy0U sEiXi1wgP92HNmgIpjPrR8YEbKg9qyHFSiR7tqA47kKjPDhvLeEP63Uoj6lgHad1Of5xpe1Q jKmiPfUYkw0182ZsN12xZFu6JRUFO+wYnOQGmtSq7nDail3alOCfXWHkZFVKU3dznwfehYkO CDLNZrmuF/qB4ymofe3ATH9b8IpmubyHkARQ1OqvlG5XMbsi2y7NEQ== Received: from EXFE101.corp.netflix.com (10.64.32.161) by exout103.netflix.com (10.64.240.73) with Microsoft SMTP Server (TLS) id 14.2.298.4; Sat, 19 Jan 2013 10:49:21 -0800 Received: from EXMB102.corp.netflix.com ([169.254.2.85]) by exfe101.corp.netflix.com ([10.64.32.161]) with mapi id 14.02.0283.003; Sat, 19 Jan 2013 10:49:18 -0800 From: Jim Cistaro To: "user@cassandra.apache.org" , Wei Zhu Subject: Re: Cassandra pending compaction tasks keeps increasing Thread-Topic: Cassandra pending compaction tasks keeps increasing Thread-Index: AQHN9bgExENXgQMIR02bJiPNrp6a5phQ/+2A Date: Sat, 19 Jan 2013 18:49:18 +0000 Message-ID: In-Reply-To: <1358539841.10972.YahooMailNeo@web160904.mail.bf1.yahoo.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.2.5.121010 x-originating-ip: [10.64.25.13] Content-Type: multipart/alternative; boundary="_000_CD202A7493C0jcistaronetflixcom_" MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org --_000_CD202A7493C0jcistaronetflixcom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable 1) In addition to iostat, dstat is a good tool to see wht kind of disck thr= ouput your are getting. That would be one thing to monitor. 2) For LCS, we also see pending compactions skyrocket. During load, LCS wi= ll create a lot of small sstables which will queue up for compaction. 3) For us the biggest concern is not how high the pending count gets, but h= ow often it gets back down near zero. If your load is something you can do= in segments or pause, then you can see how fast the cluster recovers on th= e compactions. 4) One thing which we tune per cluster is the size of the files. Increasin= g this from 5MB can sometimes improve things. But I forget if we have ever= changed this after starting data load. Is your cluster receiving read traffic during this data migration? If so, I= would say that read latency is your best measure. If the high number of S= STables waiting to compact is not hurting your reads, then you are probably= ok. Since you are on SSD, there is a good chance the compactions are not = hurting you. As for compactionthroughput, we set ours high for SSD. You u= sually wont use it all because the compactions are usually single threaded.= Dstat will help you measure this. I hope this helps, jc From: Wei Zhu > Reply-To: "user@cassandra.apache.org" >, Wei Zhu > Date: Friday, January 18, 2013 12:10 PM To: Cassandr usergroup > Subject: Cassandra pending compaction tasks keeps increasing Hi, When I run nodetool compactionstats I see the number of pending tasks keep going up steadily. I tried to increase the compactionthroughput, by using nodetool setcompactionthroughput I even tried the extreme to set it to 0 to disable the throttling. I checked iostats and we have SSD for data, the disk util is less than 5% w= hich means it's not I/O bound, CPU is also less than 10% We are using levelcompaction and in the process of migrating data. We have = 4500 writes per second and very few reads. We have about 70G data now and w= ill grow to 150G when the migration finishes. We only have one CF and right= now the number of SSTable is around 15000, write latency is still under 0= .1ms. Anything needs to be concerned? Or anything I can do to reduce the number o= f pending compaction? Thanks. -Wei --_000_CD202A7493C0jcistaronetflixcom_ Content-Type: text/html; charset="us-ascii" Content-ID: <72043CD8C78445449AB5ADF00B70EBBF@netflix.com> Content-Transfer-Encoding: quoted-printable
1) In addition to iostat, dstat is a good tool to see wht kind of disc= k throuput your are getting.  That would be one thing to monitor.
2) For LCS, we also see pending compactions skyrocket.  During lo= ad, LCS will create a lot of small sstables which will queue up for compact= ion.
3) For us the biggest concern is not how high the pending count gets, = but how often it gets back down near zero.  If your load is something = you can do in segments or pause, then you can see how fast the cluster reco= vers on the compactions.
4) One thing which we tune per cluster is the size of the files.  = ;Increasing this from 5MB can sometimes improve things.  But I forget = if we have ever changed this after starting data load.

Is your cluster receiving read traffic during this data migration? If = so, I would say that read latency is your best measure.  If the high n= umber of SSTables waiting to compact is not hurting your reads, then you ar= e probably ok.  Since you are on SSD, there is a good chance the compactions are not hurting you.  As for c= ompactionthroughput, we set ours high for SSD.  You usually wont use i= t all because the compactions are usually single threaded.  Dstat will= help you measure this.

I hope this helps,
jc

From: Wei Zhu <wz1975@yahoo.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>, We= i Zhu <wz1975@yahoo.com>
Date: Friday, January 18, 2013 12:1= 0 PM
To: Cassandr usergroup <user@cassandra.apache.org>
Subject: Cassandra pending compacti= on tasks keeps increasing

Hi,
When I run nodetool compactionstats

I see the number of pending tasks keep going up steadily. 

I tried to increase the  compactionthroughput, by using

nodetool setcompactionthroughput

I even tried the extreme to set it to 0 to disable the throttling. 

I checked iostats and we have SSD for data, the disk util is less than 5% w= hich means it's not I/O bound, CPU is also less than 10%

We are using levelcompaction and in the process of migrating data. We have = 4500 writes per second and very few reads. We have about 70G data now and w= ill grow to 150G when the migration finishes. We only have one CF and right= now the number of  SSTable is around 15000, write latency is still under 0.1ms. 

Anything needs to be concerned? Or anything I can do to reduce the number o= f pending compaction?

Thanks.
-Wei


--_000_CD202A7493C0jcistaronetflixcom_--