Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6CB4CD67C for ; Tue, 11 Sep 2012 01:58:30 +0000 (UTC) Received: (qmail 94041 invoked by uid 500); 11 Sep 2012 01:58:28 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 94014 invoked by uid 500); 11 Sep 2012 01:58:28 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 94004 invoked by uid 99); 11 Sep 2012 01:58:28 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Sep 2012 01:58:28 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a54.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 11 Sep 2012 01:58:19 +0000 Received: from homiemail-a54.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a54.g.dreamhost.com (Postfix) with ESMTP id AB9D73A4061 for ; Mon, 10 Sep 2012 18:57:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=thelastpickle.com; bh=iqcor7C0vkaaZ4FlGSGGVZT9zb I=; b=GLoCucQBE9LCXULRJ0s4N9I0adS2UrycgSMQ9zl2G+n1/s17TjAjPwV3MU 8PxRWapsJaohePy6+VQHIYOF04/HwjgHDLGvuEDEcZebtGGv8tnaxVkh6COX3vMQ RoOPm/+6O80Dw8cMVj1DgQnMzDjuSiey/2z1V1kqvoQlT5a98= Received: from [10.8.0.150] (unknown [72.28.97.147]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a54.g.dreamhost.com (Postfix) with ESMTPSA id 00FA93A4058 for ; Mon, 10 Sep 2012 18:57:56 -0700 (PDT) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_A4DDD34E-7451-481C-A62C-BDA72D93E8A9" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 6.0 \(1486\)) Subject: Re: Suggestions for tuning and monitoring. Date: Tue, 11 Sep 2012 13:57:58 +1200 References: To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1486) --Apple-Mail=_A4DDD34E-7451-481C-A62C-BDA72D93E8A9 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 > It's impossible to start new connections, or impossible to send = requests, or it just doesn't return anything when you've sent a request. If it's totally frozen it sounds like GC. How long does it freeze for? > Despite that, we occasionally get OOM exceptions, and nodes crashing, = maybe a few times per month.=20 Do you have an error stack ? > We can't find anything in the cassandra logs indicating that = something's up Is it logging dropped messages or high TP pending ? Are the freezes = associated with compaction or repair running? > and occasionally we do bulk deletion of supercolumns in a row. mmm, are you sending a batch mutation with lots-o-deletes ? Each row = mutation (insert or delete) in the batch becomes a thread pool tasks. If = you send 1,000 rows in a batch you will temporarily prevent other = requests from being served. > The config options we are unsure about are things like commit log = sizes, =85. I would try to find some indication of what's going on before tweaking. = Have you checked iostat ? Hope that helps.=20 ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 11/09/2012, at 2:05 AM, Henrik Schr=F6der wrote: > Hi all, >=20 > We're running a small Cassandra cluster (v1.0.10) serving data to our = web application, and as our traffic grows, we're starting to see some = weird issues. The biggest of these is that sometimes, a single node = becomes unresponsive. It's impossible to start new connections, or = impossible to send requests, or it just doesn't return anything when = you've sent a request. Our client library is set to retry on another = server when this happens, and what we see then is that the request is = usually served instantly. So it's not the case that some requests are = very difficult, it's that sometimes a node is just "busy", and we have = no idea why or what it's doing. >=20 > We're using MRTG and Monit to monitor the servers, and in MRTG the = average CPU usage is around 5%, on our quad-core Xeon servers with SSDs. = But occassionally through Monit we can see that the 1-min load average = goes above 4, and this usually corresponds to the above issues. Is this = common? Does this happen to everyone else? And why the spikiness in = load? We can't find anything in the cassandra logs indicating that = something's up (such as a slow GC or compaction), and there's no = corresponding traffic spike in the application either. Should we just = add more nodes if any single one gets CPU spikes? >=20 > Another explanation could also be that we've configured it wrong. = We're running pretty much default config. Each node has 16G of RAM, 4GB = of heap, no row-cache and an sizeable key-cache. Despite that, we = occasionally get OOM exceptions, and nodes crashing, maybe a few times = per month. Should we increase heap size? Or move to 1.1 and enable = off-heap caching? >=20 > We have quite a lot of traffic to the cluster. A single keyspace with = two column families, RF=3D3, compression is enabled, and we're running = the default size-tiered compaction. > Column family A has 60GB of actual data, each row has a single column, = and that column holds binary data that varies in size up to 500kB. When = we update a row, we write a new value to this single column, effectively = replacing that entire row. We do ~1000 updates/s, totalling ~10MB/s in = writes. > Column family B also has 60GB of actual data, but each row has a = variable (~100-10000) number of supercolumns, and each supercolumn has a = fixed number of columns with a fixed amount of data, totalling ~1kB. The = operations we are doing on this column family is that we add = supercolumns to rows with a rate of ~200/s, and occasionally we do bulk = deletion of supercolumns in a row. >=20 > The config options we are unsure about are things like commit log = sizes, memtable flushing thresholds, commit log syncing, compaction = throughput, etc. Are we at a point with our data size and write load = that the defaults aren't good enough anymore? Should we stick with = size-tiered compaction, even though our application is update-heavy? >=20 >=20 > Many thanks, > /Henrik --Apple-Mail=_A4DDD34E-7451-481C-A62C-BDA72D93E8A9 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252 It's impossible to start new connections, or = impossible to send requests, or it just doesn't return anything when = you've sent a request.
If it's totally frozen it sounds = like GC. How long does it freeze for?

Despite that, we occasionally get OOM exceptions, and = nodes crashing, maybe a few times per month. 
Do you = have an error stack ?

We = can't find anything in the cassandra logs indicating that something's = up
Is it logging dropped messages or high TP pending ? Are = the freezes associated with compaction or repair = running?

 and = occasionally we do bulk deletion of supercolumns in a = row.
mmm, are you sending a batch mutation with = lots-o-deletes ? Each row mutation (insert or delete) in the batch = becomes a thread pool tasks. If you send 1,000 rows in a batch you will = temporarily prevent other requests from being = served.

The config = options we are unsure about are things like commit log = sizes, =85.
I would try to find some indication of = what's going on before tweaking. Have you checked iostat = ?

Hope that helps. 

http://www.thelastpickle.com

On 11/09/2012, at 2:05 AM, Henrik Schr=F6der <skrolle@gmail.com> = wrote:

Hi all,

We're running a small Cassandra cluster = (v1.0.10) serving data to our web application, and as our traffic grows, = we're starting to see some weird issues. The biggest of these is that = sometimes, a single node becomes unresponsive. It's impossible to start = new connections, or impossible to send requests, or it just doesn't = return anything when you've sent a request. Our client library is set to = retry on another server when this happens, and what we see then is that = the request is usually served instantly. So it's not the case that some = requests are very difficult, it's that sometimes a node is just "busy", = and we have no idea why or what it's doing.

We're using MRTG and Monit to monitor the servers, and in MRTG the = average CPU usage is around 5%, on our quad-core Xeon servers with SSDs. = But occassionally through Monit we can see that the 1-min load average = goes above 4, and this usually corresponds to the above issues. Is this = common? Does this happen to everyone else? And why the spikiness in = load? We can't find anything in the cassandra logs indicating that = something's up (such as a slow GC or compaction), and there's no = corresponding traffic spike in the application either. Should we just = add more nodes if any single one gets CPU spikes?

Another explanation could also be that we've configured it wrong. = We're running pretty much default config. Each node has 16G of RAM, 4GB = of heap, no row-cache and an sizeable key-cache. Despite that, we = occasionally get OOM exceptions, and nodes crashing, maybe a few times = per month. Should we increase heap size? Or move to 1.1 and enable = off-heap caching?

We have quite a lot of traffic to the cluster. A single keyspace = with two column families, RF=3D3, compression is enabled, and we're = running the default size-tiered compaction.
Column family A has 60GB = of actual data, each row has a single column, and that column holds = binary data that varies in size up to 500kB. When we update a row, we = write a new value to this single column, effectively replacing that = entire row. We do ~1000 updates/s, totalling ~10MB/s in writes.
Column family B also has 60GB of actual data, but each row has a = variable (~100-10000) number of supercolumns, and each supercolumn has a = fixed number of columns with a fixed amount of data, totalling ~1kB. The = operations we are doing on this column family is that we add = supercolumns to rows with a rate of ~200/s, and occasionally we do bulk = deletion of supercolumns in a row.

The config options we are unsure about are things like commit log = sizes, memtable flushing thresholds, commit log syncing, compaction = throughput, etc. Are we at a point with our data size and write load = that the defaults aren't good enough anymore? Should we stick with = size-tiered compaction, even though our application is update-heavy?


Many thanks,
/Henrik

= --Apple-Mail=_A4DDD34E-7451-481C-A62C-BDA72D93E8A9--