Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 39182CDB9 for ; Wed, 16 May 2012 09:45:34 +0000 (UTC) Received: (qmail 90791 invoked by uid 500); 16 May 2012 09:45:31 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 90716 invoked by uid 500); 16 May 2012 09:45:31 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 90694 invoked by uid 99); 16 May 2012 09:45:30 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 May 2012 09:45:30 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a51.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 May 2012 09:45:24 +0000 Received: from homiemail-a51.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a51.g.dreamhost.com (Postfix) with ESMTP id AD6BB2E8062 for ; Wed, 16 May 2012 02:45:03 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=wl2vfkRar5 02FHESCK7n3bYWbjBhIBzeFjWCEBmdEQ1TnP2uErY4jPd4dSGyB67qC6feePLo65 qO5PJgxvZquVXridsiMYFCevu2ZZPFO4sf2kPlMUCNkvxz0fwe3W45MKOWCIONGl hh+Qa3uS8AEqbWJab18M1nkFZvFaFGVr0= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=9oYSsk/wfULWbRh9 MHoZoIxtS54=; b=BNZe8Z2+ZAxQBS3ZIyFXCyQFBEPn6pKP5AnX7peO7sGMsFuG VC/ANbhA0nznfKVxYPrrkUDXqNmnusPt92IiCDseTjF/LD/gcS2EG1ES/qLhnAm2 T0ulbctwitdyM/Zc6xIqWj5BwT1EqCTT+xYCdlpKUcUkCPSCyGPWW6lST0w= Received: from [172.16.1.4] (unknown [203.86.207.101]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a51.g.dreamhost.com (Postfix) with ESMTPSA id 28D572E805C for ; Wed, 16 May 2012 02:45:03 -0700 (PDT) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1257) Content-Type: multipart/alternative; boundary="Apple-Mail=_18CF90EF-AD45-4F02-9825-074E2B699CA1" Subject: Re: Tuning cassandra (compactions overall) Date: Wed, 16 May 2012 21:45:00 +1200 In-Reply-To: To: user@cassandra.apache.org References: Message-Id: <1620C181-AEFE-40A8-81B6-4909CE87AA92@thelastpickle.com> X-Mailer: Apple Mail (2.1257) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_18CF90EF-AD45-4F02-9825-074E2B699CA1 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 > 1 - I got this kind of message quite often (let's say every 30 = seconds) : You are running out of memory. Depending on the size of your schema and = the work load you will want to start with 4 or 8 GB machines. But most = people get the best results with 16Gb. On AWS the common setup is to use m1.xlarge. 2 - I shared 2 screenshot the cluster performance (via OpsCenter) and >=20 > the hardware metrics (via AWS). The other system resources and throughput will be affected if cassandra = does not have enough memory to work with.=20 There are some changes you can make to mitigate things (let me know if = you need help), but this is essentially a memory problem.=20 Cheers ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 16/05/2012, at 4:31 AM, Alain RODRIGUEZ wrote: > Hi, >=20 > I'm using a 2 node cluster in production ( 2 EC2 c1.medium, CL.ONE, RF > =3D 2, using RP) >=20 > 1 - I got this kind of message quite often (let's say every 30 = seconds) : >=20 > WARN [ScheduledTasks:1] 2012-05-15 15:44:53,083 GCInspector.java (line > 145) Heap is 0.8081418550931491 full. You may need to reduce memtable > and/or cache sizes. Cassandra will now flush up to the two largest > memtables to free up memory. Adjust flush_largest_memtables_at > threshold in cassandra.yaml if you don't want Cassandra to do this > automatically > WARN [ScheduledTasks:1] 2012-05-15 15:44:53,084 StorageService.java > (line 2645) Flushing CFS(Keyspace=3D'xxx', ColumnFamily=3D'yyy') to > relieve memory pressure >=20 > Is that a problem ? >=20 > 2 - I shared 2 screenshot the cluster performance (via OpsCenter) and > the hardware metrics (via AWS). >=20 > http://img337.imageshack.us/img337/6812/performance.png > http://img256.imageshack.us/img256/9644/aws.png >=20 > What do you think of these metrics ? Are frequents compaction normal ? > What about having a 60-70% cpu load for 600 Reads&Writes/sec with this > hardware ? Is there a way to optimize my cluster ? >=20 > Here you got the main points of my cassandra.yaml : >=20 > flush_largest_memtables_at: 0.75 > reduce_cache_sizes_at: 0.85 > reduce_cache_capacity_to: 0.6 > concurrent_reads: 32 > concurrent_writes: 32 > commitlog_total_space_in_mb: 4096 > rpc_server_type: sync (I am going to switch to hsha, because we are > using ubuntu) > #concurrent_compactors: 1 (commented, so I use default) > multithreaded_compaction: false > compaction_throughput_mb_per_sec: 16 > rpc_timeout_in_ms: 10000 >=20 > others tuning options (as many of the ones above) are default. >=20 > Any advice or comment would be appreciated :). >=20 > Alain --Apple-Mail=_18CF90EF-AD45-4F02-9825-074E2B699CA1 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1
1 - I got this kind of message quite = often (let's say every 30 seconds) :
You are = running out of memory. Depending on the size of your schema and the work = load you will want to start with 4 or 8 GB machines. But most people get = the best results with 16Gb.

On AWS the common setup = is to use m1.xlarge.

 2 - I shared 2 = screenshot the cluster performance (via OpsCenter) and
the hardware metrics (via = AWS).

The other system resources and = throughput will be affected if cassandra does not have enough memory to = work with. 

There are some changes you can = make to mitigate things (let me know if you need help), but this is = essentially a memory = problem. 

Cheers

http://www.thelastpickle.com

On 16/05/2012, at 4:31 AM, Alain RODRIGUEZ wrote:

Hi,

I'm using a 2 node cluster in production ( = 2 EC2 c1.medium, CL.ONE, RF
=3D 2, using RP)

1 - I got this = kind of message quite often (let's say every 30 seconds) :

WARN = [ScheduledTasks:1] 2012-05-15 15:44:53,083 GCInspector.java = (line
145) Heap is 0.8081418550931491 full.  You may need to = reduce memtable
and/or cache sizes.  Cassandra will now flush up = to the two largest
memtables to free up memory.  Adjust = flush_largest_memtables_at
threshold in cassandra.yaml if you don't = want Cassandra to do this
automatically
WARN [ScheduledTasks:1] = 2012-05-15 15:44:53,084 StorageService.java
(line 2645) Flushing = CFS(Keyspace=3D'xxx', ColumnFamily=3D'yyy') to
relieve memory = pressure

Is that a problem ?

2 - I shared 2 screenshot the = cluster performance (via OpsCenter) and
the hardware metrics (via = AWS).

http://im= g337.imageshack.us/img337/6812/performance.png
http://img256.images= hack.us/img256/9644/aws.png

What do you think of these metrics ? = Are frequents compaction normal ?
What about having a 60-70% cpu load = for 600 Reads&Writes/sec with this
hardware ? Is there a way to = optimize my cluster ?

Here you got the main points of my = cassandra.yaml :

flush_largest_memtables_at: = 0.75
reduce_cache_sizes_at: 0.85
reduce_cache_capacity_to: = 0.6
concurrent_reads: 32
concurrent_writes: = 32
commitlog_total_space_in_mb: 4096
rpc_server_type: sync (I am = going to switch to hsha, because we are
using = ubuntu)
#concurrent_compactors: 1 (commented, so I use = default)
multithreaded_compaction: = false
compaction_throughput_mb_per_sec: 16
rpc_timeout_in_ms: = 10000

others tuning options (as many of the ones above) are = default.

Any advice or comment would be appreciated = :).

Alain

= --Apple-Mail=_18CF90EF-AD45-4F02-9825-074E2B699CA1--