From user-return-33262-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Sat Apr 6 11:36:37 2013 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 50F0A1036B for ; Sat, 6 Apr 2013 11:36:37 +0000 (UTC) Received: (qmail 20431 invoked by uid 500); 6 Apr 2013 11:36:35 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 19920 invoked by uid 500); 6 Apr 2013 11:36:34 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 19891 invoked by uid 99); 6 Apr 2013 11:36:33 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 06 Apr 2013 11:36:33 +0000 X-ASF-Spam-Status: No, hits=3.3 required=5.0 tests=DATE_IN_PAST_06_12,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a43.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 06 Apr 2013 11:36:28 +0000 Received: from homiemail-a43.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a43.g.dreamhost.com (Postfix) with ESMTP id C59598C058 for ; Sat, 6 Apr 2013 04:36:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :content-type:message-id:mime-version:subject:date:references:to :in-reply-to; s=thelastpickle.com; bh=bwihObStonXForf5sVDbF63HbM E=; b=qZzw8g5mfcpe8wsa11vU/aMEX7jjXVcf7z2EVZnBBjL2bduSZb4H2XeLP5 C9uuC/URQuHviP+ArwzcYir6xkHI+Vz4M+7o0LhOsg555jIwxOJOV+Ym1y7CpoNM TJ3dKlTMwi6aIurYcowbG5KF+QuA6YDcSKWNWadjuVil9Iab4= Received: from [172.20.10.4] (unknown [118.148.254.8]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a43.g.dreamhost.com (Postfix) with ESMTPSA id 070F98C057 for ; Sat, 6 Apr 2013 04:36:06 -0700 (PDT) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_45312F8E-CFBE-4529-8FC5-265B39CFA178" Message-Id: <5CB189A6-BAFF-4BB5-BB7D-DEFF2EBD08CA@thelastpickle.com> Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Cassandra services down frequently [Version 1.1.4] Date: Sat, 6 Apr 2013 11:13:36 +0800 References: <20130404032742.xafvjl9adc4wwgcs@webmail.opentransfer.com> To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1499) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_45312F8E-CFBE-4529-8FC5-265B39CFA178 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > We can see from below that you've tweaked and disabled many of the = memory "safety valve" and other memory related settings.=20 Agree.=20 Also you are running with JVM heap size of 3.81GB which is non default. = For a 16GB node I would expect 8GB.=20 Try restoring the yaml values to the defaults and allowing the = cassandra-env.sh file to determine the memory size.=20 Cheers =20 ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 5/04/2013, at 12:36 PM, Bryan Talbot wrote: > On Thu, Apr 4, 2013 at 1:27 AM, wrote: >=20 > After some time (1 hour / 2 hour) cassandra shut services on one or = two nodes with follwoing errors; >=20 >=20 > Wonder what the workload and schema is like ... >=20 > We can see from below that you've tweaked and disabled many of the = memory "safety valve" and other memory related settings. Those could be = causing issues too. >=20 > =20 > hinted_handoff_throttle_delay_in_ms: 0 > flush_largest_memtables_at: 1.0 > reduce_cache_sizes_at: 1.0 > reduce_cache_capacity_to: 0.6 > rpc_keepalive: true > rpc_server_type: sync > rpc_min_threads: 16 > rpc_max_threads: 2147483647 > in_memory_compaction_limit_in_mb: 256 > compaction_throughput_mb_per_sec: 16 > rpc_timeout_in_ms: 15000 > dynamic_snitch_badness_threshold: 0.0 --Apple-Mail=_45312F8E-CFBE-4529-8FC5-265B39CFA178 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
We can see from below = that you've tweaked and disabled many of the memory "safety valve" and = other memory related = settings. 
Agree. 
Also you = are running with JVM heap size of 3.81GB which is non default. For a = 16GB node I would expect 8GB. 

Try = restoring the yaml values to the defaults and allowing the = cassandra-env.sh file to determine the memory = size. 

Cheers
 
http://www.thelastpickle.com

On 5/04/2013, at 12:36 PM, Bryan Talbot <btalbot@aeriagames.com> = wrote:

On Thu, Apr 4, 2013 at 1:27 AM, <adeel.akbar@panasiangroup.com> = wrote:

After some time (1 hour / 2 hour) cassandra shut services on = one or two nodes with follwoing errors;


Wonder what the workload = and schema is like ...

We can see = from below that you've tweaked and disabled many of the memory "safety = valve" and other memory related settings.  Those could be causing = issues too.

 
hinted_handoff_throttle_delay_in_ms: = 0
flush_largest_memtables_at: 1.0
reduce_cache_sizes_at: 1.0
reduce_cache_capacity_to: 0.6
rpc_keepalive: true
rpc_server_type: sync
rpc_min_threads: 16
rpc_max_threads: 2147483647
in_memory_compaction_limit_in_m= b: 256
compaction_throughput_mb_per_sec: = 16
rpc_timeout_in_ms: 15000
dynamic_snitch_badness_threshold: = 0.0

= --Apple-Mail=_45312F8E-CFBE-4529-8FC5-265B39CFA178--