Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 03C66D018 for ; Mon, 2 Jul 2012 23:34:11 +0000 (UTC) Received: (qmail 86404 invoked by uid 500); 2 Jul 2012 23:34:08 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 86374 invoked by uid 500); 2 Jul 2012 23:34:08 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 86365 invoked by uid 99); 2 Jul 2012 23:34:08 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2012 23:34:08 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a79.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 02 Jul 2012 23:34:00 +0000 Received: from homiemail-a79.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a79.g.dreamhost.com (Postfix) with ESMTP id D32917D406E for ; Mon, 2 Jul 2012 16:33:37 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=SW0ysndt/x X2zhFOoPfqD0uITLeRVQ6ojP0kS98Sb+xYXS7OiI4NfWBLTtMTBHv2IG7F2QBw2f JZU0sIislIrXefdxkrNk2hyxUMNmNFqY4TGdJoc5eUKVxt6FSD+4fQ27jgCv0C+t 3ttjptwts3VQ0ox8YcpGsRBg0TkWTshlM= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=b6rc8eMLx1QN5MNC f3rwQaEluvY=; b=lZ96aYAedJNBJbD4Dw/XY7boxnSOsj7ugMi99jydV9vPnhop rGiIx1/xX81YYlhQk2NRgZ+D6DZ596w5N1r92mkZrVmvN6aN8c0YHcZZckDDzW7z nysoq5xs9n0DmxasaEe6BwCq7dRCVgYPJR4oxUnFeJVyZmbGkU5RvbVwzsM= Received: from [192.168.2.189] (unknown [116.90.132.105]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a79.g.dreamhost.com (Postfix) with ESMTPSA id 24A157D406C for ; Mon, 2 Jul 2012 16:33:36 -0700 (PDT) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1278) Content-Type: multipart/alternative; boundary="Apple-Mail=_90392707-C7CD-47E7-9E77-0BA5B59F9B07" Subject: Re: Cassandra and massive TTL expirations cause HEAP issue Date: Tue, 3 Jul 2012 11:33:34 +1200 In-Reply-To: To: user@cassandra.apache.org References: Message-Id: X-Mailer: Apple Mail (2.1278) --Apple-Mail=_90392707-C7CD-47E7-9E77-0BA5B59F9B07 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 > After 10 days my cluster crashes due to a java.lang.OutOfMemoryError = during compaction of the big column family that contains roughly 95% of = the data.=20 Does this column family have very wide rows ?=20 > simply some tweaks I need to make in the yaml file. I have tried: The main things that reduce the impact compaction has on memory are: in_memory_compaction_limit_in_mb concurrent_compactors Of the top of my head I cannot think of any shortcuts taken by = compaction if/when all data in an SSTable is past TTL. I think there was = talk of something like that though.=20 Hope that helps. ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 27/06/2012, at 2:38 AM, Nils Pommerien wrote: > Hello, > I am evaluating Cassandra in a log retrieval application. My ring = conists of3 m2.xlarge instances (17.1 GB memory, 6.5 ECU (2 virtual = cores with 3.25 EC2 Compute Units each), 420 GB of local instance = storage, 64-bit platform) and I am writing at roughly 220 writes/sec. = Per day I am adding roughly 60GB of data. All of this sounds simple and = easy and all three nodes are humming along with basically no load. =20 >=20 > The issue is that I am writing all my data with a TTL of 10 days. = After 10 days my cluster crashes due to a java.lang.OutOfMemoryError = during compaction of the big column family that contains roughly 95% of = the data. So basically after 10 days my data set is 600GB and after 10 = days Cassandra would have to tombstone and purge 60GB of data at the = same rate of roughly 220 deletes/second. I am not sure if Cassandra = should be able to do it, whether I should take a partitioning approach = (one CF per day), or if there is simply some tweaks I need to make in = the yaml file. I have tried: > Decrease flush-largest-memtables-at to .4=20 > reduce_cache_sizes_at and reduce_cache_capacity_to set to 1 > Now, the issue remains the same: >=20 > WARN [ScheduledTasks:1] 2012-06-11 19:39:42,017 GCInspector.java (line = 145) Heap is 0.9920103380107628 full. You may need to reduce memtable = and/or cache sizes. Cassandra will now flush up to the two largest = memtables to free up memory. Adjust flush_largest_memtables_at = threshold in cassandra.yaml if you don't want Cassandra to do this = automatically. >=20 > Eventually it will just die with this message. This affects all nodes = in the cluster, not just one.=20 > =20 > Dump file is incomplete: file size limit > ERROR 19:39:39,695 Exception in thread Thread[ReadStage:134,5,main] > java.lang.OutOfMemoryError: Java heap space > ERROR 19:39:39,724 Exception in thread Thread[MutationStage:57,5,main] > java.lang.OutOfMemoryError: Java heap space > at = org.apache.cassandra.utils.FBUtilities.hashToBigInteger(FBUtilities.java:2= 13) > at = org.apache.cassandra.dht.RandomPartitioner.getToken(RandomPartitioner.java= :154) > at = org.apache.cassandra.dht.RandomPartitioner.decorateKey(RandomPartitioner.j= ava:47) > at = org.apache.cassandra.db.RowPosition.forKey(RowPosition.java:54) > =20 > Any help is highly appreciated. It would be cool to tweak it in a way = that I can have a moving window of 10 days in Cassandra while dropping = the old data=85 Or, if there is any other recommended way to deal with = such sliding time windows I am open for ideas. >=20 > Thank you for your help! =09 --Apple-Mail=_90392707-C7CD-47E7-9E77-0BA5B59F9B07 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252
http://www.thelastpickle.com

On 27/06/2012, at 2:38 AM, Nils Pommerien wrote:

I am evaluating = Cassandra in a log retrieval application.  My ring conists of3 = m2.xlarge instances (17.1 GB memory, 6.5 ECU (2 virtual cores with 3.25 = EC2 Compute Units each), 420 GB of local instance storage, 64-bit = platform) and I am writing at roughly 220 writes/sec.  Per day = I am adding roughly 60GB of data.  All of this sounds simple and = easy and all three nodes are humming along with basically no load. =  

The issue is that I am writing all my = data with a TTL of 10 days.  After 10 days my cluster crashes due = to a java.lang.OutOfMemoryError during compaction of the big = column family that contains roughly 95% of the data.  So basically = after 10 days my data set is 600GB and after 10 days Cassandra would = have to tombstone and purge 60GB of data at the same rate of roughly 220 = deletes/second.  I am not sure if Cassandra should be able to do = it, whether I should take a partitioning approach (one CF per day), or = if there is simply some tweaks I need to make in the yaml file.  I = have tried:
  1. Decrease flush-largest-memtables-at = to .4 
  2. reduce_cache_sizes_at = and reduce_cache_capacity_to set to 1
Now, the issue = remains the same:

WARN [ScheduledTasks:1] = 2012-06-11 19:39:42,017 GCInspector.java (line 145) Heap is = 0.9920103380107628 full.  You may need to reduce memtable and/or = cache sizes.  Cassandra will now flush up to the two largest = memtables to free up memory.  Adjust flush_largest_memtables_at = threshold in cassandra.yaml if you don't want Cassandra to do this = automatically.

Eventually it will just die with = this message.  This affects all nodes in the cluster, not just = one. 
 
Dump file is incomplete: file size = limit
ERROR 19:39:39,695 Exception in thread = Thread[ReadStage:134,5,main]
java.lang.OutOfMemoryError: Java = heap space
ERROR 19:39:39,724 Exception in thread = Thread[MutationStage:57,5,main]
java.lang.OutOfMemoryError: = Java heap space
      at = org.apache.cassandra.utils.FBUtilities.hashToBigInteger(FBUtilities.java:2= 13)
      at = org.apache.cassandra.dht.RandomPartitioner.getToken(RandomPartitioner.java= :154)
      at = org.apache.cassandra.dht.RandomPartitioner.decorateKey(RandomPartitioner.j= ava:47)
      at = org.apache.cassandra.db.RowPosition.forKey(RowPosition.java:54)
=  
Any help is highly appreciated.  It would be = cool to tweak it in a way that I can have a moving window of 10 days in = Cassandra while dropping the old data=85 Or, if there is any other = recommended way to deal with such sliding time windows I am open for = ideas.

Thank you for your help! =

= --Apple-Mail=_90392707-C7CD-47E7-9E77-0BA5B59F9B07--