From user-return-37514-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Tue Nov 12 04:16:16 2013 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 50CDD10724 for ; Tue, 12 Nov 2013 04:16:16 +0000 (UTC) Received: (qmail 41641 invoked by uid 500); 12 Nov 2013 04:16:12 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 41587 invoked by uid 500); 12 Nov 2013 04:16:11 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 41579 invoked by uid 99); 12 Nov 2013 04:16:10 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Nov 2013 04:16:10 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: local policy includes SPF record at spf.trusted-forwarder.org) Received: from [209.85.160.41] (HELO mail-pb0-f41.google.com) (209.85.160.41) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Nov 2013 04:16:03 +0000 Received: by mail-pb0-f41.google.com with SMTP id jt11so1830pbb.14 for ; Mon, 11 Nov 2013 20:15:41 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-type:message-id:mime-version :subject:date:references:to:in-reply-to; bh=KqSxJBJBPxmpGeERPmWJt8Rd5dm7k1cwsa3rI/wCQg8=; b=TCu11tudJs8sx3ymveCjGuwT4l+s/CahOuNyOnTd2j/cFT1k+H+YMhCEJiW2K2EbMH kqo17I9vykXggTwb9yqQ1HlpibwGyCQZ6X7f0oQbOu4dpOQB5WDbDETxHUq8AX0J/iI2 bwP+dQ4m9OLShDSKA8GgUBLYkLia8YpVeDoFaLQNsKuiKW5d3AppZmGa1lWqLRjHLRda pEeekkToskqM1qEzSKG2MhI7asrP/DT2CR4LWFRWn3Wx3LguzID6HabpD6stkRv4Nubc YAuf0M4UAPIpGoYKTMnZQqCK7ri9pPBojspChvw/NNQW8sAu1+/2MH5Kpmec86cYz/LJ Hvtg== X-Gm-Message-State: ALoCoQkVyAK4/YzhKQXr1nhoY47GAQ9lmI3KtpybtWXDX1IFeB8wfZ0HnAXugpQJ2puU7pn0eMm/ X-Received: by 10.68.137.9 with SMTP id qe9mr10875507pbb.133.1384229741698; Mon, 11 Nov 2013 20:15:41 -0800 (PST) Received: from [172.16.1.20] ([203.86.207.101]) by mx.google.com with ESMTPSA id hi5sm34327644pbb.43.2013.11.11.20.15.39 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 11 Nov 2013 20:15:41 -0800 (PST) From: Aaron Morton Content-Type: multipart/alternative; boundary="Apple-Mail=_6EB42C1F-A12F-495E-A085-CE9F852A8755" Message-Id: <9FD742B0-27D6-4333-A289-18E9A25FCA98@thelastpickle.com> Mime-Version: 1.0 (Mac OS X Mail 7.0 \(1822\)) Subject: Re: OOM while reading key cache Date: Tue, 12 Nov 2013 17:15:36 +1300 References: To: Cassandra User In-Reply-To: X-Mailer: Apple Mail (2.1822) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_6EB42C1F-A12F-495E-A085-CE9F852A8755 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 > -6 machines which 8gb RAM each and three 150GB disks each > -default heap configuration With 8GB the default heap is 2GB, try kicking that up to 4GB and a 600 = to 800 MB new heap.=20 I would guess for the data load you have 2GB is not enough.=20 hope that helps.=20 ----------------- Aaron Morton New Zealand @aaronmorton Co-Founder & Principal Consultant Apache Cassandra Consulting http://www.thelastpickle.com On 8/11/2013, at 11:31 pm, olek.stasiak@gmail.com wrote: > Hello, > I'm facing OOM on reading key_cache > Cluster conf is as follows: > -6 machines which 8gb RAM each and three 150GB disks each > -default heap configuration > -deafult key cache configuration > -the biggest keyspace has abt 500GB size (RF: 2, so in fact there is > 250GB of raw data). >=20 > After upgrading first of the machines from 1.2.11 to 2.0.2 i've = recieved error: > INFO [main] 2013-11-08 10:53:16,716 AutoSavingCache.java (line 114) > reading saved cache > = /home/synat/nosql_filesystem/cassandra/data/saved_caches/production_storag= e-METADATA-KeyCache-b.db > ERROR [main] 2013-11-08 10:53:16,895 CassandraDaemon.java (line 478) > Exception encountered during startup > java.lang.OutOfMemoryError: Java heap space > at = org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394) > at = org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.ja= va:355) > at = org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(C= acheService.java:352) > at = org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:= 119) > at = org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:26= 4) > at = org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFa= milyStore.java:409) > at = org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFa= milyStore.java:381) > at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:314) > at org.apache.cassandra.db.Keyspace.(Keyspace.java:268) > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:110) > at org.apache.cassandra.db.Keyspace.open(Keyspace.java:88) > at = org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:27= 4) > at = org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java= :461) > at = org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:504= ) >=20 >=20 > Error appears every start, so I've decided to disable key cache (this > was not helpful) and temporarily moved key cache out of cache folder > (file was of size 13M). That helps in starting node, but this is only > workaround and it's not demanded configuration. Anyone has any idea > what is the real cause of problem with oom? > best regards > Aleksander > ps. I've still 5 nodes to upgrade, I'll inform if problem apperas on = the rest. --Apple-Mail=_6EB42C1F-A12F-495E-A085-CE9F852A8755 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1
-6 machines which 8gb RAM = each and three 150GB disks each
-default heap = configuration
With 8GB the default heap is 2GB, try kicking = that up to 4GB and a 600 to 800 MB new = heap. 

I would guess for the data load =  you have 2GB is not enough. 

hope = that helps. 

http://www.thelastpickle.com

On 8/11/2013, at 11:31 pm, olek.stasiak@gmail.com = wrote:

Hello,
I'm facing OOM on reading key_cache
Cluster = conf is as follows:
-6 machines which 8gb RAM each and three 150GB = disks each
-default heap configuration
-deafult key cache = configuration
-the biggest keyspace has abt 500GB size (RF: 2, so in = fact there is
250GB of raw data).

After upgrading first of the = machines from 1.2.11 to 2.0.2 i've recieved error:
INFO [main] = 2013-11-08 10:53:16,716 AutoSavingCache.java (line 114)
reading saved = cache
/home/synat/nosql_filesystem/cassandra/data/saved_caches/producti= on_storage-METADATA-KeyCache-b.db
ERROR [main] 2013-11-08 = 10:53:16,895 CassandraDaemon.java (line 478)
Exception encountered = during startup
java.lang.OutOfMemoryError: Java heap space
=        at = org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:394)        at = org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.ja= va:355)
       at = org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(C= acheService.java:352)
       at = org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:= 119)
       at = org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.j= ava:264)
       at = org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFa= milyStore.java:409)
       at = org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFa= milyStore.java:381)
       at = org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:314)
=        at = org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:268)
=        at = org.apache.cassandra.db.Keyspace.open(Keyspace.java:110)
=        at = org.apache.cassandra.db.Keyspace.open(Keyspace.java:88)
=        at = org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:27= 4)
       at = org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java= :461)
       at = org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:504= )


Error appears every start, so I've decided to disable key = cache (this
was not helpful) and temporarily moved key cache out of = cache folder
(file was of size 13M). That helps in starting node, but = this is only
workaround and it's not demanded configuration. Anyone = has any idea
what is the real cause of problem with oom?
best = regards
Aleksander
ps. I've still 5 nodes to upgrade, I'll inform = if problem apperas on the = rest.

= --Apple-Mail=_6EB42C1F-A12F-495E-A085-CE9F852A8755--