Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 71FA5955D for ; Mon, 31 Oct 2011 08:14:14 +0000 (UTC) Received: (qmail 83167 invoked by uid 500); 31 Oct 2011 08:14:12 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 83145 invoked by uid 500); 31 Oct 2011 08:14:11 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 83137 invoked by uid 99); 31 Oct 2011 08:14:10 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Oct 2011 08:14:10 +0000 X-ASF-Spam-Status: No, hits=0.7 required=5.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [193.213.115.49] (HELO mail49.e.nsc.no) (193.213.115.49) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 31 Oct 2011 08:14:05 +0000 Received: from hummingbird.wever.org (ti0022a380-dhcp1004.bb.online.no [88.89.20.243]) by mail49.nsc.no (8.14.4/8.14.4) with ESMTP id p9V8DgAD007344 for ; Mon, 31 Oct 2011 09:13:42 +0100 (MET) Received: by hummingbird.wever.org (Postfix, from userid 1009) id 48091195CA6B; Mon, 31 Oct 2011 09:13:42 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on hummingbird.wever.org X-Spam-score: score=0.0 required=3.0 tests=none autolearn=no Received: from [10.0.0.2] (unknown [10.0.0.2]) by hummingbird.wever.org (Postfix) with ESMTP id D791E195C056 for ; Mon, 31 Oct 2011 09:13:29 +0100 (CET) Subject: Re: get_range_slices OOM on CompressionMetadata.readChunkOffsets(..) From: Mick Semb Wever To: user@cassandra.apache.org In-Reply-To: <1320044410.6216.3.camel@localhost> References: <1320044410.6216.3.camel@localhost> Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-F/3hRZovJsb7BmLGza+B" Date: Mon, 31 Oct 2011 09:07:53 +0100 Message-ID: <1320048473.6216.6.camel@localhost> Mime-Version: 1.0 X-Mailer: Evolution 2.32.3 X-Virus-Status: No X-Virus-Checker-Version: clamassassin 1.2.4 with clamdscan / ClamAV 0.96.5/13869/Mon Oct 31 02:11:37 2011 --=-F/3hRZovJsb7BmLGza+B Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, 2011-10-31 at 08:00 +0100, Mick Semb Wever wrote: > After an upgrade to cassandra-1.0 any get_range_slices gives me: >=20 > java.lang.OutOfMemoryError: Java heap space > at org.apache.cassandra.io.compress.CompressionMetadata.readChunkOffsets= (CompressionMetadata.java:93) > at org.apache.cassandra.io.compress.CompressionMetadata.(Compressi= onMetadata.java:66) > at org.apache.cassandra.io.compress.CompressedRandomAccessReader.metadat= a(CompressedRandomAccessReader.java:53) > at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(Co= mpressedRandomAccessReader.java:63) > at org.apache.cassandra.io.sstable.SSTableReader.openDataReader(SSTableR= eader.java:896) > at org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.= java:72) > at org.apache.cassandra.io.sstable.SSTableReader.getScanner(SSTableReade= r.java:748) > at org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFac= tory.java:88) > at org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyS= tore.java:1310) > at org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.= java:840) > at org.apache.cassandra.thrift.CassandraServer.get_range_slices(Cassandr= aServer.java:698) >=20 >=20 > I set chunk_length_kb to 16 as my rows are very skinny (typically 100b) I see now this was a bad choice. The read pattern of these rows is always in bulk so the chunk_length could have been much higher so to reduce memory usage (my largest sstable is 61G). After changing the ckunk_length is there any way to rebuild just some sstables rather than having to do a full nodetool scrub ? ~mck --=20 =E2=80=9CAn idea is a point of departure and no more. As soon as you elabor= ate it, it becomes transformed by thought.=E2=80=9D - Pablo Picasso=20 | http://semb.wever.org | http://sesat.no | | http://tech.finn.no | Java XSS Filter | --=-F/3hRZovJsb7BmLGza+B Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) iEYEABECAAYFAk6uV1kACgkQkdPrePiuutNVowCfa+qJYiLPhFfrXJ3BpwuqfM9a 6QMAoKS8iATlgkRFJXoLBJaANlKIEvSY =6ZnS -----END PGP SIGNATURE----- --=-F/3hRZovJsb7BmLGza+B--