Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Subject: Re:  OOM on CompressionMetadata.readChunkOffsets(..)
From: Mick Semb Wever <mck@apache.org>
To: user@cassandra.apache.org
In-Reply-To: <1320048473.6216.6.camel@localhost>
References: <1320044410.6216.3.camel@localhost>
	 <1320048473.6216.6.camel@localhost>
Content-Type: multipart/signed; micalg="pgp-sha1";
 protocol="application/pgp-signature"; boundary="=-n8+e09oj5MfQX6DjSQ9N"
Date: Mon, 31 Oct 2011 13:05:49 +0100
Message-ID: <1320062749.6216.20.camel@localhost>
Mime-Version: 1.0


--=-n8+e09oj5MfQX6DjSQ9N
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Mon, 2011-10-31 at 09:07 +0100, Mick Semb Wever wrote:
> The read pattern of these rows is always in bulk so the chunk_length
> could have been much higher so to reduce memory usage (my largest
> sstable is 61G).=20

Isn't CompressionMetadata.readChunkOffsets(..) rather dangerous here?

Given a 60G sstable, even with 64kb chunk_length, to read just that one
sstable requires close to 8G free heap memory...

Especially when the default for cassandra is 4G heap in total.

~mck

--=20
"Anyone who has attended a computer conference in a fancy hotel can tell
you that a sentence like "You're one of those computer people, aren't
you?" is roughly equivalent to "Look, another amazingly mobile form of
slime mold!" in the mouth of a hotel cocktail waitress." Elizabeth
Zwicky=20

| http://semb.wever.org | http://sesat.no |
| http://tech.finn.no   | Java XSS Filter |

--=-n8+e09oj5MfQX6DjSQ9N
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: This is a digitally signed message part

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)

iEYEABECAAYFAk6ujx0ACgkQkdPrePiuutOXYgCfYjbSVSgdfacrRxBGqp0wC3YU
mMoAn0AkfL8o3w3ychNln4yZyg0Xq0h7
=6Ary
-----END PGP SIGNATURE-----

--=-n8+e09oj5MfQX6DjSQ9N--