Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 75583 invoked from network); 7 Apr 2011 00:59:19 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 7 Apr 2011 00:59:19 -0000 Received: (qmail 88392 invoked by uid 500); 7 Apr 2011 00:59:17 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 88371 invoked by uid 500); 7 Apr 2011 00:59:17 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 88363 invoked by uid 99); 7 Apr 2011 00:59:17 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Apr 2011 00:59:17 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of openvictor@gmail.com designates 209.85.214.172 as permitted sender) Received: from [209.85.214.172] (HELO mail-iw0-f172.google.com) (209.85.214.172) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Apr 2011 00:59:13 +0000 Received: by iwn39 with SMTP id 39so2452059iwn.31 for ; Wed, 06 Apr 2011 17:58:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=dCEDJVWBhRAK+bX783fYf885WazbJFVm+HytQ98aFFw=; b=nxGIWMnMgYDlwJq39bzWSoBEzBggow/+0AP1BlJOIx1bqcgTDEORBwt0vcoKL9l4yD eEtQBDn8xxSwXjhM3WNeSE03Criu8KpTZ44FfYqacSErH/hPt1qSHTKXg/OZm9iQebn3 8p9pyq15g8gMrEgZP0GnZBiwF79aa22L3Bfhw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=PyJ5xwI3NyGRsz2bszcmzgmC00dysH8NS3occIlDwo0sRi61DjpdQTqMN3IS++UjfL 5EVC4ZcXxSNnP9x5WDI6eRWnyqoeWqa+A6WBofz7hKxIdpnV8fyvccST9ASHvArrQl8f pNR69PBIKwI6wPhboZy6QTCDyMsnEm8pdmcRU= MIME-Version: 1.0 Received: by 10.42.168.198 with SMTP id x6mr415939icy.273.1302137932634; Wed, 06 Apr 2011 17:58:52 -0700 (PDT) Received: by 10.43.63.142 with HTTP; Wed, 6 Apr 2011 17:58:52 -0700 (PDT) In-Reply-To: References: Date: Wed, 6 Apr 2011 20:58:52 -0400 Message-ID: Subject: Re: Abnormal memory consumption From: openvictor Open To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=90e6ba6e8a20ec5b2e04a0499c97 --90e6ba6e8a20ec5b2e04a0499c97 Content-Type: text/plain; charset=ISO-8859-1 Hello Paul, Thank you for the tip. The random port attribution policy of JMX was really making me mad ! Good to know there is a solution for that problem. Concerning the rest of the conversation, my only concern is that as an administrator and a student it is hard to constantly watch Cassandra instances so that they don't crash. As much as I love the principle of Cassandra, being constantly afraid of memory consumption is an issue in my opinion. That being said, I took a new 16 Gb server today, but I don't want Cassandra to eat up everything if it is not needed, because Cassandra will have some neighbors such as Tomcat, solR on this server. And for me it is very weird that on my small instance where I put a lot of constraints like throughput_memtableInMb to 6 Cassandra uses 600 Mb of ram for 6 Mb of data. It seems to be a little bit of an overkill to me... And so far I failed to find any information on what this massive overhead can be... Thank you for your answers and for taking the time to answer my questions. 2011/4/6 Paul Choi > You can use JMX over ssh by doing this: > http://blog.reactive.org/2011/02/connecting-to-cassandra-jmx-via-ssh.html > Basically, you use SSH -D to do dynamic application port forwarding. > > In terms of scaling, you'll be able to afford 120GB RAM/node in 3 years > if you're successful. Or, a machine with much less RAM and flash-based > storage. :) > Seriously, though, the formula in the tuning guidelines is a guideline. You > can probably get acceptable performance with much less. If not, you can > shard your app such that you host a few Cfs per cluster. I doubt you'll need > to though. > > > From: openvictor Open > Reply-To: > Date: Mon, 4 Apr 2011 18:24:25 -0400 > To: > Subject: Re: Abnormal memory consumption > > Okay, I see. But isn't there a big issue for scaling here ? > Imagine that I am the developper of a certain very successful website : At > year 1 I need 20 CF. I might need to have 8Gb of RAM. Year 2 I need 50 CF > because I added functionalities to my wonderful webiste will I need 20 Gb of > RAM ? And if at year three I had 300 Column families, will I need 120 Gb of > ram / node ? Or did I miss something about memory consuption ? > > Thank you very much, > > Victor > > 2011/4/4 Peter Schuller > >> > And about the production 7Gb or RAM is sufficient ? Or 11 Gb is the >> minimum >> > ? >> > Thank you for your inputs for the JVM I'll try to tune that >> >> Production mem reqs are mostly dependent on memtable thresholds: >> >> http://www.datastax.com/docs/0.7/operations/tuning >> >> If you enable key caching or row caching, you will have to adjust >> accordingly as well. >> >> -- >> / Peter Schuller >> > > --90e6ba6e8a20ec5b2e04a0499c97 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hello Paul,

Thank you for the tip. The random port attribution polic= y of JMX was really making me mad ! Good to know there is a solution for th= at problem.

Concerning the rest of the conversation, my only concern= is that as an administrator and a student it is hard to constantly watch= =A0 Cassandra instances so that they don't crash. As much as I love the= principle of Cassandra, being constantly afraid of memory consumption is a= n issue in my opinion. That being said, I took a new 16 Gb server today, bu= t I don't want Cassandra to eat up everything if it is not needed, beca= use Cassandra will have some neighbors such as Tomcat, solR on this server.=
And for me it is very weird that on my small instance where I put a lot of = constraints like throughput_memtableInMb to 6 Cassandra uses 600 Mb of ram = for 6 Mb of data. It seems to be a little bit of an overkill to me... And s= o far I failed to find any information on what this massive overhead can be= ...

Thank you for your answers and for taking the time to answer my questio= ns.

2011/4/6 Paul Choi = <paulchoi@plaxo.com>=
You can use JMX over ssh by doing this:
Basically, you use SSH -D to do dynamic application port forwarding.

In terms of scaling, you'll be able to afford 120GB RAM/node in 3 = years if you're successful. Or, a machine with much less RAM and flash-= based storage. :)
Seriously, though, the formula in the tuning guidelines is a guideline= . You can probably get acceptable performance with much less. If not, you c= an shard your app such that you host a few Cfs per cluster. I doubt you'= ;ll need to though.


From: openvictor Open <openvictor@gmail.com= >
Reply-To: <user@cassandra.apache.org>=
Date: Mon, 4 Apr 2011 18:24:25 -040= 0
To: <user@cassandra.apache.org>
Subject: Re: Abnormal memory consum= ption

Okay, I see. But isn't there a big issue for scaling here ?
Imagine that I am the developper of a certain very successful website : At = year 1 I need 20 CF. I might need to have 8Gb of RAM. Year 2 I need 50 CF b= ecause I added functionalities to my wonderful webiste will I need 20 Gb of= RAM ? And if at year three I had 300 Column families, will I need 120 Gb of ram / node ? Or did I miss some= thing about memory consuption ?

Thank you very much,

Victor

2011/4/4 Peter Schuller <peter.schull= er@infidyne.com>
> And about the production 7Gb or RAM is sufficient ? Or 11 Gb is t= he minimum
> ?
> Thank you for your inputs for the JVM I'll try to tune that

Production mem reqs are mostly dependent on memtable thresholds:

=A0 http://www.datastax.com/docs/0.7/operations/tuning

If you enable key caching or row caching, you will have to adjust
accordingly as well.

--
/ Peter Schuller

--90e6ba6e8a20ec5b2e04a0499c97--