Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of agundabattula@threatmetrix.com
 designates 98.129.35.9 as permitted sender)
From: Ananth Gundabattula <agundabattula@threatmetrix.com>
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Tue, 18 Jun 2013 03:22:22 -0500
Subject: Re: What is the effect of reducing the thrift message sizes on GC
Thread-Topic: What is the effect of reducing the thrift message sizes on GC
Thread-Index: Ac5r/PXYlmsDK35YQrm9g1yPs4c56Q==
Message-ID: <CDE65425.11991%agundabattula@threatmetrix.com>
In-Reply-To: <155143A8-17F5-47AD-B0C4-EED4B0102085@thelastpickle.com>
Accept-Language: en-US
Content-Language: en-US
user-agent: Microsoft-MacOutlook/14.3.4.130416
acceptlanguage: en-US
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Thanks Aaron for the insight.

One quick question:

>The buffers are not pre allocated, but once they are allocated they are
>not returned. So it's only an issue if have lots of clients connecting
>and reading a lot of data.
So to understand you correctly, the buffer is allocated per client
connection and remains all the while during the JVM and is reused for each
request ?=20
If that is the case, then I am presuming there is no much gain by playing
around with this config with respect to optimizing for Gcs.

>reduce bloom filters, index intervals =8A.
Well we have tried all the configs as advised below (and others like key
cache sizes etc ) and hit a dead end and that is the reason for a 1.2.4
move. Thanks for all your thoughts and advice on this.


Regards,
Ananth=20


On 6/18/13 5:56 PM, "aaron morton" <aaron@thelastpickle.com> wrote:

>> *thrift_framed_transport_size_in_mb & thrift_max_message_length_in_mb*
>This control the max size of a bugger allocated by thrift when processing
>requests / responses. The buffers are not pre allocated, but once they
>are allocated they are not returned. So it's only an issue if have lots
>of clients connecting and reading a lot of data.
>
>> Our system is a very short column (both in number of columns and data
>>sizes
>> ) tables but having millions/billions of rows in each column family.
>If you have over 500 million rows per node you may be running into issues
>with the bloom filters and index samples.
>
>This typically looks like the heap usage does not reduce after CMS
>compaction has completed.
>
>Ensure the bloom_file_fp_chance on the CF's is set to 0.01 for size
>tiered compaction and 0.1 for levelled compaction. If you need to change
>it  run nodetool upgradesstables
>
>Then consider increasing the index_interval in the yaml file, see the
>comments.=20
>
>Note that v 1.2 moves the bloom filters off heap, so if you upgrade to
>1.2 it will probably resolve your issues.
>
>Cheers
>
>-----------------
>Aaron Morton
>Freelance Cassandra Consultant
>New Zealand
>
>@aaronmorton
>http://www.thelastpickle.com
>
>On 18/06/2013, at 7:30 PM, Ananth Gundabattula
><agundabattula@threatmetrix.com> wrote:
>
>> We are currently running on 1.1.10 and planning to migrate to a higher
>> version 1.2.4.
>>=20
>> The question pertains to tweaking all the knobs to reduce GC related
>>issues
>> ( we have been fighting a lot of really bad GC issues on 1.1.10 and met
>>with little
>> success all the way using 1.1.10)
>>=20
>> Taking into consideration GC tuning is a black art, I was wondering if
>>we
>> can have some good effect on the GC by tweaking the following settings:
>>=20
>> *thrift_framed_transport_size_in_mb & thrift_max_message_length_in_mb*
>> *
>> *
>> Our system is a very short column (both in number of columns and data
>>sizes
>> ) tables but having millions/billions of rows in each column family.
>>The typical
>> number of columns in each column family is 4. The typical lookup
>>involves
>> specifying the row key and fetching one column most of the times. The
>> writes are also similar except for one keyspace where the number of
>>columns
>> are 50 but very small data sizes per column.
>>=20
>> Assuming we can tweak the config values :
>> *
>> *
>> * > thrift_framed_transport_size_in_mb & *
>> * >  thrift_max_message_length_in_mb *
>>=20
>> to lower values in the above context, I was wondering if it helps in
>>the GC
>> being invoked less if the thrift settings reflect our data model reads
>>and writes ?
>>=20
>> For example: What is the impact by reducing the above config values on
>>the
>> GC to say 1 mb rather than say 15 or 16 ?
>>=20
>> Thanks a lot for your inputs and thoughts.
>>=20
>>=20
>> Regards,
>> Ananth
>