incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: performance tuning - where does the slowness come from?
Date Tue, 11 May 2010 19:26:27 GMT
We're working on better GC defaults for 0.6.2.  Thanks!

On Tue, May 11, 2010 at 12:00 PM, B. Todd Burruss <bburruss@real.com> wrote:
> another note on this ... since all my nodes are very well balanced and were
> started at the same time, i notice that they all do garbage collection at
> about the same time.  this of course causes a performance issue.
>
> i also have noticed that with the default JVM options and heavy load,
> ConcMarkSweepGC can fall behind and require the JVM to unexpectedly pause
> while it plays catchup.  adding the following param can help this out, it
> says to start processing when "CMS Old Gen" memory is 88% used.
>
> -XX:CMSInitiatingOccupancyFraction=88
>
> my understanding of how the default is calculated, mine was about 92%, so i
> only lowered this 4%, but now i can see GC starting earlier and haven't had
> a pause like a saw before.
>
>
> On 05/06/2010 02:42 PM, Todd Burruss wrote:
>
> i think you will see a slow down because of large values in your columns.
> make sure you take a look at MemtableThroughputInMB in your config.  if you
> are writing 1MB of data per row, then you'll probably want to increase this
> quite a bit so you are not constantly creating sstables.  can't recall, did
> you see compaction mgr reporting a lot of pending compactions?  maybe try to
> "chunk" your data into multiple columns or multiple rows.
>
> i too see slowness that exhibits in the same manner as you guys have
> described.  i'm still trying to track it down as well.
>
> On 05/06/2010 10:56 AM, Ran Tavory wrote:
>
> Jonathan, I think it's the case of large values in the columns. The
> problematic CF is a key-value store, so it has only one column per row,
> however the value of that column can be large. It's a java serialized object
> (uncompressed) which, may be 100s of bytes, maybe even a few megs. This CF
> also suffers from zero cache hits since each time a read is for a unique
> key.
> I ran stress.py and I see much better results (reads are < 1ms) so I assume
> my cluster is healthy, so I need to fix the app. Would 1meg bytes object
> explain a 30ms (sometimes even more) read latency? The boxes aren't fancy,
> not sure exactly what hardware we have there but it's "commodity"...
> Thanks!
>
> On Thu, May 6, 2010 at 5:22 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
>>
>> columns, not CFs.
>>
>> put another way, how wide are the rows in the slow CF?
>>
>> On Wed, May 5, 2010 at 11:30 PM, Ran Tavory <rantav@gmail.com> wrote:
>> > I have a few CFs but the one I'm seeing slowness in, which is the one
>> > with
>> > plenty of cache misses has only one column per key.
>> > Latency varies b/w 10m and 60ms but I'd say average is 30ms.
>> >
>> > On Thu, May 6, 2010 at 4:25 AM, Jonathan Ellis <jbellis@gmail.com>
>> > wrote:
>> >>
>> >> How many columns are in the rows you are reading from?
>> >>
>> >> 30ms is quite high, so I suspect you have relatively large rows, in
>> >> which case decreasing the column index threshold may help.
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
>> http://riptano.com
>
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Mime
View raw message