incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: how can we get (a lot) more performance from cassandra
Date Tue, 22 May 2012 11:18:27 GMT
> I would look into the problems you are having with GC...


When ParNew runs the jvm pauses https://blogs.oracle.com/jonthecollector/entry/our_collectors
. If it's pausing for 4 seconds it's not processing queries. 

> Then check the throughput on the san and the steal on the VM's.


Check to see if the IO system is overloaded / has a long latencies see http://spyced.blogspot.co.nz/2010/01/linux-performance-basics.html

Steal time is the time your virtual cpu is waiting to get the physical cpu, you can see it
in top (http://manpages.ubuntu.com/manpages/lucid/man1/top.1.html) and vmstat

> Also try to isolate the issue to "it takes this long for a single thread to make this
call"
Things like "3-4 MB/sec" don't explain what you are doing. If you say "we are making this
sort of query, on a single thread, and it takes this long or we only get this many a second"
it's easier. 

Cheers


-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 22/05/2012, at 8:15 AM, Yiming Sun wrote:

> Hi Aaron,
> 
> I don't know if you could elaborate a bit more on each of the points you suggested. 
Thanks.
> 
> -- Y.
> 
> On Sun, May 20, 2012 at 7:29 PM, aaron morton <aaron@thelastpickle.com> wrote:
> I would look into the problems you are having with GC...
> 
>> The server log shows the GC ParNew frequently gets longer than 200ms, often in the
range of 4-5seconds.  But nowhere near 15 seconds (which is an indication that JVM heap is
being swapped out).
> 
> Then check the throughput on the san and the steal on the VM's.
> 
> Also try to isolate the issue to "it takes this long for a single thread to make this
call"
> 
> In a low write environment reads should be flying along. 
> 
> Cheers
> 
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 17/05/2012, at 1:44 PM, Yiming Sun wrote:
> 
>> Hi Aaron T.,  No, actually we haven't, but this sounds like a good suggestion.  I
can definitely try THIS before jumping into other things such as enabling row cache etc. Thanks!
>> 
>> -- Y.
>> 
>> On Wed, May 16, 2012 at 9:38 PM, Aaron Turner <synfinatic@gmail.com> wrote:
>> On Wed, May 16, 2012 at 12:59 PM, Yiming Sun <yiming.sun@gmail.com> wrote:
>> > Hello,
>> >
>> > I asked the question as a follow-up under a different thread, so I figure I
>> > should ask here instead in case the other one gets buried, and besides, I
>> > have a little more information.
>> >
>> > "We find the lack of performance disturbing" as we are only able to get
>> > about 3-4MB/sec read performance out of Cassandra.
>> >
>> > We are using cassandra as the backend for an IR repository of digital texts.
>> > It is a read-mostly repository with occasional writes.  Each row represents
>> > a book volume, and each column of a row represents a page of the volume.
>> >  Granted the data size is small -- the average size of a column text is
>> > 2-3KB, and each row has about 250 columns (varies quite a bit from one
>> > volume to another).
>> >
>> > Currently we are running a 3-node cluster, and will soon be upgraded to a
>> > 6-node setup.  Each node is a VM with 4 cores and 16GB of memory.  All VMs
>> > use SAN as disk storage.
>> >
>> > To retrieve a volume, a slice query is used via Hector that specifies the
>> > row key (the volume), and a list of column keys (pages), and the consistency
>> > level is set to ONE.  It is typical to retrieve multiple volumes per
>> > request.
>> >
>> > The read rate that I have been seeing is about 3-4 MB/sec, and that is
>> > reading the raw bytes... using string serializer the rate is even lower,
>> > about 2.2MB/sec.
>> >
>> > The server log shows the GC ParNew frequently gets longer than 200ms, often
>> > in the range of 4-5seconds.  But nowhere near 15 seconds (which is an
>> > indication that JVM heap is being swapped out).
>> >
>> > Currently we have not added JNA.  From a blog post, it seems JNA is able to
>> > increase the performance by 13%, and we are hoping to increase the
>> > performance by something more like 1300% (3-4 MB/sec is just disturbingly
>> > low).  And we are hesitant to disable swap entirely since one of the nodes
>> > is running a couple other services
>> >
>> > Do you have any suggestions on how we may boost the performance?  Thanks!
>> 
>> Have you tried using more threads on the client side?  Generally
>> speaking, when I need faster read/write performance I look for ways to
>> parallelize my requests and it scales pretty much linearly.
>> 
>> 
>> --
>> Aaron Turner
>> http://synfin.net/         Twitter: @synfinatic
>> http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & Windows
>> Those who would give up essential Liberty, to purchase a little temporary
>> Safety, deserve neither Liberty nor Safety.
>>     -- Benjamin Franklin
>> "carpe diem quam minimum credula postero"
>> 
> 
> 


Mime
View raw message