Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 84010 invoked from network); 11 Jun 2010 16:20:35 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 11 Jun 2010 16:20:35 -0000 Received: (qmail 91764 invoked by uid 500); 11 Jun 2010 16:20:34 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 91742 invoked by uid 500); 11 Jun 2010 16:20:33 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 91734 invoked by uid 99); 11 Jun 2010 16:20:33 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 16:20:33 +0000 X-ASF-Spam-Status: No, hits=2.9 required=10.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [74.125.83.172] (HELO mail-pv0-f172.google.com) (74.125.83.172) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 11 Jun 2010 16:20:27 +0000 Received: by pvf33 with SMTP id 33so280060pvf.31 for ; Fri, 11 Jun 2010 09:20:06 -0700 (PDT) MIME-Version: 1.0 Received: by 10.141.100.21 with SMTP id c21mr1582187rvm.101.1276273206540; Fri, 11 Jun 2010 09:20:06 -0700 (PDT) Received: by 10.140.162.10 with HTTP; Fri, 11 Jun 2010 09:20:06 -0700 (PDT) In-Reply-To: References: <729091.43486.qm@web53001.mail.re2.yahoo.com> Date: Fri, 11 Jun 2010 09:20:06 -0700 Message-ID: Subject: Re: Cassandra Write Performance, CPU usage From: Mike Malone To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=000e0cd139541cbc420488c38353 --000e0cd139541cbc420488c38353 Content-Type: text/plain; charset=ISO-8859-1 Jonathan, while I agree with you re: this being an unusual load for the system, it is interesting that he's found at least one use-case where Cassandra is CPU-bound, not IO-bound. I'd definitely be interested in learning what his critical path is and seeing if there's some low-hanging fruit that may improve performance overall. I have also noticed very high CPU usage during high write loads and have wondered whether write speed and throughput could be improved by improving some of the algorithms along that path. I'm nowhere near being an expert on the whole Java ecosystem, but I've had good luck with the `jvisualvm` tool that comes with Java SE 6. It's a nice lightweight CPU and memory profiling tool that can attach to a running process like Cassandra and dump stats in real time. Mike On Thu, Jun 10, 2010 at 7:39 PM, Jonathan Shook wrote: > You are testing Cassandra in a way that it was not designed to be used. > Bandwidth to disk is not a meaningful example for nearly anything > except for filesystem benchmarking and things very nearly the same as > filesystem benchmarking. > Unless the usage patterns of your application match your test data, > there is not a good reason to expect a strong correlation between this > test and actual performance. > > Cassandra is not simply shuffling data through IO when you write. > There are calculations that have to be done as writes filter their way > through various stages of processing. The point of this is to minimize > the overall effort Cassandra has to make in order to retrieve the data > again. One example would be bloom filters. Each column that is written > requires bloom filter processing and potentially auxiliary IO. Some of > these steps are allowed to happen in the background, but if you try, > you can cause them to stack up on top of the available CPU and memory > resources. > > In such a case (continuous bulk writes), you are causing all of these > costs to be taken in more of a synchronous (not delayed) fashion. You > are not allowing the background processing that helps reduce client > blocking (by deferring some processing) to do its magic. > > > > On Thu, Jun 10, 2010 at 7:42 PM, Rishi Bhardwaj > wrote: > > Hi > > I am investigating Cassandra write performance and see very heavy CPU > usage > > from Cassandra. I have a single node Cassandra instance running on a dual > > core (2.66 Ghz Intel ) Ubuntu 9.10 server. The writes to Cassandra are > being > > generated from the same server using BatchMutate(). The client makes > exactly > > one RPC call at a time to Cassandra. Each BatchMutate() RPC contains 2 MB > of > > data and once it is acknowledged by Cassandra, the next RPC is done. > > Cassandra has two separate disks, one for commitlog with a sequential b/w > of > > 130MBps and the other a solid state disk for data with b/w of 90MBps. > Tuning > > various parameters, I observe that I am able to attain a maximum write > > performance of about 45 to 50 MBps from Cassandra. I see that the > Cassandra > > java process consistently uses 100% to 150% of CPU resources (as shown by > > top) during the entire write operation. Also, iostat clearly shows that > the > > max disk bandwidth is not reached anytime during the write operation, > every > > now and then the i/o activity on "commitlog" disk and the data disk spike > > but it is never consistently maintained by cassandra close to their > peak. I > > would imagine that the CPU is probably the bottleneck here. Does anyone > have > > any idea why Cassandra beats the heck out of the CPU here? Any > suggestions > > on how to go about finding the exact bottleneck here? > > Some more information about the writes: I have 2 column families, the > data > > though is mostly written in one column family with column sizes of around > > 32k and each row having around 256 or 512 columns. I would really > appreciate > > any help here. > > Thanks, > > Rishi > > > > > --000e0cd139541cbc420488c38353 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Jonathan, while I agree with you re: this being an unusual load for the sys= tem, it is interesting that he's found at least one use-case where Cass= andra is CPU-bound, not IO-bound. I'd definitely be interested in learn= ing what his critical path is and seeing if there's some low-hanging fr= uit that may improve performance overall. I have also noticed very high CPU= usage during high write loads and have wondered whether write speed and th= roughput could be improved by improving some of the algorithms along that p= ath.

I'm nowhere near being an expert on the whole Java ecosy= stem, but I've had good luck with the `jvisualvm` tool that comes with = Java SE 6. It's a nice lightweight CPU and memory profiling tool that c= an attach to a running process like Cassandra and dump stats in real time.<= /div>

Mike

On Thu, Jun 10, = 2010 at 7:39 PM, Jonathan Shook <jshook@gmail.com> wrote:
You are testing Cassandra in a way that it was not designed to be used.
Bandwidth to disk is not a meaningful example for nearly anything
except for filesystem benchmarking and things very nearly the same as
filesystem benchmarking.
Unless the usage patterns of your application match your test data,
there is not a good reason to expect a strong correlation between this
test and actual performance.

Cassandra is not simply shuffling data through IO when you write.
There are calculations that have to be done as writes filter their way
through various stages of processing. The point of this is to minimize
the overall effort Cassandra has to make in order to retrieve the data
again. One example would be bloom filters. Each column that is written
requires bloom filter processing and potentially auxiliary IO. Some of
these steps are allowed to happen in the background, but if you try,
you can cause them to stack up on top of the available CPU and memory
resources.

In such a case (continuous bulk writes), you are causing all of these
costs to be taken in more of a synchronous (not delayed) fashion. You
are not allowing the background processing that helps reduce client
blocking (by deferring some processing) to do its magic.



On Thu, Jun 10, 2010 at 7:42 PM, Rishi Bhardwaj <khichrishi@yahoo.com> wrote:
> Hi
> I am investigating Cassandra write performance and see very heavy CPU = usage
> from Cassandra. I have a single node Cassandra instance running on a d= ual
> core (2.66 Ghz Intel ) Ubuntu 9.10 server. The writes to Cassandra are= being
> generated from the same server using BatchMutate(). The client makes e= xactly
> one RPC call at a time to Cassandra. Each BatchMutate() RPC contains 2= MB of
> data and once it is acknowledged by Cassandra, the next RPC is done. > Cassandra has two separate disks, one for commitlog with a sequential = b/w of
> 130MBps and the other a solid state disk for data with b/w of 90MBps. = Tuning
> various parameters, I observe that I am able to attain a maximum write=
> performance of about 45 to 50 MBps from Cassandra. I see=A0that the Ca= ssandra
> java process consistently uses 100% to 150% of CPU resources (as shown= by
> top) during the entire write operation. Also, iostat clearly shows tha= t the
> max disk bandwidth is not reached anytime during the write operation, = every
> now and then the i/o activity on "commitlog" disk and the da= ta disk spike
> but it is never consistently maintained by cassandra close to their pe= ak.=A0I
> would imagine that the CPU is probably the bottleneck here. Does anyon= e have
> any idea why Cassandra beats the heck out of the CPU here? Any suggest= ions
> on how to go about finding the exact bottleneck here?
> Some more information about the writes: I have 2 column families, the = data
> though is mostly written in one column family with column sizes of aro= und
> 32k and each row having around 256 or 512 columns. I would really appr= eciate
> any help here.
> Thanks,
> Rishi
>
>

--000e0cd139541cbc420488c38353--