Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CBFBB9091 for ; Mon, 21 May 2012 20:16:11 +0000 (UTC) Received: (qmail 5122 invoked by uid 500); 21 May 2012 20:16:08 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 4958 invoked by uid 500); 21 May 2012 20:16:08 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 4938 invoked by uid 99); 21 May 2012 20:16:08 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 May 2012 20:16:08 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of yiming.sun@gmail.com designates 209.85.212.172 as permitted sender) Received: from [209.85.212.172] (HELO mail-wi0-f172.google.com) (209.85.212.172) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 May 2012 20:16:03 +0000 Received: by wibhj8 with SMTP id hj8so2002975wib.7 for ; Mon, 21 May 2012 13:15:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=oV2prWbbnv3aSjefn8J3ee1NDt/sKRYqPE4WWw/N0bg=; b=Dg+WpcKeSY82SRd4m3LyEqsG27PriNy+Aer+keTVs3aYBh/u19I8ezroj+RfjQhbDO 04fu4oRrK8TJdAyS8l4hFCp+2R8WMC72qdSFoAMFA/JKsqM+9UYipU9RfcIjgT/Yil2e kA6QEl7onKJW1SeQCVA55erkLcjeslGFTyXPLCSr8LNj8X0RXb0BNkV/6yV4XIVJeyw2 6OpbLcNtcfL3tXJJ5to9GnmTI6AjsnyoT6mGjJJc8N5KeJze0BAmCEF6iG6eARKNmE3R a5LlYVT2fStYinPiXCtTQwhyVocgU15RSixvsugV/VIoheRsqWVGmC2+SQIS/FfUd189 T1Bw== Received: by 10.216.142.194 with SMTP id i44mr13971621wej.88.1337631342299; Mon, 21 May 2012 13:15:42 -0700 (PDT) MIME-Version: 1.0 Received: by 10.216.232.92 with HTTP; Mon, 21 May 2012 13:15:22 -0700 (PDT) In-Reply-To: <06DF9C1C-C699-4A44-82D0-925DF29B32C3@thelastpickle.com> References: <06DF9C1C-C699-4A44-82D0-925DF29B32C3@thelastpickle.com> From: Yiming Sun Date: Mon, 21 May 2012 16:15:22 -0400 Message-ID: Subject: Re: how can we get (a lot) more performance from cassandra To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=0016e6d99a92ff89f904c091906a X-Virus-Checked: Checked by ClamAV on apache.org --0016e6d99a92ff89f904c091906a Content-Type: text/plain; charset=ISO-8859-1 Hi Aaron, I don't know if you could elaborate a bit more on each of the points you suggested. Thanks. -- Y. On Sun, May 20, 2012 at 7:29 PM, aaron morton wrote: > I would look into the problems you are having with GC... > > The server log shows the GC ParNew frequently gets longer than 200ms, > often in the range of 4-5seconds. But nowhere near 15 seconds (which is an > indication that JVM heap is being swapped out). > > > Then check the throughput on the san and the steal on the VM's. > > Also try to isolate the issue to "it takes this long for a single thread > to make this call" > > In a low write environment reads should be flying along. > > Cheers > > ----------------- > Aaron Morton > Freelance Developer > @aaronmorton > http://www.thelastpickle.com > > On 17/05/2012, at 1:44 PM, Yiming Sun wrote: > > Hi Aaron T., No, actually we haven't, but this sounds like a good > suggestion. I can definitely try THIS before jumping into other things > such as enabling row cache etc. Thanks! > > -- Y. > > On Wed, May 16, 2012 at 9:38 PM, Aaron Turner wrote: > >> On Wed, May 16, 2012 at 12:59 PM, Yiming Sun >> wrote: >> > Hello, >> > >> > I asked the question as a follow-up under a different thread, so I >> figure I >> > should ask here instead in case the other one gets buried, and besides, >> I >> > have a little more information. >> > >> > "We find the lack of performance disturbing" as we are only able to get >> > about 3-4MB/sec read performance out of Cassandra. >> > >> > We are using cassandra as the backend for an IR repository of digital >> texts. >> > It is a read-mostly repository with occasional writes. Each row >> represents >> > a book volume, and each column of a row represents a page of the volume. >> > Granted the data size is small -- the average size of a column text is >> > 2-3KB, and each row has about 250 columns (varies quite a bit from one >> > volume to another). >> > >> > Currently we are running a 3-node cluster, and will soon be upgraded to >> a >> > 6-node setup. Each node is a VM with 4 cores and 16GB of memory. All >> VMs >> > use SAN as disk storage. >> > >> > To retrieve a volume, a slice query is used via Hector that specifies >> the >> > row key (the volume), and a list of column keys (pages), and the >> consistency >> > level is set to ONE. It is typical to retrieve multiple volumes per >> > request. >> > >> > The read rate that I have been seeing is about 3-4 MB/sec, and that is >> > reading the raw bytes... using string serializer the rate is even lower, >> > about 2.2MB/sec. >> > >> > The server log shows the GC ParNew frequently gets longer than 200ms, >> often >> > in the range of 4-5seconds. But nowhere near 15 seconds (which is an >> > indication that JVM heap is being swapped out). >> > >> > Currently we have not added JNA. From a blog post, it seems JNA is >> able to >> > increase the performance by 13%, and we are hoping to increase the >> > performance by something more like 1300% (3-4 MB/sec is just >> disturbingly >> > low). And we are hesitant to disable swap entirely since one of the >> nodes >> > is running a couple other services >> > >> > Do you have any suggestions on how we may boost the performance? >> Thanks! >> >> Have you tried using more threads on the client side? Generally >> speaking, when I need faster read/write performance I look for ways to >> parallelize my requests and it scales pretty much linearly. >> >> >> -- >> Aaron Turner >> http://synfin.net/ Twitter: @synfinatic >> http://tcpreplay.synfin.net/ - Pcap editing and replay tools for Unix & >> Windows >> Those who would give up essential Liberty, to purchase a little temporary >> Safety, deserve neither Liberty nor Safety. >> -- Benjamin Franklin >> "carpe diem quam minimum credula postero" >> > > > --0016e6d99a92ff89f904c091906a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Aaron,

I don't know if you could elaborate a bit = more on each of the points you suggested. =A0Thanks.

-- Y.

On Sun, May 20, 2012 at 7:29 PM,= aaron morton <aaron@thelastpickle.com> wrote:
I would look into the problems you are having with GC...

The server log shows the GC ParNew frequ= ently gets longer than 200ms, often in the range of 4-5seconds. =A0But nowh= ere near 15 seconds (which is an indication that JVM heap is being swapped = out).

Then check the throughput on the san= and the steal on the VM's.

Also try to isolat= e the issue to "it takes this long for a single thread to make this ca= ll"

In a low write environment reads should be flying along= .=A0

Cheers

<= div style=3D"word-wrap:break-word">
-----------------
Aaron Morton
Freelance Deve= loper
@aaronmorton

On 17/05/2012, at 1:44 PM, Yiming Sun wrote:

Hi Aaron T., =A0No, actually we haven't, but this so= unds like a good suggestion. =A0I can definitely try THIS before jumping in= to other things such as enabling row cache etc. Thanks!

-- Y.

On Wed, May 16, 2012 at 9:38 PM, Aaron Turner <synfinatic@gmail.com= > wrote:
On Wed, May 16, 2012 at 12:59 PM, Yiming Sun <yiming.sun@gmail.com> wrot= e:
> Hello,
>
> I asked the question as a follow-up under a different thread, so I fig= ure I
> should ask here instead in case the other one gets buried, and besides= , I
> have a little more information.
>
> "We find the lack of performance disturbing" as we are only = able to get
> about 3-4MB/sec read performance out of Cassandra.
>
> We are using cassandra as the backend for an IR repository of digital = texts.
> It is a read-mostly repository with=A0occasional=A0writes. =A0Each row= represents
> a book volume, and each column of a row represents a page of the volum= e.
> =A0Granted the data size is small -- the average size of a column text= is
> 2-3KB, and each row has about 250 columns (varies quite a bit from one=
> volume to another).
>
> Currently we are running a 3-node cluster, and will soon be upgraded t= o a
> 6-node setup. =A0Each node is a VM with 4 cores and 16GB of memory. = =A0All VMs
> use SAN as disk storage.
>
> To retrieve a volume, a slice query is used via Hector that specifies = the
> row key (the volume), and a list of column keys (pages), and the consi= stency
> level is set to ONE. =A0It is typical to retrieve multiple volumes per=
> request.
>
> The read rate that I have been seeing is about 3-4 MB/sec, and that is=
> reading the raw bytes... using string serializer the rate is even lowe= r,
> about 2.2MB/sec.
>
> The server log shows the GC ParNew frequently gets longer than 200ms, = often
> in the range of 4-5seconds. =A0But nowhere near 15 seconds (which is a= n
> indication that JVM heap is being swapped out).
>
> Currently we have not added JNA. =A0From a blog post, it seems JNA is = able to
> increase the performance by 13%, and we are hoping to increase the
> performance by something more like 1300% (3-4 MB/sec is just disturbin= gly
> low). =A0And we are hesitant to disable swap entirely since one of the= nodes
> is running a couple other services
>
> Do you have any suggestions on how we may boost the performance? =A0Th= anks!

Have you tried using more threads on the client side? =A0Genera= lly
speaking, when I need faster read/write performance I look for ways to
parallelize my requests and it scales pretty much linearly.


--
Aaron Turner
http://synfin.net/=A0 = =A0 =A0 =A0=A0 Twitter: @synfinatic
http://tcpreplay= .synfin.net/ - Pcap editing and replay tools for Unix & Windows
Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety.
=A0 =A0 -- Benjamin Franklin
"carpe diem quam minimum credula postero"



--0016e6d99a92ff89f904c091906a--