Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 7DA73200B4C for ; Fri, 8 Jul 2016 00:35:21 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 7C276160A72; Thu, 7 Jul 2016 22:35:21 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 7968E160A68 for ; Fri, 8 Jul 2016 00:35:20 +0200 (CEST) Received: (qmail 4321 invoked by uid 500); 7 Jul 2016 22:35:18 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 4311 invoked by uid 99); 7 Jul 2016 22:35:18 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 07 Jul 2016 22:35:18 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 65FF11A0282 for ; Thu, 7 Jul 2016 22:35:18 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.279 X-Spam-Level: * X-Spam-Status: No, score=1.279 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=kryptoncloud-com.20150623.gappssmtp.com Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id IouncXQaqVn2 for ; Thu, 7 Jul 2016 22:35:16 +0000 (UTC) Received: from mail-it0-f51.google.com (mail-it0-f51.google.com [209.85.214.51]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with ESMTPS id 29ADA5FC06 for ; Thu, 7 Jul 2016 22:35:15 +0000 (UTC) Received: by mail-it0-f51.google.com with SMTP id u186so228769ita.0 for ; Thu, 07 Jul 2016 15:35:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kryptoncloud-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=8uYgn0BcMPbjZMmbCWREd3Ob0WJKJXskSmBAQvIVsSA=; b=mHuLTQ29ZNz9HsmRYiJHwcAVTNqhpr03bUB9QdK6OIpyIxZMQ7o+2NUAgEdvVftlW/ 3zpAZFLw2ntqsv7KzI0W8a6AJu0R0q+n+PBv006aZMWrtW0Cio11kI2CZWxDXMhTc1dT x0C5u1uFVASaEXAXfUzP8KQikCTYyJef/t6TOhfOccOL+4782D8Wj0isyJIl+tdfims7 pl6mQJ+Bd5aKBsEJGA6GuzcSg8q82sV8i0BeOMA3mPujFl3jo6p+KFbxTsQtfXUBdZRc s8Iguo4AdHxcjyOh42FsqJ0Fw4LaC9CKcJMNJ264+xOujIvpAdF5O/Sn1NTHuQhUvt12 zXMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=8uYgn0BcMPbjZMmbCWREd3Ob0WJKJXskSmBAQvIVsSA=; b=NWGGTE71tHl2siPunYzBZMIDWicUBg1LaHv9ygiqr/bTmW87InQIfld/ZPmZTNtcH+ 13qqnY9b2bF3R3ag8UNEwHi5/9ubQIBKtmd0JVpamMXV43F5IrAFlCQRxfcv7Mwkq1v2 960e31CWGor5vb6O2g2feCaouOeKmjY2nwX0UhMlhZYuKMmFI0DM99xMDEpx4ZxI/rzC abZOTVDWUd1XXUpa71HQso+X1/WqUtEGHaFBLDpCKCCnVY7eaxjVz3TIqWkjrS1+5GSu ryf9iR7JT5AnlbBzjkeEkDO50mBCPghv2TuPqkQ8VJ1KCi2XKH88ZyDWqpElcJlERSX+ i2pw== X-Gm-Message-State: ALyK8tLsPQyCADGdAmWBADje3RDie/sqHEzViqwad9TKW4L+pjRzLwk1BFxJ0khwdWwaAWLpL/80B4ac3nnnyQ== X-Received: by 10.36.154.65 with SMTP id l62mr182736ite.79.1467930914235; Thu, 07 Jul 2016 15:35:14 -0700 (PDT) MIME-Version: 1.0 Received: by 10.50.203.34 with HTTP; Thu, 7 Jul 2016 15:34:34 -0700 (PDT) In-Reply-To: References: From: Yuan Fang Date: Thu, 7 Jul 2016 15:34:34 -0700 Message-ID: Subject: Re: Is my cluster normal? To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=94eb2c114852b2b3e30537134cb3 archived-at: Thu, 07 Jul 2016 22:35:21 -0000 --94eb2c114852b2b3e30537134cb3 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thanks Ben! For the post, it seems they got a little better but similar result than i did. Good to know it. I am not sure if a little fine tuning of heap memory will help or not. On Thu, Jul 7, 2016 at 2:58 PM, Ben Slater wrote: > Hi Yuan, > > You might find this blog post a useful comparison: > > https://www.instaclustr.com/blog/2016/01/07/multi-data-center-apache-spar= k-and-apache-cassandra-benchmark/ > > Although the focus is on Spark and Cassandra and multi-DC there are also > some single DC benchmarks of m4.xl clusters plus some discussion of how w= e > went about benchmarking. > > Cheers > Ben > > > On Fri, 8 Jul 2016 at 07:52 Yuan Fang wrote: > >> Yes, here is my stress test result: >> Results: >> op rate : 12200 [WRITE:12200] >> partition rate : 12200 [WRITE:12200] >> row rate : 12200 [WRITE:12200] >> latency mean : 16.4 [WRITE:16.4] >> latency median : 7.1 [WRITE:7.1] >> latency 95th percentile : 38.1 [WRITE:38.1] >> latency 99th percentile : 204.3 [WRITE:204.3] >> latency 99.9th percentile : 465.9 [WRITE:465.9] >> latency max : 1408.4 [WRITE:1408.4] >> Total partitions : 1000000 [WRITE:1000000] >> Total errors : 0 [WRITE:0] >> total gc count : 0 >> total gc mb : 0 >> total gc time (s) : 0 >> avg gc time(ms) : NaN >> stdev gc time(ms) : 0 >> Total operation time : 00:01:21 >> END >> >> On Thu, Jul 7, 2016 at 2:49 PM, Ryan Svihla wrote: >> >>> Lots of variables you're leaving out. >>> >>> Depends on write size, if you're using logged batch or not, what >>> consistency level, what RF, if the writes come in bursts, etc, etc. >>> However, that's all sort of moot for determining "normal" really you ne= ed a >>> baseline as all those variables end up mattering a huge amount. >>> >>> I would suggest using Cassandra stress as a baseline and go from there >>> depending on what those numbers say (just pick the defaults). >>> >>> Sent from my iPhone >>> >>> On Jul 7, 2016, at 4:39 PM, Yuan Fang wrote: >>> >>> yes, it is about 8k writes per node. >>> >>> >>> >>> On Thu, Jul 7, 2016 at 2:18 PM, daemeon reiydelle >>> wrote: >>> >>>> Are you saying 7k writes per node? or 30k writes per node? >>>> >>>> >>>> *.......* >>>> >>>> >>>> >>>> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 >>>> <%28%2B1%29%20415.501.0198>London (+44) (0) 20 8144 9872 >>>> <%28%2B44%29%20%280%29%2020%208144%209872>* >>>> >>>> On Thu, Jul 7, 2016 at 2:05 PM, Yuan Fang >>>> wrote: >>>> >>>>> writes 30k/second is the main thing. >>>>> >>>>> >>>>> On Thu, Jul 7, 2016 at 1:51 PM, daemeon reiydelle >>>>> wrote: >>>>> >>>>>> Assuming you meant 100k, that likely for something with 16mb of >>>>>> storage (probably way small) where the data is more that 64k hence w= ill not >>>>>> fit into the row cache. >>>>>> >>>>>> >>>>>> *.......* >>>>>> >>>>>> >>>>>> >>>>>> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 >>>>>> <%28%2B1%29%20415.501.0198>London (+44) (0) 20 8144 9872 >>>>>> <%28%2B44%29%20%280%29%2020%208144%209872>* >>>>>> >>>>>> On Thu, Jul 7, 2016 at 1:25 PM, Yuan Fang >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> I have a cluster of 4 m4.xlarge nodes(4 cpus and 16 gb memory and >>>>>>> 600GB ssd EBS). >>>>>>> I can reach a cluster wide write requests of 30k/second and read >>>>>>> request about 100/second. The cluster OS load constantly above 10. = Are >>>>>>> those normal? >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> >>>>>>> Best, >>>>>>> >>>>>>> Yuan >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> -- > =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94 > Ben Slater > Chief Product Officer > Instaclustr: Cassandra + Spark - Managed | Consulting | Support > +61 437 929 798 > --94eb2c114852b2b3e30537134cb3 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks Ben! For the post, it seems they got a little bette= r but similar result than i did. Good to know it.
I am not sure if a li= ttle fine tuning of heap memory will help or not.=C2=A0

On Thu, Jul 7, 2016 at 2:58 PM, B= en Slater <ben.slater@instaclustr.com> wrote:
Hi Yuan,

Y= ou might find this blog post a useful comparison:

Although the focus is on Spark and Cas= sandra and multi-DC there are also some single DC benchmarks of m4.xl clust= ers plus some discussion of how we went about benchmarking.

<= /div>
Cheers
Ben


On Fri, 8 Jul 2016 at 07:52 Yuan Fa= ng <yuan@kryp= toncloud.com> wrote:
Yes, here is my stress test result:
Results:
o= p rate =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 122= 00 [WRITE:12200]
partition rate =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0: 12200 [WRITE:12200]
row rate =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 12200 [WRITE:12200]
laten= cy mean =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 16.4 [WRITE:16.4]=
latency median =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 7.1 [W= RITE:7.1]
latency 95th percentile =C2=A0 : 38.1 [WRITE:38.1]
latency 99th percentile =C2=A0 : 204.3 [WRITE:204.3]
latenc= y 99.9th percentile : 465.9 [WRITE:465.9]
latency max =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 1408.4 [WRITE:1408.4]
To= tal partitions =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 1000000 [WRITE:1000000]<= /div>
Total errors =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0: 0 = [WRITE:0]
total gc count =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0: 0
total gc mb =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 : 0
total gc time (s) =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 0
avg gc time(ms) =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 : NaN
stdev= gc time(ms) =C2=A0 =C2=A0 =C2=A0 =C2=A0 : 0
Total operation time= =C2=A0 =C2=A0 =C2=A0: 00:01:21
END

On Thu, Jul 7, 2016 at 2:49= PM, Ryan Svihla <rs@foundev.pro> wrote:
Lots of variables you're leaving o= ut.

Depends on write size, if you're using log= ged batch or not, what consistency level, what RF, if the writes come in bu= rsts, etc, etc. However, that's all sort of moot for determining "= normal" really you need a baseline as all those variables end up matte= ring a huge amount.

I would suggest using Cassandr= a stress as a baseline and go from there depending on what those numbers sa= y (just pick the defaults).

Sent from my iPhone
=
On Jul 7, 2016, at 4:39 PM, Yuan Fang <yuan@kryptoncloud.com> wrote:

<= /div>
yes, it is about 8k wr= ites per node.



On Thu, Jul 7, 2016 at 2:18 PM, daemeon re= iydelle <daemeonr@gmail.com> wrote:
Are you saying 7k writes= per node? or 30k writes per node?

=
...= ....

Daemeon C.M. Reiydelle
USA (+1) 41= 5.501.0198
London (+44) (0) 20 8144 9872=

On Thu, Jul 7, 2016 at 2:05= PM, Yuan Fang <yuan@kryptoncloud.com> wrote:
writes 30k/second is the main thin= g.


On Thu, Jul 7, 2016 at 1:51 PM, daemeon reiydelle <daeme= onr@gmail.com> wrote:
Assuming you meant 100k, that likely for so= mething with 16mb of storage (probably way small) where the data is more th= at 64k hence will not fit into the row cache.

= =
.......

Daemeon C.M. Reiydelle
USA (+1) 415.501.0198
London (+44) (0) 20 8144 = 9872
<= /div>

On Thu, Jul 7, 2016 at 1:25 PM, Yuan Fang <yuan@kryptoncloud.com> wrote:


I have a cluster of 4 m4.xlarge nodes(4 cpus and 16 gb memory and 600GB ss= d EBS).
I can reach a cluster wide write requests of 30k/second and rea= d request about 100/second. The cluster OS load constantly above 10. Are th= ose normal?

Thanks!


<= /div>
Best,

Yuan=C2=A0






--
= =E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94=E2=80=94Ben Slater
Chief Product Officer
Instaclustr: Cassandra + S= park - Managed | Consulting | Support

--94eb2c114852b2b3e30537134cb3--