Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 71A9811059 for ; Fri, 18 Jul 2014 04:19:46 +0000 (UTC) Received: (qmail 88748 invoked by uid 500); 18 Jul 2014 04:19:43 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 88700 invoked by uid 500); 18 Jul 2014 04:19:43 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 88690 invoked by uid 99); 18 Jul 2014 04:19:43 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jul 2014 04:19:43 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of SRS0=xz4W5N=4N=basetechnology.com=jack@yourhostingaccount.com designates 65.254.253.78 as permitted sender) Received: from [65.254.253.78] (HELO walmailout09.yourhostingaccount.com) (65.254.253.78) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 18 Jul 2014 04:19:40 +0000 Received: from mailscan11.yourhostingaccount.com ([10.1.15.11] helo=walmailscan11.yourhostingaccount.com) by walmailout09.yourhostingaccount.com with esmtp (Exim) id 1X7zdc-0007fM-0d for user@cassandra.apache.org; Fri, 18 Jul 2014 00:19:16 -0400 Received: from impout01.yourhostingaccount.com ([10.1.55.1] helo=impout01.yourhostingaccount.com) by walmailscan11.yourhostingaccount.com with esmtp (Exim) id 1X7zdb-00086F-Kx for user@cassandra.apache.org; Fri, 18 Jul 2014 00:19:15 -0400 Received: from walauthsmtp10.yourhostingaccount.com ([10.1.18.10]) by impout01.yourhostingaccount.com with NO UCE id TgKF1o0010D2B7u01gKFNQ; Fri, 18 Jul 2014 00:19:15 -0400 X-Authority-Analysis: v=2.0 cv=Ir+cgcDg c=1 sm=1 a=UkMH5KcvGpXfM81wB0t8ug==:17 a=aQzbgH187woA:10 a=gyc7I-e8LvAA:10 a=3jZET7lWBKwA:10 a=jvYhGVW7AAAA:8 a=xF2oFCkVh6o2lxuHi5cA:9 a=QEXdDO2ut3YA:10 a=EMlJoiak7gQA:10 a=rzzP_kbwrucA:10 a=SOaqlzX7avcsrDwm:21 a=DtngFmSeG-lg0lMx:21 a=pGLkceISAAAA:8 a=mV9VRH-2AAAA:8 a=5GG0SCeFAAAA:8 a=sPLmadQcv9oKqwKU7acA:9 a=_W_S_7VecoQA:10 a=tXsnliwV7b4A:10 a=MSl-tDqOz04A:10 a=-oBvMAH-s9btIQN_:21 a=ISXsV6__4WgkBTmU:21 a=2gI1vAumJwWs0gD4Ojj1yg==:117 X-EN-OrigOutIP: 10.1.18.10 X-EN-IMPSID: TgKF1o0010D2B7u01gKFNQ Received: from 207-237-113-28.c3-0.nyr-ubr1.nyr.ny.cable.rcn.com ([207.237.113.28]:54487 helo=JackKrupansky14) by walauthsmtp10.yourhostingaccount.com with esmtpa (Exim) id 1X7zdb-0001NJ-A5 for user@cassandra.apache.org; Fri, 18 Jul 2014 00:19:15 -0400 Message-ID: <43A7F5BFFCEB4231B2D261DB7F61D648@JackKrupansky14> From: "Jack Krupansky" To: References: <62C7CCE5A334433FBC9F93F87370648F@JackKrupansky14> In-Reply-To: Subject: Re: horizontal query scaling issues follow on Date: Fri, 18 Jul 2014 00:19:15 -0400 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_1C8C_01CFA21D.E88C64E0" X-Priority: 3 X-MSMail-Priority: Normal Importance: Normal X-Mailer: Microsoft Windows Live Mail 16.4.3528.331 X-MimeOLE: Produced By Microsoft MimeOLE V16.4.3528.331 X-EN-UserInfo: e0a4b55451ed9f27313ebf02e3d4348d:931c98230c6409dcc37fa7e93b490c27 X-EN-AuthUser: jack@basetechnology.com Sender: "Jack Krupansky" X-EN-OrigIP: 207.237.113.28 X-EN-OrigHost: 207-237-113-28.c3-0.nyr-ubr1.nyr.ny.cable.rcn.com X-Virus-Checked: Checked by ClamAV on apache.org This is a multi-part message in MIME format. ------=_NextPart_000_1C8C_01CFA21D.E88C64E0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sorry I may have confused the discussion by mentioning tokens =E2=80=93 = I wasn=E2=80=99t intending to refer to vnodes or the num_tokens = property, but merely referring to the token range of a node and that the = partition key hashes to a token value. The main question is what you use for your primary key and whether you = are using a small number of partition keys and a large number of = clustering columns, or does each row have a unique partition key and no = clustering columns. -- Jack Krupansky From: Diane Griffith=20 Sent: Thursday, July 17, 2014 6:21 PM To: user=20 Subject: Re: horizontal query scaling issues follow on So do partitions equate to tokens/vnodes?=20 If so we had configured all cluster nodes/vms with num_tokens: 256 = instead of setting init_token and assigning ranges. I am still not = getting why in Cassandra 2.0, I would assign my own ranges via = init_token and this was based on the documentation and even this blog = item that made it seem right for us to always configure our cluster vms = with num_tokens: 256 in the cassandra.yaml file. =20 Also in all testing, all vms were of equal sizing so one was not more = powerful than another. =20 I didn't think I was hitting an i/o wall on the client vm (separate vm) = where we command line scripted our query call to the cassandra cluster. = I can break the client call load across vms which I tried early on. = Happy to verify that again though. So given that I was assuming the partitions were such that it wasn't a = problem. Is that an incorrect assumption and something to dig into = more? Thanks, Diane On Thu, Jul 17, 2014 at 3:01 PM, Jack Krupansky = wrote: How many partitions are you spreading those 18 million rows over? That = many rows in a single partition will not be a sweet spot for Cassandra. = It=E2=80=99s not exceeding any hard limit (2 billion), but some internal = operations may cache the partition rather than the logical row. And all those rows in a single partition would certainly not be a test = of =E2=80=9Chorizontal scaling=E2=80=9D (adding nodes to handle more = data =E2=80=93 more token values or partitions.) -- Jack Krupansky From: Diane Griffith=20 Sent: Thursday, July 17, 2014 1:33 PM To: user=20 Subject: horizontal query scaling issues follow on This is a follow on re-post to clarify what we are trying to do, = providing information that was missing or not clear. Goal: Verify horizontal scaling for random non duplicating key reads = using the simplest configuration (or minimal configuration) possible. Background: A couple years ago we did similar performance testing with Cassandra = for both read and write performance and found excellent (essentially = linear) horizontal scalability. That project got put on hold. We are = now moving forward with an operational system and are having scaling = problems. During the prior testing (3 years ago) we were using a much older = version of Cassandra (0.8 or older), the THRIFT API, and Amazon AWS = rather than OpenStack VMs. We are now using the latest Cassandra and = the CQL interface. We did try moving from OpenStack to AWS/EC2 but that = did not materially change our (poor) results. Test Procedure: a.. Inserted 54 million cells in 18 million rows (so 3 cells per = row), using randomly generated row keys. That was to be our data control = for the test.=20 b.. Spawn a client on a different VM to query 100k rows and do that = for 100 reps. Each row key queried is drawn randomly from the set of = existing row keys, and then not re-used, so all 10 million row queries = use a different (valid) row key. This test is a specific use case of = our system we are trying to show will scale=20 Result: a.. 2 nodes performed better than 1 node test but 4 nodes showed = decreased performance over 2 nodes. So that did not show horizontal = scaling=20 Notes: a.. We have replication factor set to 1 as we were trying to keep = the control test simple to prove out horizontal scaling. =20 b.. When we tried to add threading to see if it would help it had = interesting side behavior which did not prove out horizontal scaling.=20 c.. We are using CQL versus THRIFT API for Cassandra 2.0.6=20 Does anyone have any feedback that either threading or replication = factor is necessary to show horizontal scaling of Cassandra versus the = minimal way of just continue to add nodes to help throughput? Any suggestions of minimal configuration necessary to show scaling of = our query use case 100k requests for random non repeating keys = constantly coming in over a period of time? Thanks, Diane ------=_NextPart_000_1C8C_01CFA21D.E88C64E0 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable
Sorry I may have confused the discussion by mentioning tokens = =E2=80=93 I wasn=E2=80=99t=20 intending to refer to vnodes or the num_tokens property, but merely = referring to=20 the token range of a node and that the partition key hashes to a token=20 value.
 
The main question is what you use for your primary key and whether = you are=20 using a small number of partition keys and a large number of clustering = columns,=20 or does each row have a unique partition key and no clustering = columns.
 
-- Jack=20 Krupansky
 
Sent: Thursday, July 17, 2014 6:21 PM
To: user
Subject: Re: horizontal query scaling issues follow=20 on
 
So do partitions equate to tokens/vnodes?=20
 
If so we had configured all cluster nodes/vms with num_tokens: 256 = instead=20 of setting init_token and assigning ranges.  I am still not getting = why in=20 Cassandra 2.0, I would assign my own ranges via init_token and this was = based on=20 the documentation and even this = blog=20 item that made it seem right for us to always configure our cluster = vms with=20 num_tokens: 256 in the cassandra.yaml file. 
 
Also in all testing, all vms were of equal sizing so one was not = more=20 powerful than another. 
 
I didn't think I was hitting an i/o wall on the client vm (separate = vm)=20 where we command line scripted our query call to the cassandra=20 cluster.    I can break the client call load across vms = which I=20 tried early on.  Happy to verify that again though.
 
So given that I was assuming the partitions were such that it = wasn't a=20 problem.  Is that an incorrect assumption and something to dig into = more?
 
Thanks,
Diane


On Thu, Jul 17, 2014 at 3:01 PM, Jack Krupansky = <jack@basetechnology.com> wrote:
How many partitions are you spreading those 18 million rows over? = That=20 many rows in a single partition will not be a sweet spot for = Cassandra. It=E2=80=99s=20 not exceeding any hard limit (2 billion), but some internal operations = may=20 cache the partition rather than the logical row.
 
And all those rows in a single partition would certainly not be a = test of=20 =E2=80=9Chorizontal scaling=E2=80=9D (adding nodes to handle more data = =E2=80=93 more token values or=20 partitions.)
 
-- Jack=20 Krupansky
 
Sent: Thursday, July 17, 2014 1:33 PM
To: user =
Subject: horizontal query scaling issues follow=20 on
 

This is a = follow on=20 re-post to clarify what we are trying to do, providing information = that was=20 missing or not clear.

 

Goal:  = Verify=20 horizontal scaling for random non duplicating key reads using the = simplest=20 configuration (or minimal configuration) possible.

 

Background:

A couple = years ago we=20 did similar performance testing with Cassandra for both read and write = performance and found excellent (essentially linear) horizontal=20 scalability.  That project got put on hold.  We are now = moving=20 forward with an operational system and are having scaling = problems.

 

During the = prior=20 testing (3 years ago) we were using a much older version of Cassandra = (0.8 or=20 older), the THRIFT API, and Amazon AWS rather than OpenStack = VMs.  We are=20 now using the latest Cassandra and the CQL interface.  We did try = moving=20 from OpenStack to AWS/EC2 but that did not materially change our = (poor)=20 results.

 

Test=20 Procedure:

  • Inserted 54 = million=20 cells in 18 million rows (so 3 cells per row), using randomly = generated row=20 keys. That was to be our data control for the test.=20
  • Spawn a = client on a=20 different VM to query 100k rows and do that for 100 reps.  Each = row key=20 queried is drawn randomly from the set of existing row keys, and = then not=20 re-used, so all 10 million row queries use a different (valid) row=20 key.  This test is a specific use case of our system we are = trying to=20 show will scale

Result:

  • 2 nodes = performed=20 better than 1 node test but 4 nodes showed decreased performance = over 2=20 nodes.  So that did not show horizontal scaling =

 

Notes:

  • We have = replication=20 factor set to 1 as we were trying to keep the control test simple to = prove=20 out horizontal scaling. 
  • When we = tried to add=20 threading to see if it would help it had interesting side behavior = which did=20 not prove out horizontal scaling.=20
  • We are = using CQL=20 versus THRIFT API for Cassandra 2.0.6

 

 

Does anyone = have any=20 feedback that either threading or replication factor is necessary to = show=20 horizontal scaling of Cassandra versus the minimal way of just = continue to add=20 nodes to help throughput?

 

Any = suggestions of=20 minimal configuration necessary to show scaling of our query use case = 100k=20 requests for random non repeating keys constantly coming in over a = period of=20 time?


Thanks,

Diane

 
------=_NextPart_000_1C8C_01CFA21D.E88C64E0--