Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1B7C510544 for ; Tue, 20 Aug 2013 14:44:12 +0000 (UTC) Received: (qmail 17309 invoked by uid 500); 20 Aug 2013 14:44:08 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 17076 invoked by uid 500); 20 Aug 2013 14:44:06 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 17062 invoked by uid 99); 20 Aug 2013 14:44:04 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Aug 2013 14:44:04 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW X-Spam-Check-By: apache.org Received-SPF: error (athena.apache.org: local policy) Received: from [209.85.215.172] (HELO mail-ea0-f172.google.com) (209.85.215.172) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Aug 2013 14:44:00 +0000 Received: by mail-ea0-f172.google.com with SMTP id r16so269837ead.17 for ; Tue, 20 Aug 2013 07:43:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=9/at1x/9MSHotOHex/AKSbrqXA2xtdAMTK/TgR6ZOP0=; b=pLPGLb3+Krr/nF1z+j2tlFgeKBVjYWXCdmpRi77BLgt6TMTTZ96KV5Gbr0sXVS7YYl d8OPVlOwXfXf/BBn1TAmMEOdXt6FqIXUkusFoMzoW6gx6EFxys1gd8+C/a5KIxt4bp6b 56MkUkVd/Uq7yao/rRnUT15zTT5lGvitEAu8iYHPt0H3l+lMYllUyXHiO0DH3+j2dP/q WcypgsOTuuX/Qa7Qmv0N3M8C3HD54JQ8YWo2PJ8hc1FUZj2c6uQd24R2lq7NmlljoNEB 1O4UsWeX/R9QwdTIYDxhD3IU5F0XRaF9KvW1MyxvnvtMMKWwn9Odir29q/4M3wmeNR3e sd3Q== X-Gm-Message-State: ALoCoQmeljBUeOBaP5NsYI54NqsjSbvxkQVshdsggDcSbrVfHqEdSH3MjzvMbg81Oq636m11Dqop MIME-Version: 1.0 X-Received: by 10.15.75.73 with SMTP id k49mr2815854eey.36.1377009799346; Tue, 20 Aug 2013 07:43:19 -0700 (PDT) Received: by 10.223.63.18 with HTTP; Tue, 20 Aug 2013 07:43:19 -0700 (PDT) X-Originating-IP: [70.112.126.233] In-Reply-To: References: <5212A108.9050804@gmail.com> <5212D106.4050700@gmail.com> <5213757D.8030106@gmail.com> Date: Tue, 20 Aug 2013 09:43:19 -0500 Message-ID: Subject: Re: insert performance (1.2.8) From: Nate McCall To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a11c1af4cf173b604e46213c0 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c1af4cf173b604e46213c0 Content-Type: text/plain; charset=ISO-8859-1 Thanks for putting this up - sorry I missed your post the other week. I would be real curious as to your results if you added a prepared statement for those inserts. On Tue, Aug 20, 2013 at 9:14 AM, Przemek Maciolek wrote: > I had similar issues (sent a note on the list few weeks ago but nobody > responded). I think there's a serious bottleneck with using wide rows and > composite keys. I made a trivial benchmark, which you check here: > http://pastebin.com/qAcRcqbF - it's written in cql-rb, but I ran the > test using astyanax/cql3 enabled and the results were the same. > > In my case, inserting 10 000 entries took following time (seconds): > > Using composite keys > Separetely: 12.892867 > Batch: 189.731306 > > This means, I got 1000 rows/s when inserting them seperately and 52 (!!!) > when inserting them in a huge batch. > > Using just partition key and wide row > Separetely: 11.292507 > Batch: 0.093355 > > Again, 1000 rows/s when inserting them one by one. But batch obviously > improves thing and I easily got >10000 rows/s. > > Anyone else with similar experiences? > > Thanks, > Przemek > > > On Tue, Aug 20, 2013 at 4:04 PM, Nate McCall wrote: > >> John makes a good point re:prepared statements (I'd increase batch sizes >> again once you did this as well - separate, incremental runs of course so >> you can gauge the effect of each). That should take out some of the >> processing overhead of statement validation in the server (some - that load >> spike still seems high though). >> >> I'd actually be really interested as to what your results were after >> doing so - i've not tried any A/B testing here for prepared statements on >> inserts. >> >> Given your load is on the server, i'm not sure adding more async >> indirection on the client would buy you too much though. >> >> Also, at what RF and consistency level are you writing? >> >> >> On Tue, Aug 20, 2013 at 8:56 AM, Keith Freeman <8forty@gmail.com> wrote: >> >>> Ok, I'll try prepared statements. But while sending my statements >>> async might speed up my client, it wouldn't improve throughput on the >>> cassandra nodes would it? They're running at pretty high loads and only >>> about 10% idle, so my concern is that they can't handle the data any >>> faster, so something's wrong on the server side. I don't really think >>> there's anything on the client side that matters for this problem. >>> >>> Of course I know there are obvious h/w things I can do to improve server >>> performance: SSDs, more RAM, more cores, etc. But I thought the servers I >>> have would be able to handle more rows/sec than say Mysql, since write >>> speed is supposed to be one of Cassandra's strengths. >>> >>> >>> On 08/19/2013 09:03 PM, John Sanda wrote: >>> >>> I'd suggest using prepared statements that you initialize at application >>> start up and switching to use Session.executeAsync coupled with Google >>> Guava Futures API to get better throughput on the client side. >>> >>> >>> On Mon, Aug 19, 2013 at 10:14 PM, Keith Freeman <8forty@gmail.com>wrote: >>> >>>> Sure, I've tried different numbers for batches and threads, but >>>> generally I'm running 10-30 threads at a time on the client, each sending a >>>> batch of 100 insert statements in every call, using the >>>> QueryBuilder.batch() API from the latest datastax java driver, then calling >>>> the Session.execute() function (synchronous) on the Batch. >>>> >>>> I can't post my code, but my client does this on each iteration: >>>> -- divides up the set of inserts by the number of threads >>>> -- stores the current time >>>> -- tells all the threads to send their inserts >>>> -- then when they've all returned checks the elapsed time >>>> >>>> At about 2000 rows for each iteration, 20 threads with 100 inserts each >>>> finish in about 1 second. For 4000 rows, 40 threads with 100 inserts each >>>> finish in about 1.5 - 2 seconds, and as I said all 3 cassandra nodes have a >>>> heavy CPU load while the client is hardly loaded. I've tried with 10 >>>> threads and more inserts per batch, or up to 60 threads with fewer, doesn't >>>> seem to make a lot of difference. >>>> >>>> >>>> On 08/19/2013 05:00 PM, Nate McCall wrote: >>>> >>>> How big are the batch sizes? In other words, how many rows are you >>>> sending per insert operation? >>>> >>>> Other than the above, not much else to suggest without seeing some >>>> example code (on pastebin, gist or similar, ideally). >>>> >>>> On Mon, Aug 19, 2013 at 5:49 PM, Keith Freeman <8forty@gmail.com>wrote: >>>> >>>>> I've got a 3-node cassandra cluster (16G/4-core VMs ESXi v5 on 2.5Ghz >>>>> machines not shared with any other VMs). I'm inserting time-series data >>>>> into a single column-family using "wide rows" (timeuuids) and have a 3-part >>>>> partition key so my primary key is something like ((a, b, day), >>>>> in-time-uuid), x, y, z). >>>>> >>>>> My java client is feeding rows (about 1k of raw data size each) in >>>>> batches using multiple threads, and the fastest I can get it run reliably >>>>> is about 2000 rows/second. Even at that speed, all 3 cassandra nodes are >>>>> very CPU bound, with loads of 6-9 each (and the client machine is hardly >>>>> breaking a sweat). I've tried turning off compression in my table which >>>>> reduced the loads slightly but not much. There are no other updates or >>>>> reads occurring, except the datastax opscenter. >>>>> >>>>> I was expecting to be able to insert at least 10k rows/second with >>>>> this configuration, and after a lot of reading of docs, blogs, and google, >>>>> can't really figure out what's slowing my client down. When I increase the >>>>> insert speed of my client beyond 2000/second, the server responses are just >>>>> too slow and the client falls behind. I had a single-node Mysql database >>>>> that can handle 10k of these data rows/second, so I really feel like I'm >>>>> missing something in Cassandra. Any ideas? >>>>> >>>>> >>>> >>>> >>> >>> >>> -- >>> >>> - John >>> >>> >>> >> > --001a11c1af4cf173b604e46213c0 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Thanks for putting this up - sorry I missed your post the = other week. I would be real curious as to your results if you added a prepa= red statement for those inserts.=A0

On Tue, Aug 20, 2013 at 9:14 AM, Przemek Maciolek <pmaciolek@gmail.com> wrote:
I had similar issues (sent a note on the list few weeks ag= o but nobody responded). I think there's a serious bottleneck with usin= g wide rows and composite keys. I made a trivial benchmark, which you check= here:=A0http://paste= bin.com/qAcRcqbF=A0 - it's written in cql-rb, but I ran the test u= sing astyanax/cql3 enabled and the results were the same.

In my case, inserting 10 000 entries took following time (se= conds):

Using composite keys
= Separetely: 12.892867
Batch: 189.731306

This means, I got 1000 rows/s when inserting them seperately and = 52 (!!!) when inserting them in a huge batch.

Using just partition key and wide row
Separetely: 11.292507
Batch= : 0.093355
=
Again, 1000 rows/s when inserting them one by one. But batch obvi= ously improves thing and I easily got >10000 rows/s.
=
Anyone else with similar experiences?

Thanks,
Przemek


On Tue, Aug 20, 2013 at 4:04 PM, Nate McCall= <nate@thelastpickle.com> wrote:
John makes a good point re:prepared statements (I'd in= crease batch sizes again once you did this as well - separate, incremental = runs of course so you can gauge the effect of each). That should take out s= ome of the processing overhead of statement validation in the server (some = - that load spike still seems high though).=A0

I'd actually be really interested as to what your result= s were after doing so - i've not tried any A/B testing here for prepare= d statements on inserts.=A0

Given your load is on = the server, i'm not sure adding more async indirection on the client wo= uld buy you too much though.=A0

Also, at what RF and consistency level are you writing?=


On Tue, Aug 20, 2013 at 8:56 AM, Keith Freeman &= lt;8forty@gmail.com> wrote:
=20 =20 =20
Ok, I'll try prepared statements.=A0=A0 But while sending my statem= ents async might speed up my client, it wouldn't improve throughput on the cassandra nodes would it?=A0 They're running at pretty high loa= ds and only about 10% idle, so my concern is that they can't handle th= e data any faster, so something's wrong on the server side.=A0 I don&= #39;t really think there's anything on the client side that matters for this problem.

Of course I know there are obvious h/w things I can do to improve server performance: SSDs, more RAM, more cores, etc.=A0 But I thought the servers I have would be able to handle more rows/sec than say Mysql, since write speed is supposed to be one of Cassandra's strengths.


On 08/19/2013 09:03 PM, John Sanda wrote:
I'd suggest using prepared statements that you initialize at application start up and switching to use Session.executeAsync coupled with Google Guava Futures API to get better throughput on the client side.


On Mon, Aug 19, 2013 at 10:14 PM, Keith Freeman <8forty@gmail.com> wrote:
Sure, I've tried different numbers for batches and threads, but generally I'm running 10-30 threads at a time on the client, each sending a batch of 100 insert statements in every call, using the QueryBuilder.batch() API from the latest datastax java driver, then calling the Session.execute() function (synchronous) on the Batch.

I can't post my code, but my client does this on each iteration:
-- divides up the set of inserts by the number of threads
-- stores the current time
-- tells all the threads to send their inserts
-- then when they've all returned checks the elapsed time=

At about 2000 rows for each iteration, 20 threads with 100 inserts each finish in about 1 second.=A0 For 4000 rows, 40 threads with 100 inserts each finish in about 1.5 - 2 seconds, and as I said all 3 cassandra nodes have a heavy CPU load while the client is hardly loaded.=A0 I've tried with 10 threads and more inserts per batch, or up to 60 threads with fewer, doesn't seem to make a lot of difference.


On 08/19/2013 05:00 PM, Nate McCall wrote:
How big are the batch sizes? In other words, how many rows are you sending per insert operation?

Other than the above, not much else to suggest without seeing some example code (on pastebin, gist or similar, ideally).=A0
On Mon, Aug 19, 2013 at 5:49 PM, Keith Freeman <8forty@gmail.com><= /span> wrote:
I've got a 3-node cassandra cluster (16G/4-core VMs ESXi v5 on 2.5Ghz machines not shared with any other VMs). =A0I'm inserting time-serie= s data into a single column-family using "wi= de rows" (timeuuids) and have a 3-part partition key so my primary key is something like ((a, b, day), in-time-uuid), x, y, z).

My java client is feeding rows (about 1k of raw data size each) in batches using multiple threads, and the fastest I can get it run reliably is about 2000 rows/second. =A0Even at that speed, all 3 cassandra nodes are very CPU bound, with loads of 6-9 each (and the client machine is hardly breaking a sweat). =A0I've tried turning off compressi= on in my table which reduced the loads slightly but not much. =A0There are no other updates or reads occurring, except the datastax opscenter.

I was expecting to be able to insert at least 10k rows/second with this configuration, and after a lot of reading of docs, blogs, and google, can't really figur= e out what's slowing my client down. =A0When = I increase the insert speed of my client beyond 2000/second, the server responses are just too slow and the client falls behind. =A0I had a single-node Mysql database that can handle 10k of these data rows/second, so I really feel like I'm missing something in Cassandra. =A0Any ideas?






--

- John




--001a11c1af4cf173b604e46213c0--