Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of 8forty@gmail.com designates
 209.85.223.174 as permitted sender)
Message-ID: <5212D106.4050700@gmail.com>
Date: Mon, 19 Aug 2013 20:14:30 -0600
From: Keith Freeman <8forty@gmail.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:17.0) Gecko/20130623 Thunderbird/17.0.7
MIME-Version: 1.0
To: user@cassandra.apache.org
CC: Nate McCall <nate@thelastpickle.com>
Subject: Re: insert performance (1.2.8)
References: <5212A108.9050804@gmail.com>
 <CAKmMYa_tgGc=BWtO0cyBphcnUFaocPd-dGxvndENrCCEtRLjTQ@mail.gmail.com>
In-Reply-To: 
 <CAKmMYa_tgGc=BWtO0cyBphcnUFaocPd-dGxvndENrCCEtRLjTQ@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------000501030904090901060302"

This is a multi-part message in MIME format.
--------------000501030904090901060302
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Sure, I've tried different numbers for batches and threads, but 
generally I'm running 10-30 threads at a time on the client, each 
sending a batch of 100 insert statements in every call, using the 
QueryBuilder.batch() API from the latest datastax java driver, then 
calling the Session.execute() function (synchronous) on the Batch.

I can't post my code, but my client does this on each iteration:
-- divides up the set of inserts by the number of threads
-- stores the current time
-- tells all the threads to send their inserts
-- then when they've all returned checks the elapsed time

At about 2000 rows for each iteration, 20 threads with 100 inserts each 
finish in about 1 second.  For 4000 rows, 40 threads with 100 inserts 
each finish in about 1.5 - 2 seconds, and as I said all 3 cassandra 
nodes have a heavy CPU load while the client is hardly loaded.  I've 
tried with 10 threads and more inserts per batch, or up to 60 threads 
with fewer, doesn't seem to make a lot of difference.

On 08/19/2013 05:00 PM, Nate McCall wrote:
> How big are the batch sizes? In other words, how many rows are you 
> sending per insert operation?
>
> Other than the above, not much else to suggest without seeing some 
> example code (on pastebin, gist or similar, ideally).
>
> On Mon, Aug 19, 2013 at 5:49 PM, Keith Freeman <8forty@gmail.com 
> <mailto:8forty@gmail.com>> wrote:
>
>     I've got a 3-node cassandra cluster (16G/4-core VMs ESXi v5 on
>     2.5Ghz machines not shared with any other VMs).  I'm inserting
>     time-series data into a single column-family using "wide rows"
>     (timeuuids) and have a 3-part partition key so my primary key is
>     something like ((a, b, day), in-time-uuid), x, y, z).
>
>     My java client is feeding rows (about 1k of raw data size each) in
>     batches using multiple threads, and the fastest I can get it run
>     reliably is about 2000 rows/second.  Even at that speed, all 3
>     cassandra nodes are very CPU bound, with loads of 6-9 each (and
>     the client machine is hardly breaking a sweat).  I've tried
>     turning off compression in my table which reduced the loads
>     slightly but not much.  There are no other updates or reads
>     occurring, except the datastax opscenter.
>
>     I was expecting to be able to insert at least 10k rows/second with
>     this configuration, and after a lot of reading of docs, blogs, and
>     google, can't really figure out what's slowing my client down.
>      When I increase the insert speed of my client beyond 2000/second,
>     the server responses are just too slow and the client falls
>     behind.  I had a single-node Mysql database that can handle 10k of
>     these data rows/second, so I really feel like I'm missing
>     something in Cassandra.  Any ideas?
>
>


--------------000501030904090901060302
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

<html>
  <head>
    <meta content="text/html; charset=ISO-8859-1"
      http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    Sure, I've tried different numbers for batches and threads, but
    generally I'm running 10-30 threads at a time on the client, each
    sending a batch of 100 insert statements in every call, using the
    QueryBuilder.batch() API from the latest datastax java driver, then
    calling the Session.execute() function (synchronous) on the Batch.<br>
    <br>
    I can't post my code, but my client does this on each iteration:<br>
    -- divides up the set of inserts by the number of threads<br>
    -- stores the current time<br>
    -- tells all the threads to send their inserts<br>
    -- then when they've all returned checks the elapsed time<br>
    <br>
    At about 2000 rows for each iteration, 20 threads with 100 inserts
    each finish in about 1 second.&nbsp; For 4000 rows, 40 threads with 100
    inserts each finish in about 1.5 - 2 seconds, and as I said all 3
    cassandra nodes have a heavy CPU load while the client is hardly
    loaded.&nbsp; I've tried with 10 threads and more inserts per batch, or
    up to 60 threads with fewer, doesn't seem to make a lot of
    difference.<br>
    <br>
    <div class="moz-cite-prefix">On 08/19/2013 05:00 PM, Nate McCall
      wrote:<br>
    </div>
    <blockquote
cite="mid:CAKmMYa_tgGc=BWtO0cyBphcnUFaocPd-dGxvndENrCCEtRLjTQ@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div class="gmail_extra">How big are the batch sizes? In other
          words, how many rows are you sending per insert operation?</div>
        <div class="gmail_extra"><br>
        </div>
        <div class="gmail_extra">Other than the above, not much else to
          suggest without seeing some example code (on pastebin, gist or
          similar, ideally).&nbsp;<br>
          <br>
          <div class="gmail_quote">On Mon, Aug 19, 2013 at 5:49 PM,
            Keith Freeman <span dir="ltr">&lt;<a moz-do-not-send="true"
                href="mailto:8forty@gmail.com" target="_blank">8forty@gmail.com</a>&gt;</span>
            wrote:<br>
            <blockquote class="gmail_quote" style="margin:0 0 0
              .8ex;border-left:1px #ccc solid;padding-left:1ex">
              I've got a 3-node cassandra cluster (16G/4-core VMs ESXi
              v5 on 2.5Ghz machines not shared with any other VMs). &nbsp;I'm
              inserting time-series data into a single column-family
              using "wide rows" (timeuuids) and have a 3-part partition
              key so my primary key is something like ((a, b, day),
              in-time-uuid), x, y, z).<br>
              <br>
              My java client is feeding rows (about 1k of raw data size
              each) in batches using multiple threads, and the fastest I
              can get it run reliably is about 2000 rows/second. &nbsp;Even
              at that speed, all 3 cassandra nodes are very CPU bound,
              with loads of 6-9 each (and the client machine is hardly
              breaking a sweat). &nbsp;I've tried turning off compression in
              my table which reduced the loads slightly but not much.
              &nbsp;There are no other updates or reads occurring, except the
              datastax opscenter.<br>
              <br>
              I was expecting to be able to insert at least 10k
              rows/second with this configuration, and after a lot of
              reading of docs, blogs, and google, can't really figure
              out what's slowing my client down. &nbsp;When I increase the
              insert speed of my client beyond 2000/second, the server
              responses are just too slow and the client falls behind.
              &nbsp;I had a single-node Mysql database that can handle 10k of
              these data rows/second, so I really feel like I'm missing
              something in Cassandra. &nbsp;Any ideas?<br>
              <br>
            </blockquote>
          </div>
          <br>
        </div>
      </div>
    </blockquote>
    <br>
  </body>
</html>

--------------000501030904090901060302--