Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of paul.cichonski@lithium.com
 designates 208.74.204.5 as permitted sender)
From: Paul Cichonski <paul.cichonski@lithium.com>
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Wed, 11 Sep 2013 09:46:00 -0700
Subject: RE: heavy insert load overloads CPUs, with MutationStage pending
Thread-Topic: heavy insert load overloads CPUs, with MutationStage pending
Thread-Index: Ac6vCkn9mDlckaJ7SySv6Tvi+YBdtAAA1QWA
Message-ID: <541C7781A689464891C05251C07E8CCF3D9D10AA9C@farseer.lithium.local>
References: <522F32FC.2030804@gmail.com>
 <CAEDUwd248J4QZinuFOjNowd_6ep3QE_OSdECZ6FFMeVfdtGERA@mail.gmail.com>
 <522F5987.9030300@gmail.com>
 <CAKmMYa8jF+QNNtuxQCyMSfL5A8DUAAAXHsR47YoEOX60S3WMVA@mail.gmail.com>
 <52309741.9060407@gmail.com>
In-Reply-To: <52309741.9060407@gmail.com>
Accept-Language: en-US
Content-Language: en-US
acceptlanguage: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

How much of the data you are writing is going against the same row key?=20

I've experienced some issues using CQL to write a full wide-row at once (ac=
ross multiple threads) that exhibited some of the symptoms you have describ=
ed (i.e., high cpu, dropped mutations).=20

This question goes into it a bit more: http://stackoverflow.com/questions/1=
8522191/using-cassandra-and-cql3-how-do-you-insert-an-entire-wide-row-in-a-=
single-reque . I was able to solve my issue by switching to using the thrif=
t batch_mutate to write a full wide-row at once instead of using many CQL I=
NSERT statements.=20

-Paul

> -----Original Message-----
> From: Keith Freeman [mailto:8forty@gmail.com]
> Sent: Wednesday, September 11, 2013 9:16 AM
> To: user@cassandra.apache.org
> Subject: Re: heavy insert load overloads CPUs, with MutationStage pending
>=20
>=20
> On 09/10/2013 11:42 AM, Nate McCall wrote:
> > With SSDs, you can turn up memtable_flush_writers - try 3 initially (1
> > by default) and see what happens. However, given that there are no
> > entries in 'All time blocked' for such, they may be something else.
> Tried that, it seems to have reduced the loads a little after everything
> warmed-up, but not much.
> >
> > How are you inserting the data?
>=20
> A java client on a separate box using the datastax java driver, 48 thread=
s
> writing 100 records each iteration as prepared batch statements.
>=20
> At 5000 records/sec, the servers just can't keep up, so the client backs =
up.
> That's only 5M of data/sec, which doesn't seem like much.  As I mentioned=
,
> switching to SSDs didn't help much, so I'm assuming at this point that th=
e
> server overloads are what's holding up the client.