incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Cichonski <paul.cichon...@lithium.com>
Subject RE: heavy insert load overloads CPUs, with MutationStage pending
Date Wed, 11 Sep 2013 21:03:14 GMT
I was reluctant to use the thrift as well, and I spent about a week trying to get the CQL inserts
to work by partitioning the INSERTS in different ways and tuning the cluster.

However, nothing worked remotely as well as the batch_mutate when it came to writing a full
wide-row at once. I think Cassandra 2.0 makes CQL work better for these cases (CASSANDRA-4693),
but I haven't tested it yet.

-Paul

> -----Original Message-----
> From: Keith Freeman [mailto:8forty@gmail.com]
> Sent: Wednesday, September 11, 2013 1:06 PM
> To: user@cassandra.apache.org
> Subject: Re: heavy insert load overloads CPUs, with MutationStage pending
> 
> Thanks, I had seen your stackoverflow post.  I've got hundreds of
> (wide-) rows, and the writes are pretty well distributed across them.
> I'm very reluctant to drop back to the thrift interface.
> 
> On 09/11/2013 10:46 AM, Paul Cichonski wrote:
> > How much of the data you are writing is going against the same row key?
> >
> > I've experienced some issues using CQL to write a full wide-row at once
> (across multiple threads) that exhibited some of the symptoms you have
> described (i.e., high cpu, dropped mutations).
> >
> > This question goes into it a bit
> more:http://stackoverflow.com/questions/18522191/using-cassandra-and-
> cql3-how-do-you-insert-an-entire-wide-row-in-a-single-reque  . I was able to
> solve my issue by switching to using the thrift batch_mutate to write a full
> wide-row at once instead of using many CQL INSERT statements.
> >
> > -Paul
> >
> >> -----Original Message-----
> >> From: Keith Freeman [mailto:8forty@gmail.com]
> >> Sent: Wednesday, September 11, 2013 9:16 AM
> >> To:user@cassandra.apache.org
> >> Subject: Re: heavy insert load overloads CPUs, with MutationStage
> >> pending
> >>
> >>
> >> On 09/10/2013 11:42 AM, Nate McCall wrote:
> >>> With SSDs, you can turn up memtable_flush_writers - try 3 initially
> >>> (1 by default) and see what happens. However, given that there are
> >>> no entries in 'All time blocked' for such, they may be something else.
> >> Tried that, it seems to have reduced the loads a little after
> >> everything warmed-up, but not much.
> >>> How are you inserting the data?
> >> A java client on a separate box using the datastax java driver, 48
> >> threads writing 100 records each iteration as prepared batch statements.
> >>
> >> At 5000 records/sec, the servers just can't keep up, so the client backs up.
> >> That's only 5M of data/sec, which doesn't seem like much.  As I
> >> mentioned, switching to SSDs didn't help much, so I'm assuming at
> >> this point that the server overloads are what's holding up the client.


Mime
View raw message