incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Freeman <8fo...@gmail.com>
Subject Re: heavy insert load overloads CPUs, with MutationStage pending
Date Thu, 12 Sep 2013 19:10:07 GMT
Ok, your results are pretty impressive, I'm giving it a try.  I've made 
some initial attempts to use Astyanax 1.56.37, but have some troubles:

   - it's not compatible with 1.2.8 client-side ( NoSuchMethodError's on 
org.apache.cassandra.thrift.TBinaryProtocol, which changed it's 
signature since 1.2.5)
   - even switching to C* 1.2.5 servers, it's been difficult getting 
simple examples to work unless I use CF's that have "WITH COMPACT STORAGE"

How did you handle these problems?  How much effort did it take you to 
switch from datastax to astyanax?

I feel like I'm getting lost in a pretty deep rabbit-hole here.

On 09/11/2013 03:03 PM, Paul Cichonski wrote:
> I was reluctant to use the thrift as well, and I spent about a week trying to get the
CQL inserts to work by partitioning the INSERTS in different ways and tuning the cluster.
>
> However, nothing worked remotely as well as the batch_mutate when it came to writing
a full wide-row at once. I think Cassandra 2.0 makes CQL work better for these cases (CASSANDRA-4693),
but I haven't tested it yet.
>
> -Paul
>
>> -----Original Message-----
>> From: Keith Freeman [mailto:8forty@gmail.com]
>> Sent: Wednesday, September 11, 2013 1:06 PM
>> To: user@cassandra.apache.org
>> Subject: Re: heavy insert load overloads CPUs, with MutationStage pending
>>
>> Thanks, I had seen your stackoverflow post.  I've got hundreds of
>> (wide-) rows, and the writes are pretty well distributed across them.
>> I'm very reluctant to drop back to the thrift interface.
>>
>> On 09/11/2013 10:46 AM, Paul Cichonski wrote:
>>> How much of the data you are writing is going against the same row key?
>>>
>>> I've experienced some issues using CQL to write a full wide-row at once
>> (across multiple threads) that exhibited some of the symptoms you have
>> described (i.e., high cpu, dropped mutations).
>>> This question goes into it a bit
>> more:http://stackoverflow.com/questions/18522191/using-cassandra-and-
>> cql3-how-do-you-insert-an-entire-wide-row-in-a-single-reque  . I was able to
>> solve my issue by switching to using the thrift batch_mutate to write a full
>> wide-row at once instead of using many CQL INSERT statements.
>>> -Paul
>>>
>>>> -----Original Message-----
>>>> From: Keith Freeman [mailto:8forty@gmail.com]
>>>> Sent: Wednesday, September 11, 2013 9:16 AM
>>>> To:user@cassandra.apache.org
>>>> Subject: Re: heavy insert load overloads CPUs, with MutationStage
>>>> pending
>>>>
>>>>
>>>> On 09/10/2013 11:42 AM, Nate McCall wrote:
>>>>> With SSDs, you can turn up memtable_flush_writers - try 3 initially
>>>>> (1 by default) and see what happens. However, given that there are
>>>>> no entries in 'All time blocked' for such, they may be something else.
>>>> Tried that, it seems to have reduced the loads a little after
>>>> everything warmed-up, but not much.
>>>>> How are you inserting the data?
>>>> A java client on a separate box using the datastax java driver, 48
>>>> threads writing 100 records each iteration as prepared batch statements.
>>>>
>>>> At 5000 records/sec, the servers just can't keep up, so the client backs
up.
>>>> That's only 5M of data/sec, which doesn't seem like much.  As I
>>>> mentioned, switching to SSDs didn't help much, so I'm assuming at
>>>> this point that the server overloads are what's holding up the client.


Mime
View raw message