cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Boxenhorn <da...@lookin2.com>
Subject What is the optimal size of batch mutate batches?
Date Tue, 11 May 2010 11:54:02 GMT
I am saving a large amount of data to Cassandra using batch mutate. I have
found that my speed is proportional to the size of the batch. It was very
slow when I was inserting one row at a time, but when I created batches of
100 rows and mutated them together, it went 100 times faster. (OK, I didn't
measure it, but it was MUCH faster.)

My problem is that my rows are of very varying degrees of bushiness (i.e.
number of supercolums and columns per row). I inserted 592,500 rows
successfully, in a few minutes, and then I hit a batch of exceptionally
bushy rows and ran out of memory.

Does anyone have any suggestions about how to deal with this problem? I can
make my algorithm smarter by taking into account the size of the rows and
not just blindly do 100 at a time, but I want to solve this problem as
generally as possible, and not depend on trial and error, and on the
specific configuration of the machine I happen to be working on right now. I
don't even know if the critical parameter is the total size of the values,
or the number of columns, or what? Or maybe there's some optimal batch size,
and that's what I should use always?

Thanks.

Mime
View raw message