cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Rollins <and...@localytics.com>
Subject Re: EC2 small instance config?
Date Wed, 07 Jul 2010 18:15:17 GMT
On Wed, Jul 7, 2010 at 2:27 AM, Michael Dürgner <m...@duergner.de> wrote:

> Have you done some testing with small nodes already? Because from what we
> saw trying to run IO bound services on small instances is, that their IO
> performance is really bad compared to other instance types as you can read
> in several blogs.
>
> Would be interesting to hear, if a Cassandra cluster can handle that.
>

I have actually.

I tested on 10 small nodes on Amazon EC2, each with 1 EBS disk. I've been
avoiding large nodes for now since they are 4x the cost of a small, and 10
small would translate to 2.5 large nodes. We figured it's better to slice
things into more nodes, since 2 or 3 nodes would mean large chunks of data
would need to be moved if a node failed.

Under pure write loads with a fairly default config and 3x replication, we
achieved 1,000 writes per second and probably could have pushed it a little
bit more (perhaps to 2k per second). Write speed barely slowed even as we
pushed past 50 million keys. Keys were 255 bytes with a single column
containing 768 bytes.

Things got much worse when we introduced reads, however. We did a 50/50 read
write split. IO went up, and nodes failed a couple hours into the test with
out of memory errors. My theory is that the reads caused much more IO, which
caused writes to get backed up in memory.

I've had success in the past with RAID striping on EBS volumes. I was able
to get nearly 4x improvement on a small instance with MySQL, so my next
thing would be to try RAID with Cassandra.

Also, another theory is that CommitLogSync in batch mode might allow me to
effectively rate limit writing so that I don't overflow memory.

Thoughts?

- Andrew

Mime
View raw message