incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lee Parker <...@socialagency.com>
Subject cassandra instability
Date Fri, 16 Apr 2010 17:50:55 GMT
I am having major issues with stability on my cassandra nodes.  Here is the
setup:
Cassandra Cluster - 2 EC2 small instances (1.7G RAM, single 32 bit core)
with an EBS for the cassandra sstables
Cassandra 0.6.0 w/ 1G heap space and 128M/1mil Memtable Thresholds
Clients are also small EC2 webservers running php 5.3.2 and using Thrift PHP
to access the cluster

I am trying to migrate data from mysql into the cluster using the following
methodology:
1. get 500 rows (12 columns each) from mysql
2. build a batch_mutate to insert these rows into one CF (1 row = 1 row )
3. build a second batch_mutate to insert an index of those rows into a
second CF ( 1 row = 1 column )
4. loop around and do it again until all data has migrated.

This process is running on two clients each working on a separate part of
the mysql data which totals to about 70G.  Each time I start it up, it will
work fine for about 1 hour and then it will crash the servers.  The error
message on the servers is usually an out of memory error.  I will get
several time out errors on the clients and occasionally get an error telling
me that i was missing the timestamp.  The timestamp error is accompanied by
a server crashing if I use framed transport instead of buffered.  I wasn't
having the out of memory errors with 0.5.0, but had lots of timeouts and
some "unknown result" errors.  So we upgraded to 0.6.0 when it became the
stable release.

I have a development VM which has all the same code on one machine and have
had stability issues there as well.  I upgraded to the newest build of Java
(b20) on the VM and it hasn't helped.  If anything, it is more unstable.

I am at the point of abandoning cassandra as a solution for us, but wanted
to see if you all could offer me some advice first.  One of the reasons we
were trying cassandra was to scale out with smaller nodes rather than having
to run larger instances for mysql.

Lee Parker

Mime
View raw message