cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dathan Pattishall <datha...@gmail.com>
Subject Using this I think I found/verified an interesting problem with Cassandra
Date Wed, 08 Sep 2010 19:08:14 GMT
This link describe ganglia / cassandra graphing.

http://mysqldba.blogspot.com/2010/09/cassandra-and-ganglia.html

I ran into a problem illustrated here.

http://www.flickr.com/photos/dathan/4971255111/

This screen shot shows a huge spike of transport exceptions between the
hours of 12:15 - to 1:30. Why? Lets see.

http://www.flickr.com/photos/dathan/4971869002/

This link shows that the pending reads jump because the message
deserialization pool (mutex) blocks or maybe its viceversa. But Why? Lets
see.

This link shows that wait_io on the box sky rocketed.

http://www.flickr.com/photos/dathan/4971290101/

but why?

Could it be because

http://www.flickr.com/photos/dathan/4971869054/

This graph shows a massive amount of data growth for this server, then it
reduces but why? How can I tune it so that a growth of data doesn't explode
like this?



Some background information:

These servers are DELL 2950 dual quad core boxes with 48GB of Ram on a
RAID-10 EXT3 FS backed by 8 disks on a PERC-6 Controller with BBC. Each
server rougly recieves 300-400 requests per second fronted by a F5
Loadbalancer (soon to be HA-Proxy) on least connections, doing a client stat
check to verify the server is up from a client point of view.

There is only one simple key space. A Super Column is defined but not used
and uses a RandomPartitioner with NO RowCaching and mmap enabled.

Mime
View raw message