cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Drew Dahlke <>
Subject Cassandra timeouts under low load
Date Tue, 15 Jun 2010 16:00:04 GMT
Hi, I'm running cassandra .6.2 on a dedicated 4 node cluster and I
also have a dedicated 4 node hadoop cluster. I'm trying to run a
simple map reduce job against a single column family and it only takes
32 map tasks before I get floods of thrift timeouts. That would make
sense to me if the cassandra was stressing the hardware or the
network, but it's not. Each box has 8 cores/16G ram. During the job
CPU averages 150-250% (1/5 utilization on 8 cores), network IO hovers
around 15% throughput, iostat < 15%.

The hadoop machines are taking even less of a beating. The simpler I
make the job, the faster it hits cassandra, the faster it throws
timeouts & vice versa. I'm guessing there's a software/config related
bottleneck I'm hitting well before tapping out the hardware. Any idea
what that might be?

java.lang.RuntimeException: TimedOutException()
	at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.maybeInit(
	at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(
	at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.computeNext(
	at org.apache.cassandra.hadoop.ColumnFamilyRecordReader.nextKeyValue(
	at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(
	at org.apache.hadoop.mapreduce.MapContext.nextKeyValue(
	at org.apache.hadoop.mapred.MapTask.runNewMapper(
	at org.apache.hadoop.mapred.Child.main(
Caused by: TimedOutException()
	at org.apache.cassandra.thrift.Cassandra$
	at org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(
	at org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(
	at org.apache.cassandra.hadoop.ColumnFamilyRecordReader$RowIterator.maybeInit(
	... 11 more

View raw message