incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Jeltema <brian.jelt...@digitalenvoy.net>
Subject Hadoop/Cassandra 1.2 timeouts
Date Mon, 24 Jun 2013 16:10:13 GMT
I'm having problems with Hadoop job failures on a Cassandra 1.2 cluster due to 

    Caused by: TimedOutException()
    2013-06-24 11:29:11,953  INFO  Driver  - 	at org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12932)

This is running on a 6-node cluster, RF=3. If I run the job with CL=ONE, it usually runs pretty
well, with an occasional timeout. But
if I run at CL=QUORUM, the number of timeouts is often enough to kill the job. The table being
read is effectively read-only when this job runs.
It has from 5 to 10 million rows, with each row having no more than 256 columns. Each column
typically only has a few hundred bytes of data at most.

I've fiddled with the batch-range size and increasing the timeout, without a lot of luck.
I see some evidence of GC activity in the Cassandra logs, but
it's hard to see a clear correlation with the timeouts.

I could use some suggestions on an approach to pin down the root cause.

TIA

Brian
Mime
View raw message