I experienced this - it's might be GC or compaction process. Try to increase timeout value in the config file. It's not a fix, but at least something...

On Mon, Nov 9, 2009 at 11:25 PM, Chris Were <chris.were@gmail.com> wrote:

I'm getting a Timeout Exception every now and again (currently every couple of minutes or so).

Using revision 833288. Quorum set to ONE. My cassandra instance has been running for two days and the data directory is around 16GB. I'm not sure what the problem is, but let me know of any tests I can do to help reduce the problem further. There are two variations on the exception, I have pasted them both below.

ERROR [pool-1-thread-63] 2009-11-09 20:17:27,579 Cassandra.java (line org.apache.cassandra.service.Cassandra$Processor) Internal error processing get_slice
java.lang.RuntimeException: java.util.concurrent.TimeoutException: Operation timed out - received only 0 responses from  .
at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:103)
at org.apache.cassandra.service.CassandraServer.getSlice(CassandraServer.java:177)
at org.apache.cassandra.service.CassandraServer.multigetSliceInternal(CassandraServer.java:252)
at org.apache.cassandra.service.CassandraServer.get_slice(CassandraServer.java:215)
at org.apache.cassandra.service.Cassandra$Processor$get_slice.process(Cassandra.java:668)
at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:624)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.util.concurrent.TimeoutException: Operation timed out - received only 0 responses from  .
at org.apache.cassandra.service.QuorumResponseHandler.get(QuorumResponseHandler.java:79)
at org.apache.cassandra.service.StorageProxy.strongRead(StorageProxy.java:408)
at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:333)
at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:95)
... 9 more

ERROR [pool-1-thread-19] 2009-11-09 11:29:18,731 Cassandra.java (line org.apache.cassandra.service.Cassandra$Processor) Internal error processing get_slice
java.lang.RuntimeException: java.util.concurrent.TimeoutException: Operation timed out - received only 1 responses from /10.121.217.5 .
        at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:103)
        at org.apache.cassandra.service.CassandraServer.getSlice(CassandraServer.java:177)
        at org.apache.cassandra.service.CassandraServer.multigetSliceInternal(CassandraServer.java:252)
        at org.apache.cassandra.service.CassandraServer.get_slice(CassandraServer.java:215)
        at org.apache.cassandra.service.Cassandra$Processor$get_slice.process(Cassandra.java:668)
        at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:624)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: java.util.concurrent.TimeoutException: Operation timed out - received only 1 responses from /10.121.217.5 .
        at org.apache.cassandra.service.QuorumResponseHandler.get(QuorumResponseHandler.java:79)
        at org.apache.cassandra.service.StorageProxy.strongRead(StorageProxy.java:408)
        at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:333)
        at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:95)
        ... 9 more

Cheers,
Chris