There's no error on the source node other than the Timeout.
It appears to be occurring across multiple CF's (the majority of which are normal columns).
I don't know an exact number but some of the CF's would have ~3million rows.
It seems odd that the error sometimes says received 1 response, but it still times out, as I only have one node.
As for load, CPU usage is certainly not a bottleneck.
"top" consistently shows ~ 10-20% waiting,

Chris.

On Mon, Nov 9, 2009 at 9:22 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
What's causing the timeout?  An error on the source node, or just
slowness?  If the latter, how many rows are in your multiget?

On Mon, Nov 9, 2009 at 10:25 PM, Chris Were <chris.were@gmail.com> wrote:
>
> I'm getting a Timeout Exception every now and again (currently every couple
> of minutes or so).
> Using revision 833288. Quorum set to ONE. My cassandra instance has been
> running for two days and the data directory is around 16GB. I'm not sure
> what the problem is, but let me know of any tests I can do to help reduce
> the problem further. There are two variations on the exception, I have
> pasted them both below.
> ERROR [pool-1-thread-63] 2009-11-09 20:17:27,579 Cassandra.java (line
> org.apache.cassandra.service.Cassandra$Processor) Internal error processing
> get_slice
> java.lang.RuntimeException: java.util.concurrent.TimeoutException: Operation
> timed out - received only 0 responses from  .
> at
> org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:103)
> at
> org.apache.cassandra.service.CassandraServer.getSlice(CassandraServer.java:177)
> at
> org.apache.cassandra.service.CassandraServer.multigetSliceInternal(CassandraServer.java:252)
> at
> org.apache.cassandra.service.CassandraServer.get_slice(CassandraServer.java:215)
> at
> org.apache.cassandra.service.Cassandra$Processor$get_slice.process(Cassandra.java:668)
> at
> org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:624)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out -
> received only 0 responses from  .
> at
> org.apache.cassandra.service.QuorumResponseHandler.get(QuorumResponseHandler.java:79)
> at
> org.apache.cassandra.service.StorageProxy.strongRead(StorageProxy.java:408)
> at
> org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:333)
> at
> org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:95)
> ... 9 more
> ERROR [pool-1-thread-19] 2009-11-09 11:29:18,731 Cassandra.java (line
> org.apache.cassandra.service.Cassandra$Processor) Internal error processing
> get_slice
> java.lang.RuntimeException: java.util.concurrent.TimeoutException: Operation
> timed out - received only 1 responses from /10.121.217.5 .
>         at
> org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:103)
>         at
> org.apache.cassandra.service.CassandraServer.getSlice(CassandraServer.java:177)
>         at
> org.apache.cassandra.service.CassandraServer.multigetSliceInternal(CassandraServer.java:252)
>         at
> org.apache.cassandra.service.CassandraServer.get_slice(CassandraServer.java:215)
>         at
> org.apache.cassandra.service.Cassandra$Processor$get_slice.process(Cassandra.java:668)
>         at
> org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:624)
>         at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:636)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out -
> received only 1 responses from /10.121.217.5 .
>         at
> org.apache.cassandra.service.QuorumResponseHandler.get(QuorumResponseHandler.java:79)
>         at
> org.apache.cassandra.service.StorageProxy.strongRead(StorageProxy.java:408)
>         at
> org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:333)
>         at
> org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:95)
>         ... 9 more
> Cheers,
> Chris