cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: TimedOutException()
Date Wed, 04 Jan 2012 09:52:36 GMT
Look at the nodetool tpstats when you get the TimedOutException, to work out which nodes are
backing up with pending messages. Then try to identify why. Check the server logs for GC,
and the CPU and IO usage. 

Somehow the cluster is getting overwhelmed and cannot respond. Either the clients have started
whacking it harder than before, the environment (memory, io etc) has changed or cassandra
cannot keep up with it's maintenance tasks any more (e.g. compacting lots of wide rows that
hogs IO).   

I *think* the DEBUG message is a client disconnecting and it can be ignored. 

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 4/01/2012, at 12:44 PM, Daning Wang wrote:

> Hi All,
> 
> We are getting TimedOutException() when inserting data into Cassandra, it was working
fine for a few months, but suddenly got this problem. I have increase rpc_timout_in_ms to
30000, but it still timed out in 30 secs.
> 
> I turned on debug, I saw many of this error in the log
> 
> DEBUG [pool-2-thread-420] 2012-01-03 15:25:43,689 CustomTThreadPoolServer.java (line
197) Thrift transport error occurred during processing of message.
> org.apache.thrift.transport.TTransportException: Cannot read. Remote side has closed.
Tried to read 4 bytes, but only got 0 bytes. (This is often indicative of an internal error
on the server side. Please ch
> eck your server logs.)
>         at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
>         at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
>         at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
>         at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
>         at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2877)
>         at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>         at java.lang.Thread.run(Thread.java:722)
> 
> 
> We are on 0.8.6. Any idea how to fix this? Your help is much appreciated.
> 
> Daning
> 


Mime
View raw message