cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daning Wang <dan...@netseer.com>
Subject Cassandra Exception
Date Wed, 21 Mar 2012 17:24:21 GMT
Hi All,


We got lots of Exception in the log, and later the server crashed. any idea
what is happening and how to fix it?

ERROR [RequestResponseStage:4] 2012-03-21 04:16:30,482
AbstractCassandraDaemon.java (line 139) Fatal exception in thread
Thread[RequestResponseStage:4,5,main]
java.io.IOError: java.io.EOFException
        at
org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
        at
org.apache.cassandra.service.ReadCallback.response(ReadCallback.java:125)
        at
org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
        at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
Caused by: java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:197)
        at
org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:104)
        at
org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82)
        at
org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64)
        ... 6 more
ERROR [RequestResponseStage:2] 2012-03-21 04:16:30,480
AbstractCassandraDaemon.java (line 139) Fatal exception in thread
Thread[RequestResponseStage:2,5,main]
java.io.IOError:
org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find
cfId=-387130991
        at
org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:71)
        at
org.apache.cassandra.service.AsyncRepairCallback.response(AsyncRepairCallback.java:47)
        at
org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:49)
        at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.cassandra.db.UnserializableColumnFamilyException:
Couldn't find cfId=-387130991
        at
org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:123)
        at org.apache.cassandra.db.RowSerializer.deserialize(Row.java:69)
        at
org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:113)
        at
org.apache.cassandra.db.ReadResponseSerializer.deserialize(ReadResponse.java:82)
        at
org.apache.cassandra.service.AbstractRowResolver.preprocess(AbstractRowResolver.java:64



The is the exception before server crashes.


ERROR [ReadRepairStage:299] 2012-03-21 05:02:53,808
AbstractCassandraDaemon.java (line 139) Fatal exception in thread
Thread[ReadRepairStage:299,5,main]
java.lang.RuntimeException:
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
shut down
        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
Caused by: java.util.concurrent.RejectedExecutionException:
ThreadPoolExecutor has shut down
        at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
        at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816)
        at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337)
        at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:490)
        at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:388)
        at
org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:346)
        at
org.apache.cassandra.service.RowRepairResolver.maybeScheduleRepairs(RowRepairResolver.java:121)
        at
org.apache.cassandra.service.RowRepairResolver.resolve(RowRepairResolver.java:85)
        at
org.apache.cassandra.service.AsyncRepairCallback$1.runMayThrow(AsyncRepairCallback.java:54)
        at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        ... 3 more

Thank you in advance,

Daning

Mime
View raw message