cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Hendry <dan.hendry.j...@gmail.com>
Subject Re: Various exceptions on 0.7
Date Sat, 04 Dec 2010 22:58:00 GMT
To be clear, I had to interrupt a clean operation earlier in the day be
killing the cassandra process. Now the node works for awhile,
but continually logging the "Error in row mutation" errors then eventually
logs a "Fatal exception in thread" error. After which, the process stays
alive but there seem to be problems reading from the node. At the very
least, read performance is massively degraded.

On Sat, Dec 4, 2010 at 5:52 PM, Dan Hendry <dan.hendry.junk@gmail.com>wrote:

> One of my Cassandra nodes is giving me a number of errors then effectively
> dying. I think it was somehow caused by interrupting a nodetool clean
> operation. Running a recent 0.7 build out of svn.
>
> ERROR [MutationStage:26] 2010-12-04 16:23:04,395
> RowMutationVerbHandler.java (line 83) Error in row mutation
> java.io.EOFException
>         at java.io.DataInputStream.readFully(DataInputStream.java:180)
>         at
> org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:264)
>         at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:76)
>         at
> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:363)
>         at
> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:312)
>         at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
>         at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:120)
>         at
> org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:383)
>         at
> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:393)
>         at
> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:351)
>         at
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:52)
>         at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> ERROR [MutationStage:13] 2010-12-04 16:25:04,061
> RowMutationVerbHandler.java (line 83) Error in row mutation
> org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find
> cfId=524288
>         at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:117)
>         at
> org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:383)
>         at
> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:393)
>         at
> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:351)
>         at
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:52)
>         at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> ERROR [MutationStage:20] 2010-12-04 16:25:25,216
> DebuggableThreadPoolExecutor.java (line 103) Error in ThreadPoolExecutor
> java.lang.NullPointerException
>         at org.apache.cassandra.db.Table.apply(Table.java:398)
>         at
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:73)
>         at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> ERROR [MutationStage:20] 2010-12-04 16:25:25,216
> AbstractCassandraDaemon.java (line 90) Fatal exception in thread
> Thread[MutationStage:20,5,main]
> java.lang.NullPointerException
>         at org.apache.cassandra.db.Table.apply(Table.java:398)
>         at
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:73)
>         at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> ERROR [COMMIT-LOG-WRITER] 2010-12-04 16:25:25,216
> AbstractCassandraDaemon.java (line 90) Fatal exception in thread
> Thread[COMMIT-LOG-WRITER,5,main]
> java.lang.RuntimeException: java.lang.NullPointerException
>         at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>         at java.lang.Thread.run(Thread.java:662)
> Caused by: java.lang.NullPointerException
>         at
> org.apache.cassandra.db.commitlog.CommitLogSegment.write(CommitLogSegment.java:92)
>         at
> org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:509)
>         at
> org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:52)
>         at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>         ... 1 more
>
>

Mime
View raw message