incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Various exceptions on 0.7
Date Sun, 05 Dec 2010 03:44:46 GMT
At least one of your nodes is sending garbage to the others.

Either there's a bug in the bleeding edge code you are running (did
you try rc1?) or you do have nodes on different versions or you have a
hardware problem.

On Sat, Dec 4, 2010 at 5:51 PM, Dan Hendry <dan.hendry.junk@gmail.com> wrote:
> Here are two other errors which appear frequently:
> ERROR [MutationStage:29] 2010-12-04 17:47:46,931 RowMutationVerbHandler.java
> (line 83) Error in row mutation
> java.io.IOException: Invalid localDeleteTime read: 0
>         at
> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:355)
>         at
> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:312)
>         at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
>         at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:120)
>         at
> org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:383)
>         at
> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:393)
>         at
> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:351)
>         at
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:52)
>         at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
>
> ERROR [MutationStage:15] 2010-12-04 17:48:33,216 RowMutationVerbHandler.java
> (line 83) Error in row mutation
> org.apache.cassandra.db.ColumnSerializer$CorruptColumnException: invalid
> column name length 0
>         at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:68)
>         at
> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:363)
>         at
> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:312)
>         at
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
>         at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:120)
>         at
> org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:383)
>         at
> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:393)
>         at
> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:351)
>         at
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:52)
>         at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
>
> On Sat, Dec 4, 2010 at 6:29 PM, Dan Hendry <dan.hendry.junk@gmail.com>
> wrote:
>>
>> No, all nodes are running very recent (< 2 day old) code out of the 0.7
>> branch. This cluster has always had 0.7 RC1(+) code running on it
>>
>> On Sat, Dec 4, 2010 at 6:24 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
>>>
>>> Are you mixing different Cassandra versions?
>>>
>>> On Sat, Dec 4, 2010 at 4:58 PM, Dan Hendry <dan.hendry.junk@gmail.com>
>>> wrote:
>>> > To be clear, I had to interrupt a clean operation earlier in the day be
>>> > killing the cassandra process. Now the node works for awhile,
>>> > but continually logging the "Error in row mutation" errors then
>>> > eventually
>>> > logs a "Fatal exception in thread" error. After which, the process
>>> > stays
>>> > alive but there seem to be problems reading from the node. At the very
>>> > least, read performance is massively degraded.
>>> >
>>> > On Sat, Dec 4, 2010 at 5:52 PM, Dan Hendry <dan.hendry.junk@gmail.com>
>>> > wrote:
>>> >>
>>> >> One of my Cassandra nodes is giving me a number of errors then
>>> >> effectively
>>> >> dying. I think it was somehow caused by interrupting a nodetool clean
>>> >> operation. Running a recent 0.7 build out of svn.
>>> >> ERROR [MutationStage:26] 2010-12-04 16:23:04,395
>>> >> RowMutationVerbHandler.java (line 83) Error in row mutation
>>> >> java.io.EOFException
>>> >>         at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:264)
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:76)
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:363)
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:312)
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:129)
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:120)
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:383)
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:393)
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:351)
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:52)
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63)
>>> >>         at
>>> >>
>>> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> >>         at
>>> >>
>>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> >>         at java.lang.Thread.run(Thread.java:662)
>>> >> ERROR [MutationStage:13] 2010-12-04 16:25:04,061
>>> >> RowMutationVerbHandler.java (line 83) Error in row mutation
>>> >> org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't
>>> >> find
>>> >> cfId=524288
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:117)
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:383)
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:393)
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:351)
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:52)
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63)
>>> >>         at
>>> >>
>>> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> >>         at
>>> >>
>>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> >>         at java.lang.Thread.run(Thread.java:662)
>>> >> ERROR [MutationStage:20] 2010-12-04 16:25:25,216
>>> >> DebuggableThreadPoolExecutor.java (line 103) Error in
>>> >> ThreadPoolExecutor
>>> >> java.lang.NullPointerException
>>> >>         at org.apache.cassandra.db.Table.apply(Table.java:398)
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:73)
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63)
>>> >>         at
>>> >>
>>> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> >>         at
>>> >>
>>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> >>         at java.lang.Thread.run(Thread.java:662)
>>> >> ERROR [MutationStage:20] 2010-12-04 16:25:25,216
>>> >> AbstractCassandraDaemon.java (line 90) Fatal exception in thread
>>> >> Thread[MutationStage:20,5,main]
>>> >> java.lang.NullPointerException
>>> >>         at org.apache.cassandra.db.Table.apply(Table.java:398)
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:73)
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:63)
>>> >>         at
>>> >>
>>> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>> >>         at
>>> >>
>>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>> >>         at java.lang.Thread.run(Thread.java:662)
>>> >> ERROR [COMMIT-LOG-WRITER] 2010-12-04 16:25:25,216
>>> >> AbstractCassandraDaemon.java (line 90) Fatal exception in thread
>>> >> Thread[COMMIT-LOG-WRITER,5,main]
>>> >> java.lang.RuntimeException: java.lang.NullPointerException
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>>> >>         at java.lang.Thread.run(Thread.java:662)
>>> >> Caused by: java.lang.NullPointerException
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.db.commitlog.CommitLogSegment.write(CommitLogSegment.java:92)
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:509)
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:52)
>>> >>         at
>>> >>
>>> >> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>> >>         ... 1 more
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Jonathan Ellis
>>> Project Chair, Apache Cassandra
>>> co-founder of Riptano, the source for professional Cassandra support
>>> http://riptano.com
>>
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Mime
View raw message