cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maxime Lamothe-Brassard (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-7144) CassandraDaemon RowMutation exception
Date Fri, 23 May 2014 06:03:02 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-7144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14006847#comment-14006847
] 

Maxime Lamothe-Brassard commented on CASSANDRA-7144:
----------------------------------------------------

The writes were not using prepared statements. The batch contain average INSERT/UPDATEs, but
I also have a counter update interleaved between the statements. Here is a sample:

{code}
self.queries.append( ( ( 'UPDATE met SET n = n + 1 WHERE id = %(id)s', { 'id' : k } ) )
self.queries.append( ( '''BEGIN UNLOGGED BATCH
                                                  INSERT INTO om ( id, obj, otype ) VALUES
( %(id)s, %(obj)s, %(type)s );
                                                  UPDATE loc SET last = %(ts)s WHERE sid =
%(sid)s AND otype = %(type)s AND id = %(id)s;
                                                  INSERT INTO lid ( id, sid ) VALUES ( %(id)s,
%(sid)s );
                                                  APPLY BATCH;''', { 'id' : k,
                                                                     'obj' : unicode( obj[
0 ] ).lower(),
                                                                     'type' : obj[ 1 ],
                                                                     'ts' : ts,
                                                                     'sid' : ag } ) )
{code}

> CassandraDaemon RowMutation exception
> -------------------------------------
>
>                 Key: CASSANDRA-7144
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7144
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu 12.04 w/ Oracle JVM, 5 nodes cluster. Nodes 2GB / 2 Cores
in DigitalOcean.
>            Reporter: Maxime Lamothe-Brassard
>            Assignee: Tyler Hobbs
>
> First time reporting a bug here, apologies if I'm not posting it in the right space.
> At what seems like random interval, on random nodes in random situations I will get the
following exception. After this the hinted handoff start timing out and the node stops participating
in the cluster.
> I started seeing these after switching to the Cassandra Python-Driver from the Python-CQL
driver.
> {noformat}
> ERROR [WRITE-/10.128.180.108] 2014-05-03 13:45:12,843 CassandraDaemon.java (line 198)
Exception in thread Thread[WRITE-/10.128.180.108,5,main]
> java.lang.AssertionError
> 	at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
> 	at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259)
> 	at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120)
> 	at org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251)
> 	at org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203)
> 	at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151)
> ERROR [WRITE-/10.128.194.70] 2014-05-03 13:45:12,843 CassandraDaemon.java (line 198)
Exception in thread Thread[WRITE-/10.128.194.70,5,main]
> java.lang.AssertionError
> 	at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
> 	at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259)
> 	at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:120)
> 	at org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:251)
> 	at org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:203)
> 	at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:151)
> ERROR [MutationStage:118] 2014-05-03 13:45:15,048 CassandraDaemon.java (line 198) Exception
in thread Thread[MutationStage:118,5,main]
> java.lang.AssertionError
> 	at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
> 	at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259)
> 	at org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654)
> 	at org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137)
> 	at org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908)
> 	at org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881)
> 	at org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)
> ERROR [MutationStage:117] 2014-05-03 13:45:15,048 CassandraDaemon.java (line 198) Exception
in thread Thread[MutationStage:117,5,main]
> java.lang.AssertionError
> 	at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:271)
> 	at org.apache.cassandra.db.RowMutation$RowMutationSerializer.serialize(RowMutation.java:259)
> 	at org.apache.cassandra.utils.FBUtilities.serialize(FBUtilities.java:654)
> 	at org.apache.cassandra.db.HintedHandOffManager.hintFor(HintedHandOffManager.java:137)
> 	at org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:908)
> 	at org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:881)
> 	at org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:1981)
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:744)
> {noformat}
> The service must be restarted for the node to come back online. Let me know any additional
configuration details needed.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message