cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Shuler (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-6589) Gossip AssertionError: Added cell does not sort as the last cell
Date Wed, 15 Jan 2014 16:56:21 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-6589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Michael Shuler updated CASSANDRA-6589:
--------------------------------------

    Description: 
The super_counter_test.py dtest sometimes, but not always, hangs until killed (^C) when running
with vnodes.  Reproduction (again, sometimes this is successful - keep trying until it hangs
for a few minutes, which is about 1/3-1/2 of the time for me):
{noformat}
$ ENABLE_VNODES=true nosetests -v super_counter_test.py
{noformat}
If it does hang up on you, in another shell:
{noformat}
$ tail -F /tmp/dtest-*/test/node*/logs/system.log
{noformat}
Scrolling errors similar to:
{noformat}
ERROR [GossipStage:54] 2014-01-15 10:39:30,229 CassandraDaemon.java:139 - Exception in thread
Thread[GossipStage:54,5,main]
java.lang.AssertionError: Added cell does not sort as the last cell
        at org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:116)
~[main/:na]
        at org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:118) ~[main/:na]
        at org.apache.cassandra.db.ColumnFamily.addIfRelevant(ColumnFamily.java:112) ~[main/:na]
        at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:214)
~[main/:na]
        at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:123)
~[main/:na]
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:81)
~[main/:na]
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:73)
~[main/:na]
        at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:296)
~[main/:na]
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:52)
~[main/:na]
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1522)
~[main/:na]
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1341)
~[main/:na]
        at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:327) ~[main/:na]
        at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
        at org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:223)
~[main/:na]
        at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:241)
~[main/:na]
        at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:57)
~[main/:na]
        at org.apache.cassandra.cql3.QueryProcessor.processInternal(QueryProcessor.java:261)
~[main/:na]
        at org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:457)
~[main/:na]
        at org.apache.cassandra.net.OutboundTcpConnectionPool.<init>(OutboundTcpConnectionPool.java:48)
~[main/:na]
        at org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:491)
~[main/:na]
        at org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:505)
~[main/:na]
        at org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:637)
~[main/:na]
        at org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:611)
~[main/:na]
        at org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40) ~[main/:na]
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:60) ~[main/:na]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_45]
        at java.lang.Thread.run(Thread.java:744) ~[na:1.7.0_45]
{noformat}
along with node1 repeatedly marking node3 up and down.

  was:
The super_counter_test.py sometimes, but not always, hangs until killed (^C) when running
with vnodes.  Reproduction (again, sometimes this is successful - keep trying until it hangs
for a few minutes, which is about 1/3-1/2 of the time for me):
{noformat}
$ ENABLE_VNODES=true nosetests -v super_counter_test.py
{noformat}
If it does hang up on you, in another shell:
{noformat}
$ tail -F /tmp/dtest-*/test/node*/logs/system.log
{noformat}
Scrolling errors similar to:
{noformat}
ERROR [GossipStage:54] 2014-01-15 10:39:30,229 CassandraDaemon.java:139 - Exception in thread
Thread[GossipStage:54,5,main]
java.lang.AssertionError: Added cell does not sort as the last cell
        at org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:116)
~[main/:na]
        at org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:118) ~[main/:na]
        at org.apache.cassandra.db.ColumnFamily.addIfRelevant(ColumnFamily.java:112) ~[main/:na]
        at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:214)
~[main/:na]
        at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:123)
~[main/:na]
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:81)
~[main/:na]
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:73)
~[main/:na]
        at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:296)
~[main/:na]
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:52)
~[main/:na]
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1522)
~[main/:na]
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1341)
~[main/:na]
        at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:327) ~[main/:na]
        at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
        at org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:223)
~[main/:na]
        at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:241)
~[main/:na]
        at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:57)
~[main/:na]
        at org.apache.cassandra.cql3.QueryProcessor.processInternal(QueryProcessor.java:261)
~[main/:na]
        at org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:457)
~[main/:na]
        at org.apache.cassandra.net.OutboundTcpConnectionPool.<init>(OutboundTcpConnectionPool.java:48)
~[main/:na]
        at org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:491)
~[main/:na]
        at org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:505)
~[main/:na]
        at org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:637)
~[main/:na]
        at org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:611)
~[main/:na]
        at org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40) ~[main/:na]
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:60) ~[main/:na]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_45]
        at java.lang.Thread.run(Thread.java:744) ~[na:1.7.0_45]
{noformat}
along with node1 repeatedly marking node3 up and down.


> Gossip AssertionError: Added cell does not sort as the last cell
> ----------------------------------------------------------------
>
>                 Key: CASSANDRA-6589
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6589
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Debian Wheezy amd64
> Oracle JDK 1.7_45
> Branch: trunk
>            Reporter: Michael Shuler
>
> The super_counter_test.py dtest sometimes, but not always, hangs until killed (^C) when
running with vnodes.  Reproduction (again, sometimes this is successful - keep trying until
it hangs for a few minutes, which is about 1/3-1/2 of the time for me):
> {noformat}
> $ ENABLE_VNODES=true nosetests -v super_counter_test.py
> {noformat}
> If it does hang up on you, in another shell:
> {noformat}
> $ tail -F /tmp/dtest-*/test/node*/logs/system.log
> {noformat}
> Scrolling errors similar to:
> {noformat}
> ERROR [GossipStage:54] 2014-01-15 10:39:30,229 CassandraDaemon.java:139 - Exception in
thread Thread[GossipStage:54,5,main]
> java.lang.AssertionError: Added cell does not sort as the last cell
>         at org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:116)
~[main/:na]
>         at org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:118) ~[main/:na]
>         at org.apache.cassandra.db.ColumnFamily.addIfRelevant(ColumnFamily.java:112)
~[main/:na]
>         at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:214)
~[main/:na]
>         at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:123)
~[main/:na]
>         at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:81)
~[main/:na]
>         at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:73)
~[main/:na]
>         at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:296)
~[main/:na]
>         at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:52)
~[main/:na]
>         at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1522)
~[main/:na]
>         at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1341)
~[main/:na]
>         at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:327) ~[main/:na]
>         at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59)
~[main/:na]
>         at org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:223)
~[main/:na]
>         at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:241)
~[main/:na]
>         at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:57)
~[main/:na]
>         at org.apache.cassandra.cql3.QueryProcessor.processInternal(QueryProcessor.java:261)
~[main/:na]
>         at org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:457)
~[main/:na]
>         at org.apache.cassandra.net.OutboundTcpConnectionPool.<init>(OutboundTcpConnectionPool.java:48)
~[main/:na]
>         at org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:491)
~[main/:na]
>         at org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:505)
~[main/:na]
>         at org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:637)
~[main/:na]
>         at org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:611)
~[main/:na]
>         at org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40)
~[main/:na]
>         at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:60)
~[main/:na]
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_45]
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_45]
>         at java.lang.Thread.run(Thread.java:744) ~[na:1.7.0_45]
> {noformat}
> along with node1 repeatedly marking node3 up and down.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message