cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Subramanya M (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-4981) Error when starting a node with vnodes while counter-add operations underway
Date Thu, 25 Aug 2016 20:04:21 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-4981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15437546#comment-15437546
] 

Subramanya M commented on CASSANDRA-4981:
-----------------------------------------

Its still happening. I have setup 2 nodes on Ubuntu on VM. Had powered off VM when nodes are
UP.

   Now when I restart the VM and bring up only one node. insert into simple table works fine.
But when I bring up the other node and try the insert, it fails with below error in cassandra
node 1:

ERROR 18:45:56 Unexpected exception during request; channel = [id: 0xbdb30e19, L:/127.0.0.1:9042
- R:/127.0.0.1:39944]
java.lang.AssertionError: null
	at org.apache.cassandra.locator.TokenMetadata.firstTokenIndex(TokenMetadata.java:946) ~[apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.locator.TokenMetadata.firstToken(TokenMetadata.java:960) ~[apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:107)
~[apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:3612)
~[apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.service.StorageProxy.getLiveSortedEndpoints(StorageProxy.java:1832)
~[apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.service.StorageProxy$RangeIterator.computeNext(StorageProxy.java:1910)
~[apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.service.StorageProxy$RangeIterator.computeNext(StorageProxy.java:1880)
~[apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.7.jar:3.7]
	at com.google.common.collect.Iterators$PeekingImpl.hasNext(Iterators.java:1149) ~[guava-18.0.jar:na]
	at org.apache.cassandra.service.StorageProxy$RangeMerger.computeNext(StorageProxy.java:1932)
~[apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.service.StorageProxy$RangeMerger.computeNext(StorageProxy.java:1917)
~[apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.service.StorageProxy$RangeCommandIterator.computeNext(StorageProxy.java:2044)
~[apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.service.StorageProxy$RangeCommandIterator.computeNext(StorageProxy.java:2010)
~[apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72) ~[apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.cql3.statements.SelectStatement.process(SelectStatement.java:735)
~[apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.cql3.statements.SelectStatement.processResults(SelectStatement.java:410)
~[apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:363)
~[apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:237)
~[apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:78)
~[apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:208) ~[apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:239) ~[apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:224) ~[apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:115) ~[apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) [apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) [apache-cassandra-3.7.jar:3.7]
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.36.Final.jar:4.0.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:292)
[netty-all-4.0.36.Final.jar:4.0.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:32)
[netty-all-4.0.36.Final.jar:4.0.36.Final]
	at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:283)
[netty-all-4.0.36.Final.jar:4.0.36.Final]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_101]
	at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
[apache-cassandra-3.7.jar:3.7]
	at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.7.jar:3.7]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
INFO  19:22:48 Redistributing index summaries
INFO  19:28:17 ConcurrentMarkSweep GC in 221ms.  CMS Old Gen: 46231032 -> 23371736; Par
Eden Space: 44030248 -> 0; Par Survivor Space: 3730896 -> 0

cqlsh:arc> select * from user;
ServerError: <ErrorMessage code=0000 [Server error] message="java.lang.AssertionError">

Cassandra version 3.7.

When only first node is started:
$ ./nodetool -h 127.0.0.1 -p 7199 status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens       Owns    Host ID                               Rack
UN  127.0.0.1  236.88 KiB  256          ?       e2c88dd6-18f5-447b-a05c-ac4a2112969b  rack1

Note: Non-system keyspaces don't have the same replication settings, effective ownership information
is meaningless

After starting 2nd node$ ./nodetool -h 127.0.0.2 -p 7199 status
nodetool: Failed to connect to '127.0.0.2:7199' - ConnectException: 'Connection refused'.



> Error when starting a node with vnodes while counter-add operations underway
> ----------------------------------------------------------------------------
>
>                 Key: CASSANDRA-4981
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4981
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: 2-node cluster on ec2, ubuntu, cassandra-1.2.0 commit a32eb9f7d2f2868e8154d178e96e045859e1d855
>            Reporter: Tyler Patterson
>            Assignee: Ryan McGuire
>            Priority: Minor
>              Labels: qa-resolved
>         Attachments: system.log
>
>
> Start both nodes, start stress on one node like this: "cassandra-stress --replication-factor=2
--operation=COUNTER_ADD"
> While that is running: On the other node, kill cassandra, wait for "nodetool status"
to show the node as down, and restart cassandra. I sometimes have to kill and restart cassandra
several times to get the problem to happen.
> I get this error several times in the log:
> {code}
> ERROR 15:39:33,198 Exception in thread Thread[MutationStage:16,5,main]
> java.lang.AssertionError
> 	at org.apache.cassandra.locator.TokenMetadata.firstTokenIndex(TokenMetadata.java:748)
> 	at org.apache.cassandra.locator.TokenMetadata.firstToken(TokenMetadata.java:762)
> 	at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:95)
> 	at org.apache.cassandra.service.StorageService.getNaturalEndpoints(StorageService.java:2426)
> 	at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:396)
> 	at org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:755)
> 	at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:53)
> 	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:662)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message