cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10926) AssertionError when removing node
Date Tue, 22 Dec 2015 22:04:46 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15068796#comment-15068796
] 

Kai Wang commented on CASSANDRA-10926:
--------------------------------------

Joel,

Thanks. This is a long story.

First of all, nodetool status from a good node:
{noformat}
--  Address       Load       Tokens       Owns    Host ID                               Rack
UN  192.168.0.10  225.89 GB  256          ?       2014f9be-423a-49cd-b8ef-252983c8f615  rack1
UN  192.168.0.11  242.84 GB  256          ?       b822b46f-9981-4105-8308-c3feebd16e32  rack1
UJ  192.168.0.12  29.22 MB   256          ?       ca519cf7-813f-4503-863a-7b5fb93cc3a8  rack1
{noformat}

I guess that pretty much explained the AssertionError.

I had a 3-node cluster. I rebuilt one of them from scratch. When I copied C* config to it
and tried to bootstrap, I got an error saying I can't add because this node was already in
the cluster. I needed to replace the node instead. So I did it by adding -Dcassandra.replace_address.
Then I found the bootstrap didn't start (can't remember what exactly happened). The other
two nodes have 250GB data but the new node only has 12MB.

Then I tried to delete the data and commit log folders from the new node, remove the -Dcassandra.replace_address.
I wanted to add the new node as a brand new one.

But then I got streaming failure error. From the new node's system.log, I saw this:
{noformat}
ERROR [STREAM-IN-/192.168.0.10] 2015-12-22 16:46:57,107 StreamSession.java:524 - [Stream #6d9603f0-a8f5-11e5-8eb5-67d8099a8b91]
Streaming error occurred
java.nio.channels.ClosedChannelException: null
        at sun.nio.ch.SocketChannelImpl.ensureReadOpen(Unknown Source) ~[na:1.8.0_65]
        at sun.nio.ch.SocketChannelImpl.read(Unknown Source) ~[na:1.8.0_65]
        at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:53)
~[apache-cassandra-2.2.4.jar:2.2.4]
...
{noformat}

and 

{noformat}
ERROR [main] 2015-12-22 16:46:57,144 StorageService.java:1245 - Error while waiting on bootstrap
to complete. Bootstrap will have to be restarted.
java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream
failed
...
        at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1240)
[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:920)
[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:705)
[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:581)
[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:315) [apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:529)
[apache-cassandra-2.2.4.jar:2.2.4]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:638) [apache-cassandra-2.2.4.jar:2.2.4]
{noformat}

>From the node I streamed from, I saw this:
{noformat}
org.apache.cassandra.io.FSReadError: java.io.IOException: An existing connection was forcibly
closed by the remote host
	at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:144) ~[apache-cassandra-2.2.4.jar:2.2.4]
	at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:79)
~[apache-cassandra-2.2.4.jar:2.2.4]
	at org.apache.cassandra.streaming.compress.CompressedStreamWriter$1.apply(CompressedStreamWriter.java:76)
~[apache-cassandra-2.2.4.jar:2.2.4]
	at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.applyToChannel(BufferedDataOutputStreamPlus.java:297)
~[apache-cassandra-2.2.4.jar:2.2.4]
	at org.apache.cassandra.streaming.compress.CompressedStreamWriter.write(CompressedStreamWriter.java:75)
~[apache-cassandra-2.2.4.jar:2.2.4]
	at org.apache.cassandra.streaming.messages.OutgoingFileMessage.serialize(OutgoingFileMessage.java:90)
~[apache-cassandra-2.2.4.jar:2.2.4]
	at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:48)
~[apache-cassandra-2.2.4.jar:2.2.4]
	at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:40)
~[apache-cassandra-2.2.4.jar:2.2.4]
	at org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:47)
~[apache-cassandra-2.2.4.jar:2.2.4]
	at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:363)
~[apache-cassandra-2.2.4.jar:2.2.4]
	at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:335)
~[apache-cassandra-2.2.4.jar:2.2.4]
{noformat}

I was pretty much stuck at that point. All I wanted to do is just bootstrap the new node as
if it was a brand new one. That's why I tried to remove it using removenode and added it back
to see if it makes any difference.

> AssertionError when removing node
> ---------------------------------
>
>                 Key: CASSANDRA-10926
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10926
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: CentOS 7 x64, Java 1.8.0.65
>            Reporter: Kai Wang
>            Assignee: Joel Knighton
>
> {noformat}
> [root@centos-2 ~]# nodetool -u xxx -pw xxxx removenode 97a9042d-ea14-49a9-9f22-2dab4b762673
> error: null
> -- StackTrace --
> java.lang.AssertionError
>         at org.apache.cassandra.locator.TokenMetadata.getTokens(TokenMetadata.java:474)
>         at org.apache.cassandra.service.StorageService.removeNode(StorageService.java:3793)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>         at java.lang.reflect.Method.invoke(Unknown Source)
>         at sun.reflect.misc.Trampoline.invoke(Unknown Source)
>         at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>         at java.lang.reflect.Method.invoke(Unknown Source)
>         at sun.reflect.misc.MethodUtil.invoke(Unknown Source)
>         at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
>         at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
>         at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)
>         at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source)
>         at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
>         at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)
>         at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)
>         at com.sun.jmx.remote.security.MBeanServerAccessController.invoke(Unknown Source)
>         at javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
>         at javax.management.remote.rmi.RMIConnectionImpl.access$300(Unknown Source)
>         at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown
Source)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown
Source)
>         at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown Source)
>         at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>         at java.lang.reflect.Method.invoke(Unknown Source)
>         at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
>         at sun.rmi.transport.Transport$1.run(Unknown Source)
>         at sun.rmi.transport.Transport$1.run(Unknown Source)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at sun.rmi.transport.Transport.serviceCall(Unknown Source)
>         at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
>         at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
>         at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$81(Unknown
Source)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>         at java.lang.Thread.run(Unknown Source)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message