cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rafał Furmański (JIRA) <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-8306) exception in nodetool enablebinary
Date Mon, 17 Nov 2014 07:21:34 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-8306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14214368#comment-14214368
] 

Rafał Furmański commented on CASSANDRA-8306:
--------------------------------------------

start_native_transport is set to true.

The whole joining process looks as follows:
1. Install cassandra 2.1.1 (from datastax debian repo)
2. stop cassandra
3. rm -rf /var/lib/cassandra/*
4. Modify cassandra.yaml and cassandra-rackdc.properties
5. Start cassandra

After struggling for a couple of days and several restarts of stubborn node, it finally managed
to join the cluster. I don't know why though. I didn't do anything specific. Binary protocol
is up and running now. That's just weird!

> exception in nodetool enablebinary
> ----------------------------------
>
>                 Key: CASSANDRA-8306
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8306
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Rafał Furmański
>         Attachments: system.log.zip
>
>
> I was trying to add new node (db4) to existing cluster - with no luck. I can't see any
errors in system.log. nodetool status shows, that node is joining into cluster (for many hours).
Attaching error and cluster info:
> {code}
> root@db4:~# nodetool enablebinary
> error: Error starting native transport: null
> -- StackTrace --
> java.lang.RuntimeException: Error starting native transport: null
> 	at org.apache.cassandra.service.StorageService.startNativeTransport(StorageService.java:350)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
> 	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
> 	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> 	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> 	at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> 	at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> 	at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> 	at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
> 	at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
> 	at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
> 	at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
> 	at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
> 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
> 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> {code}
> {code}
> root@db4:~# nodetool describecluster
> Cluster Information:
> 	Name: Production Cluster
> 	Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> 	Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> 	Schema versions:
> 		b7e98bb9-717f-3f59-bac4-84bc19544e90: [10.195.15.163, 10.195.15.162, 10.195.15.167,
10.195.15.166]
> {code}
> {code}
> root@db4:~# nodetool status
> Datacenter: Ashburn
> ===================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address        Load       Tokens  Owns    Host ID                               Rack
> UN  10.195.15.163  12.05 GB   256     ?       0a9f478c-80b5-4c15-8b2e-e27df6684c69  RAC1
> UN  10.195.15.162  12.8 GB    256     ?       c18d2218-ef84-4165-9c3a-05f592f512e9  RAC1
> UJ  10.195.15.167  18.61 GB   256     ?       0d3999d9-1e33-4407-bbbd-10cf0a93b3ba  RAC1
> UN  10.195.15.166  13.67 GB   256     ?       df8df3b7-da17-48de-8cf6-1c718dc2fde8  RAC1
> {code}
> I can't even connect to cassandra using cqlsh:
> {code}
> root@db4:~# cqlsh 10.195.15.167
> Connection error: ('Unable to connect to any servers', {'10.195.15.167': error(111, "Tried
connecting to [('10.195.15.167', 9042)]. Last error: Connection refused")})
> {code}
> In OpsCenter I can see that node has status: Active - Joining and Gossip is active. Any
ideas what could be wrong with this node?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message