cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Shuler (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-8306) exception in nodetool enablebinary
Date Fri, 14 Nov 2014 19:31:34 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-8306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14212694#comment-14212694
] 

Michael Shuler commented on CASSANDRA-8306:
-------------------------------------------

Could you explain exactly your process of how you were adding this node to your cluster? 
The provided log is 171480 lines of tombstone processing and only 54 lines of INFO,WARN normal
log entries.

> exception in nodetool enablebinary
> ----------------------------------
>
>                 Key: CASSANDRA-8306
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8306
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Rafał Furmański
>            Assignee: Russ Hatch
>         Attachments: system.log.zip
>
>
> I was trying to add new node (db4) to existing cluster - with no luck. I can't see any
errors in system.log. nodetool status shows, that node is joining into cluster (for many hours).
Attaching error and cluster info:
> {code}
> root@db4:~# nodetool enablebinary
> error: Error starting native transport: null
> -- StackTrace --
> java.lang.RuntimeException: Error starting native transport: null
> 	at org.apache.cassandra.service.StorageService.startNativeTransport(StorageService.java:350)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
> 	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
> 	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> 	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> 	at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> 	at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> 	at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> 	at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
> 	at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
> 	at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
> 	at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1420)
> 	at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:556)
> 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:811)
> 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> {code}
> {code}
> root@db4:~# nodetool describecluster
> Cluster Information:
> 	Name: Production Cluster
> 	Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> 	Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> 	Schema versions:
> 		b7e98bb9-717f-3f59-bac4-84bc19544e90: [10.195.15.163, 10.195.15.162, 10.195.15.167,
10.195.15.166]
> {code}
> {code}
> root@db4:~# nodetool status
> Datacenter: Ashburn
> ===================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> --  Address        Load       Tokens  Owns    Host ID                               Rack
> UN  10.195.15.163  12.05 GB   256     ?       0a9f478c-80b5-4c15-8b2e-e27df6684c69  RAC1
> UN  10.195.15.162  12.8 GB    256     ?       c18d2218-ef84-4165-9c3a-05f592f512e9  RAC1
> UJ  10.195.15.167  18.61 GB   256     ?       0d3999d9-1e33-4407-bbbd-10cf0a93b3ba  RAC1
> UN  10.195.15.166  13.67 GB   256     ?       df8df3b7-da17-48de-8cf6-1c718dc2fde8  RAC1
> {code}
> I can't even connect to cassandra using cqlsh:
> {code}
> root@db4:~# cqlsh 10.195.15.167
> Connection error: ('Unable to connect to any servers', {'10.195.15.167': error(111, "Tried
connecting to [('10.195.15.167', 9042)]. Last error: Connection refused")})
> {code}
> In OpsCenter I can see that node has status: Active - Joining and Gossip is active. Any
ideas what could be wrong with this node?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message