cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Zarutin (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-5689) NPE shutting down Cassandra trunk (cassandra-1.2.5-989-g70dfb70)
Date Wed, 03 Jul 2013 04:14:21 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-5689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Alex Zarutin updated CASSANDRA-5689:
------------------------------------

    Attachment: init3
                init2
                init1
                CASSANDRA-5689.txt

General comments:
I use CCM based installed C* cluster, so I do not use init script to stop and start it. So,
what I did, I creaated subset of ..../debian/init script to be able to run it from the shell
against the cluster started by CCM.

Comment regarding this particular issue. Node1 went down after I run $nodetool drain

I) logs to see after running $nodetool disablegossip

==> /Users/alexzarutin/.ccm/test-1.2.6/node1/logs/system.log <==
 WARN [RMI TCP Connection(2)-10.10.10.130] 2013-07-02 20:45:21,896 StorageService.java (line
288) Stopping gossip by operator request
 INFO [RMI TCP Connection(2)-10.10.10.130] 2013-07-02 20:45:21,897 Gossiper.java (line 1108)
Announcing shutdown

==> /Users/alexzarutin/.ccm/test-1.2.6/node2/logs/system.log <==
 INFO [GossipStage:1] 2013-07-02 20:45:23,899 Gossiper.java (line 785) InetAddress /127.0.0.1
is now DOWN

==> /Users/alexzarutin/.ccm/test-1.2.6/node3/logs/system.log <==
 INFO [GossipStage:1] 2013-07-02 20:45:23,899 Gossiper.java (line 785) InetAddress /127.0.0.1
is now DOWN
 INFO [HANDSHAKE-/127.0.0.1] 2013-07-02 20:45:24,078 OutboundTcpConnection.java (line 399)
Handshaking version with /127.0.0.1

==> /Users/alexzarutin/.ccm/test-1.2.6/node2/logs/system.log <==
 INFO [HANDSHAKE-/127.0.0.1] 2013-07-02 20:45:24,523 OutboundTcpConnection.java (line 399)
Handshaking version with /127.0.0.1

II) logs after sleeping 5 seconds and running $nodetool disablebinary

==> /Users/alexzarutin/.ccm/test-1.2.6/node1/logs/system.log <==
 INFO [RMI TCP Connection(6)-10.10.10.130] 2013-07-02 20:48:04,149 Server.java (line 148)
Stop listening for CQL clients


III) logs after running $nodetool disablethrift

==> /Users/alexzarutin/.ccm/test-1.2.6/node1/logs/system.log <==
INFO [RMI TCP Connection(8)-10.10.10.130] 2013-07-02 20:49:08,008 ThriftServer.java (line
116) Stop listening to thrift clients


IV) logs after running $nodetool drain

==> /Users/alexzarutin/.ccm/test-1.2.6/node1/logs/system.log <==
 INFO [RMI TCP Connection(10)-10.10.10.130] 2013-07-02 20:50:40,713 StorageService.java (line
897) DRAINING: starting drain process
 INFO [RMI TCP Connection(10)-10.10.10.130] 2013-07-02 20:50:40,714 Gossiper.java (line 1108)
Announcing shutdown
 INFO [RMI TCP Connection(10)-10.10.10.130] 2013-07-02 20:50:42,715 MessagingService.java
(line 685) Waiting for messaging service to quiesce
 INFO [ACCEPT-/127.0.0.1] 2013-07-02 20:50:42,716 MessagingService.java (line 895) MessagingService
shutting down server thread.
 INFO [RMI TCP Connection(10)-10.10.10.130] 2013-07-02 20:50:42,718 ColumnFamilyStore.java
(line 630) Enqueuing flush of Memtable-local@788901055(52/52 serialized/live bytes, 4 ops)
 INFO [RMI TCP Connection(10)-10.10.10.130] 2013-07-02 20:50:42,718 ColumnFamilyStore.java
(line 630) Enqueuing flush of Memtable-peers@1810922770(410/410 serialized/live bytes, 32
ops)
 INFO [FlushWriter:2] 2013-07-02 20:50:42,719 Memtable.java (line 461) Writing Memtable-local@788901055(52/52
serialized/live bytes, 4 ops)
 INFO [FlushWriter:2] 2013-07-02 20:50:42,728 Memtable.java (line 495) Completed flushing
/Users/alexzarutin/.ccm/test-1.2.6/node1/data/system/local/system-local-ic-11-Data.db (84
bytes) for commitlog position ReplayPosition(segmentId=1372823015605, position=81617)
 INFO [FlushWriter:2] 2013-07-02 20:50:42,729 Memtable.java (line 461) Writing Memtable-peers@1810922770(410/410
serialized/live bytes, 32 ops)
 INFO [FlushWriter:2] 2013-07-02 20:50:42,735 Memtable.java (line 495) Completed flushing
/Users/alexzarutin/.ccm/test-1.2.6/node1/data/system/peers/system-peers-ic-4-Data.db (323
bytes) for commitlog position ReplayPosition(segmentId=1372823015605, position=81617)
 INFO [CompactionExecutor:22] 2013-07-02 20:50:42,736 CompactionTask.java (line 105) Compacting
[SSTableReader(path='/Users/alexzarutin/.ccm/test-1.2.6/node1/data/system/peers/system-peers-ic-2-Data.db'),
SSTableReader(path='/Users/alexzarutin/.ccm/test-1.2.6/node1/data/system/peers/system-peers-ic-1-Data.db'),
SSTableReader(path='/Users/alexzarutin/.ccm/test-1.2.6/node1/data/system/peers/system-peers-ic-3-Data.db'),
SSTableReader(path='/Users/alexzarutin/.ccm/test-1.2.6/node1/data/system/peers/system-peers-ic-4-Data.db')]
 INFO [CompactionExecutor:22] 2013-07-02 20:50:42,743 CompactionTask.java (line 262) Compacted
4 sstables to [/Users/alexzarutin/.ccm/test-1.2.6/node1/data/system/peers/system-peers-ic-5,].
 986 bytes to 394 (~39% of original) in 7ms = 0.053678MB/s.  7 total rows, 2 unique.  Row
merge counts were {1:0, 2:0, 3:1, 4:1, }
 INFO [RMI TCP Connection(10)-10.10.10.130] 2013-07-02 20:50:42,893 StorageService.java (line
897) DRAINED

Pay attention to the following line. On this point node1 is down

INFO [ACCEPT-/127.0.0.1] 2013-07-02 20:50:42,716 MessagingService.java (line 895) MessagingService
shutting down server thread.

So running ./init2 and ./init3 will fully stop the cluster. There are no NullPointer or any
other exceptions.

Full repro steps and init1, init2, init3 scripts stripped to the functionality of stoping
cluster are attached. 

                
> NPE shutting down Cassandra trunk (cassandra-1.2.5-989-g70dfb70)
> ----------------------------------------------------------------
>
>                 Key: CASSANDRA-5689
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5689
>             Project: Cassandra
>          Issue Type: Bug
>          Components: API
>    Affects Versions: 1.2.0
>         Environment: Ubuntu Precise with Oracle Java 7u25.
>            Reporter: Blair Zajac
>            Assignee: Alex Zarutin
>            Priority: Trivial
>         Attachments: CASSANDRA-5689.txt, init1, init2, init3
>
>
> I built Cassandra from git trunk at cassandra-1.2.5-989-g70dfb70 using the debian/ package.
 I have a shell script to shut down Cassandra:
> {code}
>       $nodetool disablegossip
>       sleep 5
>       $nodetool disablebinary
>       $nodetool disablethrift
>       $nodetool drain
>       /etc/init.d/cassandra stop
> {code}
> Shutting it down I get this exception on all three nodes:
> {code}
> Exception in thread "main" java.lang.NullPointerException
> 	at org.apache.cassandra.transport.Server.close(Server.java:156)
> 	at org.apache.cassandra.transport.Server.stop(Server.java:107)
> 	at org.apache.cassandra.service.StorageService.stopNativeTransport(StorageService.java:347)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
> 	at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
> 	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
> 	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
> 	at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
> 	at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> 	at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
> 	at com.sun.jmx.remote.security.MBeanServerAccessController.invoke(MBeanServerAccessController.java:468)
> 	at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1487)
> 	at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:97)
> 	at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1328)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1427)
> 	at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:848)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> 	at sun.rmi.transport.Transport$1.run(Transport.java:177)
> 	at sun.rmi.transport.Transport$1.run(Transport.java:174)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> 	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:724)
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message