cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeff Jirsa (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13178) Cassandra crash if a table is dropped while joining
Date Fri, 03 Feb 2017 07:44:51 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15851189#comment-15851189
] 

Jeff Jirsa commented on CASSANDRA-13178:
----------------------------------------

Why can't we watch for that exception and then skip streaming those sstables (or, delete them
if they've already been streamed) instead of killing the bootstrap entirely? That is - why
do we need to make operators resume a bootstrap instead of protecting them? 


> Cassandra crash if a table is dropped while joining
> ---------------------------------------------------
>
>                 Key: CASSANDRA-13178
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13178
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Streaming and Messaging
>         Environment: Cassandra-2.1.9
>            Reporter: Brian Gallew
>
> I just had three different nodes crash while joining a cluster.  They've been streaming
data for 2 days now, and each one exited with this error:
> {code}
> ERROR [STREAM-IN-/XX.XX.144.63] 2017-02-02 20:42:45,275 StreamSession.java:505 - [Stream
#b4279df0-e7de-11e6-aa5a-452e7f2ad12b] Streaming error occurred
> java.io.IOException: CF 8280bcb0-cdf7-11e6-b227-b97a72c7aa0d was dropped during streaming
> 	at org.apache.cassandra.streaming.compress.CompressedStreamReader.read(CompressedStreamReader.java:71)
~[apache-cassandra-2.1.9.jar:2.1.9-SNAPSHOT]
> 	at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:48)
~[apache-cassandra-2.1.9.jar:2.1.9-SNAPSHOT]
> 	at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:38)
~[apache-cassandra-2.1.9.jar:2.1.9-SNAPSHOT]
> 	at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:56)
~[apache-cassandra-2.1.9.jar:2.1.9-SNAPSHOT]
> 	at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:250)
~[apache-cassandra-2.1.9.jar:2.1.9-SNAPSHOT]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_91]
> INFO  [STREAM-IN-/XX.XX.144.63] 2017-02-02 20:42:45,275 StreamResultFuture.java:180 -
[Stream #b4279df0-e7de-11e6-aa5a-452e7f2ad12b] Session with /XX.XX.144.63 is complete
> WARN  [STREAM-IN-/XX.XX.144.63] 2017-02-02 20:42:45,277 StreamResultFuture.java:207 -
[Stream #b4279df0-e7de-11e6-aa5a-452e7f2ad12b] Stream failed
> ERROR [main] 2017-02-02 20:42:45,278 CassandraDaemon.java:579 - Exception encountered
during startup
> java.lang.RuntimeException: Error during boostrap: Stream failed
> 	at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:86) ~[apache-cassandra-2.1.9.jar:2.1.9-SNAPSHOT]
> 	at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1148) ~[apache-cassandra-2.1.9.jar:2.1.9-SNAPSHOT]
> 	at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:938)
~[apache-cassandra-2.1.9.jar:2.1.9-SNAPSHOT]
> 	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:734) ~[apache-cassandra-2.1.9.jar:2.1.9-SNAPSHOT]
> 	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:611) ~[apache-cassandra-2.1.9.jar:2.1.9-SNAPSHOT]
> 	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:387) [apache-cassandra-2.1.9.jar:2.1.9-SNAPSHOT]
> 	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:562) [apache-cassandra-2.1.9.jar:2.1.9-SNAPSHOT]
> 	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:651) [apache-cassandra-2.1.9.jar:2.1.9-SNAPSHOT]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> 	at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[apache-cassandra-2.1.9.jar:2.1.9-SNAPSHOT]
> 	at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na]
> 	at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
~[guava-16.0.jar:na]
> 	at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-16.0.jar:na]
> 	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-16.0.jar:na]
> 	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
~[guava-16.0.jar:na]
> 	at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:208)
~[apache-cassandra-2.1.9.jar:2.1.9-SNAPSHOT]
> 	at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:184)
~[apache-cassandra-2.1.9.jar:2.1.9-SNAPSHOT]
> 	at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:415)
~[apache-cassandra-2.1.9.jar:2.1.9-SNAPSHOT]
> 	at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:510) ~[apache-cassandra-2.1.9.jar:2.1.9-SNAPSHOT]
> 	at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:268)
~[apache-cassandra-2.1.9.jar:2.1.9-SNAPSHOT]
> 	at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_91]
> WARN  [StorageServiceShutdownHook] 2017-02-02 20:42:45,286 Gossiper.java:1450 - No local
state or state is in silent shutdown, not announcing shutdown
> INFO  [StorageServiceShutdownHook] 2017-02-02 20:42:45,286 MessagingService.java:735
- Waiting for messaging service to quiesce
> INFO  [ACCEPT-/XX.XX.146.177] 2017-02-02 20:42:45,293 MessagingService.java:1011 - MessagingService
has terminated the accept() thread
> INFO  [CompactionExecutor:6837] 2017-02-02 20:42:45,323 ColumnFamilyStore.java:900 -
Enqueuing flush of compactions_in_progress: 1687 (0%) on-heap, 0 (0%) off-heap
> INFO  [MemtableFlushWriter:5703] 2017-02-02 20:42:45,324 Memtable.java:347 - Writing
Memtable-compactions_in_progress@689094870(0.184KiB serialized bytes, 11 ops, 0%/0% of on/off-heap
limit)
> INFO  [MemtableFlushWriter:5703] 2017-02-02 20:42:45,324 Memtable.java:382 - Completed
flushing /mnt/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-tmp-ka-5779-Data.db
(0.000KiB) for commitlog position ReplayPosition(segmentId=1485885417287, position=23644862)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message