cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carl Yeksigian (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-11670) Error while waiting on bootstrap to complete. Bootstrap will have to be restarted. Stream failed
Date Thu, 12 May 2016 17:30:13 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15281802#comment-15281802
] 

Carl Yeksigian commented on CASSANDRA-11670:
--------------------------------------------

This points to a more general issue with MV potentially creating large mutations, not just
during streaming.

It would be possible to create a situation where the non-MV mutation is under the max size,
some set of replicas will accept the write (thus accepting the write into the cluster), but
a replica with a different set of data would fail and never be able to apply that mutation.
This wouldn't be an issue with normal batchlogs as they would universally reject that batch
on all replicas.

[~pauloricardomg] Where did you run into problems with the multi-batch solution? I think that
would solve this problem more universally for MV.

> Error while waiting on bootstrap to complete. Bootstrap will have to be restarted. Stream
failed
> ------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-11670
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11670
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Configuration, Streaming and Messaging
>            Reporter: Anastasia Osintseva
>            Assignee: Paulo Motta
>             Fix For: 3.0.5
>
>
> I have in cluster 2 DC, in each DC - 2 Nodes. I wanted to add 1 node to each DC. One
node has been added successfully after I had made scrubing. 
> Now I'm trying to add node to another DC, but get error: org.apache.cassandra.streaming.StreamException:
Stream failed. 
> After scrubing and repair I get the same error.  
> {noformat}
> ERROR [StreamReceiveTask:5] 2016-04-27 00:33:21,082 Keyspace.java:492 - Unknown exception
caught while attempting to update MaterializedView! messages_dump.messages
> java.lang.IllegalArgumentException: Mutation of 34974901 bytes is too large for the maxiumum
size of 33554432
> 	at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:264) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:469) [apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) [apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) [apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) [apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager.java:146) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:724) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.view.ViewManager.pushViewReplicaUpdates(ViewManager.java:149)
~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:487) [apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) [apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) [apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) [apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Mutation.applyUnsafe(Mutation.java:236) [apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:169)
[apache-cassandra-3.0.5.jar:3.0.5]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_11]
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_11]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_11]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_11]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_11]
> ERROR [StreamReceiveTask:5] 2016-04-27 00:33:21,082 StreamReceiveTask.java:214 - Error
applying streamed data: 
> java.lang.IllegalArgumentException: Mutation of 34974901 bytes is too large for the maxiumum
size of 33554432
> 	at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:264) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:469) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager.java:146) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:724) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.view.ViewManager.pushViewReplicaUpdates(ViewManager.java:149)
~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:487) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Mutation.applyUnsafe(Mutation.java:236) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:169)
~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_11]
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_11]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_11]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_11]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_11]
> ERROR [StreamReceiveTask:5] 2016-04-27 00:33:21,082 StreamSession.java:520 - [Stream
#f849ffe0-0bee-11e6-9b5f-d16a1b9764ab] Streaming error occurred
> java.lang.IllegalArgumentException: Mutation of 34974901 bytes is too large for the maxiumum
size of 33554432
> 	at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:264) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:469) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager.java:146) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:724) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.view.ViewManager.pushViewReplicaUpdates(ViewManager.java:149)
~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:487) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.db.Mutation.applyUnsafe(Mutation.java:236) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:169)
~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_11]
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_11]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_11]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_11]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_11]
> DEBUG [StreamReceiveTask:5] 2016-04-27 00:33:21,082 ConnectionHandler.java:110 - [Stream
#f849ffe0-0bee-11e6-9b5f-d16a1b9764ab] Closing stream connection handler on /88.9.99.92
> DEBUG [STREAM-OUT-/88.9.99.92] 2016-04-27 00:33:21,082 ConnectionHandler.java:341 - [Stream
#f849ffe0-0bee-11e6-9b5f-d16a1b9764ab] Sending Session Failed
> INFO  [StreamReceiveTask:5] 2016-04-27 00:33:21,082 StreamResultFuture.java:182 - [Stream
#f849ffe0-0bee-11e6-9b5f-d16a1b9764ab] Session with /88.9.99.92 is complete
> WARN  [StreamReceiveTask:5] 2016-04-27 00:33:21,182 StreamResultFuture.java:209 - [Stream
#f849ffe0-0bee-11e6-9b5f-d16a1b9764ab] Stream failed
> ERROR [main] 2016-04-27 00:33:21,259 StorageService.java:1300 - Error while waiting on
bootstrap to complete. Bootstrap will have to be restarted.
> java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException:
Stream failed
> 	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
~[guava-18.0.jar:na]
> 	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
~[guava-18.0.jar:na]
> 	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-18.0.jar:na]
> 	at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1295) [apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:971)
[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:745) [apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.service.StorageService.initServer(StorageService.java:610) [apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:333) [apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551) [apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:679) [apache-cassandra-3.0.5.jar:3.0.5]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> 	at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
> 	at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
~[guava-18.0.jar:na]
> 	at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-18.0.jar:na]
> 	at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
> 	at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
~[guava-18.0.jar:na]
> 	at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:210)
~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:186)
~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:430)
~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:525) ~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:216)
~[apache-cassandra-3.0.5.jar:3.0.5]
> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_11]
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_11]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_11]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_11]
> 	at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_11]
> {noformat}
> I set commitlog_segment_size_in_mb: 128, but it didn't help. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message