Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 0D234200AC6 for ; Fri, 6 May 2016 16:08:15 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 0BAE2160A0D; Fri, 6 May 2016 14:08:15 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 0FCFB160A0C for ; Fri, 6 May 2016 16:08:13 +0200 (CEST) Received: (qmail 58434 invoked by uid 500); 6 May 2016 14:08:13 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 58413 invoked by uid 99); 6 May 2016 14:08:13 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 06 May 2016 14:08:13 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id F048A2C1F61 for ; Fri, 6 May 2016 14:08:12 +0000 (UTC) Date: Fri, 6 May 2016 14:08:12 +0000 (UTC) From: =?utf-8?Q?Alexander_Hei=C3=9F_=28JIRA=29?= To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CASSANDRA-11670) Error while waiting on bootstrap to complete. Bootstrap will have to be restarted. Stream failed MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Fri, 06 May 2016 14:08:15 -0000 [ https://issues.apache.org/jira/browse/CASSANDRA-11670?page=3Dcom.atla= ssian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId= =3D15274075#comment-15274075 ]=20 Alexander Hei=C3=9F commented on CASSANDRA-11670: -------------------------------------------- In our cassandra.yaml we have a *commitlog_segment_size_in_mb* of 32 (the d= efault) and no *max_mutation_size_in_kb* In the system.log it says *max_mutation_size_in_kb=3Dnull* and *commitlog_s= egment_size_in_mb=3D32* The Config file and log output is the same on every Server in the Cluster. > Error while waiting on bootstrap to complete. Bootstrap will have to be r= estarted. Stream failed > -------------------------------------------------------------------------= ----------------------- > > Key: CASSANDRA-11670 > URL: https://issues.apache.org/jira/browse/CASSANDRA-1167= 0 > Project: Cassandra > Issue Type: Bug > Components: Configuration, Streaming and Messaging > Reporter: Anastasia Osintseva > Fix For: 3.0.5 > > > I have in cluster 2 DC, in each DC - 2 Nodes. I wanted to add 1 node to e= ach DC. One node has been added successfully after I had made scrubing.=20 > Now I'm trying to add node to another DC, but get error: org.apache.cassa= ndra.streaming.StreamException: Stream failed.=20 > After scrubing and repair I get the same error. =20 > {noformat} > ERROR [StreamReceiveTask:5] 2016-04-27 00:33:21,082 Keyspace.java:492 - U= nknown exception caught while attempting to update MaterializedView! messag= es_dump.messages > java.lang.IllegalArgumentException: Mutation of 34974901 bytes is too lar= ge for the maxiumum size of 33554432 > =09at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:264)= ~[apache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:469) [apache-c= assandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) [apache-c= assandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) [ap= ache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) [apache-c= assandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager= .java:146) ~[apache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.jav= a:724) ~[apache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.view.ViewManager.pushViewReplicaUpdates(Vie= wManager.java:149) ~[apache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:487) [apache-c= assandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) [apache-c= assandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) [ap= ache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) [apache-c= assandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Mutation.applyUnsafe(Mutation.java:236) [ap= ache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnab= le.run(StreamReceiveTask.java:169) [apache-cassandra-3.0.5.jar:3.0.5] > =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:= 511) [na:1.8.0_11] > =09at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_= 11] > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1142) [na:1.8.0_11] > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:617) [na:1.8.0_11] > =09at java.lang.Thread.run(Thread.java:745) [na:1.8.0_11] > ERROR [StreamReceiveTask:5] 2016-04-27 00:33:21,082 StreamReceiveTask.jav= a:214 - Error applying streamed data:=20 > java.lang.IllegalArgumentException: Mutation of 34974901 bytes is too lar= ge for the maxiumum size of 33554432 > =09at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:264)= ~[apache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:469) ~[apache-= cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) ~[apache-= cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) ~[a= pache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) ~[apache-= cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager= .java:146) ~[apache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.jav= a:724) ~[apache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.view.ViewManager.pushViewReplicaUpdates(Vie= wManager.java:149) ~[apache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:487) ~[apache-= cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) ~[apache-= cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) ~[a= pache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) ~[apache-= cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Mutation.applyUnsafe(Mutation.java:236) ~[a= pache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnab= le.run(StreamReceiveTask.java:169) ~[apache-cassandra-3.0.5.jar:3.0.5] > =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:= 511) [na:1.8.0_11] > =09at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_= 11] > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1142) [na:1.8.0_11] > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:617) [na:1.8.0_11] > =09at java.lang.Thread.run(Thread.java:745) [na:1.8.0_11] > ERROR [StreamReceiveTask:5] 2016-04-27 00:33:21,082 StreamSession.java:52= 0 - [Stream #f849ffe0-0bee-11e6-9b5f-d16a1b9764ab] Streaming error occurred > java.lang.IllegalArgumentException: Mutation of 34974901 bytes is too lar= ge for the maxiumum size of 33554432 > =09at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:264)= ~[apache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:469) ~[apache-= cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) ~[apache-= cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) ~[a= pache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) ~[apache-= cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager= .java:146) ~[apache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.jav= a:724) ~[apache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.view.ViewManager.pushViewReplicaUpdates(Vie= wManager.java:149) ~[apache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:487) ~[apache-= cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:384) ~[apache-= cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:205) ~[a= pache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Mutation.apply(Mutation.java:217) ~[apache-= cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.db.Mutation.applyUnsafe(Mutation.java:236) ~[a= pache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnab= le.run(StreamReceiveTask.java:169) ~[apache-cassandra-3.0.5.jar:3.0.5] > =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:= 511) [na:1.8.0_11] > =09at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_= 11] > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1142) [na:1.8.0_11] > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:617) [na:1.8.0_11] > =09at java.lang.Thread.run(Thread.java:745) [na:1.8.0_11] > DEBUG [StreamReceiveTask:5] 2016-04-27 00:33:21,082 ConnectionHandler.jav= a:110 - [Stream #f849ffe0-0bee-11e6-9b5f-d16a1b9764ab] Closing stream conne= ction handler on /88.9.99.92 > DEBUG [STREAM-OUT-/88.9.99.92] 2016-04-27 00:33:21,082 ConnectionHandler.= java:341 - [Stream #f849ffe0-0bee-11e6-9b5f-d16a1b9764ab] Sending Session F= ailed > INFO [StreamReceiveTask:5] 2016-04-27 00:33:21,082 StreamResultFuture.ja= va:182 - [Stream #f849ffe0-0bee-11e6-9b5f-d16a1b9764ab] Session with /88.9.= 99.92 is complete > WARN [StreamReceiveTask:5] 2016-04-27 00:33:21,182 StreamResultFuture.ja= va:209 - [Stream #f849ffe0-0bee-11e6-9b5f-d16a1b9764ab] Stream failed > ERROR [main] 2016-04-27 00:33:21,259 StorageService.java:1300 - Error whi= le waiting on bootstrap to complete. Bootstrap will have to be restarted. > java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.S= treamException: Stream failed > =09at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(Abst= ractFuture.java:299) ~[guava-18.0.jar:na] > =09at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractF= uture.java:286) ~[guava-18.0.jar:na] > =09at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture= .java:116) ~[guava-18.0.jar:na] > =09at org.apache.cassandra.service.StorageService.bootstrap(StorageServic= e.java:1295) [apache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.service.StorageService.joinTokenRing(StorageSe= rvice.java:971) [apache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.service.StorageService.initServer(StorageServi= ce.java:745) [apache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.service.StorageService.initServer(StorageServi= ce.java:610) [apache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.= java:333) [apache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaem= on.java:551) [apache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.j= ava:679) [apache-cassandra-3.0.5.jar:3.0.5] > Caused by: org.apache.cassandra.streaming.StreamException: Stream failed > =09at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.on= Failure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.0.5.jar:3.0.5] > =09at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) = ~[guava-18.0.jar:na] > =09at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.exec= ute(MoreExecutors.java:457) ~[guava-18.0.jar:na] > =09at com.google.common.util.concurrent.ExecutionList.executeListener(Exe= cutionList.java:156) ~[guava-18.0.jar:na] > =09at com.google.common.util.concurrent.ExecutionList.execute(ExecutionLi= st.java:145) ~[guava-18.0.jar:na] > =09at com.google.common.util.concurrent.AbstractFuture.setException(Abstr= actFuture.java:202) ~[guava-18.0.jar:na] > =09at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(Str= eamResultFuture.java:210) ~[apache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComp= lete(StreamResultFuture.java:186) ~[apache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSes= sion.java:430) ~[apache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.= java:525) ~[apache-cassandra-3.0.5.jar:3.0.5] > =09at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnab= le.run(StreamReceiveTask.java:216) ~[apache-cassandra-3.0.5.jar:3.0.5] > =09at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:= 511) ~[na:1.8.0_11] > =09at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0= _11] > =09at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto= r.java:1142) ~[na:1.8.0_11] > =09at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut= or.java:617) ~[na:1.8.0_11] > =09at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_11] > {noformat} > I set commitlog_segment_size_in_mb: 128, but it didn't help.=20 -- This message was sent by Atlassian JIRA (v6.3.4#6332)