Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 5D8BD200A5B for ; Wed, 25 May 2016 09:47:41 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 5C488160A18; Wed, 25 May 2016 07:47:41 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 571EF160A17 for ; Wed, 25 May 2016 09:47:40 +0200 (CEST) Received: (qmail 94130 invoked by uid 500); 25 May 2016 07:47:38 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 94120 invoked by uid 99); 25 May 2016 07:47:38 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 May 2016 07:47:38 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 089A4180530 for ; Wed, 25 May 2016 07:47:38 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.198 X-Spam-Level: * X-Spam-Status: No, score=1.198 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=textkernel.nl Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id MMs6wHr7uQ6v for ; Wed, 25 May 2016 07:47:35 +0000 (UTC) Received: from mail-vk0-f47.google.com (mail-vk0-f47.google.com [209.85.213.47]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with ESMTPS id 409D95F3F2 for ; Wed, 25 May 2016 07:47:35 +0000 (UTC) Received: by mail-vk0-f47.google.com with SMTP id y2so53660267vka.3 for ; Wed, 25 May 2016 00:47:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=textkernel.nl; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to; bh=Qdr4n2TAOGbRRDk/1Y/DP5oMKWCBkSleRNJ/lNeiguk=; b=covKUXqqscRzUU17XTdDxB8meIfvt1adAo8ITWX/+Ls4huHi4ibrxi+6QcpI8QrgS+ e0WWtP0x4AST3Vr/qnGsTagbcvq6CFAzVrHA+dNsvGj5nXW55FrjxEo8npaatXsHPY/v DGM0ADk1SFkgDXlGMGph1G8kN/A50g2wFR/U4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to; bh=Qdr4n2TAOGbRRDk/1Y/DP5oMKWCBkSleRNJ/lNeiguk=; b=C8RYOvbv1DOdBlcfrjs2gqldeA8u9ZlAVzvocvMwbhNWTSsZrjRb2nqWfO7mha2dog u5ASIFsCsKcJXYyliXPwgjF/htHC0ZPZICQ0SeychSbf0tEm/CkYq+HSXuyj2E8T5nb/ RRd3i62CZ65m+2C0krsWNJHbuqb04uaqbdic416bT+6xSnsS6JdSFnwvjjnPia+v5B0X 5hS1T3IJmyK4tUOEmC7L8CIp8YPzPgNbr61WtDUlmr16rxrQUb7ODuFZyzs2svQ2h6ve 3dYeFtNJJd50TvRnwpW6U5lDJa/xJTWJe9gj1VrspATSPOku1nkyOvbQvkrfHWSr0GDe 3C2A== X-Gm-Message-State: ALyK8tIIwz9R+ViUtWtSng4hFklghrxzZDS4vFwl4Z6B7e/PWJJO14t05BB10TcrISwjzdSf4/ElfXOu8PKKGlOt MIME-Version: 1.0 X-Received: by 10.31.155.149 with SMTP id d143mr1477022vke.96.1464162448169; Wed, 25 May 2016 00:47:28 -0700 (PDT) Received: by 10.31.165.67 with HTTP; Wed, 25 May 2016 00:47:28 -0700 (PDT) In-Reply-To: References: Date: Wed, 25 May 2016 09:47:28 +0200 Message-ID: Subject: Re: Error while rebuilding a node: Stream failed From: George Sigletos To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a113fcd6c9df16e0533a5e250 archived-at: Wed, 25 May 2016 07:47:41 -0000 --001a113fcd6c9df16e0533a5e250 Content-Type: text/plain; charset=UTF-8 Hi Mike, Yes I am using NetworkTopologyStrategy. I checked cassandra-rackdc.properties on the new node: dc=DCamazon-1 rack=RACamazon-1 I also checked the jira link you sent me. My network topology seems correct: I have 4 nodes in DC1 and 1 node in DCamazon-1 and I can verify that when running "nodetool status". Now I am running a full repair on the amazon node. I have given up rebuilding Kind regards, George On Wed, May 25, 2016 at 8:50 AM, Mike Yeap wrote: > Hi George, are you using NetworkTopologyStrategy as the replication > strategy for your keyspace? If yes, can you check the > cassandra-rackdc.properties of this new node? > > https://issues.apache.org/jira/browse/CASSANDRA-8279 > > > Regards, > Mike Yeap > > On Wed, May 25, 2016 at 2:31 PM, George Sigletos > wrote: > >> I am getting this error repeatedly while I am trying to add a new DC >> consisting of one node in AWS to my existing cluster. I have tried 5 times >> already. Running Cassandra 2.1.13 >> >> I have also set: >> streaming_socket_timeout_in_ms: 3600000 >> in all of my nodes >> >> Does anybody have any idea how this can be fixed? Thanks in advance >> >> Kind regards, >> George >> >> P.S. >> The complete stack trace: >> -- StackTrace -- >> java.lang.RuntimeException: Error while rebuilding node: Stream failed >> at >> org.apache.cassandra.service.StorageService.rebuild(StorageService.java:1076) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) >> at java.lang.reflect.Method.invoke(Unknown Source) >> at sun.reflect.misc.Trampoline.invoke(Unknown Source) >> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) >> at java.lang.reflect.Method.invoke(Unknown Source) >> at sun.reflect.misc.MethodUtil.invoke(Unknown Source) >> at >> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source) >> at >> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Unknown Source) >> at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown >> Source) >> at com.sun.jmx.mbeanserver.PerInterface.invoke(Unknown Source) >> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source) >> at >> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source) >> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source) >> at >> javax.management.remote.rmi.RMIConnectionImpl.doOperation(Unknown Source) >> at >> javax.management.remote.rmi.RMIConnectionImpl.access$300(Unknown Source) >> at >> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Unknown >> Source) >> at >> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown >> Source) >> at javax.management.remote.rmi.RMIConnectionImpl.invoke(Unknown >> Source) >> at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) >> at java.lang.reflect.Method.invoke(Unknown Source) >> at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source) >> at sun.rmi.transport.Transport$2.run(Unknown Source) >> at sun.rmi.transport.Transport$2.run(Unknown Source) >> at java.security.AccessController.doPrivileged(Native Method) >> at sun.rmi.transport.Transport.serviceCall(Unknown Source) >> at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown >> Source) >> at >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source) >> at >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.access$400(Unknown >> Source) >> at >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(Unknown Source) >> at >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(Unknown Source) >> at java.security.AccessController.doPrivileged(Native Method) >> at >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source) >> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown >> Source) >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown >> Source) >> at java.lang.Thread.run(Unknown Source) >> > > --001a113fcd6c9df16e0533a5e250 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Mike,

Yes I am us= ing NetworkTopologyStrategy. I checked cassandra-rackdc.properties on the n= ew node:
dc=3DDCamazon-1
rack=3DRACamazon-1

I also check= ed the jira link you sent me. My network topology seems correct: I have 4 n= odes in DC1 and 1 node in DCamazon-1 and I can verify that when running &qu= ot;nodetool status".

Now I am running a full repair on th= e amazon node. I have given up rebuilding

Kind regards,
George



On Wed, May 25, 2016 = at 8:50 AM, Mike Yeap <wkk1020@gmail.com> wrote:
Hi George, are you using = NetworkTopologyStrategy as the replication strategy for your keyspace? If y= es, can you check the cassandra-rackdc.properties of this new node?

= https://issues.apache.org/jira/browse/CASSANDRA-8279

Regards,
Mike Yeap

On Wed, = May 25, 2016 at 2:31 PM, George Sigletos <sigletos@textkernel.nl&= gt; wrote:
<= div>
I am getting this error repeatedly while I am = trying to add a new DC consisting of one node in AWS to my existing cluster= . I have tried 5 times already. Running Cassandra 2.1.13

I hav= e also set:
streaming_socket_timeout_in_ms: 3600000
in all of m= y nodes

Does anybody have any idea how this can be fixed? Than= ks in advance

Kind regards,
George

P.S.<= br>
The complete stack trace:
-- StackTrace --
java.lang.Runtime= Exception: Error while rebuilding node: Stream failed
=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.cassandra.service.StorageService.reb= uild(StorageService.java:1076)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at sun.reflect.NativeMethodAccessor= Impl.invoke(Unknown Source)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 a= t sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.lang.reflect.Method.invoke(Unk= nown Source)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at sun.reflect.m= isc.Trampoline.invoke(Unknown Source)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at sun.reflect.DelegatingMethod= AccessorImpl.invoke(Unknown Source)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 at java.lang.reflect.Method.invoke(Unknown Source)
=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at sun.reflect.misc.MethodUtil.invoke(Unknow= n Source)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at com.sun.jmx.mbea= nserver.StandardMBeanIntrospector.invokeM2(Unknown Source)
=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at com.sun.jmx.mbeanserver.StandardMBeanIntr= ospector.invokeM2(Unknown Source)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at com.sun.jmx.mbeanserver.P= erInterface.invoke(Unknown Source)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)
= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at com.sun.jmx.interceptor.Defau= ltMBeanServerInterceptor.invoke(Unknown Source)
=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown= Source)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at javax.management.= remote.rmi.RMIConnectionImpl.doOperation(Unknown Source)
=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at javax.management.remote.rmi.RMIConnectionImp= l.access$300(Unknown Source)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(Un= known Source)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at javax.manage= ment.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(Unknown Source)
= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at javax.management.remote.rmi.R= MIConnectionImpl.invoke(Unknown Source)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at sun.reflect.DelegatingMet= hodAccessorImpl.invoke(Unknown Source)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 at java.lang.reflect.Method.invoke(Unknown Source)
=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at sun.rmi.server.UnicastServerRef.dispat= ch(Unknown Source)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at sun.rmi= .transport.Transport$2.run(Unknown Source)
=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 at sun.rmi.transport.Transport$2.run(Unknown Source)
=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.security.AccessController.d= oPrivileged(Native Method)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at= sun.rmi.transport.Transport.serviceCall(Unknown Source)
=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 at sun.rmi.transport.tcp.TCPTransport.handleMes= sages(Unknown Source)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at sun.= rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at sun.rmi.transport.tcp.TCPTran= sport$ConnectionHandler.access$400(Unknown Source)
=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandl= er$1.run(Unknown Source)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at s= un.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(Unknown Source)=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.security.AccessControl= ler.doPrivileged(Native Method)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Sou= rce)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.util.concurrent.= ThreadPoolExecutor.runWorker(Unknown Source)
=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unkno= wn Source)
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at java.lang.Threa= d.run(Unknown Source)


--001a113fcd6c9df16e0533a5e250--