Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id AD9E4200C5D for ; Fri, 7 Apr 2017 17:12:39 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id AC0E4160B97; Fri, 7 Apr 2017 15:12:39 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 5949D160B93 for ; Fri, 7 Apr 2017 17:12:38 +0200 (CEST) Received: (qmail 12528 invoked by uid 500); 7 Apr 2017 15:12:36 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 12510 invoked by uid 99); 7 Apr 2017 15:12:36 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Apr 2017 15:12:36 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 69699C047D for ; Fri, 7 Apr 2017 15:12:36 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.681 X-Spam-Level: * X-Spam-Status: No, score=1.681 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001, URIBL_BLOCKED=0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id gqgXq0ElbNKI for ; Fri, 7 Apr 2017 15:12:33 +0000 (UTC) Received: from mail-it0-f54.google.com (mail-it0-f54.google.com [209.85.214.54]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 5118F5F27E for ; Fri, 7 Apr 2017 15:12:33 +0000 (UTC) Received: by mail-it0-f54.google.com with SMTP id a140so44313119ita.0 for ; Fri, 07 Apr 2017 08:12:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=/embbPE6NHlzdpt2wxPHQEggqhmeXRMEJz5QyJFLaT0=; b=jIO3bIY+L3a9m1T/Q+nPUU3H1PbknyeZHnojZY1G17bj4wCY7BN0Wy+vMgnAF0yj6t UI1c/HjV5d86FrEapOSZq87JhQ98YRfM+3Op7asAjP95bzYJVCzL/E8u4gK0z1GG2ek9 s6Fbvkos4h8p8G51qL7P0kdp21wuF36Tk4luj/Z4bde3iHwYBX8tGsG8mRsmrcbyPFgN zigpbQezOR+3pz87tWVQPqJLmGtLerjguOx3xWoTTWjsKT5iSmOMs4BAdoiFImqUVhja sCaQjebtYVx/SkKKy01CtVKPBJCQATRCXxojaf5QQLZl/NE7XfnGTRSfbRXGOz1aE2ev 9ncQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=/embbPE6NHlzdpt2wxPHQEggqhmeXRMEJz5QyJFLaT0=; b=HKeOLxg5z0RnGBr0SnWEOqlLZSqAJeGjnTDoHF7rN6JE2WU7rba6yHr0wilrfF9mcv eugGo96Ti8m8drqx1xn/HFMMPj2uJ3PmZjOcsqXd/5JP5cjf9e+XQxqzhiGlWQnEimik 2dXgAbycJai/4C1s7R2oIXUoQSnQJvfo5L183trbs1A8qezaD+a5BY/MfEhPGFe9FfGg ACXq+V3RaR/6hMH9+piYrpSmvE8lKP/MbMDDHUXSIezdmn2MTz2gdVXw3q9PjQYzoRQF paOUKKBDLR+K4dYlNsjGqTzdvU7i5QgJY2+GjVphCDeB2uzRfuh6DANCKz3MILRk3dkl g7Eg== X-Gm-Message-State: AN3rC/66xCLv3JrtuOaWTdMTDvH+lRYndeKFECHAcHrMx5fZIT3T4ZeFbjucL2YehGljAQR34XthYs6JDCZ0VA== X-Received: by 10.36.67.148 with SMTP id s142mr5845830itb.85.1491577946806; Fri, 07 Apr 2017 08:12:26 -0700 (PDT) MIME-Version: 1.0 Received: by 10.79.120.87 with HTTP; Fri, 7 Apr 2017 08:11:46 -0700 (PDT) In-Reply-To: <1491571403.3561.6.camel@willhaben.at> References: <1491559208.3561.2.camel@willhaben.at> <1491563805.3561.4.camel@willhaben.at> <1491571403.3561.6.camel@willhaben.at> From: Jacob Shadix Date: Fri, 7 Apr 2017 11:11:46 -0400 Message-ID: Subject: Re: cassandra node stops streaming data during nodetool rebuild To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a113fa268acd8c4054c950d9f archived-at: Fri, 07 Apr 2017 15:12:39 -0000 --001a113fa268acd8c4054c950d9f Content-Type: text/plain; charset=UTF-8 I don't see an issue with the size of the data / node. You can attempt the rebuild again and play around with throughput if your network can handle it. It can be changed on-the-fly with nodetool: nodetool setstreamthroughput This article is also worth a read - https://support.datastax.com/hc/en-us/articles/205409646-How-to-performance-tune-data-streaming-activities-like-repair-and-bootstrap -- Jacob Shadix On Fri, Apr 7, 2017 at 9:23 AM, Roland Otta wrote: > good point! > > on the source side i can see the following error > > ERROR [STREAM-OUT-/192.168.0.114:34094] 2017-04-06 17:18:56,532 > StreamSession.java:529 - [Stream #41606030-1ad9-11e7-9f16-51230e2be4e9] > Streaming error occurred on session with peer 10.192.116.1 through 192.168. > 0.114 > org.apache.cassandra.io.FSReadError: java.io.IOException: Broken pipe > at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:145) > ~[apache-cassandra-3.7.jar:3.7] > at org.apache.cassandra.streaming.compress. > CompressedStreamWriter.lambda$write$0(CompressedStreamWriter.java:90) > ~[apache-cassandra-3.7.jar:3.7] > at org.apache.cassandra.io.util.BufferedDataOutputStreamPlus. > applyToChannel(BufferedDataOutputStreamPlus.java:350) > ~[apache-cassandra-3.7.jar:3.7] > at org.apache.cassandra.streaming.compress. > CompressedStreamWriter.write(CompressedStreamWriter.java:90) > ~[apache-cassandra-3.7.jar:3.7] > at org.apache.cassandra.streaming.messages. > OutgoingFileMessage.serialize(OutgoingFileMessage.java:91) > ~[apache-cassandra-3.7.jar:3.7] > at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1. > serialize(OutgoingFileMessage.java:48) ~[apache-cassandra-3.7.jar:3.7] > at org.apache.cassandra.streaming.messages.OutgoingFileMessage$1. > serialize(OutgoingFileMessage.java:40) ~[apache-cassandra-3.7.jar:3.7] > at org.apache.cassandra.streaming.messages. > StreamMessage.serialize(StreamMessage.java:48) > ~[apache-cassandra-3.7.jar:3.7] > at org.apache.cassandra.streaming.ConnectionHandler$ > OutgoingMessageHandler.sendMessage(ConnectionHandler.java:370) > ~[apache-cassandra-3.7.jar:3.7] > at org.apache.cassandra.streaming.ConnectionHandler$ > OutgoingMessageHandler.run(ConnectionHandler.java:342) > ~[apache-cassandra-3.7.jar:3.7] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_77] > Caused by: java.io.IOException: Broken pipe > at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) > ~[na:1.8.0_77] > at sun.nio.ch.FileChannelImpl.transferToDirectlyInternal(FileChannelImpl.java:428) > ~[na:1.8.0_77] > at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:493) > ~[na:1.8.0_77] > at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:608) > ~[na:1.8.0_77] > at org.apache.cassandra.io.util.ChannelProxy.transferTo(ChannelProxy.java:141) > ~[apache-cassandra-3.7.jar:3.7] > ... 10 common frames omitted > DEBUG [STREAM-OUT-/192.168.0.114:34094] 2017-04-06 17:18:56,532 > ConnectionHandler.java:110 - [Stream #41606030-1ad9-11e7-9f16-51230e2be4e9] > Closing stream connection handler on /10.192.116.1 > INFO [STREAM-OUT-/192.168.0.114:34094] 2017-04-06 17:18:56,532 > StreamResultFuture.java:187 - [Stream #41606030-1ad9-11e7-9f16-51230e2be4e9] > Session with /10.192.116.1 is complete > WARN [STREAM-OUT-/192.168.0.114:34094] 2017-04-06 17:18:56,532 > StreamResultFuture.java:214 - [Stream #41606030-1ad9-11e7-9f16-51230e2be4e9] > Stream failed > > > the dataset is approx 300GB / Node. > > does that mean that cassandra does not try to reconnect (for streaming) in > case of short network dropouts? > > On Fri, 2017-04-07 at 08:53 -0400, Jacob Shadix wrote: > > Did you look at the logs on the source DC as well? How big is the dataset? > > -- Jacob Shadix > > On Fri, Apr 7, 2017 at 7:16 AM, Roland Otta > wrote: > > Hi! > > we are on 3.7. > > we have some debug messages ... but i guess they are not related to that > issue > DEBUG [GossipStage:1] 2017-04-07 13:11:00,440 FailureDetector.java:456 - > Ignoring interval time of 2002469610 for /192.168.0.27 > DEBUG [GossipStage:1] 2017-04-07 13:11:00,441 FailureDetector.java:456 - > Ignoring interval time of 2598593732 for /10.192.116.4 > DEBUG [GossipStage:1] 2017-04-07 13:11:00,441 FailureDetector.java:456 - > Ignoring interval time of 2002612298 for /10.192.116.5 > DEBUG [GossipStage:1] 2017-04-07 13:11:00,441 FailureDetector.java:456 - > Ignoring interval time of 2002660534 for /10.192.116.9 > DEBUG [GossipStage:1] 2017-04-07 13:11:00,465 FailureDetector.java:456 - > Ignoring interval time of 2027212880 for /10.192.116.3 > DEBUG [GossipStage:1] 2017-04-07 13:11:00,465 FailureDetector.java:456 - > Ignoring interval time of 2027279042 for /192.168.0.188 > DEBUG [GossipStage:1] 2017-04-07 13:11:00,465 FailureDetector.java:456 - > Ignoring interval time of 2027313992 for /10.192.116.10 > > beside that the debug.log is clean > > all the mentioned cassandra.yml parameters are the shipped defaults ( > streaming_socket_timeout_in_ms does not exist at all in my cassandra.yml) > i also checked the pending compactions. there are no pending compactions > at the moment. > > bg - roland otta > > On Fri, 2017-04-07 at 06:47 -0400, Jacob Shadix wrote: > > What version are you running? Do you see any errors in the system.log > (SocketTimeout, for instance)? > > And what values do you have for the following in cassandra.yaml: > - - stream_throughput_outbound_megabits_per_sec > - - compaction_throughput_mb_per_sec > - - streaming_socket_timeout_in_ms > > -- Jacob Shadix > > On Fri, Apr 7, 2017 at 6:00 AM, Roland Otta > wrote: > > hi, > > we are trying to setup a new datacenter and are initalizing the data > with nodetool rebuild. > > after some hours it seems that the node stopped streaming (at least > there is no more streaming traffic on the network interface). > > nodetool netstats shows that the streaming is still in progress > > Mode: NORMAL > Bootstrap 6918dc90-1ad6-11e7-9f16-51230e2be4e9 > Rebuild 41606030-1ad9-11e7-9f16-51230e2be4e9 > /192.168.0.26 > Receiving 257 files, 145444246572 bytes total. Already received > 1 files, 1744027 bytes total > bds/adcounter_total 76456/47310255 bytes(0%) received from > idx:0/192.168.0.26 > bds/upselling_event 1667571/1667571 bytes(100%) received > from idx:0/192.168.0.26 > /192.168.0.188 > /192.168.0.27 > Receiving 169 files, 79355302464 bytes total. Already received > 1 files, 81585975 bytes total > bds/ad_event_history 81585975/81585975 bytes(100%) received > from idx:0/192.168.0.27 > /192.168.0.189 > Receiving 140 files, 19673034809 bytes total. Already received > 1 files, 5996604 bytes total > bds/adcounter_per_day 5956840/42259846 bytes(14%) received > from idx:0/192.168.0.189 > bds/user_event 39764/39764 bytes(100%) received from > idx:0/192.168.0.189 > Read Repair Statistics: > Attempted: 0 > Mismatch (Blocking): 0 > Mismatch (Background): 0 > Pool Name Active Pending Completed Dropped > Large messages n/a 2 3 0 > Small messages n/a 0 68632465 0 > Gossip messages n/a 0 217661 0 > > > > it is in that state for approx 15 hours now > > does it make sense waiting for the streaming to finish or do i have to > restart the node, discard data and restart the rebuild? > > > > > --001a113fa268acd8c4054c950d9f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I don't see an issue with the size of the data / node.= You can attempt the rebuild again and play around with throughput if your = network can handle it.

It can be changed on-the-fly with= nodetool:

=C2=A0nodetool setstreamthroughput=C2= =A0


-- Jacob Shadix=C2= =A0

On Fri, Apr 7, 2017 at 9:23 AM, Roland Otta = <Roland.Otta@willhaben.at> wrote:
good point!

on the source side i can see the following error

ERROR [STREAM-OUT-/192.168.0.114:34094] 2017-04-06 17:18:56,532 StreamSession.j= ava:529 - [Stream #41606030-1ad9-11e7-9f16-51230e2be4e9] Streaming err= or occurred on session with peer 10.192.116.1 through 192.168.
0.114
org.apache.cassandra.io= .FSReadError: java.io.IOException: Broken pipe
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0at org.apache.cassandr= a.io.util.ChannelProxy.transferTo(ChannelProxy.java:145) ~[apache= -cassandra-3.7.jar:3.7]
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0at org.apache.cassandr= a.streaming.compress.CompressedStreamWriter.lambda$write$0(<= wbr>CompressedStreamWriter.java:90) ~[apache-cassandra-3.7.jar:3.= 7]
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0at org.apache.cassandr= a.io.util.BufferedDataOutputStreamPlus.applyToChannel(Buffer= edDataOutputStreamPlus.java:350) ~[apache-cassandra-3.7.jar:3.7]<= /div>
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0at org.apache.cassandr= a.streaming.compress.CompressedStreamWriter.write(Compressed= StreamWriter.java:90) ~[apache-cassandra-3.7.jar:3.7]
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0at org.apache.cassandr= a.streaming.messages.OutgoingFileMessage.serialize(OutgoingF= ileMessage.java:91) ~[apache-cassandra-3.7.jar:3.7]
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0at org.apache.cassandr= a.streaming.messages.OutgoingFileMessage$1.serialize(Outgoin= gFileMessage.java:48) ~[apache-cassandra-3.7.jar:3.7]
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0at org.apache.cassandr= a.streaming.messages.OutgoingFileMessage$1.serialize(Outgoin= gFileMessage.java:40) ~[apache-cassandra-3.7.jar:3.7]
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0at org.apache.cassandr= a.streaming.messages.StreamMessage.serialize(StreamMessage.j= ava:48) ~[apache-cassandra-3.7.jar:3.7]
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0at org.apache.cassandr= a.streaming.ConnectionHandler$OutgoingMessageHandler.sendMes= sage(ConnectionHandler.java:370) ~[apache-cassandra-3.7.jar:3.7]<= /div>
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0at org.apache.cassandr= a.streaming.ConnectionHandler$OutgoingMessageHandler.run(Con= nectionHandler.java:342) ~[apache-cassandra-3.7.jar:3.7]
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0at java.lang.Thread.ru= n(Thread.java:745) [na:1.8.0_77]
Caused by: java.io.IOException: Broken pipe
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0at sun.nio.ch.FileChan= nelImpl.transferTo0(Native Method) ~[na:1.8.0_77]
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0at sun.nio.ch.FileChan= nelImpl.transferToDirectlyInternal(FileChannelImpl.java:428) ~[na= :1.8.0_77]
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0at sun.nio.ch.FileChan= nelImpl.transferToDirectly(FileChannelImpl.java:493) ~[na:1.8.0_7= 7]
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0at sun.nio.ch.FileChan= nelImpl.transferTo(FileChannelImpl.java:608) ~[na:1.8.0_77]
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0at org.apache.cassandr= a.io.util.ChannelProxy.transferTo(ChannelProxy.java:141) ~[apache= -cassandra-3.7.jar:3.7]
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0... 10 common frames o= mitted
DEBUG [STREAM-OUT-/192.168.0.114:34094] 2017-04-06 17:18:56,532 ConnectionHandl= er.java:110 - [Stream #41606030-1ad9-11e7-9f16-51230e2be4e9] Closing s= tream connection handler on /10.192.116.1
INFO=C2=A0=C2=A0[STREAM-OUT-/192.168.0.114:34094] 2017-04-06 17:18:56,532 Strea= mResultFuture.java:187 - [Stream #41606030-1ad9-11e7-9f16-51230e2be4e9= ] Session with /10.192.11= 6.1 is complete
WARN=C2=A0=C2=A0[STREAM-OUT-/192.168.0.114:34094] 2017-04-06 17:18:56,532 Strea= mResultFuture.java:214 - [Stream #41606030-1ad9-11e7-9f16-51230e2be4e9= ] Stream failed


the dataset is approx 300GB / Node.

does that mean that cassandra does not try to reconnect (for streaming= ) in case of short network dropouts?

On Fri, 2017-04-07 at 08:53 -0400, Jacob Shadix wrote:
Did you look at the logs on the source DC as well? How big= is the dataset?

-- Jacob Shad= ix=C2=A0

On Fri, Apr 7, 2017 at 7:16 AM, Roland Otta <Roland.Ot= ta@willhaben.at> wrote:
Hi!

we are on 3.7.

we have some debug messages ... but i guess they are not related to th= at issue
DEBUG [GossipStage:1] 2017-04-07 13:11:00,440 FailureDetector.java:456= - Ignoring interval time of 2002469610 for /192.168.0.27
DEBUG [GossipStage:1] 2017-04-07 13:11:00,441 FailureDetector.java:456= - Ignoring interval time of 2598593732 for /10.192.116.4
DEBUG [GossipStage:1] 2017-04-07 13:11:00,441 FailureDetector.java:456= - Ignoring interval time of 2002612298 for /10.192.116.5
DEBUG [GossipStage:1] 2017-04-07 13:11:00,441 FailureDetector.java:456= - Ignoring interval time of 2002660534 for /10.192.116.9
DEBUG [GossipStage:1] 2017-04-07 13:11:00,465 FailureDetector.java:456= - Ignoring interval time of 2027212880 for /10.192.116.3
DEBUG [GossipStage:1] 2017-04-07 13:11:00,465 FailureDetector.java:456= - Ignoring interval time of 2027279042 for /192.168.0.188
DEBUG [GossipStage:1] 2017-04-07 13:11:00,465 FailureDetector.java:456= - Ignoring interval time of 2027313992 for /10.192.116.10

beside that the debug.log is clean

all the mentioned cassandra.yml parameters are the shipped defaults (<= span style=3D"color:rgb(64,64,64);font-family:helvetica,sans-serif;font-siz= e:14px">streaming_socket_timeout_in_ms does not exist at all in my cas= sandra.yml)
i also checked the pending compactions. there are no pending compactio= ns at the moment.

bg - roland otta

On Fri, 2017-04-07 at 06:47 -0400, Jacob Shadix wrote:
What version are you running? Do you see any errors in the= system.log (SocketTimeout, for instance)?=C2=A0

And what values do you have for the following in cassandra.yaml:=C2=A0
- - stream_throughput_outbound_megabits_per_sec
- - compaction_throughput_mb_per_sec
- - streaming_socket_timeout_in_ms

-- Jacob Shad= ix=C2=A0

On Fri, Apr 7, 2017 at 6:00 AM, Roland Otta <Roland.Ot= ta@willhaben.at> wrote:
hi,

we are trying to setup a new datacenter and are initalizing the data
with nodetool rebuild.

after some hours it seems that the node stopped streaming (at least
there is no more streaming traffic on the network interface).

nodetool netstats shows that the streaming is still in progress

Mode: NORMAL
Bootstrap 6918dc90-1ad6-11e7-9f16-51230e2be4e9
Rebuild 41606030-1ad9-11e7-9f16-51230e2be4e9
=C2=A0=C2=A0=C2=A0=C2=A0/192.168.0.26
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Receiving 257 files, 145444= 246572 bytes total. Already received
1 files, 1744027 bytes total
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0bds= /adcounter_total 76456/47310255 bytes(0%) received from
idx:0/= 192.168.0.26
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0bds= /upselling_event 1667571/1667571 bytes(100%) received
from idx:0/192.168.0.26
=C2=A0=C2=A0=C2=A0=C2=A0/192.168.0.188
=C2=A0=C2=A0=C2=A0=C2=A0/192.168.0.27
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Receiving 169 files, 793553= 02464 bytes total. Already received
1 files, 81585975 bytes total
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0bds= /ad_event_history 81585975/81585975 bytes(100%) received
from idx:0/192.168.0.27
=C2=A0=C2=A0=C2=A0=C2=A0/192.168.0.189
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Receiving 140 files, 196730= 34809 bytes total. Already received
1 files, 5996604 bytes total
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0bds= /adcounter_per_day 5956840/42259846 bytes(14%) received
from idx:0/192.168.0.189
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0bds= /user_event 39764/39764 bytes(100%) received from
idx:0/192.168.0.189
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool Name=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Active=C2=A0=C2= =A0=C2=A0Pending=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Completed=C2=A0=C2=A0= =C2=A0Dropped
Large messages=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0n/a=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A02=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A03=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A00
Small messages=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0n/a=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A00=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A068632465=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A00<= br> Gossip messages=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0n/a=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A00=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0217661=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A00



it is in that state for approx 15 hours now

does it make sense waiting for the streaming to finish or do i have to
restart the node, discard data and restart the rebuild?





--001a113fa268acd8c4054c950d9f--